Mac OS X and iOS Internals : To the Apple's Core

by Levin, Jonathan

• ISBN13: 9781118057650
• ISBN10: 1118057651
• Format: Paperback
• Copyright: 2012-11-06
• Publisher: Wrox

Summary

An in-depth look into Mac OS X and iOS kernels With the widespread adoption of Macs and iOS there is a growing need for a detailed description of how the system is actually built, from the ground up, and how to take full advantage of all it has to offer. With this indispensible guide, you will examine the internal works of Mac OS X and iOS as well as the OS from user mode to kernel mode, all while focusing on the structure of various components along the way. Provides you with an in-depth exploration of the internal works of Mac OS X and iOS Looks at the underlying design and implementation of XNU, with emphases on the sub components such as Mach, BSD, and IOKit Reviews various tools and techniques, including Instruments, Shark, and DTrace framework Covers system startup and shutdown, booting EFI, booting the kernel, and user mode startup From architecture to implementation, this book is essential reading if you want to get serious about the internal workings of Mac OS X and iOS.

Author Biography

Jonathan Levin is a longtime trainer and consultant focusing on the system and kernel levels of the 'Big Three'—Windows, Linux, and OS X, as well as their mobile derivatives. He is the founder and CTO of Technologeeks.com, a partnership of experts delivering advanced training on systems/kernel programming, debugging, and profiling.

Table of Contents

INTRODUCTION xxv

PART I: FOR POWER USERS

CHAPTER 1: DARWINISM: THE EVOLUTION OF OS X 3

The Pre-Darwin Era: Mac OS Classic 3

The Prodigal Son: NeXTSTEP 4

Enter: OS X 4

OS X Versions, to Date 5

10.0 — Cheetah and the First Foray 5

10.1 — Puma — a Stronger Feline, but . . . 6

10.2 — Jaguar — Getting Better 6

10.3 — Panther and Safari 6

10.4 — Tiger and Intel Transition 6

10.5 — Leopard and UNIX 7

10.6 — Snow Leopard 7

10.7 — Lion 8

10.8 — Mountain Lion 9

iOS — OS X Goes Mobile 10

1.x — Heavenly and the First iPhone 11

2.x — App Store, 3G and Corporate Features 11

3.x — Farewell, 1st gen, Hello iPad 11

4.x — iPhone 4, Apple TV, and the iPad 2 11

5.x — To the iPhone 4S and Beyond 12

iOS vs. OS X 12

The Future of OS X 15

Summary 16

References 16

CHAPTER 2: E PLURIBUS UNUM: ARCHITECTURE OF OS X AND IOS 17

OS X Architectural Overview 17

The User Experience Layer 19

Aqua 19

Quicklook 20

Spotlight 21

Darwin — The UNIX Core 22

The Shell 22

The File System 23

UNIX System Directories 24

OS X–Specifi c Directories 25

iOS File System Idiosyncrasies 25

Interlude: Bundles 26

Applications and Apps 26

Info.plist 28

Resources 30

NIB Files 30

Internationalization with .lproj Files 31

Icons (.icns) 31

CodeResources 31

Frameworks 34

Framework Bundle Format 34

List of OS X and iOS Public Frameworks 37

Libraries 44

Other Application Types 46

System Calls 48

POSIX 48

Mach System Calls 48

A High-Level View of XNU 51

Mach 51

The BSD Layer 51

libkern 52

I/O Kit 52

Summary 52

References 53

CHAPTER 3: ON THE SHOULDERS OF GIANTS: OS X

AND IOS TECHNOLOGIES 55

BSD Heirlooms 55

sysctl 56

kqueues 57

Auditing (OS X) 59

Mandatory Access Control 62

OS X- and iOS-Specifi c Technologies 65

User and Group Management (OS X) 65

System Confi guration 67

Logging 69

Apple Events and AppleScript 72

FSEvents 74

Notifi cations 78

Additional APIs of interest 79

OS X and iOS Security Mechanisms 79

Code Signing 80

Compartmentalization (Sandboxing) 81

Entitlements: Making the Sandbox Tighter Still 83

Enforcing the Sandbox 89

Summary 90

References 90

CHAPTER 4: PARTS OF THE PROCESS: MACH-O,

PROCESS, AND THREAD INTERNALS 91

A Nomenclature Refresher 91

Processes and Threads 91

The Process Lifecycle 92

UNIX Signals 95

Executables 98

Universal Binaries 99

Mach-O Binaries 102

Load Commands 106

Dynamic Libraries 111

Launch-Time Loading of Libraries 111

Runtime Loading of Libraries 122

dyld Features 124

Process Address Space 130

The Process Entry Point 130

Address Space Layout Randomization 131

32-Bit (Intel) 132

64-Bit 132

32-Bit (iOS) 133

Experiment: Using vmmap(1) to Peek Inside a Process’s

Address Space 135

Process Memory Allocation (User Mode) 138

Heap Allocations 139

Virtual Memory — The sysadmin Perspective 140

Threads 143

Unraveling Threads 143

References 146

CHAPTER 5: NON SEQUITUR:

PROCESS TRACING AND DEBUGGING 147

DTrace 147

The D Language 147

dtruss 150

How DTrace Works 152

Other Profi ling mechanisms 154

The Decline and Fall of CHUD 154

AppleProfi leFamily: The Heir Apparent 155

Process Information 156

sysctl 156

proc_info 156

Process and System Snapshots 159

system_profi ler(8) 159

sysdiagnose(1) 159

allmemory(1) 160

stackshot(1) 160

The stack_snapshot System Call 162

kdebug 165

kdebug-based Utilities 165

kdebug codes 166

Writing kdebug messages 168

Reading kdebug messages 169

Application Crashes 170

Application Hangs and Sampling 173

Memory Corruption Bugs 174

Memory Leaks 176

heap(1) 177

leaks(1) 177

malloc_history(1) 178

Standard UNIX Tools 178

Process listing with ps(1) 179

System-Wide View with top(1) 179

File Diagnostics with lsof(1) and fuser(1) 180

Using GDB 181

GDB Darwin Extensions 181

GDB on iOS 182

LLDB 182

Summary 182

References and Further Reading 182

CHAPTER 6: ALONE IN THE DARK:

THE BOOT PROCESS: EFI AND IBOOT 183

Traditional Forms of Boot 183

EFI Demystifi ed 185

Basic Concepts of EFI 186

The EFI Services 188

NVRAM Variables 192

OS X and boot.efi 194

Flow of boot.efi 195

Booting the Kernel 201

Kernel Callbacks into EFI 203

Boot.efi Changes in Lion 204

Boot Camp 204

Count Your Blessings 204

Experiment: Running EFI Programs on a Mac 206

iOS and iBoot 210

Precursor: The Boot ROM 210

Normal Boot 211

Recovery Mode 212

Device Firmware Update (DFU) Mode 213

Downgrade and Replay Attacks 213

Installation Images 214

OS X Installation Process 214

iOS File System Images (.ipsw) 219

Summary 225

References and Further Reading 225

CHAPTER 7: THE ALPHA AND THE OMEGA — LAUNCHD 227

launchd 227

Starting launchd 227

System-Wide Versus Per-User launchd 228

Daemons and Agents 229

The Many Faces of launchd 229

Lists of LaunchDaemons 241

GUI Shells 246

Finder (OS X) 247

SpringBoard (iOS) 248

XPC (Lion and iOS) 253

Summary 257

References and Further Reading 258

PART II: THE KERNEL

CHAPTER 8: SOME ASSEMBLY REQUIRED:

KERNEL ARCHITECTURES 261

Kernel Basics 261

Kernel Architectures 262

User Mode versus Kernel Mode 266

Intel Architecture — Rings 266

ARM Architecture: CPSR 267

Kernel/User Transition Mechanisms 268

Trap Handlers on Intel 269

Voluntary kernel transition 278

System Call Processing 283

POSIX/BSD System calls 284

Mach Traps 287

Machine Dependent Calls 292

Diagnostic calls 292

XNU and hardware abstraction 295

Summary 297

References 297

CHAPTER 9: FROM THE CRADLE TO THE GRAVE —

KERNEL BOOT AND PANICS 299

The XNU Sources 299

Getting the Sources 299

Making XNU 300

One Kernel, Multiple Architectures 302

The XNU Source Tree 305

Booting XNU 308

The Bird’s Eye View 309

OS X: vstart 310

iOS: start 310

[i386|arm]_init 311

i386_init_slave() 313

machine_startup 314

kernel_bootstrap 314

kernel_bootstrap_thread 318

bsd_init 320

bsdinit_task 325

Sleeping and Waking Up 328

Boot Arguments 329

Kernel Debugging 332

“Don’t Panic” 333

Implementation of Panic 334

Panic Reports 336

Summary 340

References 341

CHAPTER 10: THE MEDIUM IS THE MESSAGE: MACH PRIMITIVES 343

Introducing: Mach 344

The Mach Design Philosophy 344

Mach Design Goals 345

Mach Messages 346

Simple Messages 346

Complex messages 347

Sending Messages 348

Ports 349

The Mach Interface Generator (MIG) 351

IPC, in Depth 357

Behind the Scenes of Message Passing 359

Synchronization Primitives 360

Lock Group Objects 361

Mutex Object 362

Read-Write Lock Object 363

Spinlock Object 364

Semaphore Object 364

Lock Set Object 366

Machine Primitives 367

Clock Object 378

Processor Object 380

Processor Set Object 384

Summary 388

References 388

CHAPTER 11: TEMPUS FUGIT — MACH SCHEDULING 389

Scheduling Primitives 389

Threads 390

Tasks 395

Task and Thread APIs 399

Task APIs 399

Thread APIs 404

Scheduling 408

The High-Level View 408

Priorities 409

Run Queues 412

Mach Scheduler Specifi cs 415

Asynchronous Software Traps (ASTs) 423

Scheduling Algorithms 427

Timer Interrupts 431

Interrupt-Driven Scheduling 431

Timer Interrupt Processing in XNU 432

Exceptions 436

The Mach Exception Model 436

Implementation Details 437

Experiment: Mach Exception Handling 440

Summary 446

References 446

CHAPTER 12: COMMIT TO MEMORY:

MACH VIRTUAL MEMORY 447

Virtual Memory Architecture 447

The 30,000-Foot View of Virtual Memory 448

The Bird’s Eye View 449

The User Mode View 452

Physical Memory Management 462

Mach Zones 467

The Mach Zone Structure 468

Zone Setup During Boot 470

Zone Garbage Collection 471

Zone Debugging 473

Kernel Memory Allocators 473

kernel_memory_allocate() 473

kmem_alloc() and Friends 477

kalloc 477

OSMalloc 479

Mach Pagers 480

The Mach Pager interface 480

Universal Page Lists 484

Pager Types 486

Paging Policy Management 494

The Pageout Daemon 495

Handling Page Faults 497

The dynamic_pager(8) (OS X) 498

Summary 499

References 500

CHAPTER 13: BS”D — THE BSD LAYER 501

Introducing BSD 501

One Ring to Bind Them 502

What’s in the POSIX Standard? 503

Implementing BSD 503

XNU Is Not Fully BSD 504

Processes and Threads 504

BSD Process Structs 504

Process Lists and Groups 507

Threads 508

Mapping to Mach 510

Process Creation 512

The User Mode Perspective 512

The Kernel Mode Perspective 513

Loading and Executing Binaries 516

Mach-O Binaries 522

Process Control and Tracing 525

ptrace (#26) 525

proc_info (#336) 527

Policies 527

Process Suspension/Resumption 529

Signals 529

The UNIX Exception Handler 529

Hardware-Generated Signals 534

Software-Generated Signals 535

Signal Handling by the Victim 536

Summary 536

References 537

CHAPTER 14: SOMETHING OLD, SOMETHING NEW:

ADVANCED BSD ASPECTS 539

Memory Management 539

POSIX Memory and Page Management System Calls 540

BSD Internal Memory Functions 541

Memory Pressure 545

Jetsam (iOS) 546

Kernel Address Space Layout Randomization 548

Work Queues 550

BSD Heirlooms Revisited 552

Sysctl 552

Kqueues 555

Auditing (OS X) 556

Mandatory Access Control 558

Apple’s Policy Modules 560

Summary 563

References 563

CHAPTER 15: FEE, FI-FO, FILE: FILE SYSTEMS AND THE VFS 565

Prelude: Disk Devices and Partitions 565

Partitioning Schemes 567

Generic File System Concepts 577

Files 577

Extended Attributes 577

Permissions 577

Timestamps 578

Shortcuts and Links 578

File Systems in the Apple Ecosystem 579

Native Apple File Systems 579

DOS/Windows File Systems 580

CD/DVD File Systems 581

Network-Based File Systems 582

Pseudo File Systems 583

Mounting File Systems (OS X only) 587

Disk Image Files 589

Booting from a Disk Image (Lion) 590

The Virtual File System Switch 591

The File System Entry 591

The Mount Entry 592

The vnode Object 595

FUSE — File Systems in USEr Space 597

File I/O from Processes 600

Summary 605

References and Further Reading 605

CHAPTER 16: TO B (-TREE) OR NOT TO BE —

THE HFS+ FILE SYSTEMS 607

HFS+ File System Concepts 607

Timestamps 607

Access Control Lists 608

Extended Attributes 608

Forks 611

Compression 612

Unicode Support 617

Finder integration 617

Case Sensitivity (HFSX) 619

Journaling 619

Dynamic Resizing 620

Metadata Zone 620

Hot Files 621

Dynamic Defragmentation 622

HFS+ Design Concepts 624

B-Trees: The Basics 624

Components 630

The HFS+ Volume Header 631

The Catalog File 633

The Extent Overfl ow 640

The Attribute B-Tree 640

The Hot File B-Tree 641

The Allocation File 642

HFS Journaling 642

VFS and Kernel Integration 645

fsctl(2) integration 645

sysctl(2) integration 646

File System Status Notifi cations 647

Summary 647

References 648

CHAPTER 17: ADHERE TO PROTOCOL: THE NETWORKING STACK 649

User Mode Revisited 650

UNIX Domain Sockets 651

IPv4 Networking 651

Routing Sockets 652

Network Driver Sockets 652

IPSec Key Management Sockets 654

IPv6 Networking 654

System Sockets 655

Socket and Protocol Statistics 658

Layer V: Sockets 660

Socket Descriptors 660

mbufs 661

Sockets in Kernel Mode 667

Layer IV: Transport Protocols 668

Domains and Protosws 669

Initializing Domains 673

Layer III: Network Protocols 676

Layer II: Interfaces 678

Interfaces in OS X and iOS 678

The Data Link Interface Layer 680

The ifnet Structure 680

Case Study: utun 682

Putting It All Together: The Stack 686

Receiving Data 686

Sending Data 690

Packet Filtering 693

Socket Filters 694

ipfw(8) 696

The PF Packet Filter (Lion and iOS) 697

IP Filters 698

Interface Filters 701

The Berkeley Packet Filter 701

Traffi c Shaping and QoS 705

The Integrated Services Model 706

The Diff erentiated Services Model 706

Implementing dummynet 706

Controlling Parameters from User Mode 707

Summary 707

References and Further Reading 708

CHAPTER 18: MODU(LU)S OPERANDI — KERNEL EXTENSIONS 711

Extending the Kernel 711

Securing Modular Architecture 712

Kernel Extensions (Kexts) 713

Kext Structure 717

Kext Security Requirements 718

Working with Kernel Extensions 719

Kernelcaches 719

Multi-Kexts 723

A Programmer’s View of Kexts 724

Kernel Kext Support 725

Summary 735

References 735

CHAPTER 19: DRIVING FORCE — I/O KIT 737

Introducing I/O Kit 738

Device Driver Programming Constraints 738

What I/O Kit Is 738

What I/O Kit Isn’t 741

LibKern: The I/O Kit Base Classes 742

The I/O Registry 743

I/O Kit from User Mode 746

I/O Registry Access 747

Getting/Setting Driver Properties 749

Plug and Play (Notifi cation Ports) 750

I/O Kit Power Management 751

Other I/O Kit Subsystems 753

I/O Kit Diagnostics 753

I/O Kit Kernel Drivers 755

Driver Matching 755

The I/O Kit Families 757

The I/O Kit Driver Model 761

The IOWorkLoop 764

Interrupt Handling 765

I/O Kit Memory Management 769

BSD Integration 769

Summary 771

References and Further Reading 771

APPENDIX: WELCOME TO THE MACHINE 773

INDEX 793