Foreword

xxi

Preface

xxiii

About the Authors

xxxi

Acknowledgments

xxxiii

PART ONE Observability Methods

(232)

Introduction to Observability Tools

(8)

Observability Tools

(3)

Kstat Tools

(1)

Process Tools

(1)

Forensic Tools

(1)

Drill-Down Analysis

(1)

About Part One

(3)

Chapter Layout

(3)

CPUs

(24)

Tools for CPU Analysis

(2)

vmstat Tool

(1)

CPU Utilization

(1)

CPU Saturation

(1)

psrinfo Command

(1)

uptime Command

(1)

sar Command

(3)

sar Default Output

(1)

sar-q

(1)

Capturing Historical Data

(1)

Clock Tick Woes

(1)

mpstat Command

(3)

Who Is Using the CPU?

(1)

CPU Run Queue Latency

(2)

CPU Statistics Internals

(3)

usr, sys, idl Times

(2)

Load Averages

(1)

pr_pctcpu Field

(1)

Using DTrace to Explain Events from Performance Tools

(2)

DTrace Versions of runq-sz, %runocc

(2)

DTrace Probes for CPU States

(2)

Processes

(32)

Tools for Process Analysis

(2)

Process Statistics Summary: prstat

(4)

Thread Summary: prstat -L

(1)

Process Microstates: prstat-m

(1)

Sorting by a Key: prstat -s

(1)

User Summary: prstat -t

(1)

Project Summary: prstat -J

(1)

Zone Summary: prstat -Z

(1)

Process Status: ps

(4)

/usr/bin/ps Command

(2)

/usr/ucb/ps

(1)

Tools for Listing and Controlling Processes

(2)

Process Tree: ptree

(1)

Grepping for Processes: pgrep

(1)

Killing Processes: pkill

(1)

Temporarily Stop a Process: pstop

(1)

Making a Process Runnable: prun

(1)

Wait for Process Completion: pwait

(1)

Reap a Zombie Process: preap

(1)

Process Introspection Commands

(5)

Process Stack: pstack

(1)

Process Memory Map: pmap -x

(1)

Process File Table: pfiles

(1)

Execution Time Statistics for a Process: ptime

(1)

Process Signal Disposition: psig

(1)

Process Libraries: pldd

(1)

Process Flags: pflags

(1)

Process Credentials: pcred

(1)

Process Arguments: pargs

(1)

Process Working Directory: pwdx

(1)

Examining User-Level Locks in a Process

(1)

Tracing Processes

(7)

Using truss to Trace Processes

(1)

Using apptrace to Trace Processes

(2)

Using DTrace to Trace Process Functions

(1)

Using DTrace to Aggregate Process Functions

(1)

Using DTrace to Peer Inside Processes

(1)

Using DTrace to Sample Stack Backtraces

(1)

Java Processes

(7)

Process Stack on a Java Virtual Machine: pstack

(1)

JVM Profiling

(1)

Tuning Java Garbage Collection

(2)

Using DTrace on Java Applications

(3)

Disk Behavior and Analysis

(42)

Terms for Disk Analysis

(2)

Random vs. Sequential I/O

(1)

Demonstration of Sequential Disk Activity

(1)

Demonstration of Random Disk Activity

(1)

Storage Arrays

(1)

Sector Zoning

(1)

Max I/O Size

(1)

iostat Utility

(1)

Disk Utilization

(1)

Disk Saturation

(1)

Disk Throughput

(1)

iostat Reference

(6)

iostat Default

(1)

iostat -D

(1)

iostat -l n

(1)

iostat -n

(1)

iostat -x

(1)

iostat -p, -p

(1)

iostat -e

(1)

iostat -E

(1)

Reading iostat

(3)

Event Size Ratio

(1)

Service Times

(2)

iostat Internals

(2)

sar -d

(1)

Trace Normal Form (TNF) Tracing for I/O

(1)

DTrace for I/O

(9)

io Probes

(1)

I/O Size One-Liners

(1)

A More Elaborate Example

(1)

I/O Size Aggregation

(3)

I/O Seek Aggregation

(2)

I/O File Names

(1)

Disk I/O Time

(4)

Simple Disk Event

(1)

Concurrent Disk Events

(1)

Adaptive Disk I/O Time Algorithm

(1)

Other Response Times

100

(1)

Time by Layer

101

(1)

DTraceToolkit Commands

101

(7)

iotop Script

102

(2)

iosnoop Script

104

(1)

Plotting Disk Activity

105

(1)

Plotting Concurrent Activity

106

(2)

DTraceTazTool

108

(1)

File Systems

109

(26)

Layers of File System and I/O

109

(2)

Observing Physical I/O

111

(1)

File System Latency

112

(2)

Causes of Read/Write File System Latency

114

(4)

Disk I/O Wait

115

(1)

Block or Metadata Cache Misses

116

(1)

I/O Breakup

116

(1)

Locking in the File System

117

(1)

Metadata Updates

117

(1)

Observing File System ``Top End'' Activity

118

(1)

File System Caches

119

(14)

Page Cache

120

(4)

Bypassing the Page Cache with Direct I/O

124

(2)

The Directory Name Lookup Cache

126

(2)

Block Buffer Cache

128

(2)

UFS Inode Cache

130

(2)

Monitoring UFS Caches with fcachestat

132

(1)

NFS Statistics

133

(2)

NFS Client Statistics: nfsstat -c

133

(1)

NFS Server Statistics: nfsstat -s

134

(1)

Memory

135

(38)

Tools for Memory Analysis

135

(2)

vmstat(1M) Command

137

(1)

Types of Paging

138

(4)

File I/O Paging: ``Good'' Paging

139

(2)

Anonymous Memory Paging: ``Bad'' Paging

141

(1)

Per-Zone Paging Statistics

141

(1)

Physical Memory Allocation

142

(2)

Total Physical Memory

142

(1)

Free Memory

142

(1)

Using the memstat Command in MDB

143

(1)

Relieving Memory Pressure

144

(2)

Scan Rate as a Memory Health Indicator

146

(2)

Using prstat to Estimate Memory Slowdowns

148

(1)

Process Virtual and Resident Set Size

148

(1)

Using pmap to Inspect Process Memory Usage

149

(1)

Calculating Process Memory Usage with ps and pmap

150

(3)

Displaying Page-Size Information with pmap

153

(1)

Using DTrace for Memory Analysis

154

(3)

Using DTrace to Estimate Memory Slowdowns

155

(2)

Obtaining Memory Kstats

157

(1)

Using the Perl Kstat API to Look at Memory Statistics

158

(1)

System Memory Allocation Kstats

158

(2)

Kernel Memory with kstat

160

(1)

System Paging Kstats

161

(2)

Observing MMU Performance Impact with trapstat

163

(1)

Swap Space

164

(9)

Swap Allocation

164

(2)

Swap Statistics

166

(1)

Swap Summary: swap -s

166

(1)

Listing Physical Swap Devices: swap -l

167

(1)

Determining Swapped-Out Threads

167

(1)

Monitoring Physical Swap Activity

168

(1)

MemTool prtswap

169

(1)

Display of Swap Reservations with pmap

170

(3)

Networks

173

(30)

Terms for Network Analysis

173

(2)

Packets Are Not Bytes

175

(1)

Network Utilization

176

(1)

Network Saturation

177

(1)

Network Errors

177

(1)

Misconfigurations

177

(1)

Systemwide Statistics

178

(11)

netstat Command

178

(1)

kstat Command

179

(1)

nx.se Tool

180

(1)

nicstat Tool

181

(1)

SNMP

182

(1)

checkcable Tool

183

(2)

ping Tool

185

(1)

traceroute Tool

185

(1)

snoop Tool

186

(1)

TTCP

186

(1)

pathchar Tool

187

(1)

ntop Tool

187

(1)

NFS Client Statistics: nfsstat -c

188

(1)

NFS Server Statistics: nfsstat -s

188

(1)

Per-Process Network Statistics

189

(2)

tcptop Tool

189

(2)

tcpsnoop Tool

191

(1)

TCP Statistics

191

(5)

TCP Statistics Internals

192

(1)

TCP Statistics from Kstat

193

(1)

TCP Statistics Reference

194

(1)

TCP Statistics from DTrace

195

(1)

IP Statistics

196

(3)

IP Statistics Internals

197

(1)

IP Statistics from Kstat

197

(1)

IP Statistics Reference

197

(2)

IP Statistics from DTrace

199

(1)

ICMP Statistics

199

(4)

ICMP Statistics from Kstat

199

(1)

ICMP Statistics Reference

200

(1)

ICMP Statistics from DTrace

201

(1)

Tracing Raw Network Functions

201

(2)

Performance Counters

203

(18)

Introducing CPU Caches

203

(3)

cpustat Command

206

(9)

Cache Hit Ratio, Cache Misses

206

(2)

Listing PICs and Events

208

(1)

PIC Examples: UltraSPARC IIi

209

(1)

PIC Examples: The UltraSPARC T1 Processor

210

(2)

Event Multiplexing

212

(1)

Using cpustat with Multiple CPUs

213

(1)

Cycles per Instruction

213

(1)

PIC Examples: UltraSPARC IV

214

(1)

cputrack Command

215

(1)

busstat Command

216

(5)

Listing Supported Buses

216

(1)

Listing Bus Events

217

(1)

Monitoring Bus Events

218

(1)

Event Multiplexing

219

(1)

Example: UltraSPARC T1

220

(1)

Kernel Monitoring

221

(12)

Tools for Kernel Monitoring

221

(1)

Profiling the Kernel and Drivers

222

(1)

Profiling the Kernel with lockstat -I

222

(1)

Analyzing Kernel Locks

223

(4)

Adaptive Locks

224

(1)

Spin Locks

224

(1)

Reader/Writer Locks

224

(1)

Thread Locks

225

(1)

Analyzing Locks with lockstat

225

(2)

DTrace lockstat Provider

227

(2)

DTrace Kernel Profiling

229

(1)

Interrupt Statistics: vmstat -i

230

(1)

Interrupt Analysis: intrstat

230

(3)

PART TWO Observability Infrastructure

233

(92)

Dynamic Tracing

235

(60)

Introduction to DTrace

235

(1)

The Basics

236

(21)

D Program Structure

239

(1)

Providers and Probes

240

(2)

Aggregations

242

(3)

Variables

245

(3)

Probe Arguments

248

(3)

Mixing Providers

251

(2)

Accessing Global Kernel Data

253

(2)

Assorted Actions of Interest

255

(2)

Inspecting Java Applications with DTrace

257

(8)

Inspecting Applications with the DTrace jstack Action

262

(2)

Adding Probes to Pre-Mustang Releases

264

(1)

DTrace Architecture

265

(6)

Summary

271

(1)

Probe Reference

271

(23)

The I/O Provider

271

(5)

Virtual Memory Provider Probes

276

(2)

The Sched Provider

278

(4)

DTrace Lockstat Provider

282

(4)

The Java Virtual Machine Provider

286

(8)

MDB Reference

294

(1)

Kernel Statistics

295

(30)

C-Level Kstat Interface

295

(12)

Data Structure Overview

296

(2)

Getting Started

298

(1)

Data Types

299

(4)

Kstat Names

303

(1)

Functions

304

(1)

Management of Chain Updates

305

(1)

Putting It All Together

306

(1)

Command-Line Interface

307

(4)

The kstat Command

308

(1)

Real-World Example That Uses kstat and nawk

309

(2)

Using Perl to Access kstats

311

(6)

The Tied-Hash Interface to the kstat Facility

312

(1)

The update () Method

312

(1)

64-Bit Values

313

(1)

Getting Started with Perl

313

(1)

netstatMulti Implemented in Perl

314

(3)

Snooping a Program's kstat Use with DTrace

317

(1)

Adding Statistics to the Solaris Kernel

317

(6)

A kstat Provider Walkthrough

318

(3)

I/O Statistics

321

(2)

Additional Information

323

(2)

PART THREE Debugging

325

(104)

The Modular Debugger

327

(8)

Introduction to the Modular Debugger

327

(3)

MDB

328

(1)

MDB Features

328

(1)

Terms

329

(1)

MDB Concepts

330

(5)

Building Blocks

330

(1)

Targets

330

(1)

Debugger Commands

331

(1)

Walker

331

(1)

Debugger Modules

331

(1)

Macros

332

(1)

Modularity

332

(3)

An MDB Tutorial

335

(32)

Invoking MDB

335

(1)

Logging Output to a File

336

(1)

MDB Command Syntax

336

(17)

Expressions

338

(2)

Symbols

340

(1)

Formatting Metacharacters

341

(1)

Formatting Characters

341

(3)

dcmds

344

(2)

Walkers

346

(1)

Macros

347

(1)

Pipelines

347

(1)

Piping to UNIX Commands

348

(1)

Obtaining Symbolic Type Information

349

(2)

Variables

351

(1)

Walkers, Variables, and Expressions Combined

352

(1)

Working with Debugging Targets

353

(4)

Displaying Stacks

355

(1)

Displaying Registers

355

(1)

Disassembling the Target

356

(1)

Setting Breakpoints

356

(1)

GDB-to-MDB Reference

357

(2)

dcmd and Walker Reference

359

(8)

Commands

359

(1)

Comments

359

(1)

Expressions

360

(1)

Symbols

360

(1)

dcmds

360

(1)

Variables

360

(1)

Read Formats

361

(1)

Write Formats

361

(1)

Search Formats

361

(1)

General dcmds

362

(1)

Target-Related dcmds

362

(1)

CTF-Related

362

(1)

Kernel: proc-Related

363

(1)

Kernel: Thread-Related

363

(1)

Kernel: Synchronization-Related

363

(1)

Kernel: CPU-Related

363

(1)

Kernel: Memory-Related

364

(1)

Kernel: kmem-Related

364

(1)

Process: Target-Related

364

(1)

Kernel: kmdb-Related

365

(2)

Debugging Kernels

367

(62)

Working with Kernel Cores

367

(15)

Locating and Attaching the Target

367

(1)

Examining Kernel Core Summary Information

368

(1)

Examining the Message Buffer

369

(1)

Obtaining a Stack Trace of the Running Thread

369

(1)

Which Process?

370

(1)

Disassembling the Suspect Code

371

(1)

Displaying General-Purpose Registers

372

(1)

Navigating the Stack Backtrace

372

(3)

Looking at the Status of the CPUs

375

(2)

Traversing Stack Frames in SPARC Architectures

377

(1)

Listing Processes and Process Stacks

378

(2)

Global Memory Summary

380

(1)

Listing Network Connections

380

(1)

Listing All Kernel Threads

381

(1)

Other Notable Kernel dcmds

381

(1)

Examining User Process Stacks within a Kernel Image

382

(3)

Enabling Process Pages in a Dump

383

(1)

Invoking MDB to Examine the Kernel Image

383

(1)

Locating the Target Process

383

(1)

Extracting the User-Mode Stack Frame Pointers

384

(1)

Switching MDB to Debug a Specific Process

385

(3)

Constructing the Process Stack

385

(2)

Examining the Process Memory

387

(1)

kmdb, the Kernel Modular Debugger

388

(7)

Diagnosing with kmdb and moddebug

388

(1)

Booting with the Kernel Debugger

389

(1)

Configuring a tty Console on x86

390

(1)

Investigating Hangs

390

(1)

Collecting Information about Panics

391

(1)

Working with Debugging Targets

392

(1)

Setting Breakpoints

393

(1)

Forcing a Crash Dump with halt -d

394

(1)

Forcing a Dump with kmdb

394

(1)

Kernel Built-In MDB dcmds

395

(6)

APPENDICES

Appendix A Tunables and Settings

401

(6)

Appendix B DTrace One-Liners

407

(2)

Appendix C Java DTrace Scripts

409

(4)

Appendix D Sample Perl Kstat Utilities

413

(16)

Bibliography

429

(4)

Index

433

Welcome to the second edition ofSolaris Internalsand its companion volume,Solaris Performance and Tools. It has been almost five years since the release of the first edition, during which time we have had the opportunity to communicate with a great many Solaris users, software developers, system administrators, database administrators, performance analysts, and even the occasional kernel hacker. We are grateful for all the feedback, and we have made specific changes to the format and content of this edition based on reader input. Read on to learn what is different. We look forward to continued communication with the Solaris community. About These Books These books are about the internals of Sun's Solaris Operating Systemspecifically, the SunOS kernel. Other components of Solaris, such as windowing systems for desktops, are not covered. The first edition ofSolaris Internalscovered Solaris releases 2.5.1, 2.6, and Solaris 7. These volumes focus on Solaris 10, with updated information for Solaris 8 and 9. In the first edition, we wanted not only to describe the internal components that make the Solaris kernel tick, but also to provide guidance on putting the information to practical use. These same goals apply to this work, with further emphasis on the use of bundled (and in some cases unbundled) tools and utilities that can be used to examine and probe a running system. Our ability to illustrate more of the kernel's inner workings with observability tools is facilitated in no small part by the inclusion of some revolutionary and innovative technology in Solaris 10DTrace, a dynamic kernel tracing framework. DTrace is one of many new technologies in Solaris 10, and is used extensively throughout this text. In working on the second edition, we enlisted the help of several friends and colleagues, many of whom are part of Solaris kernel engineering. Their expertise and guidance contributed significantly to the quality and content of these books. We also found ourselves expanding topics along the way, demonstrating the use of dtrace(1) , mdb(1) , kstat(1) , and other bundled tools. So much so that we decided early on that some specific coverage of these tools was necessary, and chapters were written to provide readers with the required background information on the tools and utilities. From this, an entire chapter on using the tools for performance and behavior analysis evolved. As we neared completion of the work, and began building the entire manuscript, we ran into a bit of a problemthe size. The book had grown to over 1,500 pages. This, we discovered, presented some problems in the publishing and production of the book. After some discussion with the publisher, it was decided we should break the work up into two volumes. Solaris Internals.This represents an update to the first edition, including a significant amount of new material. All major kernel subsystems are included: the virtual memory (VM) system, processes and threads, the kernel dispatcher and scheduling classes, file systems and the virtual file system (VFS) framework, and core kernel facilities. New Solaris facilities for resource management are covered as well, along with a new chapter on networking. New features in Solaris 8 and Solaris 9 are called out as appropriate throughout the text. Examples of Solaris utilities and tools for performance and analysis work, described in the companion volume, are used throughout the text. Solaris Performance and Tools.This book contains chapters on the tools and utilities bundled with Solaris 10: dtrace(1) , mdb(1) , kstat(1) , etc. There are also extensive chapters on using the tools to analyze the performance andbehavior of a Solaris system. The two texts a