Linux Troubleshooting for System Administrators and Power Users

by Kirkland, James; Carmichael, David; Tinker, Christopher L.; Tinker, Gregory L.

• ISBN13: 9780131855151
• ISBN10: 0131855158
• Edition: 1st
• Format: Paperback
• Copyright: 2006-04-27
• Publisher: Prentice Hall

Summary

Provides system administrators and end users with the tools and techniques to effectively troubleshoot Linux system problems.

Author Biography

James Kirkland is a Senior Consultant for Racemi David Carmichael works for Hewlett-Packard as a Technical Problem Manager in Alpharetta, Georgia Chris Tinker is a Senior Software Business Recovery Specialist for HP Greg Tinker's primary role is as a Storage Business Recovery Specialist

Table of Contents

Preface	xvii
System Boot, Startup, and Shutdown Issues	1	(50)
Bootloaders	2	(20)
GRUB	2	(4)
Editing the Menu Choices with GRUB	6	(9)
LILO	15	(5)
Booting When GRUB or LILO Doesn't Work	20	(2)
The init Process and/etc/inittab File	22	(8)
Startup in Multiuser Mode	26	(3)
init Errors	29	(1)
rc Scripts	30	(11)
Confirm Mode	36	(1)
Startup Problems in re Scripts	37	(4)
Fixing Problems with the Root Filesystem	41	(7)
Booting from a Second Hard Disk	42	(1)
Booting from a Rescue CD	42	(3)
Reset Lost Root Password Using a Knoppix CD	45	(1)
Reinstall GRUB Using Knoppix CD	46	(2)
Booting from a Floppy Rescue Disk	48	(1)
Summary	48	(1)
Endnotes	49	(2)
System Hangs and Panics	51	(28)
OS Hangs	52	(14)
Troubleshooting Interruptible Hangs	53	(11)
Troubleshooting Non-Interruptible Hangs	64	(2)
OS Panics	66	(8)
Troubleshooting OS Panics	67	(4)
Troubleshooting Panics Resulting from Oops	71	(3)
Hardware Machine Checks	74	(1)
Summary	75	(4)
Performance Tools	79	(28)
top	80	(9)
Adding and Removing Fields	81	(2)
Output Explained	83	(2)
Saving Customization	85	(2)
Batch Mode	87	(2)
sar	89	(10)
sar Data Collector	89	(2)
CPU Statistics	91	(3)
Disk I/O Statistics	94	(1)
Networking Statistics	95	(4)
vmstat	99	(3)
iostat	102	(3)
free	105	(1)
Summary	105	(1)
Endnotes	105	(2)
Performance	107	(52)
Start Troubleshooting at the Lowest Layer Possible	107	(24)
Binding a Raw Device to a Block Device Using the raw Command	109	(1)
Raw Device Performance	110	(1)
Using the dd Command to Determine Sequential I/O Speed	111	(1)
Using sar and iostat to Measure Disk Performance	112	(2)
Understanding the Importance of I/O Block Size When Testing Performance	114	(7)
Multiple Threads (Processes) of I/O to a Disk	121	(1)
Using a Striped Ivol to Reduce Disk I/O Strain	122	(4)
Multipath I/O	126	(5)
Filesystems	131	(26)
Journaling to a Separate Disk	131	(2)
Determining I/O Size for Filesystem Requests	133	(1)
Loading a Filesystem with Small Block I/O Transfers	134	(4)
Utilizing Key Benefits of a Filesystem	138	(1)
Linux and Windows Performance and Tuning Sector Alignments	139	(2)
Performance Tuning and Benchmarking Using bonnie++	141	(3)
Assessing Application CPU Utilization Issues	144	(5)
Using Oracle statspak	149	(3)
Troubleshooting ``No Space Left on Device'' Errors When Allocating Shared Memory	152	(2)
Additional Performance Tools	154	(1)
isag	155	(2)
Summary	157	(2)
Adding New Storage via SAN with Reference to PCMCIA and USB	159	(26)
Configuration	160	(1)
Kernel Module	161	(9)
Adding LUNs via PCI	170	(11)
Adding Storage via PCMCIA/USB	181	(3)
Summary	184	(1)
Disk Partitions and Filesystems	185	(44)
Background	186	(3)
IDE and SCSI	186	(1)
Bit Calculation	186	(3)
Partition Table/Master Boot Record: Location	189	(3)
Partition Table/Master Boot Record: CHS Addressing	192	(7)
Defining a Primary Partition	192	(3)
Determining Whether Additional Partitions Can Be Created	195	(4)
Partition Table/Master Boot Record: Logical/Extended	199	(4)
Partition Table/Master Boot Record: Logical Block Addressing (LBA)	203	(2)
Partition Table/Master Boot Record: Bootloader	205	(6)
Byte Review on a Used Drive	208	(2)
BIOS Initializing the Bootloader	210	(1)
Partition Table/Master Boot Record: Backup	211	(12)
Partition Recovery Walkthrough	212	(2)
Demonstrating a Failure	214	(2)
Mounting a Partition	216	(1)
Recovering Superblock and Inode Table on ext Filesystems	217	(6)
Further Scenarios	223	(5)
Summary	228	(1)
Device Failure and Replacement	229	(24)
Supported Devices	229	(2)
Where to Look for Errors	231	(4)
Identifying Failed Devices	235	(10)
Replacement of a Failed Device	245	(7)
Summary	252	(1)
Linux Processes: Structure, Hangs, and Core Dumps	253	(32)
Process Structure and Life Cycle	253	(5)
Process/Task Overview	254	(1)
Process Relationships	255	(1)
Linux Process Creation	255	(3)
Linux Process Termination	258	(1)
Linux Threads	258	(9)
Identifying Threads	261	(6)
Identifying Process Hangs	267	(8)
Process Cores	275	(9)
Signals	275	(4)
Limits	279	(2)
The Core File	281	(3)
Summary	284	(1)
Backup/Recovery	285	(30)
Media	285	(13)
Magnetic Tape	286	(11)
Optical Disk	297	(1)
Hard Disk	298	(1)
Backup Scope	298	(2)
Basic Backup and Recovery Commands	300	(8)
tar	300	(3)
cpio	303	(2)
dump and Restore	305	(1)
dd	306	(1)
mkisofs	307	(1)
rsync	307	(1)
Bare Metal Recovery	308	(1)
I Have a Tape, and I Don't Know What It Is...	309	(1)
How Can I Tell Whether My Tape Problem Is Hardware or Software?	309	(4)
Summary	313	(2)
cron and at	315	(30)
cron	316	(13)
cron Daemon	324	(3)
kcron	327	(2)
anacron	329	(3)
at	332	(3)
at Daemon and atrun	333	(2)
Troubleshooting cron	335	(9)
Methodology	342	(2)
Summary	344	(1)
Printing and Printers	345	(38)
What Is a Spooler?	345	(13)
Using the Spooler Commands	347	(3)
Spooler Plumbing	350	(4)
Term Definitions	354	(4)
Printer Types	358	(2)
Connectivity Types	360	(19)
Local Serial Printing	360	(5)
Local USB Printing	365	(4)
Local Parallel Printing	369	(1)
Remote Printing	369	(6)
Raw Network Socket Printing	375	(2)
IPP	377	(2)
Terminal Servers (Networked Serial)	379	(1)
Page Description Languages	379	(1)
General Printing Troubleshooting	380	(2)
Mapping the Spooling Environment	380	(1)
Breakpoints	381	(1)
Summary	382	(1)
System Security	383	(40)
What Is System Security?	383	(6)
Host Versus Network Security	383	(1)
What Is Vulnerability?	384	(1)
Classes of Host Vulnerabilities	385	(2)
Types of Vulnerabilities and Exposures	387	(1)
General Steps to Increase Host Security	388	(1)
Prevention	389	(32)
Encryption with SSH	390	(13)
System Hardening with netfilter/iptables	403	(18)
Patching	421	(1)
Recovery After Being Hacked	421	(1)
Summary	421	(2)
Network Problems	423	(72)
An Introduction to the OSI and TCP/IP Layers	423	(2)
Troubleshooting the Network Layers	425	(68)
Troubleshooting the TCP/IP Physical Network Access Layer	426	(17)
Troubleshooting the Network Layer (OSI Third Layer, TCP/IP Second Layer)	443	(25)
Troubleshooting at the Transport Layer (TCP and UDP)	468	(24)
Troubleshooting at the Application Level: The Final Layer of the TCP/IP Model	492	(1)
Summary	493	(2)
Login Problems	495	(32)
/etc/passwd,/etc/shadow, and Password Aging	496	(11)
/etc/passwd and /etc/shadow	496	(2)
change, passwd, and usermod	498	(8)
/etc/passwd and /etc/shadow Corruption	506	(1)
pwck	507	(1)
Login Failures Due to Linux Configuration	507	(3)
/etc/securetty	508	(1)
/etc/nologin	509	(1)
PAM	510	(11)
Function	511	(1)
Priority	511	(1)
Module Name	512	(1)
Arguments	512	(1)
/etc/pam.d	512	(1)
/etc/pam.conf	513	(1)
/lib/security	514	(1)
Linux-PAM Resources	515	(1)
Troubleshooting PAM	516	(3)
Validate Modules	519	(1)
Bugs in PAM	519	(2)
Shell Problems	521	(1)
User Profile Files	521	(1)
Password Problems	522	(3)
Summary	525	(1)
Endnote	525	(2)
X Windows Problems	527	(1)
X Background	527	(2)
X Components	529	(13)
X Server Component	529	(11)
X Client Component	540	(2)
X Display Manager	542	(2)
X Desktop Manager (Environment)	544	(1)
X Troubleshooting Scenarios	544	(4)
Summary	548	(1)
Endnotes	549

Excerpts

Preface Preface My good friend, James Kirkland, sent me an instant message one day asking if I wanted to write a Linux troubleshooting book with him. James has been heavily involved in Linux at the HP Response Center for several years. While troubleshooting Linux issues for customers, he realized there was not a good troubleshooting reference available. I remember a meeting discussing Linux troubleshooting. Someone asked what the most valuable Linux troubleshooting tool was. The answer was immediate. Google. If you have ever spent time trying to find a solution for a Linux problem, you know what that engineer was talking about. A wealth of great Linux information can be found on the Internet, but you can't always rely on this strategy. Some of the Linux information is outdated. A lot of it can't be understood without a good foundation of subject knowledge, and some of it is incorrect. We wanted to write this book so the Linux administrator will know how Linux works and how to approach and resolve common issues. This book contains the information we wish we had when we started troubleshooting Linux. Greg and Chris are identical twins and serious Linux hobbyists. They have been Linux advocates within HP for years. Yes, they both run Linux on their laptops. Chris is a member of the Superdome Server team ( http://www.hp.com/products1/servers/scalableservers/superdome/index.html ). Greg works for the XP storage team ( http://h18006.www1.hp.com/storage/xparrays.html ). Their Linux knowledge is wide and deep. They have worked through SAN storage issues and troubleshot process hangs, Linux crashes, performance issues, and everything else for our customers, and they have put their experience into the book. I am a member of the HP escalations team. I've primarily spent my time resolving HPUX issues. I've been a Linux hobbyist for a few years, and I've started working Linux escalations, but I'm definitely newer to Linux than the rest of the team. I try to give the book the perspective of someone who is fairly new to Linux. I tried to remember the questions I had when I first started troubleshooting Linux issues and included them in the book. We sincerely hope our effort is helpful to you. --Dave Carmichael Chapter Summaries These chapter summaries will give you an idea of how the book is organized and a bit of an overview of the content of each chapter. Chapter 1: System Boot, Startup, and Shutdown Issues Chapter 1 discusses the different subsystems that comprise Linux startup. These include the bootloaders GRUB and LILO, the init process, and the rc startup and shutdown scripts. We explain how GRUB and LILO work along with the important features of each. The reader will learn how to boot when there are problems with the bootloader. There are numerous examples. We explain how init works and what part it plays in starting Linux. The rc scripts are explained in detail as well. The reader will learn how to boot to single user mode, emergency mode, and confirm mode. Examples are included of using a recovery CD when Linux won't boot from disk. Chapter 2: System Hangs and Panics This chapter explains interruptible and non-interruptible OS hangs, kernel panics, and IA64 hardware machine checks. A Linux hang takes one of two forms. An interruptible hang is when Linux seems frozen but does respond to some events, such as a ping request. Non-interruptible hangs do not respond to any actions. We show how to use the Magic SysReq keystroke to generate a stack trace to troubleshoot an interruptible hang. We explain how to force a panic when Linux is in a non-