The Practice of System and Network Administration

by Limoncelli, Thomas A.; Hogan, Christina J.

• ISBN13: 9780201702712
• ISBN10: 0201702711
• Edition: 1st
• Format: Paperback
• Copyright: 2002-01-01
• Publisher: Addison-Wesley Professional

Summary

"Your organization needs this book!" --Peter Salus, Chief Knowledge Officer, Matrix.Net, "The Bookworm"This book describes the best practices of system and network administration, independent of specific platforms or technologies. It features six key principles of site design and support practices: simplicity, clarity, generality, automation, communication, and basics first. It examines the major areas of responsibility for system administrators within the context of these principles. The book also discusses change management and revision control, server upgrades, maintenance windows, and service conversions. You will find experience-based advice on topics such as: bull; bull;The key elements your networks/systems need that will make all other services run better bull;Building and running reliable, scalable services, including email, printing, and remote access bull;Creating security policies and enforcing them bull;Upgrading thousands of hosts without creating havoc bull;Planning for and performing flawless scheduled maintenance windows bull;Superior helpdesks, customer care, and avoiding the temporary fix trap bull;Building data centers that prevent problems bull;Designing networks for speed and reliability bull;Email scaling and security issues bull;Why building a backup system isn't about backups bull;Monitoring what you have and predicting what you will need bull;How to stay technical and how not to be pushed into management And there's more! When was the last time you read a book that dealt with: bull; bull;Real-world technical management issues, including morale, organization building, coaching, maintaining positive visibility, and communicating with nontechnical management bull;Personal skill techniques, including our secrets for getting more done each day, dealing with less technical people, ethical dilemmas, managing your boss, and loving your job bull;System administration salary negotiation tips--the first book that includes this topic! Chapters are divided into The Basics and The Icing. The Basics are those key elements that, when done right, make every other aspect of the job easier. Things like starting all new hosts with the same configuration and picking the right things to automate first. The Icing sections contain all those powerful things that can be done on top of the basics to wow customers and managers. Do the basics first. The icing is a vision for the future that usually only comes with decades of experience. 0201702711B07232001

Table of Contents

Preface	xix
Acknowledgments	xxvii
About the Authors	xxix
Introduction	xxxi
Do These Now!	xxxii
Use a Trouble-Ticket System	xxxii
Manage Quick Requests Right	xxxii
Start Every New Host in a Known State	xxxiv
Conclusion	xxxv
PART I The Principles	1	(192)
Desktops	3	(26)
The Basics	7	(17)
Loading the System Software and Applications Initially	8	(6)
Updating the System Software and Applications	14	(3)
Network Configuration	17	(4)
Dynamic DNS with DHCP	21	(3)
The Icing	24	(2)
High Confidence in Completion	24	(1)
Involve Customers in the Standardization Process	25	(1)
A Variety of Standard Configurations	25	(1)
Conclusion	26	(3)
Exercises	27	(2)
Servers	29	(20)
The Basics	29	(12)
Buy Server Hardware for Servers	30	(1)
Vendors Known for Reliable Products	31	(1)
Does Server Hardware Really Cost More?	32	(1)
Maintenance Contracts and Spare Parts	33	(2)
Data Backups	35	(1)
Servers Live in the Data Center	36	(1)
Same, Different, or a Stripped-Down OS on Clients	36	(1)
Remote Administration Access	37	(2)
Mirrored Root Disks	39	(2)
The Icing	41	(4)
Server Appliances	41	(1)
Redundant Power Supplies	41	(2)
Full and n + 1 Redundancy	43	(1)
Hot-swap Components	43	(1)
Separate Networks for Administrative Functions	44	(1)
Opposing View: Many Inexpensive Workstations	45	(1)
Conclusion	46	(3)
Exercises	46	(3)
Services	49	(30)
The Basics	50	(22)
Customer Requirements	52	(1)
Operational Requirements	53	(3)
Open Architecture	56	(4)
Simplicity	60	(1)
Vendor Relations	61	(1)
Machine Independence	61	(1)
Environment	62	(1)
Restricted Access	63	(1)
Reliability	64	(3)
Single or Multiple Servers	67	(1)
Centralization and Standards	67	(1)
Performance	68	(3)
Monitoring	71	(1)
Service Rollout	71	(1)
The Icing	72	(3)
Dedicated Machines	72	(1)
Full Redundancy	73	(2)
Conclusion	75	(4)
Exercises	76	(3)
Debugging	79	(12)
The Basics	80	(6)
Learn the Customer's Problem	80	(1)
Find the Problem's Cause and Fix It	81	(2)
Have the Right Tools	83	(3)
The Icing	86	(3)
Better Tools	86	(1)
Formal Training on the Tools	87	(1)
End-to-End Understanding of the System	87	(2)
Conclusion	89	(2)
Exercises	89	(2)
Fixing Things Once	91	(10)
The Basics	92	(6)
Fix Things Once, Rather Than Over and Over	92	(2)
Avoid the Temporary Fix Trap	94	(2)
Learning from Carpenters	96	(2)
The Icing	98	(1)
Conclusion	99	(2)
Exercises	100	(1)
Namespaces	101	(16)
The Basics	102	(12)
Namespaces Need Policies	102	(10)
Namespaces Need Change Procedures	112	(1)
Namespace Management Should Be Centralized	113	(1)
The Icing	114	(2)
One Huge Database That Drives Everything	114	(1)
Further Automation	114	(1)
Customers Do Many of the Updates	115	(1)
``Next-Level'' Namespace Ubiquity	115	(1)
Conclusion	116	(1)
Exercises	116	(1)
Security Policy	117	(46)
The Basics	119	(36)
Build Security Using a Solid Infrastructure	120	(1)
Ask the Right Questions	121	(3)
Document the Company's Security Policies	124	(6)
Basics for the Technical Staff	130	(12)
Management and Organizational Issues	142	(13)
The Icing	155	(3)
Make Security Pervasive	156	(1)
Stay Up-to-Date: Contacts and Technologies	157	(1)
Produce Metrics	157	(1)
Organization Profiles	158	(3)
Small Company	158	(1)
Medium-Size Company	159	(1)
Large Company	159	(1)
E-commerce Site	160	(1)
University	160	(1)
Conclusion	161	(2)
Exercises	162	(1)
Disaster Recovery and Data Integrity	163	(12)
The Basics	164	(6)
What Is a Disaster?	164	(1)
Risk Analysis	164	(2)
Legal Obligations	166	(1)
Damage Limitation	166	(1)
Preparation	167	(2)
Data Integrity	169	(1)
The Icing	170	(2)
Redundant Site	170	(1)
Security Disasters	171	(1)
Media Relations	171	(1)
Conclusion	172	(3)
Exercises	172	(3)
Ethics	175	(18)
The Basics	175	(12)
Informed Consent	176	(1)
Professional Code of Conduct	176	(2)
Network/Computer User Code of Conduct	178	(1)
Privileged Access Code of Conduct	179	(2)
Copyright Adherence	181	(1)
Working with Law Enforcement	182	(5)
The Icing	187	(2)
Setting Expectations on Privacy and Monitoring	187	(1)
Being Told to Do Something Illegal/Unethical	188	(1)
Conclusion	189	(4)
Exercises	190	(3)
PART II The Processes	193	(90)
Change Management and Revision Control	195	(18)
The Basics	196	(9)
Technical Issues	196	(4)
Communications Structure	200	(1)
Scheduling	201	(3)
Process and Documentation	204	(1)
Quiet Times	205	(1)
The Icing	205	(5)
Automated Front-Ends	205	(2)
Change Management Meetings	207	(3)
Streamline the Process	210	(1)
Conclusion	210	(3)
Exercises	210	(3)
Server Upgrades	213	(18)
The Basics	214	(10)
The Steps in Detail	214	(10)
The Icing	224	(4)
Add and Remove Services at the Same Time	224	(1)
Fresh Installs	224	(1)
Reusing the Tests	225	(1)
System Changelog	225	(1)
A Dress Rehearsal	225	(1)
Install Old and New Versions on the Same Machine	226	(1)
Minimal Changes From the Base	226	(2)
Conclusion	228	(3)
Excercises	229	(2)
Maintenance Windows	231	(24)
The Basics	233	(15)
Scheduling	234	(1)
Planning	235	(1)
Flight Director	235	(1)
Change Proposals	236	(2)
The Master Plan	238	(1)
Disabling Access	239	(1)
Mechanics and Coordination	240	(4)
Deadlines for Change Completion	244	(1)
Comprehensive System Testing	245	(1)
Postmaintenance Communication	246	(1)
Re-enable Remote Access	247	(1)
Visible Presence the Next Morning	247	(1)
Postmortem	247	(1)
The Icing	248	(2)
Mentoring a New Flight Director	248	(1)
Trending of Historical Data	248	(1)
Providing Limited Availability	249	(1)
High-Availability Sites	250	(2)
The Similarities	250	(1)
The Differences	251	(1)
Conclusion	252	(3)
Exercises	253	(2)
Service Conversions	255	(12)
The Basics	256	(7)
Small Groups First, Then Expand	256	(1)
Communication	257	(1)
Minimize Intrusiveness	257	(2)
Layers Versus Pillars	259	(1)
Avoid Flash-Cuts	260	(2)
Successful Flash-Cuts	262	(1)
Back-Out Plan	263	(1)
The Icing	263	(3)
Instant Roll-Back	264	(1)
Avoid Explicit Conversions	265	(1)
Vendor Support	266	(1)
Conclusion	266	(1)
Exercises	266	(1)
Centralization and Decentralization	267	(16)
The Basics	268	(8)
Guiding Principles	268	(3)
Candidates for Centralization	271	(3)
Candidates for Decentralization	274	(2)
The Icing	276	(5)
Consolidate Purchasing	276	(2)
Outsourcing	278	(3)
Conclusion	281	(2)
Exercises	281	(2)
PART III The Practices	283	(240)
Helpdesks	285	(16)
The Basics	286	(9)
Have a Helpdesk	286	(2)
A Friendly Face	288	(1)
Staff Sizing	288	(1)
Defined Scope of Coverage	289	(3)
Defined Processes for Staff	292	(1)
An Escalation Process	292	(1)
Helpdesk Software	293	(2)
The Icing	295	(3)
Statistical Improvements	295	(1)
Out of Hours and 24 x 7 Coverage	296	(1)
Better Advertising for the Helpdesk	297	(1)
Different ``Desks'' for Service Provision Versus Problem Resolution	297	(1)
Conclusion	298	(3)
Exercises	299	(2)
Customer Care	301	(24)
The Basics	303	(14)
Ticket Tracking Software	304	(1)
Phase A: The Greeting	304	(1)
Phase B: Problem Identification (``What's Wrong?'')	305	(5)
Phase C: Planning and Execution (``Fix It'')	310	(3)
Phase D: Verification (``Verify It'')	313	(2)
Perils of Skipping a Step	315	(2)
Team of One	317	(1)
The Icing	317	(4)
Training Based on the Model	317	(1)
The Single Point of Contact	317	(1)
Increasing Customer Familiarity	318	(1)
Special Announcements for Major Outages	318	(1)
Trend Analysis	319	(1)
Customers That Know the Process	320	(1)
Architectural Decisions That Match the Process	321	(1)
Conclusion	321	(4)
Exercises	323	(2)
Data Centers	325	(48)
The Basics	326	(36)
Picking a Location	327	(1)
Access	328	(1)
Security	329	(2)
Power and Air	331	(9)
Fire Suppression	340	(1)
Racks	341	(7)
Wiring	348	(6)
Labeling	354	(2)
Communication	356	(1)
Console Servers	356	(3)
Workbench	359	(1)
Tools and Supplies	359	(2)
Parking Spaces	361	(1)
The Icing	362	(2)
Greater Redundancy	362	(2)
More Space	364	(1)
Ideal Data Centers	364	(6)
Tom's Dream Data Center	365	(3)
Christine's Dream Data Center	368	(2)
Conclusion	370	(3)
Exercises	371	(2)
Networks	373	(32)
The Basics	374	(26)
The OSI Model	374	(1)
Clean Architecture	375	(1)
Network Topologies	376	(7)
Intermediate Distribution Frame	383	(6)
Main Distribution Frame	389	(2)
Demarcation Points	391	(1)
Documentation	391	(2)
Simple Host Routing	393	(2)
Use Network Devices	395	(1)
Overlay Networks	396	(1)
Number of Vendors	397	(1)
Standards-Based Protocols	398	(1)
Monitoring	398	(1)
Single Administrative Domain	399	(1)
The Icing	400	(2)
Leading-Edge Versus Reliability	401	(1)
Multiple Administrative Domains	401	(1)
Conclusion	402	(3)
Exercises	403	(2)
Email Service	405	(20)
The Basics	405	(13)
Privacy Policy	406	(1)
Namespaces	406	(2)
Reliability	408	(1)
Simplicity	409	(2)
Generality	411	(1)
Automation	412	(1)
Basic Monitoring	413	(1)
Redundancy	413	(1)
Scaling	414	(2)
Security Issues	416	(1)
Communication	417	(1)
The Icing	418	(4)
Encryption	418	(1)
Backup Policy	419	(1)
Advanced Monitoring	420	(1)
High-Volume List Processing	420	(2)
Conclusion	422	(3)
Exercises	422	(3)
Print Service	425	(16)
The Basics	426	(9)
Select the Level of Centralization	426	(2)
Print Architecture Policy	428	(3)
Designing the System	431	(1)
Documentation	432	(1)
Monitoring	433	(1)
Environmental Issues	434	(1)
The Icing	435	(3)
Automatic Fail-Over and Load Balancing	435	(1)
Dedicated Clerical Support	436	(1)
Shredding	436	(1)
Dealing with Printer Abuse	437	(1)
Conclusion	438	(3)
Exercises	438	(3)
Backup and Restore	441	(32)
The Basics	442	(21)
Three Reasons for Restores	443	(6)
The Backup Schedule	449	(6)
Time and Capacity Planning	455	(1)
Consumables Planning	456	(2)
The Restore Process	458	(1)
Backup Automation	459	(2)
Centralization	461	(1)
Tape Inventory	462	(1)
The Icing	463	(6)
Firedrills	463	(1)
Backup Media and Off-Site Storage	464	(3)
High DB Availability	467	(1)
Technology Changes	468	(1)
Conclusion	469	(4)
Exercises	471	(2)
Remote Access Service	473	(14)
The Basics	474	(8)
Remote Access Requirements	474	(2)
Define a Remote Access Policy	476	(1)
Define Service Levels	476	(2)
Centralization	478	(1)
Outsourcing	478	(3)
Authentication	481	(1)
Perimeter Security	481	(1)
The Icing	482	(2)
Home Office	482	(1)
Cost Analysis and Reduction	483	(1)
New Technologies	484	(1)
Conclusion	484	(3)
Exercises	485	(2)
Software Depot Service	487	(20)
The Basics	489	(12)
Understand the Justification	489	(1)
Understand the Technical Expectations	490	(1)
Set the Policy	491	(1)
Selecting Depot Software	492	(1)
Create the Process Manual	492	(1)
A Unix Example	493	(5)
A Windows Example	498	(3)
The Icing	501	(3)
Different Configurations for Different Hosts	501	(1)
Local Replication	501	(1)
Including Commercial Software in the Depot	502	(1)
Handling Second-Class Citizens	503	(1)
Conclusion	504	(3)
Exercises	504	(3)
Service Monitoring	507	(16)
The Basics	508	(7)
Historical Data	509	(1)
Real-Time Monitoring	510	(5)
The Icing	515	(5)
Accessibility	515	(1)
Pervasive Monitoring	516	(1)
Device Discovery	516	(1)
End-to-End Tests	517	(1)
Application Response Time Monitoring	518	(1)
Scaling	518	(2)
Conclusion	520	(3)
Exercises	521	(2)
PART IV Management	523	(162)
Organizational Structures	525	(22)
The Basics	526	(14)
Sizing	526	(3)
Cost Centers	529	(2)
Management Chain	531	(2)
Appropriate Skills	533	(1)
Infrastructure Teams	534	(2)
Customer Support	536	(2)
Helpdesk	538	(1)
Outsourcing	538	(2)
The Icing	540	(1)
Consultants and Contractors	540	(1)
Sample Organizational Structures	541	(3)
Small Company	541	(1)
Medium Company	541	(1)
Large Company	542	(1)
E-commerce Site	542	(1)
Universities and Non-Profit Organizations	543	(1)
Conclusion	544	(3)
Exercises	545	(2)
Perception and Visibility	547	(22)
The Basics	548	(11)
A Good First Impression	548	(3)
Attitude, Perception, and Customers	551	(2)
Align Your Priorities with Customer Expectations	553	(1)
Be the System Advocate	554	(5)
The Icing	559	(7)
The System Status Web Page	560	(1)
Management Meetings	561	(1)
Be Visible	561	(1)
Town Meetings	561	(2)
Newsletters	563	(1)
Mail to All Customers	564	(1)
Lunch	565	(1)
Conclusion	566	(3)
Exercises	567	(2)
Being Happy	569	(38)
The Basics	570	(18)
Organizing for Excellent Follow-Through	570	(2)
Time Management	572	(9)
Communication Skills	581	(5)
Constant Professional Development	586	(1)
Staying Technical	587	(1)
The Icing	588	(16)
Learn To Negotiate	588	(6)
Loving Your Job	594	(6)
Managing Your Manager	600	(4)
Further Reading	604	(1)
Conclusion	604	(3)
Exercises	605	(2)
A Guide for Technical Managers	607	(30)
The Basics	607	(26)
Responsibilities	608	(12)
Working with Nontechnical Managers	620	(3)
Working with Your Employees	623	(5)
Decisions	628	(5)
The Icing	633	(2)
Make Your Team Even Stronger	633	(1)
Sell Your Department to Senior Management	634	(1)
Work on Your Own Career Growth	634	(1)
Do Something You Enjoy	634	(1)
Conclusion	635	(2)
Exercises	635	(2)
A Guide for Nontechnical Managers	637	(16)
The Basics	637	(9)
Morale	638	(2)
Communication	640	(1)
Staff Meetings	641	(1)
Look for One-Year Plans	642	(1)
Technical Staff and the Budget Process	643	(2)
Professional Development	645	(1)
The Icing	646	(5)
Have a Five-Year Vision	647	(1)
Meetings with Single Point of Contact	648	(2)
Understand the Technical Staff's Work	650	(1)
Conclusion	651	(2)
Exercises	651	(2)
Hiring System Administrators	653	(22)
The Basics	653	(19)
Job Description	654	(2)
Skill Level	656	(1)
Recruiting	656	(2)
Timing Is Everything	658	(2)
Team Considerations	660	(3)
Select the Interview Team	663	(2)
Interview Process	665	(2)
Technical Interviewing	667	(1)
Nontechnical Interviewing	668	(2)
Sell the Position	670	(1)
Employee Retention	670	(2)
The Icing	672	(1)
Get Noticed	672	(1)
Conclusion	672	(3)
Exercises	673	(2)
Firing System Administrators	675	(10)
The Basics	676	(4)
Follow Your Corporate HR Policy	676	(1)
Remove Physical Access	676	(1)
Remove Remote Access	677	(1)
Remove Service Access	677	(3)
Fewer Access Databases	680	(1)
The Icing	680	(2)
A Single Authentication Database	680	(1)
Monitoring System File Changes	681	(1)
Conclusion	682	(3)
Exercises	683	(2)
Epilogue	685	(2)
Appendix A: The Many Roles of a System Administrator	687	(26)
Appendix B: What to Do When. . .	713	(16)
Appendix C: Acronyms	729	(4)
Bibliography	733	(12)
Index	745

Excerpts

The goal of this book is to write down all the things that we've learned from our mentors and our real-world experiences. These are the things that are beyond what the manuals and the usual system administration books teach. System administrators (SAs) often find themselves swamped with work, struggling to keep the site running, and faced with requests for new technologies from their customers. Servers are overloaded or unreliable, but fixing the problem requires weeks of planning and painstakingly untangling a mess of services so that they can be moved to new machines. Hidden dependencies are lurking around every corner, and getting bitten by one can be catastrophic. In the meantime, repetitive day-to-day tasks still need to be done. The challenges seem insurmountable. Most sites grow organically, with little thought given to the big picture as each little change is implemented. Haphazardly, SAs learn about the fundamentals of good site design and support practices. They are taught by mentors, if at all, about the importance of simplicity, clarity, generality, automation, communication, and doing the basics first. These six principles are recurring themes in this book. Simplicitymeans that the smallest solution that solves the entire problem is the best solution. It keeps the systems easy to understand and reduces complex interactions between components that can cause debugging nightmares. Claritymeans that the solution is not convoluted. It can be easily explained to someone on the project or even outside the project. Clarity makes it easier to change the system, as well as to maintain and debug it. Generalitymeans that the solution solves many problems at once. Sometimes the most general solution is the simplest. It also means using vendor-independent open standard protocols that make systems more exible and make it easier to link software packages together for better services. Automationis critical. Manual processes cannot be repeated accurately nor do they scale as well as automated processes. Automation is key to easing the system administration burden, and it eliminates tedious repetitive tasks and gives SAs more time to improve services. Communicationbetween the right people can solve more problems than hardware or software. You need to communicate well with other SAs and with your customers. It is your responsibility to initiate communication. Communication ensures that everyone is working toward the same goals. Lack of communication leaves people concerned and annoyed. Communication also includes documentation: document customers needs to make sure you agree on them, document design decisions you make, document maintenance procedures. Documentation makes systems easier to maintain and upgrade. Good communication and proper documentation also make it easier to hand off projects and maintenance when you leave or take on a new role. Doing thebasics firstmeans that you build the site on strong foundations by identifying and solving the basic problems before trying to attack more advanced ones. Doing the basics first makes adding advanced features considerably easier, and it makes services more robust. A good basic infrastructure can be repeatedly leveraged to improve the site with relatively little effort. Sometimes we see SAs at other sites making a huge effort to solve a problem that wouldn't exist, or would be a simple enhancement, if the site had a basic infrastructure in place. This book will help you identify what the basics are and show you how the other five principles apply. Each chapter looks at the basics of a given area. Get the fundamentals right, and everything else will fall into place. These principles are universal. They apply at all levels of the system. They apply to physical networks and to computer hardware. They apply to all operating systems running at the site, all protocols