The Shellcoder's Handbook: Discovering and Exploiting Security Holes

by Jack Koziol; David Litchfield; Dave Aitel; Chris Anley; Sinan noir Eren; Neel Mehta; Riley Hassell

• ISBN13: 9780764544682
• ISBN10: 0764544683
• Format: Paperback
• Copyright: 2004-04-01
• Publisher: WILEY

Summary

Examines where security holes come from, how to discover them, how hackers exploit them and take control of systems on a daily basis, and most importantly, how to close these security holes so they never occur again A unique author team-a blend of industry and underground experts- explain the techniques that readers can use to uncover security holes in any software or operating system Shows how to pinpoint vulnerabilities in popular operating systems (including Windows, Linux, and Solaris) and applications (including MS SQL Server and Oracle databases) Details how to deal with discovered vulnerabilities, sharing some previously unpublished advanced exploits and techniques

Author Biography

Jack Koziol, the lead author of The Shellcoder’s Handbook, is a Senior Instructor and Security Program Manager at InfoSec Institute, a provider of advanced ethical hacking training. He regularly is called upon to train members of the United States intelligence community, military, and federal law enforcement agencies. Additionally, Jack provides training for Fortune 500 companies, such as Microsoft, HP, and Citibank, on how to better secure their networks and applications. When not teaching hacking classes, Jack regularly performs penetration tests and application security assessments for a number of clients. He has years of private vulnerability development and exploitation experience for his customers and himself.
Jack is also the author of Intrusion Detection with Snort, one of the best-selling security books in its first year of publication (2003). The book has been translated into several languages, including French and Japanese, and has received rave reviews from Linux Journal, Slashdot, and Information Security magazine.
Jack has appeared in USA Today, CNN, MSNBC, First Business, and other media outlets for his expert opinions on information security. He lives in Oak Park, Illinois, in the shadow of Frank Lloyd Wright’s home and studio, with his girlfriend Tracy and dog Quasi.

David Litchfield is the world’s leading computer security vulnerability researcher and one of the five founding members of NGSSoftware. David has discovered and published over 100 major security vulnerabilities in many different products, including most notably Apache, Microsoft Internet Information Server, Oracle, and Microsoft SQL Server. With his vast experience of network and application penetration testing, David is a permanent presenter to the Black Hat Briefings. He is also the lead author of SQL Security (Osborne/ McGraw-Hill).

Dave Aitel is the author of SPIKE and the founder of the NYC-based Internet security company Immunity, Inc. His research has incorporated exploitation of both Windows and Unix vulnerabilities, and advanced methodologies for finding new vulnerabilities.

Chris Anley is a Director of Next Generation Security Software, a U.K.-based security consulting, research, and software company. Chris is actively involved in vulnerability research and has published several white papers and security advisories on a number of products, including PGP, Windows, SQL Server, and Oracle. He splits his time evenly between research, coding, consulting, and drinking, and hopes at some point to add sleeping to the list.

Sinan Eren is a security researcher based in the Bay Area. He has done extensive work regarding exploitation of Unix vulnerabilities, developed advanced and robust methodologies for exploiting Kernel-level holes, and found many high-profile bugs in commercial and open source Unix software.

Neel Mehta works as an application vulnerability researcher at ISS X-Force, and, like many other security researchers, comes from a reverse-engineering background. His reverse-engineering experience was cultivated through extensive consulting work in the copy protection field, and has more recently been focused on application security. Neel has done extensive research into binary and source-code auditing and has applied this knowledge to find many vulnerabilities in critical and widely deployed network applications.

Riley Hassell, a Senior Researcher Engineer at eEye Digital Security, is responsible for the design and implementation of eEye Digital Security’s QA and research tool suite. He is responsible for the discovery of several highly exposed vulnerabilities released by eEye Digital Security.

Table of Contents

About the Authors	v
Credits	vii
Acknowledgments	xi
Part 1 Introduction to Exploitation: Linux on x86	1	(102)
Chapter 1 Before You Begin	3	(8)
Basic Concepts	3	(5)
Memory Management	4	(2)
Assembly	6	(7)
Registers	7	(1)
Recognizing C++ Code Constructs in Assembly	8	(2)
Conclusion	10	(1)
Chapter 2 Stack Overflows	11	(24)
Buffers	12	(1)
The Stack	13	(5)
Functions and the Stack	15	(3)
Overflowing Buffers on the Stack	18	(4)
Controlling EIP	20	(2)
Using an Exploit to Get Root Privileges	22	(7)
The Address Problem	24	(3)
The NOP Method	27	(2)
Defeating a Non-Executable Stack	29	(1)
Return to libc	30	(3)
Conclusion	33	(2)
Chapter 3 Shellcode	35	(20)
Understanding System Calls	36	(2)
Writing Shellcode for the exit() Syscall	38	(4)
Injectable Shellcode	42	(2)
Spawning a Shell	44	(9)
Conclusion	53	(2)
Chapter 4 Introduction to Format String Bugs	55	(28)
Prerequisites	55	(1)
What Is a Format String?	56	(1)
What Is a Format String Bug?	57	(5)
Format String Exploits	62	(7)
Crashing Services	63	(1)
Information Leakage	64	(5)
Controlling Execution for Exploitation	69	(10)
Why Did This Happen?	79	(1)
Format String Technique Roundup	79	(3)
Conclusion	82	(1)
Chapter 5 Introduction to Heap Overflows	83	(20)
What Is a Heap?	84	(2)
How a Heap Works	85	(1)
Finding Heap Overflows	86	(15)
Basic Heap Overflows	87	(6)
Intermediate Heap Overflows	93	(6)
Advanced Heap Overflow Exploitation	99	(1)
What to Overwrite	100	(1)
Conclusion	101	(2)
Part 2 Exploiting More Platforms: Windows, Solaris, and Tru64	103	(228)
Chapter 6 The Wild World of Windows	105	(18)
How Does Windows Differ from Linux?	105	(3)
Win32 API and PE-COFF	106	(2)
Heaps	108	(2)
Threading	109	(1)
The Genius and Idiocy of the Distributed Common Object Model and DCE-RPC	110	(8)
Recon	112	(2)
Exploitation	114	(1)
Tokens and Impersonation	114	(2)
Exception Handling under Win32	116	(2)
Debugging Windows	118	(3)
Bugs in Win32	118	(1)
Writing Windows Shellcode	119	(1)
A Hacker's Guide to the Win32 API	119	(1)
A Windows Family Tree from the Hacker's Perspective	120	(1)
Conclusion	121	(2)
Chapter 7 Windows Shellcode	123	(26)
Syntax and Filters	123	(2)
Setting Up	125	(1)
Parsing the PEB	126	(16)
Heapoverflow.c Analysis	126	(16)
Searching with Windows Exception Handling	142	(5)
Popping a Shell	147	(1)
Why You Should Never Pop a Shell on Windows	147	(2)
Conclusion	148	(1)
Chapter 8 Windows Overflows	149	(48)
Stack-Based Buffer Overflows	150	(11)
Frame-Based Exception Handlers	150	(5)
Abusing Frame-Based Exception Handling on Windows 2003 Server	155	(5)
Abusing an Existing Handler	156	(2)
Find a block of code in an address not associated with a module that will get us back to our buffer	158	(1)
Find a block of code in the address space of a module that does not have a Load Configuration Directory	159	(1)
A Final Note about Frame-Based Handler Overwrites	160	(1)
Stack Protection and Windows 2003 Server	161	(6)
Heap-Based Buffer Overflows	167	(5)
The Process Heap	167	(1)
Dynamic Heaps	167	(1)
Working with the Heap	168	(1)
How the Heap Works	168	(4)
Exploiting Heap-Based Overflows	172	(16)
Overwrite Pointer to RtlEnterCriticalSection in the PEB	172	(3)
Overwrite Pointer to First Vectored Handler at 77FC3210	175	(3)
Overwrite pointer to Unhandled Exception Filter	178	(6)
Overwrite Pointer to Exception Handler in Thread Environment Block	184	(1)
Repairing the Heap	185	(2)
Other Aspects of Heap-Based Overflows	187	(1)
COM Objects and the Heap	187	(1)
Overflowing Logic Program Control Data	188	(1)
Wrapping Up the Heap	188	(1)
Other Overflows	188	(3)
.data section overflows	188	(2)
TEB/PEB Overflows	190	(1)
Exploiting Buffer Overflows and Non-Executable Stacks	191	(5)
Conclusion	196	(1)
Chapter 9 Overcoming Filters	197	(18)
Writing Exploits for Use with an Alphanumeric Filter	197	(4)
Writing Exploits for Use with a Unicode Filter	201	(2)
What Is Unicode?	202	(1)
Converting from ASCII to Unicode	202	(1)
Exploiting Unicode-Based Vulnerabilities	203	(2)
The Available Instruction Set in Unicode Exploits	204	(1)
The Venetian Method	205	(5)
An ASCII Venetian Implementation	207	(3)
Decoder and Decoding	210	(3)
The Decoder Code	211	(1)
Getting a Fix on the Buffer Address	212	(1)
Conclusion	213	(2)
Chapter 10 Introduction to Solaris Exploitation	215	(54)
Introduction to the SPARC Architecture	216	(4)
Registers and Register Windows	216	(3)
The Delay Slot	219	(1)
Synthetic Instructions	219	(1)
Solaris/SPARC Shellcode Basics	220	(3)
Self-Location Determination and SPARC Shellcode	220	(1)
Simple SPARC exec Shellcode	221	(1)
Useful System Calls on Solaris	222	(1)
NOP and Padding Instructions	222	(1)
Solaris/SPARC Stack Frame Introduction	223	(1)
Stack-Based Overflow Methodologies	224	(4)
Arbitrary Size Overflow	224	(1)
Register Windows and Stack Overflow Complications	225	(1)
Other Complicating Factors	225	(1)
Possible Solutions	225	(1)
Off-By-One Stack Overflow Vulnerabilities	226	(1)
Shellcode Locations	227	(1)
Stack Overflow Exploitation in Action	228	(5)
The Vulnerable Program	228	(2)
The Exploit	230	(3)
Heap-Based Overflows on Solaris/SPARC	233	(21)
Solaris System V Heap Introduction	234	(1)
Heap Tree Structure	234	(20)
Basic Exploit Methodology (t_delete)	254	(7)
Standard Heap Overflow Limitations	257	(1)
Targets for Overwrite	258	(3)
The Bottom Chunk	259	(1)
Small Chunk Corruption	260	(1)
Other Heap-Related Vulnerabilities	261	(1)
Off-by-One Overflows	261	(1)
Double Free Vulnerabilities	261	(1)
Arbitrary Free Vulnerabilities	262	(1)
Heap Overflow Example	262	(4)
The Vulnerable Program	262	(4)
Other Solaris Exploitation Techniques	266	(2)
Static Data Overflows	267	(1)
Bypassing the Non-Executable Stack Protection	267	(1)
Conclusion	268	(1)
Chapter 11 Advanced Solaris Exploitation	269	(32)
Single Stepping the Dynamic Linker	271	(15)
Various Style Tricks for Solaris SPARC Heap Overflows	286	(2)
Advanced Solaris/SPARC Shellcode	288	(12)
Conclusion	300	(1)
Chapter 12 HP Tru64 Unix Exploitation	301	(30)
The Alpha Architecture	302	(4)
Alpha Registers	302	(1)
Instruction Set	303	(2)
Calling Conventions	305	(1)
Retrieving the Program Counter (GetPC)	306	(2)
System Call Invocation	308	(1)
XOR Decoder for Shellcode	308	(2)
.end main setuid + execve Shellcode	310	(6)
Code the setuid(0) + execve("/bin/sh", ...) systemcalls	310	(1)
Compile the Assembly Code and Extract the Main Function	311	(1)
Encode the Extracted opcodes with the XOR Key	312	(1)
Plug the Encoded Code into the XOR Decoder	313	(1)
Compile and Extract the Final Shellcode	314	(2)
Connect-Back Shellcode	316	(1)
Find-Socket Shellcode	317	(2)
Bind-Socket Shellcode	319	(1)
Stack Overflow Exploitation	320	(2)
Defeating the Non-Executable Stack	321	(1)
Exploiting rpc.ttdbserver	322	(8)
Conclusion	330	(1)
Part 3 Vulnerability Discovery	331	(142)
Chapter 13 Establishing a Working Environment	333	(16)
What You Need for Reference	334	(1)
What You Need for Code	334	(2)
gcc	334	(1)
gdb	335	(1)
NASM	335	(1)
WinDbg	335	(1)
OllyDbg	335	(1)
SoftICE	335	(1)
Visual C++	336	(1)
Python	336	(1)
What You Need for Investigation	336	(4)
Useful Custom Scripts/Tools	336	(2)
An Offset Finder	336	(1)
Generic Fuzzers	337	(1)
The Debug Trick	337	(1)
All Platforms	338	(1)
Unix	338	(1)
ltrace	339	(1)
strace	339	(1)
fstat (BSD)	339	(1)
tcpdump	339	(1)
Ethereal	339	(1)
Windows	339	(1)
IDA Pro Disassembler	340	(1)
What You Need to Know	340	(3)
Paper Archives	343	(1)
Optimizing Shellcode Development	343	(5)
Plan the Exploit	343	(1)
Write the Shellcode in Inline Assembler	344	(1)
Maintain a Shellcode Library	345	(1)
Make It Continue Nicely	346	(1)
Make the Exploit Stable	347	(1)
Make It Steal the Connection	347	(1)
Conclusion	348	(1)
Chapter 14 Fault Injection	349	(14)
Design Overview	350	(10)
Input Generation	351	(3)
Manual Generation	352	(1)
Automated Generation	352	(1)
Live Capture	353	(1)
"Fuzz" Generation	353	(1)
Fault Injection	354	(1)
Modification Engines	354	(4)
Delimiting Logic	354	(3)
Getting around Input Sanitization	357	(1)
Fault Delivery	358	(1)
Nagel Algorithm	359	(1)
Timing	359	(1)
Heuristics	359	(1)
Stateless versus State-Based Protocols	360	(1)
Fault Monitoring	360	(1)
Using a Debugger	360	(1)
FaultMon	361	(1)
Putting It Together	361	(1)
Conclusion	362	(1)
Chapter 15 The Art of Fuzzing	363	(20)
General Theory of Fuzzing	363	(7)
Static Analysis versus Fuzzing	368	(1)
Fuzzing Is Scalable	368	(2)
Weaknesses in Fuzzers	370	(1)
Modeling Arbitrary Network Protocols	371	(1)
Other Fuzzer Possibilities	371	(2)
Bit Flipping	372	(1)
Modifying Open Source Programs	372	(1)
Fuzzing with Dynamic Analysis	372	(1)
SPIKE	373	(8)
What Is a Spike?	373	(1)
Why Use the SPIKE Data Structure to Model Network Protocols?	374	(10)
Various Programs Included with SPIKE 374
SPIKE Example: dtlogin	374	(7)
Other Fuzzers	381	(1)
Conclusion	382	(1)
Chapter 16 Source Code Auditing: Finding Vulnerabilities in C-Based Languages	383	(22)
Tools	384	(2)
Cscope	384	(1)
Ctags	385	(1)
Editors	385	(1)
Cbrowser	385	(1)
Automated Source Code Analysis Tools	386	(1)
Methodology	386	(2)
Top-Down (Specific) Approach	387	(1)
Bottom-Up Approach	387	(1)
Selective Approach	387	(1)
Vulnerability Classes	388	(14)
Generic Logic Errors	388	(1)
(Almost) Extinct Bug Classes	388	(1)
Format Strings	389	(1)
Generic Incorrect Bounds-Checking	390	(2)
Loop Constructs	392	(1)
Off-by-One Vulnerabilities	392	(1)
Non-Null Termination Issues	393	(1)
Skipping Null-Termination Issues	394	(1)
Signed Comparison Vulnerabilities	395	(1)
Integer-Related Vulnerabilities	396	(2)
Different-Sized Integer Conversions	398	(2)
Double Free Vulnerabilities	400	(1)
Out-of-Scope Memory Usage Vulnerabilities	400	(1)
Uninitialized Variable Usage	400	(1)
Use After Free Vulnerabilities	401	(1)
Multithreaded Issues and Re-Entrant Safe Code	402	(1)
Beyond Recognition: A Real Vulnerability versus a Bug	402	(1)
Conclusion	403	(2)
Chapter 17 Instrumented Investigation: A Manual Approach	405	(22)
Philosophy	405	(1)
Oracle extproc Overflow	406	(4)
Common Architectural Failures	410	(5)
Problems Happen at Boundaries	410	(1)
A Process Calling into an External Process on the Same Host	410	(1)
A Process Calling into an External, Dynamically Loaded Library	410	(1)
A Process Calling into a Function on a Remote Host	411	(1)
Problems Happen When Data Is Translated	411	(2)
Problems Cluster in Areas of Asymmetry	413	(1)
Problems Occur When Authentication and Authorization Are Confused	414	(1)
Problems Occur in the Dumbest Places	414	(1)
Bypassing Input Validation and Attack Detection	415	(6)
Stripping Bad Data	415	(1)
Using Alternate Encodings	415	(1)
Using File-Handling Features	416	(3)
Required String Is Present in Path	416	(1)
Prohibited String Not Present in Path	417	(1)
Incorrect Behavior Based on File Extension	417	(2)
Evading Attack Signatures	419	(1)
Defeating Length Limitations	419	(9)
Sea Monkey Data	419	(1)
Harmful Truncation-Severing Escape Characters	420	(1)
Multiple Attempts	421	(1)
Context-Free Length Limits	421	(1)
Windows 2000 SNMP DOS	421	(1)
Finding DOS Attacks	422	(1)
SQL-UDP	423	(1)
Conclusion	424	(3)
Chapter 18 Tracing for Vulnerabilities	427	(24)
Overview	428	(22)
A Vulnerable Program	429	(2)
Component Design	431	(9)
Process Injection	432	(1)
Machine-Code Analysis	432	(4)
Function Hooking	436	(3)
Data Collection	439	(1)
Building VulnTrace	440	(5)
VTInject	440	(3)
VulnTrace.dll	443	(2)
Using VulnTrace	445	(3)
Advanced Techniques	448	(4)
Fingerprint Systems	448	(1)
More Vulnerability Classes	449	(1)
Conclusion	450	(1)
Chapter 19 Binary Auditing: Hacking Closed Source Software	451	(22)
Binary versus Source-Code Auditing: The Obvious Differences	452	(1)
IDA Pro-The Tool of the Trade	452	(2)
Features: A Quick Crash Course	453	(1)
Debugging Symbols	454	(1)
Binary Auditing Introduction	454	(10)
Stack Frames	454	(2)
Traditional BP-Based Stack Frames	455	(1)
Functions without a Frame Pointer	455	(1)
Non-Traditional BP-Based Stack Frames	456	(1)
Calling Conventions	456	(2)
The C Calling Convention	457	(1)
The Stdcall Calling Convention	457	(1)
Compiler-Generated Code	458	(4)
Function Layouts	458	(1)
If Statements	458	(1)
For and While Loops	459	(1)
Switch Statements	460	(2)
memcpy-Like Code Constructs	462	(1)
strlen-Like Code Constructs	462	(1)
C++ Code Constructs	463	(1)
The this Pointer	463	(1)
Reconstructing Class Definitions	464	(2)
vtables	465	(1)
Quick but Useful Tidbits	465	(1)
Manual Binary Analysis	466	(2)
Quick Examination of Library Calls	466	(1)
Suspicious Loops and Write Instructions	466	(1)
Higher-Level Understanding and Logic Bugs	467	(1)
Graphical Analysis of Binaries	468	(1)
Manual Decompilation	468	(1)
Binary Vulnerability Examples	468	(4)
Microsoft SQL Server Bugs	469	(1)
LSD's RPC-DCOM Vulnerability	469	(1)
IIS WebDAV Vulnerability	470	(2)
Conclusion	472	(1)
Part 4 Advanced Materials	473	(108)
Chapter 20 Alternative Payload Strategies	475	(24)
Modifying the Program	476	(1)
The SQL Server 3-Byte Patch	477	(4)
The MySQL 1-Bit Patch	481	(2)
OpenSSH RSA Authentication Patch	483	(1)
Other Runtime Patching Ideas	484	(2)
GPG 1.2.2 Randomness Patch	485	(1)
Upload and Run (or Proglet Server)	486	(1)
Syscall Proxies	486	(3)
Problems with Syscall Proxies	489	(9)
Conclusion	498	(1)
Chapter 21 Writing Exploits that Work in the Wild	499	(10)
Factors in Unreliability	499	(9)
Magic Numbers	499	(1)
Versioning	500	(1)
Shellcode Problems	501	(2)
Network Related	501	(1)
Privilege Related	501	(1)
Configuration Related	502	(1)
Host IDS Related	502	(1)
Thread Related	502	(1)
Countermeasures	503	(18)
Preparation	504	(1)
Brute Forcing	504	(1)
Local Exploits	505	(1)
OS/Application Fingerprinting	505	(2)
Information Leaks	507	(1)
Conclusion	508	(1)
Chapter 22 Attacking Database Software	509	(20)
Network Layer Attacks	510	(10)
Application Layer Attacks	520	(1)
Running Operating System Commands	521	(4)
Microsoft SQL Server	521	(1)
Oracle	522	(1)
IBM DB2	523	(2)
Exploiting Overruns at the SQL Level	525	(3)
SQL Functions	526	(12)
Using the CHR/CHAR Function	526	(2)
Conclusion	528	(1)
Chapter 23 Kernel Overflows	529	(20)
Kernel Vulnerability Types	529	(9)
Oday Kernel Vulnerabilities	538	(6)
OpenBSD exec_ibcs2_coff_prep_zmagic() Stack Overflow	538	(2)
The Vulnerability	540	(4)
Solaris vfs_getvfssw() Loadable Kernel Module
Traversal Vulnerability	544	(5)
The sysfs() System Call	546	(1)
The mount() System Call	547	(1)
Conclusion	548	(1)
Chapter 24 Exploiting Kernel Vulnerabilities	549	(32)
The exec_ibcs2_coff_prep_zmagic() Vulnerability	549	(25)
Calculating Offsets and Breakpoints	554	(3)
Overwriting the Return Address and Redirecting Execution	557	(1)
Locating the Process Descriptor (or the Proc Structure)	558	(2)
Stack Lookup	558	(1)
sysctl() Syscall	558	(2)
Kernel Mode Payload Creation	560	(1)
p_cred and u_cred	560	(1)
Breaking chroot	561	(1)
Returning Back from Kernel Payload	561	(6)
Return to User Mode: iret Technique	562	(2)
Return to Kernel Code: sidt Technique and _kernel_text Search	564	(3)
Getting root (uid=0)	567	(7)
Solaris vfs_getvfssw() Loadable Kernel Module Path Traversal Exploit	574	(6)
Crafting the Exploit	574	(2)
The Kernel Module to Load	576	(3)
Getting root (uid=0)	579	(1)
Conclusion	580	(1)
Index	581