9780387303031

Numerical Optimization

by ;
  • ISBN13:

    9780387303031

  • ISBN10:

    0387303030

  • Edition: 2nd
  • Format: Hardcover
  • Copyright: 6/1/2006
  • Publisher: Springer Verlag
  • Purchase Benefits
  • Free Shipping On Orders Over $59!
    Your order must be $59 or more to qualify for free economy shipping. Bulk sales, PO's, Marketplace items, eBooks and apparel do not qualify for this offer.
  • Get Rewarded for Ordering Your Textbooks! Enroll Now
  • We Buy This Book Back!
    In-Store Credit: $15.75
    Check/Direct Deposit: $15.00
List Price: $79.95

Summary

Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

Table of Contents

Preface xvii
Preface to the Second Edition xxi
Introduction
1(9)
Mathematical Formulation
2(2)
Example: A Transportation Problem
4(1)
Continuous versus Discrete Optimization
5(1)
Constrained and Unconstrained Optimization
6(1)
Global and Local Optimization
6(1)
Stochastic and Deterministic Optimization
7(1)
Convexity
7(1)
Optimization Algorithms
8(2)
Notes and References
9(1)
Fundamentals of Unconstrained Optimization
10(20)
What Is a Solution?
12(6)
Recognizing a Local Minimum
14(3)
Nonsmooth Problems
17(1)
Overview of Algorithms
18(12)
Two Strategies: Line Search and Trust Region
19(1)
Search Directions for Line Search Methods
20(5)
Models for Trust-Region Methods
25(1)
Scaling
26(1)
Exercises
27(3)
Line Search Methods
30(36)
Step Length
31(6)
The Wolfe Conditions
33(3)
The Goldstein Conditions
36(1)
Sufficient Decrease and Backtracking
37(1)
Convergence of Line Search Methods
37(4)
Rate of Convergence
41(7)
Convergence Rate of Steepest Descent
42(2)
Newton's Method
44(2)
Quasi-Newton Methods
46(2)
Newton's Method with Hessian Modification
48(8)
Eigenvalue Modification
49(2)
Adding a Multiple of the Identity
51(1)
Modified Cholesky Factorization
52(2)
Modified Symmetric Indefinite Factorization
54(2)
Step-Length Selection Algorithms
56(10)
Interpolation
57(2)
Initial Step Length
59(1)
A Line Search Algorithm for the Wolfe Conditions
60(2)
Notes and References
62(1)
Exercises
63(3)
Trust-Region Methods
66(35)
Outline of the Trust-Region Approach
68(3)
Algorithms Based on the Cauchy Point
71(6)
The Cauchy Point
71(2)
Improving on the Cauchy Point
73(1)
The Dogleg Method
73(3)
Two-Dimensional Subspace Minimization
76(1)
Global Convergence
77(6)
Reduction Obtained by the Cauchy Point
77(2)
Convergence to Stationary Points
79(4)
Iterative Solution of the Subproblem
83(9)
The Hard Case
87(2)
Proof of Theorem 4.1
89(2)
Convergence of Algorithms Based on Nearly Exact Solutions
91(1)
Local Convergence of Trust-Region Newton Methods
92(3)
Other Enhancements
95(6)
Scaling
95(2)
Trust Regions in Other Norms
97(1)
Notes and References
98(1)
Exercises
98(3)
Conjugate Gradient Methods
101(34)
The Linear Conjugate Gradient Method
102(19)
Conjugate Direction Methods
102(5)
Basic Properties of the Conjugate Gradient Method
107(4)
A Practical Form of the Conjugate Gradient Method
111(1)
Rate of Convergence
112(6)
Preconditioning
118(2)
Practical Preconditioners
120(1)
Nonlinear Conjugate Gradient Methods
121(14)
The Fletcher--Reeves Method
121(1)
The Polak-Ribiere Method and Variants
122(2)
Quadratic Termination and Restarts
124(1)
Behavior of the Fletcher-Reeves Method
125(2)
Global Convergence
127(4)
Numerical Performance
131(1)
Notes and References
132(1)
Exercises
133(2)
Quasi-Newton Methods
135(29)
The BFGS Method
136(8)
Properties of the BFGS Method
141(1)
Implementation
142(2)
The SRI Method
144(5)
Properties of SRI Updating
147(2)
The Broyden Class
149(4)
Convergence Analysis
153(11)
Global Convergence of the BFGS Method
153(3)
Superlinear Convergence of the BFGS Method
156(4)
Convergence Analysis of the SRI Method
160(1)
Notes and References
161(1)
Exercises
162(2)
Large-Scale Unconstrained Optimization
164(29)
Inexact Newton Methods
165(11)
Local Convergence of Inexact Newton Methods
166(2)
Line Search Newton--CG Method
168(2)
Trust-Region Newton--CG Method
170(4)
Preconditioning the Trust-Region Newton--CG Method
174(1)
Trust-Region Newton--Lanczos Method
175(1)
Limited-Memory Quasi-Newton Methods
176(9)
Limited-Memory BFGS
177(3)
Relationship with Conjugate Gradient Methods
180(1)
General Limited-Memory Updating
181(1)
Compact Representation of BFGS Updating
181(3)
Unrolling the Update
184(1)
Sparse Quasi-Newton Updates
185(1)
Algorithms for Partially Separable Functions
186(3)
Perspectives and Software
189(4)
Notes and References
190(1)
Exercises
191(2)
Calculating Derivatives
193(27)
Finite-Difference Derivative Approximations
194(10)
Approximating the Gradient
195(2)
Approximating a Sparse Jacobian
197(4)
Approximating the Hessian
201(1)
Approximating a Sparse Hessian
202(2)
Automatic Differentiation
204(16)
An Example
205(1)
The Forward Mode
206(1)
The Reverse Mode
207(3)
Vector Functions and Partial Separability
210(2)
Calculating Jacobians of Vector Functions
212(1)
Calculating Hessians: Forward Mode
213(2)
Calculating Hessians: Reverse Mode
215(1)
Current Limitations
216(1)
Notes and References
217(1)
Exercises
217(3)
Derivative-Free Optimization
220(25)
Finite Differences and Noise
221(2)
Model-Based Methods
223(6)
Interpolation and Polynomial Bases
226(1)
Updating the Interpolation Set
227(1)
A Method Based on Minimum-Change Updating
228(1)
Coordinate and Pattern-Search Methods
229(5)
Coordinate Search Method
230(1)
Pattern-Search Methods
231(3)
A Conjugate-Direction Method
234(4)
Nelder-Mead Method
238(2)
Implicit Filtering
240(5)
Notes and References
242(1)
Exercises
242(3)
Least-Squares Problems
245(25)
Background
247(3)
Linear Least-Squares Problems
250(4)
Algorithms for Nonlinear Least-Squares Problems
254(11)
The Gauss-Newton Method
254(1)
Convergence of the Gauss--Newton Method
255(3)
The Levenberg--Marquardt Method
258(1)
Implementation of the Levenberg--Marquardt Method
259(2)
Convergence of the Levenberg--Marquardt Method
261(1)
Methods for Large-Residual Problems
262(3)
Orthogonal Distance Regression
265(5)
Notes and References
267(2)
Exercises
269(1)
Nonlinear Equations
270(34)
Local Algorithms
274(11)
Newton's Method for Nonlinear Equations
274(3)
Inexact Newton Methods
277(2)
Broyden's Method
279(4)
Tensor Methods
283(2)
Practical Methods
285(11)
Merit Functions
285(2)
Line Search Methods
287(3)
Trust-Region Methods
290(6)
Continuation/Homotopy Methods
296(8)
Motivation
296(1)
Practical Continuation Methods
297(5)
Notes and References
302(1)
Exercises
302(2)
Theory of Constrained Optimization
304(51)
Local and Global Solutions
305(1)
Smoothness
306(1)
Examples
307(8)
A Single Equality Constraint
308(2)
A Single Inequality Constraint
310(3)
Two Inequality Constraints
313(2)
Tangent Cone and Constraint Qualifications
315(5)
First-Order Optimality Conditions
320(3)
First-Order Optimality Conditions: Proof
323(7)
Relating the Tangent Cone and the First-Order Feasible Direction Set
323(2)
A Fundamental Necessary Condition
325(1)
Farkas' Lemma
326(3)
Proof of Theorem 12.1
329(1)
Second-Order Conditions
330(8)
Second-Order Conditions and Projected Hessians
337(1)
Other Constraint Qualifications
338(2)
A Geometric Viewpoint
340(1)
Lagrange Multipliers and Sensitivity
341(2)
Duality
343(12)
Notes and References
349(2)
Exercises
351(4)
Linear Programming: The Simplex Method
355(37)
Linear Programming
356(2)
Optimality and Duality
358(4)
Optimality Conditions
358(1)
The Dual Problem
359(3)
Geometry of the Feasible Set
362(4)
Bases and Basic Feasible Points
362(3)
Vertices of the Feasible Polytope
365(1)
The Simplex Method
366(6)
Outline
366(4)
A Single Step of the Method
370(2)
Linear Algebra in the Simplex Method
372(3)
Other Important Details
375(7)
Pricing and Selection of the Entering Index
375(3)
Starting the Simplex Method
378(3)
Degenerate Steps and Cycling
381(1)
The Dual Simplex Method
382(3)
Presolving
385(3)
Where Does the Simplex Method Fit?
388(4)
Notes and References
389(1)
Exercises
389(3)
Linear Programming: Interior-Point Methods
392(29)
Primal-Dual Methods
393(14)
Outline
393(4)
The Central Path
397(2)
Central Path Neighborhoods and Path-Following Methods
399(8)
Practical Primal-Dual Algorithms
407(6)
Corrector and Centering Steps
407(2)
Step Lengths
409(1)
Starting Point
410(1)
A Practical Algorithm
411(1)
Solving the Linear Systems
411(2)
Other Primal-Dual Algorithms and Extensions
413(3)
Other Path-Following Methods
413(1)
Potential-Reduction Methods
414(1)
Extensions
415(1)
Perspectives and Software
416(5)
Notes and References
417(1)
Exercises
418(3)
Fundamentals of Algorithms for Nonlinear Constrained Optimization
421(27)
Categorizing Optimization Algorithms
422(2)
The Combinatorial Difficulty of Inequality-Constrained Problems
424(2)
Elimination of Variables
426(9)
Simple Elimination using Linear Constraints
428(3)
General Reduction Strategies for Linear Constraints
431(3)
Effect of Inequality Constraints
434(1)
Merit Functions and Filters
435(5)
Merit Functions
435(2)
Filters
437(3)
The Maratos Effect
440(3)
Second-Order Correction and Nonmonotone Techniques
443(5)
Nonmonotone (Watchdog) Strategy
444(2)
Notes and References
446(1)
Exercises
446(2)
Quadratic Programming
448(49)
Equality-Constrained Quadratic Programs
451(3)
Properties of Equality-Constrained QPs
451(3)
Direct Solution of the KKT System
454(5)
Factoring the Full KKT System
454(1)
Schur-Complement Method
455(2)
Null-Space Method
457(2)
Iterative Solution of the KKT System
459(4)
CG Applied to the Reduced System
459(2)
The Projected CG Method
461(2)
Inequality-Constrained Problems
463(4)
Optimality Conditions for Inequality-Constrained Problems
464(1)
Degeneracy
465(2)
Active-Set Methods for Convex QPs
467(13)
Specification of the Active-Set Method for Convex QP
472(4)
Further Remarks on the Active-Set Method
476(1)
Finite Termination of Active-Set Algorithm on Strictly Convex QPs
477(1)
Updating Factorizations
478(2)
Interior-Point Methods
480(5)
Solving the Primal-Dual System
482(1)
Step Length Selection
483(1)
A Practical Primal-Dual Method
484(1)
The Gradient Projection Method
485(5)
Cauchy Point Computation
486(2)
Subspace Minimization
488(2)
Perspectives and Software
490(7)
Notes and References
492(1)
Exercises
492(5)
Penalty and Augmented Lagrangian Methods
497(32)
The Quadratic Penalty Method
498(9)
Motivation
498(3)
Algorithmic Framework
501(1)
Convergence of the Quadratic Penalty Method
502(3)
Conditioning and Reformulations
505(2)
Nonsmooth Penalty Functions
507(7)
A Practical l1 Penalty Method
511(2)
A General Class of Nonsmooth Penalty Methods
513(1)
Augmented Lagrangian Method: Equality Constraints
514(5)
Motivation and Algorithmic Framework
514(3)
Properties of the Augmented Lagrangian
517(2)
Practical Augmented Lagrangian Methods
519(6)
Bound-Constrained Formulation
519(3)
Linearly Constrained Formulation
522(1)
Unconstrained Formulation
523(2)
Perspectives and Software
525(4)
Notes and References
526(1)
Exercises
527(2)
Sequential Quadratic Programming
529(34)
Local SQP Method
530(3)
SQP Framework
531(1)
Inequality Constraints
532(1)
Preview of Practical SQP Methods
533(2)
IQP and EQP
533(1)
Enforcing Convergence
534(1)
Algorithmic Development
535(10)
Handling Inconsistent Linearizations
535(1)
Full Quasi-Newton Approximations
536(2)
Reduced-Hessian Quasi-Newton Approximations
538(2)
Merit Functions
540(3)
Second-Order Correction
543(2)
A Practical Line Search SQP Method
545(1)
Trust-Region SQP Methods
546(8)
A Relaxation Method for Equality-Constrained Optimization
547(2)
Sl1QP (Sequential l1 Quadratic Programming)
549(2)
Sequential Linear-Quadratic Programming (SLQP)
551(2)
A Technique for Updating the Penalty Parameter
553(1)
Nonlinear Gradient Projection
554(2)
Convergence Analysis
556(4)
Rate of Convergence
557(3)
Perspectives and Software
560(3)
Notes and References
561(1)
Exercises
561(2)
Interior-Point Methods for Nonlinear Programming
563(35)
Two Interpretations
564(2)
A Basic Interior-Point Algorithm
566(3)
Algorithmic Development
569(8)
Primal vs. Primal-Dual System
570(1)
Solving the Primal-Dual System
570(2)
Updating the Barrier Parameter
572(1)
Handling Nonconvexity and Singularity
573(2)
Step Acceptance: Merit Functions and Filters
575(1)
Quasi-Newton Approximations
575(1)
Feasible Interior-Point Methods
576(1)
A Line Search Interior-Point Method
577(1)
A Trust-Region Interior-Point Method
578(5)
An Algorithm for Solving the Barrier Problem
578(2)
Step Computation
580(1)
Lagrange Multipliers Estimates and Step Acceptance
581(1)
Description of a Trust-Region Interior-Point Method
582(1)
The Primal Log-Barrier Method
583(4)
Global Convergence Properties
587(4)
Failure of the Line Search Approach
587(2)
Modified Line Search Methods
589(1)
Global Convergence of the Trust-Region Approach
589(2)
Superlinear Convergence
591(1)
Perspectives and Software
592(6)
Notes and References
593(1)
Exercises
594(4)
A. Background Material
598(37)
Elements of Linear Algebra
598(19)
Vectors and Matrices
598(2)
Norms
600(2)
Subspaces
602(1)
Eigenvalues, Eigenvectors, and the Singular-Value Decomposition
603(2)
Determinant and Trace
605(1)
Matrix Factorizations: Cholesky, LU, QR
606(4)
Symmetric Indefinite Factorization
610(2)
Sherman--Morrison--Woodbury Formula
612(1)
Interlacing Eigenvalue Theorem
613(1)
Error Analysis and Floating-Point Arithmetic
613(3)
Conditioning and Stability
616(1)
Elements of Analysis, Geometry, Topology
617(18)
Sequences
617(2)
Rates of Convergence
619(1)
Topology of the Euclidean Space Rn
620(1)
Convex Sets in Rn
621(2)
Continuity and Limits
623(2)
Derivatives
625(3)
Directional Derivatives
628(1)
Mean Value Theorem
629(1)
Implicit Function Theorem
630(1)
Order Notation
631(2)
Root-Finding for Scalar Equations
633(2)
B. A Regularization Procedure
635(2)
References 637(16)
Index 653

Rewards Program

Write a Review