Further Topics on Discrete-Time Markov Control Processes

by Hernandez-Lerma, Onesimo; Lasserre, Jean-Bernard

• ISBN13: 9780387986944
• ISBN10: 0387986944
• Format: Hardcover
• Copyright: 1999-06-01
• Publisher: Springer Verlag

Summary

This book is devoted to a systematic exposition of some recent developments in the theory of discrete-time Markov control processes. Interest is mainly confined to MCPs with Borel state and control spaces, and possibly unbounded costs. The book follows on from the authors earlier volume in this area, however, an important feature of the present volume is that it is essentially self-contained and can be read independently of the first volume, because although both volumes deal with similar classes of markov control processes the assumptions on the control models are usually different. This volume allows cost functions to take positive or negative values, as needed in some applications. The control model studied is sufficiently general to include virtually all the usual discrete-time stochastic control models that appear in applications to engineering, economics, mathematical population processes, operations research, and management science.

Table of Contents

Preface	vii
7 Ergodicity and Poisson's Equation	1	(38)
7.1 Introduction	1	(1)
7.2 Weighted norms and signed kernels	2	(5)
A. Weighted-norm spaces	2	(1)
B. Signed kernels	3	(2)
C. Contraction maps	5	(2)
7.3 Recurrence concepts	7	(10)
A. Irreducibility and recurrence	7	(1)
B. Invariant measures	8	(1)
C. Conditions for irreducibility and recurrence	9	(2)
D. (Omega)-Geometric ergodicity	11	(6)
7.4 Examples on (Omega)-geometric ergodicity	17	(7)
7.5 Possition's equation	24	(15)
A. The multichain case	26	(5)
B. The unichain P.E.	31	(3)
C. Examples	34	(5)
8 Discounted Dynamic Programming with Weighted Norms	39	(36)
8.1 Introduction	39	(1)
8.2 The control model and control policies	40	(3)
8.3 The optimality equation	43	(12)
A. Assumptions	44	(3)
B. The discounted-cost optimality equation	47	(1)
C. The dynamic programming operator	48	(3)
D. Proof of Theorem 8.3.6	51	(4)
8.4 Further analysis of value iteration	55	(10)
A. Asymptotic discount optimality	56	(1)
B. Estimates of VI convergence	57	(1)
C. Rolling horizon procedures	58	(1)
D. Forecast horizons and elimination of non-optimal actions	59	(6)
8.5 The weakly continuous case	65	(3)
8.6 Examples	68	(5)
8.7 Further remarks	73	(2)
9 The Expected Total Cost Criterion	75	(42)
9.1 Introduction	75	(1)
9.2 Preliminaries	76	(3)
A. Extended real numbers	76	(2)
B. Integrability	78	(1)
9.3 The expected total cost	79	(5)
9.4 Occupation measures	84	(9)
A. Expected occupation measures	85	(3)
B. The sufficiency problem	88	(5)
9.5 The optimality equation	93	(10)
A. The optimality equation	93	(2)
B. Optimality criteria	95	(5)
C. Deterministic stationary policies	100	(3)
9.6 The transient case	103	(14)
A. Transient models	103	(6)
B. Optimality conditions	109	(1)
C. Reduction to deterministic policies	110	(3)
D. The policy iteration algorithm	113	(4)
10 Undiscounted Cost Criteria	117	(46)
10.1 Introduction	117	(3)
A. Undiscounted criteria	117	(2)
B. AC criteria	119	(1)
C. Outline of the chapter	120	(1)
10.2 Preliminaries	120	(6)
A. Assumptions	121	(2)
B. Corollaries	123	(1)
C. Discussion	124	(2)
10.3 From AC-optimality to undiscounted criteria	126	(11)
A. The AC optimality inequality	128	(1)
B. The AC optimality equation	129	(2)
C. Uniqueness of the ACOE	131	(1)
D. Bias-optimal policies	132	(3)
E. Undiscounted criteria	135	(2)
10.4 Proof of Theorem 10.3.1	137	(6)
A. Preliminary lemmas	137	(4)
B. Completion of the proof	141	(2)
10.5 Proof of Theorem 10.3.6	143	(6)
A. Proof of part (a)	143	(3)
B. Proof of part (b)	146	(1)
C. Policy iteration	146	(3)
10.6 Proof of Theorem 10.3.7	149	(1)
10.7 Proof of Theorem 10.3.10	150	(4)
10.8 Proof of Theorem 10.3.11	154	(2)
10.9 Examples	156	(7)
11 Sample Path Average Cost	163	(40)
11.1 Introduction	163	(4)
A. Definitions	163	(3)
B. Outline of the chapter	166	(1)
11.2 Preliminaries	167	(8)
A. Positive Harris recurrence	167	(1)
B. Limiting average variance	168	(7)
11.3 The (Omega)-geometrically ergodic case	175	(13)
A. Optimality in II(DS)	177	(1)
B. Optimality in II	178	(2)
C. Variance minimization	180	(1)
D. Proof of Theorem 11.3.5	181	(4)
E. Proof of Theorem 11.3.8	185	(3)
11.4 Strictly unbounded costs	188	(8)
11.5 Examples	196	(7)
12 The Linear Programming Approach	203	(48)
12.1 Introduction	203	(2)
A. Outline of the chapter	204	(1)
12.2 Preliminaries	205	(13)
A. Dual pairs of vector spaces	205	(7)
B. Infinite linear programming	212	(2)
C. Approximation of linear programs	214	(1)
D. Tightness and invariant measures	215	(3)
12.3 Linear programs for the AC problem	218	(15)
A. The linear programs	219	(3)
B. Solvability of (P)	222	(2)
C. Absence of duality gap	224	(2)
D. The Farkas alternative	226	(7)
12.4 Approximating sequences and strong duality	233	(5)
A. Minimizing sequences for (P)	233	(1)
B. Maximizing sequences for (P*)	234	(4)
12.5 Finite LP approximations	238	(4)
A. Aggregation	238	(1)
B. Aggregation-relaxation	239	(1)
C. Aggregation-relaxion-inner approximations	240	(2)
12.6 Proof of Theorems 12.5.3, 12.5.5, 12.5.7	242	(9)
References	251	(12)
Abbreviations	263	(2)
Glossary of notation	265	(6)
Index	271