Continuous-Time Markov Decision Processes

by Guo, Xianping; Hernandez-Lerma, Onesimo

• ISBN13: 9783642025464
• ISBN10: 3642025463
• Format: Hardcover
• Copyright: 2009-10-01
• Publisher: Springer Nature

Summary

Continuous-time Markov decision processes (MDPs), also known as controlled Markov chains, are used for modeling decision-making problems that arise in operations research (for instance, inventory, manufacturing, and queueing systems), computer science, communications engineering, control of populations (such as fisheries and epidemics), and management science, among many other fields. This volume provides a unified, systematic, self-contained presentation of recent developments on the theory and applications of continuous-time MDPs. The MDPs in this volume include most of the cases that arise in applications, because they allow unbounded transition and reward/cost rates. Much of the material appears for the first time in book form.

Table of Contents

Introduction and Summary	p. 1
Introduction	p. 1
Preliminary Examples	p. 1
Summary of the Following Chapters	p. 6
Continuous-Time Markov Decision Processes	p. 9
Introduction	p. 9
The Control Model	p. 10
Continuous-Time Markov Decision Processes	p. 13
Basic Optimality Criteria	p. 16
Average Optimality for Finite Models	p. 19
Introduction	p. 19
n-bias Optimality Criteria	p. 20
Difference Formulas of n-biases	p. 23
Characterization of n-bias Policies	p. 29
Computation of n-bias Optimal Policies	p. 36
The Policy Iteration Algorithm for Average Optimality	p. 36
The 0-bias Policy Iteration Algorithm	p. 39
n-bias Policy Iteration Algorithms	p. 43
The Linear Programming Approach	p. 46
Linear Programming for Ergodic Models	p. 46
Linear Programming for Multichain Models	p. 49
Notes	p. 52
Discount Optimality for Nonnegative Costs	p. 55
Introduction	p. 55
The Nonnegative Model	p. 55
Preliminaries	p. 56
The Discounted Cost Optimality Equation	p. 60
Existence of Optimal Policies	p. 63
Approximation Results	p. 63
The Policy Iteration Approach	p. 66
Examples	p. 68
Notes	p. 69
Average Optimality for Nonnegative Costs	p. 71
Introduction	p. 71
The Average-Cost Criterion	p. 72
The Minimum Nonnegative Solution Approach	p. 73
The Average-Cost Optimality Inequality	p. 76
The Average-Cost Optimality Equation	p. 80
Examples	p. 81
Notes	p. 84
Discount Optimality for Unbounded Rewards	p. 87
Introduction	p. 87
The Discounted-Reward Optimality Equation	p. 89
Discount Optimal Stationary Policies	p. 95
A Value Iteration Algorithm	p. 98
Examples	p. 98
Notes	p. 102
Average Optimality for Unbounded Rewards	p. 105
Introduction	p. 105
Exponential Ergodicity Conditions	p. 106
The Existence of AR Optimal Policies	p. 109
The Policy Iteration Algorithm	p. 113
Examples	p. 119
Notes	p. 124
Average Optimality for Pathwise Rewards	p. 127
Introduction	p. 127
The Optimal Control Problem	p. 129
Optimality Conditions and Preliminaries	p. 129
The Existence of PAR Optimal Policies	p. 131
Policy and Value Iteration Algorithms	p. 138
An Example	p. 139
Notes	p. 142
Advanced Optimality Criteria	p. 143
Bias and Weakly Overtaking Optimality	p. 143
Sensitive Discount Optimality	p. 147
Blackwell Optimality	p. 159
Notes	p. 160
Variance Minimization	p. 163
Introduction	p. 163
Preliminaries	p. 164
Computation of the Average Variance	p. 164
Variance Minimization	p. 170
Examples	p. 171
Notes	p. 173
Constrained Optimality for Discount Criteria	p. 175
The Model with a Constraint	p. 175
Preliminaries	p. 177
Proof of Theorem 11.4	p. 182
An Example	p. 184
Notes	p. 186
Constrained Optimality for Average Criteria	p. 187
Average Optimality with a Constraint	p. 187
Preliminaries	p. 188
Proof of Theorem 12.4	p. 192
An Example	p. 192
Notes	p. 194
	p. 195
Limit Theorems	p. 195
Results from Measure Theory	p. 197
	p. 203
Continuous-Time Markov Chains	p. 203
Stationary Distributions and Ergodicity	p. 206
	p. 209
The Construction of Transition Functions	p. 209
Ergodicity Based on the Q-Matrix	p. 214
Dynkin's Formula	p. 218
References	p. 221
Index	p. 229
Table of Contents provided by Ingram. All Rights Reserved.