Preface 

xiii  

Introduction to Probability 


1  (57) 

Introduction: Why Study Probability? 


1  (1) 

The Different Kinds of Probability 


2  (3) 


2  (1) 

Probability as the Ratio of Favorable to Total Outcomes (Classical Theory) 


3  (1) 

Probability as a Measure of Frequency of Occurrence 


4  (1) 

Probability Based on an Axiomatic Theory 


5  (1) 

Misuses, Miscalculations, and Paradoxes in Probability 


5  (2) 


7  (4) 

Examples of Sample Spaces 


7  (4) 

Axiomatic Definition of Probability 


11  (5) 

Joint, Conditional, and Total Probabilities; Independence 


16  (6) 

Bayes' Theorem and Applications 


22  (2) 


24  (8) 


28  (2) 

Extensions and Applications 


30  (2) 

Bernoulli TrialsBinomial and Multinomial Probability Laws 


32  (7) 

Multinomial Probability Law 


36  (3) 

Asymptotic Behavior of the Binomial Law: The Poisson Law 


39  (6) 

Normal Approximation to the Binomial Law 


45  (2) 


47  (11) 


48  (9) 


57  (1) 


58  (58) 


58  (1) 

Definition of a Random Variable 


59  (3) 

Probability Distribution Function 


62  (4) 

Probability Density Function (PDF) 


66  (9) 

Four Other Common Density Functions 


71  (3) 

More Advanced Density Functions 


74  (1) 

Continuous, Discrete, and Mixed Random Variables 


75  (5) 

Examples of Probability Mass Functions 


77  (3) 

Conditional and Joint Distributions and Densities 


80  (25) 


105  (3) 


108  (8) 


109  (6) 


115  (1) 


115  (1) 

Functions of Random Variables 


116  (53) 


116  (4) 

Functions of a Random Variable (Several Views) 


119  (1) 

Solving Problems of the Type Y = g(X) 


120  (14) 

General Formula of Determining the pdf of Y = g(X) 


129  (5) 

Solving Problems of the Type Z = g(X, Y) 


134  (18) 

Solving Problems of the Type V = g(X, Y), W = h(X, Y) 


152  (5) 


152  (2) 

Obtaining fVW Directly from fXY 


154  (3) 


157  (4) 


161  (8) 


162  (6) 


168  (1) 


168  (1) 

Expectation and Introduction to Estimation 


169  (75) 

Expected Value of a Random Variable 


169  (14) 

On the Validity of Equation 4.18 


172  (11) 


183  (9) 

Conditional Expectation as a Random Variable 


190  (2) 


192  (13) 


196  (2) 

Properties of Uncorrelated Random Variables 


198  (3) 

Jointly Gaussian Random Variables 


201  (2) 

Contours of Constant Density of the Joint Gaussian pdf 


203  (2) 

Chebyshev and Schwarz Inequalities 


205  (6) 

Random Variables with Nonnegative Values 


207  (1) 


208  (3) 

MomentGenerating Functions 


211  (3) 


214  (2) 


216  (14) 

Joint Characteristic Functions 


222  (3) 

The Central Limit Theorem 


225  (5) 

Estimators for the Mean and Variance of the Normal Law 


230  (6) 

Confidence Intervals for the Mean 


231  (3) 

Confidence Interval for the Variance 


234  (2) 


236  (8) 


237  (6) 


243  (1) 


243  (1) 

Random Vectors and Parameter Estimation 


244  (60) 

Joint Distribution and Densities 


244  (4) 

Multiple Transformation of Random Variables 


248  (3) 

Expectation Vectors and Covariance Matrices 


251  (3) 

Properties of Covariance Matrices 


254  (5) 

Simultaneous Diagonalization of Two Covariance Matrices and Applications in Pattern Recognition 


259  (10) 


262  (1) 

Maximization of Quadratic Forms 


263  (6) 

The Multidimensional Gaussian Law 


269  (8) 

Characteristic Functions of Random Vectors 


277  (5) 

The Characteristic Function of the Normal Law 


280  (2) 


282  (4) 


284  (2) 

Estimation of Vector Means and Covariance Matrices 


286  (4) 


286  (1) 

Estimation of the Covariance K 


287  (3) 

Maximum Likelihood Estimators 


290  (4) 

Linear Estimation of Vector Parameters 


294  (3) 


297  (7) 


298  (5) 


303  (1) 


303  (1) 


304  (97) 


304  (30) 

InfiniteLength Bernoulli Trials 


310  (5) 

Continuity of Probability Measure 


315  (2) 

Statistical Specification of a Random Sequence 


317  (17) 

Basic Principles of DiscreteTime Linear Systems 


334  (6) 

Random Sequences and Linear Systems 


340  (8) 


348  (14) 


351  (1) 

Interpretation of the PSD 


352  (3) 

Synthesis of Random Sequences and DiscreteTime Simulation 


355  (3) 


358  (1) 


359  (3) 


362  (10) 


365  (1) 


366  (6) 

Vector Random Sequences and State Equations 


372  (3) 

Convergence of Random Sequences 


375  (8) 


383  (4) 


387  (14) 


388  (11) 


399  (2) 


401  (86) 


402  (4) 

Some Important Random Processes 


406  (24) 

Asynchronous Binary Signaling 


406  (2) 


408  (4) 

Alternative Derivation of Poisson Process 


412  (2) 


414  (2) 

Digital Modulation Using PhaseShift Keying 


416  (2) 

Wiener Process or Brownian Motion 


418  (3) 


421  (4) 

BirthDeath Markov Chains 


425  (4) 

ChapmanKolmogorov Equations 


429  (1) 

Random Process Generated from Random Sequences 


430  (1) 

ContinuousTime Linear Systems with Random Inputs 


430  (7) 


436  (1) 

Some Useful Classifications of Random Processes 


437  (2) 


437  (2) 

WideSense Stationary Processes and LSI Systems 


439  (19) 

WideSense Stationary Case 


440  (3) 


443  (1) 

An Interpretation of the psd 


444  (4) 


448  (7) 

Stationary Processes and Differential Equations 


455  (3) 

Periodic and Cyclostationary Processes 


458  (6) 

Vector Processes and State Equations 


464  (5) 


466  (3) 


469  (18) 


469  (17) 


486  (1) 

Advanced Topics in Random Processes 


487  (65) 

MeanSquare (m.s.) Calculus 


487  (15) 

Stochastic Continuity and Derivatives [81] 


487  (10) 

Further Results on m.s. Convergence [81] 


497  (5) 

m.s. Stochastic Integrals 


502  (4) 

m.s. Stochastic Differential Equations 


506  (5) 


511  (7) 

KarhunenLoeve Expansion [85] 


518  (6) 

Representation of Bandlimited and Periodic Processes 


524  (11) 


525  (3) 

Bandpass Random Processes 


528  (2) 


530  (3) 

Fourier Series for WSS Processes 


533  (2) 


535  (1) 

Appendix: Integral Equations 


535  (17) 


536  (4) 


540  (11) 


551  (1) 

Applications to Statistical Signal Processing 


552  (89) 

Estimation of Random Variables 


552  (18) 

More on the Conditional Mean 


558  (2) 

Orthogonality and Linear Estimation 


560  (8) 

Some Properties of the Operator E 


568  (2) 

Innovation Sequences and Kalman Filtering 


570  (15) 

Predicting Gaussian Random Sequences 


574  (1) 

Kalman Predictor and Filter 


575  (6) 

ErrorCovariance Equations 


581  (4) 

Wiener Filters for Random Sequences 


585  (4) 

Unrealizable Case (Smoothing) 


585  (2) 


587  (2) 

ExpectationMaximization Algorithm 


589  (11) 

LogLikelihood for the Linear Transformation 


592  (2) 

Summary of the EM algorithm 


594  (1) 

EM Algorithm for Exponential Probability Functions 


594  (1) 

Application to Emission Tomography 


595  (3) 

Loglikelihood Function of Complete Data 


598  (1) 


598  (1) 


599  (1) 

Hidden Markov Models (HMM) 


600  (10) 


601  (3) 

Application to Speech Processing 


604  (1) 

Efficient Computation of P[EM] with a Recursive Algorithm 


605  (2) 

Viterbi Algorithm and the Most Likely State Sequence for the Observations 


607  (3) 


610  (13) 


611  (3) 

Bartlett's ProcedureAveraging Periodograms 


614  (2) 

Parametric Spectral Estimate 


616  (4) 

Maximum Entropy Spectral Density 


620  (3) 


623  (10) 


624  (1) 

Noncausal GaussMarkov Models 


625  (4) 


629  (1) 


630  (3) 


633  (8) 


635  (4) 


639  (2) 
Appendix A Review of Relevant Mathematics 

641  (17) 


641  (3) 


641  (1) 


642  (1) 


643  (1) 


643  (1) 

A.2 Continuous Mathematics 


644  (5) 

Definite and Indefinite Integrals 


645  (1) 

Differentiation of Integrals 


645  (1) 


646  (1) 


647  (1) 


647  (1) 


648  (1) 

A.3 Residue Method for Inverse Fourier Transformation 


649  (7) 


650  (3) 

Inverse Fourier Transform for psd of Random Sequence 


653  (3) 

A.4 Mathematical Induction [A4] 


656  (2) 


656  (1) 


657  (1) 
Appendix B Gamma and Delta Functions 

658  (4) 


658  (1) 


659  (3) 
Appendix C Functional Transformations and Jacobians 

662  (6) 


662  (1) 


663  (1) 

C.3 Jacobian for General n 


664  (4) 
Appendix D Measure and Probability 

668  (4) 

D.1 Introduction and Basic Ideas 


668  (2) 

Measurable Mappings and Functions 


670  (1) 

D.2 Application of Measure Theory to Probability 


670  (2) 


671  (1) 
Appendix E Sampled Analog Waveforms and Discretetime Signals 

672  (2) 
Index 

674  