Nonlinear System Identification

by Nelles, Oliver

• ISBN13: 9783540673699
• ISBN10: 3540673695
• Format: Hardcover
• Copyright: 2000-12-01
• Publisher: Springer Verlag

Summary

The book covers the most common and important approaches for the identification of nonlinear static and dynamic systems. Additionally, it provides the reader with the necessary background on optimization techniques making the book self-contained. The emphasis is put on modern methods based on neural networks and fuzzy systems without neglecting the classical approaches. The entire book is written from an engineering point-of-view, focusing on the intuitive understanding of the basic relationships. This is supported by many illustrative figures. Advanced mathematics is avoided. Thus, the book is suitable for last year undergraduate and graduate courses as well as research and development engineers in industries. The new edition includes exercises.

Table of Contents

Introduction	1	(22)
Relevance of Nonlinear System Identification	1	(5)
Linear or Nonlinear?	1	(1)
Prediction	2	(1)
Simulation	3	(1)
Optimization	4	(1)
Analysis	4	(1)
Control	4	(1)
Fault Detection	5	(1)
Tasks in Nonlinear System Identification	6	(9)
Choice of the Model Inputs	8	(1)
Choice of the Excitation Signals	9	(1)
Choice of the Model Architecture	10	(1)
Choice of the Dynamics Representation	11	(1)
Choice of the Model Order	11	(1)
Choice of the Model Structure and Complexity	11	(1)
Choice of the Model Parameters	12	(1)
Model Validation	13	(1)
The Role of Fiddle Parameters	13	(2)
White Box, Black Box, and Gray Box Models	15	(1)
Outline of the Book and Some Reading Suggestions	16	(2)
Terminology	18	(5)
Part I. Optimization Techniques
Introduction to Optimization	23	(12)
Overview of Optimization Techniques	25	(1)
Kangaroos	25	(3)
Loss Functions for Supervised Methods	28	(6)
Maximum Likelihood Method	30	(2)
Maximum A-Posteriori and Bayes Method	32	(2)
Loss Functions for Unsupervised Methods	34	(1)
Linear Optimization	35	(44)
Least Squares (LS)	36	(24)
Covariance Matrix of the Parameter Estimate	44	(1)
Errorbars	45	(3)
Orthogonal Regressors	48	(1)
Regularization / Ridge Regression	49	(5)
Noise Assumptions	54	(1)
Weighted Least Squares (WLS)	55	(2)
Least Squares with Equality Constraints	57	(1)
Smoothing Kernels	58	(2)
Recursive Least Squares (RLS)	60	(6)
Reducing the Computational Complexity	63	(1)
Tracking Time-Variant Processes	64	(1)
Relationship between the RLS and the Kalman Filter	65	(1)
Linear Optimization with Inequality Constraints	66	(1)
Subset Selection	67	(10)
Methods for Subset Selection	68	(4)
Orthogonal Least Squares (OLS) for Forward Selection	72	(3)
Ridge Regression or Subset Selection?	75	(2)
Summary	77	(2)
Nonlinear Local Optimization	79	(34)
Batch and Sample Adaptation	81	(2)
Initial Parameters	83	(3)
Direct Search Algorithms	86	(4)
Simplex Search Method	86	(2)
Hooke-Jeeves Method	88	(2)
General Gradient-Based Algorithms	90	(12)
Line Search	91	(1)
Finite Difference Techniques	92	(1)
Steepest Descent	93	(3)
Newton's Method	96	(2)
Quasi-Newton Methods	98	(2)
Conjugate Gradient Methods	100	(2)
Nonlinear Least Squares Problems	102	(5)
Gauss-Newton Method	104	(1)
Levenberg-Marquardt Method	105	(2)
Constrained Nonlinear Optimization	107	(3)
Summary	110	(3)
Nonlinear Global Optimization	113	(24)
Simulated Annealing (SA)	116	(4)
Evolutionary Algorithms (EA)	120	(13)
Evolution Strategies (ES)	123	(3)
Genetic Algorithms (GA)	126	(6)
Genetic Programming (GP)	132	(1)
Branch and Bound (B&B)	133	(2)
Tabu Search (TS)	135	(1)
Summary	135	(2)
Unsupervised Learning Techniques	137	(20)
Principal Component Analysis (PCA)	139	(3)
Clustering Techniques	142	(13)
K-Means Algorithm	143	(3)
Fuzzy C-Means (FCM) Algorithm	146	(2)
Gustafson-Kessel Algorithm	148	(1)
Kohonen's Self-Organizing Map (SOM)	149	(3)
Neural Gas Network	152	(1)
Adaptive Resonance Theory (ART) Network	153	(1)
Incorporating Information about the Output	154	(1)
Summary	155	(2)
Model Complexity Optimization	157	(46)
Introduction	157	(1)
Bias/Variance Tradeoff	158	(9)
Bias Error	160	(1)
Variance Error	161	(3)
Tradeoff	164	(3)
Evaluating the Test Error and Alternatives	167	(9)
Training, Validation, and Test Data	168	(1)
Cross Validation	169	(2)
Information Criteria	171	(1)
Multi-Objective Optimization	172	(2)
Statistical Tests	174	(2)
Correlation-Based Methods	176	(1)
Explicit Structure Optimization	176	(3)
Regularization: Implicit Structure Optimization	179	(10)
Effective Parameters	179	(1)
Regularization by Non-Smoothness Penalties	180	(2)
Regularization by Early Stopping	182	(2)
Regularization by Constraints	184	(2)
Regularization by Staggered Optimization	186	(1)
Regularization by Local Optimization	187	(2)
Structured Models for Complexity Reduction	189	(11)
Curse of Dimensionality	190	(2)
Hybrid Structures	192	(3)
Projection-Based Structures	195	(1)
Additive Structures	196	(1)
Hierarchical Structures	197	(1)
Input Space Decomposition with Tree Structures	198	(2)
Summary	200	(3)
Summary of Part I	203	(6)
Part II. Static Models
Introduction to Static Models	209	(10)
Multivariable Systems	209	(1)
Basis Function Formulation	210	(5)
Global and Local Basis Functions	211	(1)
Linear and Nonlinear Parameters	212	(3)
Extended Basis Function Formulation	215	(1)
Static Test Process	216	(1)
Evaluation Criteria	216	(3)
Linear, Polynomial, and Look-Up Table Models	219	(20)
Linear Models	219	(2)
Polynomial Models	221	(3)
Look-Up Table Models	224	(13)
One-Dimensional Look-Up Tables	225	(2)
Two-Dimensional Look-Up Tables	227	(2)
Optimization of the Heights	229	(2)
Optimization of the Grid	231	(1)
Optimization of the Complete Look-Up Table	232	(1)
Incorporation of Constraints	232	(3)
Properties of Look-Up Table Models	235	(2)
Summary	237	(2)
Neural Networks	239	(60)
Construction Mechanisms	242	(4)
Ridge Construction	242	(2)
Radial Construction	244	(1)
Tensor Product Construction	245	(1)
Multilayer Perceptron (MLP) Network	246	(18)
MLP Neuron	247	(2)
Network Structure	249	(3)
Backpropagation	252	(1)
MLP Training	253	(3)
Simulation Examples	256	(4)
MLP Properties	260	(1)
Multiple Hidden Layers	261	(1)
Projection Pursuit Regression (PPR)	262	(2)
Radial Basis Function (RBF) Networks	264	(22)
RBF Neuron	264	(3)
Network Structure	267	(2)
RBF Training	269	(8)
Simulation Examples	277	(2)
RBF Properties	279	(2)
Regularization Theory	281	(2)
Normalized Radial Basis Function (NRBF) Networks	283	(3)
Other Neural Networks	286	(10)
General Regression Neural Network (GRNN)	286	(2)
Cerebellar Model Articulation Controller (CMAC)	288	(4)
Delaunay Networks	292	(1)
Just-in-Time Models	293	(3)
Summary	296	(3)
Fuzzy and Neuro-Fuzzy Models	299	(42)
Fuzzy Logic	299	(5)
Membership Functions	300	(2)
Logic Operators	302	(1)
Rule Fulfillment	303	(1)
Accumulation	303	(1)
Types of Fuzzy Systems	304	(6)
Linguistic Fuzzy Systems	304	(3)
Singleton Fuzzy Systems	307	(2)
Takagi-Sugeno Fuzzy Systems	309	(1)
Neuro-Fuzzy (NF) Networks	310	(13)
Fuzzy Basis Functions	311	(1)
Equivalence between RBF and Fuzzy Models	312	(1)
What to Optimize?	313	(3)
Interpretation of Neuro-Fuzzy Networks	316	(4)
Incorporating and Preserving Prior Knowledge	320	(1)
Simulation Examples	321	(2)
Neuro-Fuzzy Learning Schemes	323	(16)
Nonlinear Local Optimization	323	(2)
Nonlinear Global Optimization	325	(1)
Orthogonal Least Squares Learning	325	(2)
Fuzzy Rule Extraction by a Genetic Algorithm	327	(10)
Adaptive Spline Modeling of Observation Data	337	(2)
Summary	339	(2)
Local Linear Neuro-Fuzzy Models: Fundamentals	341	(50)
Basic Ideas	342	(9)
Illustration of Local Linear Neuro-Fuzzy Models	343	(3)
Interpretation of the Local Linear Model Offsets	346	(1)
Interpretation as Takagi-Sugeno Fuzzy System	347	(2)
Interpretation as Extended NRBF Network	349	(2)
Parameter Optimization of the Rule Consequents	351	(11)
Global Estimation	351	(1)
Local Estimation	352	(4)
Global Versus Local Estimation	356	(5)
Data Weighting	361	(1)
Structure Optimization of the Rule Premises	362	(27)
Local Linear Model Tree (LOLIMOT) Algorithm	365	(7)
Structure and Parameter Optimization	372	(2)
Smoothness Optimization	374	(2)
Splitting Ration Optimization	376	(2)
Merging of Local Models	378	(2)
Flat and Hierarchical Model Structures	380	(3)
Principal Component Analysis for Preprocessing	383	(2)
Models with Multiple Outputs	385	(4)
Summary	389	(2)
Local Linear Neuro-Fuzzy Models: Advanced Aspects	391	(60)
Different Input Spaces	391	(6)
Identification of Direction Dependent Behavior	395	(2)
More Complex Local Models	397	(7)
From Local Neuro-Fuzzy Models to Polynomials	397	(3)
Local Quadratic Models for Input Optimization	400	(2)
Different Types of Local Models	402	(2)
Structure Optimization of the Rule Consequents	404	(4)
Interpolation and Extrapolation Behavior	408	(8)
Interpolation Behavior	408	(3)
Extrapolation Behavior	411	(5)
Global and Local Linearization	416	(4)
Online Learning	420	(10)
Online Adaptation of the Rule Consequents	421	(7)
Online Construction of the Rule Premise Structure	428	(2)
Errorbars and Design of Excitation Signals	430	(7)
Errorbars	431	(3)
Detecting Extrapolation	434	(1)
Design of Excitation Signals	435	(1)
Active Learning	436	(1)
Hinging Hyperplanes	437	(7)
Hinging Hyperplanes	438	(1)
Smooth Hinging Hyperplanes	439	(2)
Hinging Hyperplane Trees (HHT)	441	(2)
Comparison with Local Linear Neuro-Fuzzy Models	443	(1)
Summary and Conclusions	444	(7)
Summary of Part II	451	(6)
Part III. Dynamic Models
Linear Dynamic System Identification	457	(90)
Overview of Linear System Identification	458	(1)
Excitation Signals	459	(3)
General Model Structure	462	(16)
Terminology and Classification	465	(6)
Optimal Predictor	471	(3)
Some Remarks on the Optimal Predictor	474	(2)
Prediction Error Methods	476	(2)
Time Series Models	478	(4)
Autoregressive (AR)	479	(1)
Moving Average (MA)	480	(1)
Autoregressive Moving Average (ARMA)	481	(1)
Models with Output Feedback	482	(27)
Autoregressive with Exogenous Input (ARX)	482	(10)
Autoregressive Moving Average with Exogenous Input	492	(4)
Autoregressive Autoregressive with Exogenous Input	496	(3)
Output Error (OE)	499	(4)
Box-Jenkins (BJ)	503	(2)
State Space Models	505	(1)
Simulation Example	506	(3)
Models without Output Feedback	509	(15)
Finite Impulse Response (FIR)	510	(2)
Orthonormal Basis Functions (OBF)	512	(8)
Simulation Example	520	(4)
Some Advanced Aspects	524	(7)
Initial Conditions	524	(2)
Consistency	526	(1)
Frequency-Domain Interpretation	526	(2)
Relationship between Noise Model and Filtering	528	(1)
Offsets	529	(2)
Recursive Algorithms	531	(5)
Recursive Least Squares (RLS) Method	532	(1)
Recursive Instrumental Variables (RIV) Method	532	(1)
Recursive Extended Least Squares (RELS) Method	533	(1)
Recursive Prediction Error Methods (RPEM)	534	(2)
Determination of Dynamic Orders	536	(1)
Multivariable Systems	537	(4)
P-Canonical Model	539	(1)
Matrix Polynomial Model	540	(1)
Subspace Methods	541	(1)
Closed-Loop Identification	541	(5)
Direct Methods	542	(2)
Indirect Methods	544	(1)
Identification for Control	545	(1)
Summary	546	(1)
Nonlinear Dynamic System Identification	547	(32)
From Linear to Nonlinear System Identification	547	(2)
External Dynamics	549	(14)
Illustration of the External Dynamics Approach	550	(5)
Series-Parallel and Parallel Models	555	(2)
Nonlinear Dynamic Input/Output Model Classes	557	(5)
Restrictions of Nonlinear Input/Output Models	562	(1)
Internal Dynamics	563	(1)
Parameter Scheduling Approach	564	(1)
Training Recurrent Structures	564	(4)
Backpropagation-Through-Time (BPTT) Algorithm	565	(2)
Real Time Recurrent Learning	567	(1)
Multivariable Systems	568	(1)
Excitation Signals	569	(5)
Determination of Dynamic Orders	574	(2)
Summary	576	(3)
Classical Polynomial Approaches	579	(8)
Properties of Dynamic Polynomial Models	580	(1)
Kolmogorov-Gabor Polynomial Models	581	(1)
Volterra-Series Models	582	(1)
Parametric Volterra-Series Models	583	(1)
NDE Models	583	(1)
Hammerstein Models	584	(1)
Wiener Models	585	(2)
Dynamic Neural and Fuzzy Models	587	(14)
Curse of Dimensionality	587	(2)
MLP Networks	588	(1)
RBF Networks	588	(1)
Singleton Fuzzy and NRBF Models	588	(1)
Interpolation and Extrapolation Behavior	589	(2)
Training	591	(2)
MLP Networks	592	(1)
RBF Networks	592	(1)
Singleton Fuzzy and NRBF Models	592	(1)
Integration of a Linear Model	593	(1)
Simulation Examples	594	(6)
MLP Networks	595	(2)
RBF Networks	597	(2)
Singleton Fuzzy and NRBF Models	599	(1)
Summary	600	(1)
Dynamic Local Linear Neuro-Fuzzy Models	601	(44)
One-Step Prediction Error Versus Simulation Error	604	(2)
Determination of the Rule Premises	606	(2)
Linearization	608	(5)
Static and Dynamic Linearization	608	(2)
Dynamics of the Linearized Model	610	(2)
Different Rule Consequent Structures	612	(1)
Model Stability	613	(5)
Influence of Rule Premise Inputs on Stability	614	(2)
Lyapunov Stability and Linear Matrix Inequalities	616	(1)
Ensuring Stable Extrapolation	617	(1)
Dynamic LOLIMOT Simulation Studies	618	(8)
Nonlinear Dynamic Test Processes	618	(2)
Hammerstein Process	620	(4)
Wiener Process	624	(1)
NDE Process	625	(1)
Dynamic Nonlinearity Process	625	(1)
Advanced Local Linear Methods and Models	626	(5)
Local Linear Instrumental Variables (IV) Method	628	(2)
Local Linear Output Error (OE) Models	630	(1)
Local Linear ARMAX Models	631	(1)
Local Linear Orthonormal Basis Functions Models	631	(5)
Structure Optimization of the Rule Consequents	636	(4)
Summary and Conclusions	640	(5)
Neural Networks with Internal Dynamics	645	(10)
Fully Recurrent Networks	645	(1)
Partially Recurrent Networks	646	(1)
State Recurrent Networks	647	(1)
Locally Recurrent Globally Feedforward Networks	648	(2)
Internal Versus External Dynamics	650	(5)
Part IV. Applications
Applications of Static Models	655	(22)
Driving Cycle	655	(4)
Process Description	656	(1)
Smoothing of a Driving Cycle	657	(1)
Improvements and Extensions	658	(1)
Differentiation	659	(1)
Modeling and Optimization of Combustion Engine Exhaust	659	(15)
The Role of Look-Up Tables	660	(3)
Modeling of Exhaust Gases	663	(3)
Optimization of Exhaust Gases	666	(6)
Outlook: Dynamic Models	672	(2)
Summary	674	(3)
Applications of Dynamic Models	677	(32)
Cooling Blast	677	(6)
Process Description	677	(2)
Experimental Results	679	(4)
Diesel Engine Turbocharger	683	(8)
Process Description	684	(1)
Experimental Results	685	(6)
Thermal Plant	691	(16)
Process Description	692	(1)
Transport Process	693	(5)
Tubular Heat Exchanger	698	(4)
Cross-Flow Heat Exchanger	702	(5)
Summary	707	(2)
Applications of Advanced Methods	709	(26)
Nonlinear Model Predictive Control	709	(4)
Online Adaption	713	(10)
Variable Forgetting Factor	714	(1)
Control and Adaptation Models	715	(2)
Parameter Transger	717	(1)
Systems with Multiple Inputs	718	(1)
Experimental Results	719	(4)
Fault Detection	723	(6)
Methodology	723	(3)
Experimental Results	726	(3)
Fault Diagnosis	729	(3)
Methodology	729	(2)
Experimental Results	731	(1)
Reconfiguration	732	(3)
A. Vectors and Matrices	735	(4)
Vector and Matrix Derivatives	735	(2)
Gradient, Hessian, and Jacobian	737	(2)
B. Statistics	739	(18)
Deterministic and Random Variables	739	(2)
Probability Density Function (pdf)	741	(2)
Stochastic Processes and Ergodicity	743	(2)
Expectation	745	(3)
Variance	748	(1)
Correlation and Covariance	749	(4)
Properties of Estimators	753	(4)
References	757	(22)
Index	779