Recurrent Neural Networks for Prediction Learning Algorithms, Architectures and Stability

by Mandic, Danilo P.; Chambers, Jonathon A.

• ISBN13: 9780471495178
• ISBN10: 0471495174
• Edition: 1st
• Format: Hardcover
• Copyright: 2001-09-05
• Publisher: WILEY

Summary

New technologies in engineering, physics and biomedicine are demanding increasingly complex methods of digital signal processing. By presenting the latest research work the authors demonstrate how real-time recurrent neural networks (RNNs) can be implemented to expand the range of traditional signal processing techniques and to help combat the problem of prediction. Within this text neural networks are considered as massively interconnected nonlinear adaptive filters. ? Analyses the relationships between RNNs and various nonlinear models and filters, and introduces spatio-temporal architectures together with the concepts of modularity and nesting ? Examines stability and relaxation within RNNs ? Presents on-line learning algorithms for nonlinear adaptive filters and introduces new paradigms which exploit the concepts of a priori and a posteriori errors, data-reusing adaptation, and normalisation ? Studies convergence and stability of on-line learning algorithms based upon optimisation techniques such as contraction mapping and fixed point iteration ? Describes strategies for the exploitation of inherent relationships between parameters in RNNs ? Discusses practical issues such as predictability and nonlinearity detecting and includes several practical applications in areas such as air pollutant modelling and prediction, attractor discovery and chaos, ECG signal processing, and speech processing Recurrent Neural Networks for Prediction offers a new insight into the learning algorithms, architectures and stability of recurrent neural networks and, consequently, will have instant appeal. It provides an extensive background for researchers, academics and postgraduates enabling them to apply such networks in new applications. VISIT OUR COMMUNICATIONS TECHNOLOGY WEBSITE! http://www.wiley.co.uk/commstech/ VISIT OUR WEB PAGE! http://www.wiley.co.uk/

Author Biography

Danilo Mandic from the Imperial College London, London, UK was named Fellow of the Institute of Electrical and Electronics Engineers in 2013 for contributions to multivariate and nonlinear learning systems.

Jonathon A. Chambers is the author of Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability, published by Wiley.

Table of Contents

Preface	xv
Introduction	1	(8)
Some Important Dates in the History of Connectionism	2	(1)
The Structure of Neural Networks	2	(2)
Perspective	4	(1)
Neural Networks for Prediction: Perspective	5	(1)
Structure of the Book	6	(2)
Readership	8	(1)
Fundamentals	9	(22)
Perspective	9	(1)
Chapter Summary	9	(1)
Adaptive Systems	9	(3)
Configurations of Adaptive Systems Used in Signal Processing	10	(2)
Blind Adaptive Techniques	12	(1)
Gradient-Based Learning Algorithms	12	(2)
A General Class of Learning Algorithms	14	(1)
Quasi-Newton Learning Algorithms	14	(1)
A Step-by-Step Derivation of the Least Mean Square (LMS) Algorithm	15	(3)
The Wiener Filter	16	(1)
Further Perspective on the Least Mean Square (LMS) Algorithm	17	(1)
On Gradient Descent for Nonlinear Structures	18	(1)
Extension to a General Neural Network	19	(1)
On Some Important Notions From Learning Theory	19	(5)
Relationship Between the Error and the Error Function	19	(1)
The Objective Function	20	(1)
Types of Learning with Respect to the Training Set and Objective Function	20	(1)
Deterministic, Stochastic and Adaptive Learning	21	(1)
Constructive Learning	21	(1)
Transformation of Input Data, Learning and Dimensionality	22	(2)
Learning Strategies	24	(1)
General Framework for the Training of Recurrent Networks by Gradient-Descent-Based Algorithms	24	(2)
Adaptive Versus Nonadaptive Training	24	(1)
Performance Criterion, Cost Function, Training Function	25	(1)
Recursive Versus Nonrecursive Algorithms	25	(1)
Iterative Versus Noniterative Algorithms	25	(1)
Supervised Versus Unsupervised Algorithms	25	(1)
Pattern Versus Batch Learning	26	(1)
Modularity Within Neural Networks	26	(3)
Summary	29	(2)
Network Architectures for Prediction	31	(16)
Perspective	31	(1)
Introduction	31	(1)
Overview	32	(1)
Prediction	33	(2)
Building Blocks	35	(2)
Linear Filters	37	(2)
Nonlinear Predictors	39	(2)
Feedforward Neural Networks: Memory Aspects	41	(2)
Recurrent Neural Networks: Local and Global Feedback	43	(1)
State-Space Representation and Canonical Form	44	(1)
Summary	45	(2)
Activation Functions Used in Neural Networks	47	(22)
Perspective	47	(1)
Introduction	47	(4)
Overview	51	(1)
Neural Networks and Universal Approximation	51	(3)
Other Activation Functions	54	(3)
Implementation Driven Choice of Activation Functions	57	(3)
MLP versus RBF Networks	60	(1)
Complex Activation Functions	60	(5)
Complex Valued Neural Networks as Modular Groups of Compositions of Mobius Transformations	65	(3)
Mobius Transformation	65	(1)
Activation Functions and Mobius Transformations	65	(2)
Existence and Uniqueness of Fixed Points in a Complex Neural Network via Theory of Modular Groups	67	(1)
Summary	68	(1)
Recurrent Neural Networks Architectures	69	(22)
Perspective	69	(1)
Introduction	69	(3)
Overview	72	(1)
Basic Modes of Modelling	72	(2)
Parametric versus Nonparametric Modelling	72	(1)
White, Grey and Black Box Modelling	73	(1)
NARMAX Models and Embedding Dimension	74	(1)
How Dynamically Rich are Nonlinear Neural Models?	75	(2)
Feedforward versus Recurrent Networks for Nonlinear Modelling	76	(1)
Wiener and Hammerstein Models and Dynamical Neural Networks	77	(4)
Overview of Block-Stochastic Models	77	(1)
Connection Between Block-Stochastic Models and Neural Networks	78	(3)
Recurrent Neural Network Architectures	81	(3)
Hybrid Neural Network Architectures	84	(2)
Nonlinear ARMA Models and Recurrent Networks	86	(3)
Summary	89	(2)
Neural Networks as Nonlinear Adaptive Filters	91	(24)
Perspective	91	(1)
Introduction	91	(1)
Overview	92	(1)
Neural Networks and Polynomial Filters	92	(3)
Neural Networks and Nonlinear Adaptive Filters	95	(6)
Training Algorithms for Recurrent Neural Networks	101	(1)
Learning Strategies for a Neural Predictor/Identifier	101	(4)
Learning Strategies for a Neural Adaptive Recursive Filter	103	(1)
Equation Error Formulation	104	(1)
Output Error Formulation	104	(1)
Filter Coefficient Adaptation for IIR Filters	105	(2)
Equation Error Coefficient Adaptation	107	(1)
Weight Adaptation for Recurrent Neural Networks	107	(2)
Teacher Forcing Learning for a Recurrent Perceptron	108	(1)
Training Process for a NARMA Neural Predictor	109	(1)
The Problem of Vanishing Gradients in Training of Recurrent Neural Networks	109	(2)
Learning Strategies in Different Engineering Communities	111	(1)
Learning Algorithms and the Bias/Variance Dilemma	111	(2)
Recursive and Iterative Gradient Estimation Techniques	113	(1)
Exploiting Redundancy in Neural Network Design	113	(1)
Summary	114	(1)
Stability Issues in RNN Architectures	115	(20)
Perspective	115	(1)
Introduction	115	(3)
Overview	118	(1)
A Fixed Point Interpretation of Convergence in Networks with a Sigmoid Nonlinearity	118	(6)
Some Properties of the Logistic Function	118	(3)
Logistic Function, Rate of Convergence and Fixed Point Theory	121	(3)
Convergence of Nonlinear Relaxation Equations Realised Through a Recurrent Perceptron	124	(3)
Relaxation in Nonlinear Systems Realised by an RNN	127	(3)
The Iterative Approach and Nesting	130	(3)
Upper Bounds for GAS Relaxation within FCRNNs	133	(1)
Summary	133	(2)
Data-Reusing Adaptive Learning Algorithms	135	(14)
Perspective	135	(1)
Introduction	135	(3)
Towards an A Posteriori Nonlinear Predictor	136	(1)
Note on the Computational Complexity	137	(1)
Chapter Summary	138	(1)
A Class of Simple A Posteriori Algorithms	138	(4)
The Case of a Recurrent Neural Filter	140	(1)
The Case of a General Recurrent Neural Network	141	(1)
Example for the Logistic Activation Function	141	(1)
An Iterated Data-Reusing Learning Algorithm	142	(1)
The Case of a Recurrent Predictor	143	(1)
Convergence of the A Posteriori Approach	143	(1)
A Posteriori Error Gradient Descent Algorithm	144	(2)
A Posteriori Error Gradient Algorithm for Recurrent Neural Networks	146	(1)
Experimental Results	146	(1)
Summary	147	(2)
A Class of Normalised Algorithms for Online Training of Recurrent Neural Networks	149	(12)
Perspective	149	(1)
Introduction	149	(1)
Overview	150	(1)
Derivation of the Normalised Adaptive Learning Rate for a Simple Feedforward Nonlinear Filter	151	(5)
A Normalised Algorithm for Online Adaptation of Recurrent Neural Networks	156	(4)
Summary	160	(1)
Convergence of Online Learning Algorithms in Neural Networks	161	(10)
Perspective	161	(1)
Introduction	161	(3)
Overview	164	(1)
Convergence Analysis of Online Gradient Descent Algorithms for Recurrent Neural Adaptive Filters	164	(3)
Mean-Squared and Steady-State Mean-Squared Error Convergence	167	(2)
Convergence in the Mean Square	168	(1)
Steady-State Mean-Squared Error	169	(1)
Summary	169	(2)
Some Practical Considerations of Predictability and Learning Algorithms for Various Signals	171	(28)
Perspective	171	(1)
Introduction	171	(3)
Detecting Nonlinearity in Signals	173	(1)
Overview	174	(1)
Measuring the Quality of Prediction and Detecting Nonlinearity within a Signal	174	(7)
Deterministic Versus Stochastic Plots	175	(1)
Variance Analysis of Delay Vectors	175	(1)
Dynamical Properties of NO2 Air Pollutant Time Series	176	(5)
Experiments on Heart Rate Variability	181	(14)
Experimental Results	181	(14)
Prediction of the Lorenz Chaotic Series	195	(2)
Bifurcations in Recurrent Neural Networks	197	(1)
Summary	198	(1)
Exploiting Inherent Relationships Between Parameters in Recurrent Neural Networks	199	(22)
Perspective	199	(1)
Introduction	199	(5)
Overview	204	(1)
Static and Dynamic Equivalence of Two Topologically Identical RNNs	205	(4)
Dynamic Equivalence of Two Isomorphic RNNs	205	(3)
Extension to a General RTRL Trained RNN	208	(1)
Extension to Other Commonly Used Activation Functions	209	(1)
Extension to Other Commonly Used Learning Algorithms for Recurrent Neural Networks	209	(1)
Relationships Between β and n for the Backpropagation Through Time Algorithm	210	(3)
Results for the Recurrent Backpropagation Algorithm	211	(1)
Results for Algorithms with a Momentum Term	211	(1)
Simulation Results	212	(1)
Summary of Relationships Between β and n for General Recurrent Neural Networks	213	(1)
Relationship Between n and β for Modular Neural Networks: Perspective	214	(1)
Static Equivalence Between an Arbitrary and a Referent Modular Neural Network	214	(1)
Dynamic Equivalence Between an Arbitrary and a Referent Modular Network	215	(1)
Dynamic Equivalence for a GD Learning Algorithm	216	(3)
Dynamic Equivalence Between Modular Recurrent Neural Networks for the ERLS Learning Algorithm	217	(1)
Equivalence Between an Arbitrary and the Referent PRNN	218	(1)
Note on the β-n-W Relationships and Contractivity	218	(1)
Summary	219	(2)
Appendix A The O Notation and Vector and Matrix Differentiation	221	(2)
A.1 The O Notation	221	(1)
A.2 Vector and Matrix Differentiation	221	(2)
Appendix B Concepts from the Approximation Theory	223	(4)
Appendix C Complex Sigmoid Activation Functions, Holomorphic Mappings and Modular Groups	227	(4)
C.1 Complex Sigmoid Activation Functions	227	(4)
C.1.1 Modular Groups	228	(3)
Appendix D Learning Algorithms for RNNs	231	(8)
D.1 The RTRL Algorithm	231	(3)
D.1.1 Teacher Forcing Modification of the RTRL Algorithm	234	(1)
D.2 Gradient Descent Learning Algorithm for the PRNN	234	(2)
D.3 The ERLS Algorithm	236	(3)
Appendix E Terminology Used in the Field of Neural Networks	239	(2)
Appendix F On the A Posteriori Approach in Science and Engineering	241	(4)
F.1 History of A Posteriori Techniques	241	(1)
F.2 The Usage of A Posteriori	242	(3)
F.2.1 A Posteriori Techniques in the RNN Framework	242	(1)
F.2.2 The Geometric Interpretation of A Posteriori Error Learning	243	(2)
Appendix G Contraction Mapping Theorems	245	(6)
G.1 Fixed Points and Contraction Mapping Theorems	245	(1)
G.1.1 Contraction Mapping Theorem in R	245	(1)
G.1.2 Contraction Mapping Theorem in RN	246	(1)
G.2 Lipschitz Continuity and Contraction Mapping	246	(1)
G.3 Historical Perspective	247	(4)
Appendix H Linear GAS Relaxation	251	(12)
H.1 Relaxation in Linear Systems	251	(2)
H.1.1 Stability Result for	253	(1)
H.2 Examples	253	(10)
Appendix I The Main Notions in Stability Theory	263	(2)
Appendix J Deseasonalising Time Series	265	(2)
References	267	(14)
Index	281