Pulsed Neural Networks

by Maass, Wolfgang; Bishop, Christopher M.

• ISBN13: 9780262133500
• ISBN10: 0262133504
• Format: Hardcover
• Copyright: 1998-11-20
• Publisher: Bradford Books

Summary

Most practical applications of artificial neural networks are based on a computational model involving the propagation of continuous variables from one processing unit to the next. In recent years, data from neurobiological experiments have made it increasingly clear that biological neural networks, which communicate through pulses, use the timing of the pulses to transmit information and perform computation. This realization has stimulated significant research on pulsed neural networks, including theoretical analyses and model development, neurobiological modeling, and hardware implementation. This book presents the complete spectrum of current research in pulsed neural networks and includes the most important work from many of the key scientists in the field. Terrence J. Sejnowski's foreword, "Neural Pulse Coding," presents an overview of the topic. The first half of the book consists of longer tutorial articles spanning neurobiology, theory, algorithms, and hardware. The second half contains a larger number of shorter research chapters that present more advanced concepts. The contributors use consistent notation and terminology throughout the book. Contributors: Peter S. Burge, Stephen R. Deiss, Rodney J. Douglas, John G. Elias, Wulfram Gerstner, Alister Hamilton, David Horn, Axel Jahnke, Richard Kempter, Wolfgang Maass, Alessandro Mortara, Alan F. Murray, David P. M. Northmore, Irit Opher, Kostas A. Papathanasiou, Michael Recce, Barry J. P. Rising, Ulrich Roth, Tim Schouml;nauer, Terrence J. Sejnowski, John Shawe-Taylor, Max R. van Daalen, J. Leo van Hemmen, Philippe Venier, Hermann Wagner, Adrian M. Whatley, Anthony M. Zador.

Table of Contents

Foreword	xiii	(12)
Terrence J. Sejnowski
Preface	xxv	(2)
Contributors to the book	xxvii
Basic Concepts and Models	1	(132)
1 Spiking Neurons	3	(52)
1.1 The Problem of Neural Coding	3	(13)
1.1.1 Motivation	3	(4)
1.1.2 Rate Codes	7	(4)
1.1.3 Candidate Pulse Codes	11	(4)
1.1.4 Discussion: Spikes or Rates?	15	(1)
1.2 Neuron Models	16	(31)
1.2.1 Simple Spiking Neuron Model	17	(3)
1.2.2 First Steps towards Coding by Spikes	20	(3)
1.2.3 Threshold-Fire Models	23	(11)
1.2.4 Conductance-Based Models	34	(11)
1.2.5 Rate Models	45	(2)
1.3 Conclusions	47	(1)
References	48	(7)
2 Computing with Spiking Neurons	55	(32)
2.1 Introduction	55	(1)
2.2 A Formal Computational Model for a Network of Spiking Neurons	55	(2)
2.3 McCulloch-Pitts Neurons versus Spiking Neurons	57	(2)
2.4 Computing with Temporal Patterns	59	(9)
2.4.1 Conincidence Detection	59	(3)
2.4.2 RBF-Units in the Temporal Domain	62	(1)
2.4.3 Computing a Weighted Sum in Temporal Coding	63	(2)
2.4.4 Universal Approximation of Continuous Functions with Spiking Neurons	65	(2)
2.4.5 Other Computations with Temporal Patterns in Networks of Spiking Neurons	67	(1)
2.5 Computing with a Space-Rate Code	68	(5)
2.6 Computing with Firing Rates	73	(1)
2.7 Firing Rates and Temporal Correlations	74	(5)
2.8 Networks of Spiking Neurons for Storing and Retrieving Information	79	(1)
2.9 Computing on Spike Trains	80	(1)
2.10 Conclusions	80	(1)
References	81	(6)
3 Pulse-Based Computation in VLSI Neural Networks	87	(24)
3.1 Background	87	(1)
3.2 Pulsed Coding: A VLSI Perspective	88	(5)
3.2.1 Pulse Amplitude Modulation	90	(1)
3.2.2 Pulse Width Modulation	90	(1)
3.2.3 Pulse Frequency Modulation	91	(1)
3.2.4 Phase or Delay Modulation	91	(1)
3.2.5 Noise, Robustness, Accuracy and Speed	92	(1)
3.3 A MOSFET Introduction	93	(4)
3.3.1 Subthreshold Circuits for Neural Networks	96	(1)
3.4 Pulse Generation in VLSI	97	(2)
3.4.1 Pulse Intercommunication	99	(1)
3.5 Pulsed Arithmetic in VLSI	99	(5)
3.5.1 Addition of Pulse Stream Signals	99	(2)
3.5.2 Multiplication of Pulse Stream Signals	101	(1)
3.5.3 MOS Transconductance Multiplier	102	(1)
3.5.4 MOSFET Analog Multiplier	102	(2)
3.6 Learning in Pulsed Systems	104	(1)
3.7 Summary and Issues Raised	105	(2)
References	107	(4)
4 Encoding Information in Neuronal Activity	111	(22)
4.1 Introduction	111	(3)
4.2 Synchronization and Oscillations	114	(2)
4.3 Temporal Binding	116	(2)
4.4 Phase Coding	118	(1)
4.5 Dynamic Range and Firing Rate Codes	119	(1)
4.6 Interspike Interval Variability	120	(4)
4.7 Synapses and Rate Coding	124	(1)
4.8 Summary and Implications	125	(2)
References	127	(6)
Implementations	133	(126)
5 Building Silicon Nervous Systems with Dendritic Tree Neuromorphs	135	(22)
5.1 Introduction	135	(3)
5.1.1 Why Spikes?	135	(1)
5.1.2 Dendritic Processing of Spikes	136	(1)
5.1.3 Tunability	137	(1)
5.2 Implementation in VLSI	138	(8)
5.2.1 Artificial Dendrites	138	(1)
5.2.2 Synapses	139	(2)
5.2.3 Dendritic Non-Linearities	141	(1)
5.2.4 Spike-Generating Soma	142	(1)
5.2.5 Excitability Control	143	(1)
5.2.6 Spike Distribution -- Virtual Wires	144	(2)
5.3 Neuromorphs in Action	146	(7)
5.3.1 Feedback to Threshold-Setting Synapses	147	(1)
5.3.2 Discrimination of Complex Spatio-Temporal Patterns	147	(2)
5.3.3 Processing of Temporally Encoded Information	149	(4)
5.4 Conclusions	153	(1)
References	154	(3)
6 A Pulse-Coded Communications Infrastructure	157	(22)
6.1 Introduction	157	(1)
6.2 Neuromorphic Computational Nodes	158	(1)
6.3 Neuromorphic a VLSI Neurons	158	(4)
6.4 Address Event Representation (AER)	162	(3)
6.5 Implementations of AER	165	(1)
6.6 Silicon Cortex	166	(4)
6.6.1 Basic Layout	168	(2)
6.7 Functional Tests of Silicon Cortex	170	(4)
6.7.1 An Example Neuronal Network	170	(2)
6.7.2 An Example of Sensory Input to SCX	172	(2)
6.8 Future Research on AER Neuromorphic Systems	174	(2)
References	176	(3)
7 Analog VLSI Pulsed Networks for Perceptive Processing	179	(38)
7.1 Introduction	179	(1)
7.2 Analog Perceptive Nets Communication Requirements	180	(3)
7.2.1 Coding Information with Pulses	180	(1)
7.2.2 Multiplexing of the Signals Issued by Each Neuron	181	(1)
7.2.3 Non-Arbitered PFM Communication	182	(1)
7.3 Analysis of the NAPFM Communication Systems	183	(10)
7.3.1 Statistical Assumptions	183	(1)
7.3.2 Detection	184	(2)
7.3.3 Performance	186	(3)
7.3.4 Data Dependency of System Performance	189	(2)
7.3.5 Discussion	191	(2)
7.4 Address Coding	193	(1)
7.5 Silicon Retina Equipped with the NAPFM Communication System	194	(5)
7.5.1 Circuit Description	194	(3)
7.5.2 Noise Measurement Results	197	(2)
7.6 Projective Field Generation	199	(8)
7.6.1 Overview	199	(2)
7.6.2 Anisotropic Current Pulse Spreading in a Nonlinear Network	201	(1)
7.6.3 Analysis of the Spatial Response of the Nonlinear Network	202	(2)
7.6.4 Analysis of the Size and Shape of the Bubbles Generable by the Nonlinear Network	204	(3)
7.7 Description of the Integrated Circuit for Orientation Enhancement	207	(4)
7.7.1 Overview	207	(1)
7.7.2 Circuit Description	207	(1)
7.7.3 System Measurement Results	208	(1)
7.7.4 Other Applications	209	(2)
7.8 Display Interface	211	(3)
7.9 Conclusion	214	(1)
References	215	(2)
8 Preprocessing for Pulsed Neural VLSI Systems	217	(20)
8.1 Introduction	217	(1)
8.2 A Sound Segmentation System	217	(2)
8.3 Signal Processing in Analog VLSI	219	(5)
8.3.1 Continuous Time Active Filters	220	(1)
8.3.2 Sampled Data Active Switched Capacitor (SC) Filters	220	(1)
8.3.3 Sampled Data Active Switched Current (SI) Filters	221	(2)
8.3.4 Discussion	223	(1)
8.4 Palmo -- Pulse Based Signal Processing	224	(9)
8.4.1 Basic Palmo Concepts	224	(2)
8.4.2 A CMOS Analog Palmo Cell Implementation	226	(2)
8.4.3 Interconnecting Analog Palmo Cells	228	(3)
8.4.4 Results from a Palmo VLSI Device	231	(1)
8.4.5 Digital Processing of Palmo Signals	232	(1)
8.4.6 CMOS Analog Palmo Cell: Performance	233	(1)
8.5 Conclusions	233	(1)
8.6 Further Work	234	(1)
8.7 Acknowledgements	234	(1)
References	235	(2)
9 Digital Simulation of Spiking Neural Networks	237	(22)
9.1 Introduction	237	(1)
9.2 Implementation Issues of Pulse-Coded Neural Networks	238	(4)
9.2.1 Discrete-Time Simulation	239	(1)
9.2.2 Requisite Arithmetic Precision	240	(2)
9.2.3 Basic Procedures of Network Computation	242	(1)
9.3 Programming Environment	242	(2)
9.4 Concepts of Efficient Simulation	244	(3)
9.5 Mapping Neural Networks on Parallel Computers	247	(4)
9.5.1 Neuron-Parallelism	248	(1)
9.5.2 Synapse-Parallelism	248	(1)
9.5.3 Pattern-Parallelism	249	(1)
9.5.4 Partitioning of the Network	249	(2)
9.6 Performance Study	251	(5)
9.6.1 Single PE Workstations	251	(1)
9.6.2 Neurocomputer	251	(2)
9.6.3 Parallel Computers	253	(1)
9.6.4 Results of the Performance Study	254	(1)
9.6.5 Conclusions	255	(1)
References	256	(3)
Design and Analysis of Pulsed Neural Systems	259
10 Populations of Spiking Neurons	261	(36)
10.1 Introduction	261	(2)
10.2 Model	263	(2)
10.3 Population Activity Equation	265	(3)
10.3.1 Integral Equation for the Dynamics	265	(2)
10.3.2 Normalization	267	(1)
10.4 Noise-Free Population Dynamics	268	(1)
10.5 Locking	269	(7)
10.5.1 Locking Condition	270	(3)
10.5.2 Graphical Interpretation	273	(3)
10.6 Transients	276	(3)
10.7 Incoherent Firing	279	(12)
10.7.1 Determination of the Activity	280	(3)
10.7.2 Stability of Asynchronous Firing	283	(8)
10.8 Conclusions	291	(2)
References	293	(4)
11 Collective Excitation Phenomena and Their Applications	297	(24)
11.1 Introduction	297	(2)
11.1.1 Two Variable Formulation of IAF Neurons	298	(1)
11.2 Synchronization of Pulse Coupled Oscillators	299	(3)
11.3 Clustering via Temporal Segmentation	302	(2)
11.4 Limits on Temporal Segmentation	304	(3)
11.5 Image Analysis	307	(4)
11.5.1 Image Segmentation	308	(1)
11.5.2 Edge Detection	309	(2)
11.6 Solitary Waves	311	(3)
11.7 The Importance of Noise	314	(1)
11.8 Conclusions	315	(2)
References	317	(4)
12 Computing and Learning with Dynamic Synapses	321	(16)
12.1 Introduction	321	(1)
12.2 Biological Data on Dynamic Synapses	322	(4)
12.3 Quantitative Models	326	(3)
12.4 On the Computational Role of Dynamic Synapses	329	(4)
12.5 Implications for Learning in Pulsed Neural Nets	333	(1)
12.6 Conclusions	334	(1)
References	335	(2)
13 Stochastic Bit-Stream Neural Networks	337	(16)
13.1 Introduction	338	(1)
13.2 Basic Neural Modelling	338	(3)
13.3 Feedforward Networks and Learning	341	(2)
13.3.1 Probability Level Learning	341	(1)
13.3.2 Bit-Stream Level Learning	342	(1)
13.4 Generalization Analysis	343	(1)
13.5 Recurrent Networks	344	(1)
13.6 Applications to Graph Colouring	344	(2)
13.7 Hardware Implementation	346	(3)
13.7.1 The Stochastic Neuron	346	(2)
13.7.2 Calculating Output Derivatives	348	(1)
13.7.3 Generating Stochastic Bit-Streams	348	(1)
13.7.4 Recurrent Networks	349	(1)
13.8 Conclusions	349	(2)
References	351	(2)
14 Hebbian Learning of Pulse Timing in the Barn Owl Auditory System	353
14.1 Introduction	353	(1)
14.2 Hebbian Learning	354	(7)
14.2.1 Review of Standard Formulations	354	(1)
14.2.2 Spike-Based Learning	355	(3)
14.2.3 Example	358	(1)
14.2.4 Learning Window	359	(2)
14.3 Barn Owl Auditory System	361	(3)
14.3.1 The Localization Task	361	(1)
14.3.2 Auditory Localization Pathway	362	(2)
14.4 Phase Locking	364	(4)
14.4.1 Neuron Model	364	(1)
14.4.2 Phase Locking -- Schematic	365	(1)
14.4.3 Simulation Results	366	(2)
14.5 Delay Tuning by Hebbian Learning	368	(3)
14.5.1 Motivation	368	(1)
14.5.2 Selection of Delays	369	(2)
14.6 Conclusions	371	(2)
References	373