PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory and information processing in neuromorphic systems

Giacomo Indiveri, Shih-Chii Liu

arXiv:1506.03264v1cs.NE

TL;DR

The paper addresses how neuromorphic architectures can organize memory and processing to support brain-inspired computation beyond conventional von Neumann designs. It surveys systems ranging from programmable digital platforms to mixed-signal and biologically plausible processors, comparing their capabilities and trade-offs. These systems distribute or co-localize memory with processing and support models spanning deep networks, cortical networks, neural dynamics, and plasticity, while remaining subject to implementation constraints.

  • Problem

    The paper examines alternatives to von Neumann architectures whose separated processors and memories create limited throughput and energy-related constraints for neural computation.

  • Method

    The paper surveys neuromorphic systems supporting deep networks, cortical networks, and brain-inspired cognitive architectures across digital, analog, mixed-signal, programmable, and custom hardware approaches.

  • Results

    The surveyed systems span architectures with co-localized memory and computation, biologically plausible neural dynamics, realistic plasticity, asynchronous event processing, and large-scale neural simulation capabilities.

  • Takeaways & Limitations

    Neuromorphic architectures offer different trade-offs among memory requirements, processing speed, bandwidth, biological computational primitives, configurability, and power consumption.

  • Takeaways & Limitations

    Custom hardware can restrict neural-model exploration: NeuroGrid and BrainScales use hard-wired models, while SpiNNaker is constrained by memory, resolution, and fixed-point representation.

Abstract

from arXiv · show

A striking difference between brain-inspired neuromorphic processors and current von Neumann processors architectures is the way in which memory and processing is organized. As Information and Communication Technologies continue to address the need for increased computational power through the increase of cores within a digital processor, neuromorphic engineers and scientists can complement this need by building processor architectures where memory is distributed with the processing. In this paper we present a survey of brain-inspired processor architectures that support models of cortical networks and deep neural networks. These architectures range from serial clocked implementations of multi-neuron systems to massively parallel asynchronous ones and from purely digital systems to mixed analog/digital systems which implement more biological-like models of neurons and synapses together with a suite of adaptation and learning mechanisms analogous to the ones found in biological nervous systems. We describe the advantages of the different approaches being pursued and present the challenges that need to be addressed for building artificial neural processing systems that can display the richness of behaviors seen in biological systems.

I. INTRODUCTION

Neuromorphic systems organize memory and processing according to biological nervous systems, offering alternatives to von Neumann architectures. The paper surveys approaches spanning loosely biology-inspired hardware to systems that integrate computation, memory, neural dynamics, and plasticity.

  • I. INTRODUCTION: Neuromorphic information-processing systems use biological nervous-system design principles in mixed-mode analog/digital CMOS circuits fabricated with standard VLSI processes.They process information using energy-efficient asynchronous, event-driven methods and are often adaptive and fault-tolerant.
  • I. INTRODUCTION: Biological nervous systems motivate alternative non-von-Neumann architectures despite using slow, inhomogeneous, stochastic, and faulty memory and computing elements.Their ability to perform robust computation makes these paradigms attractive for neuromorphic engineering.
  • I. INTRODUCTION: The field has expanded from a small number of research laboratories to academic and industrial groups developing nanoscale synaptic devices and multi-neuron computing platforms.These efforts target pattern recognition and machine-learning applications while exploring materials that emulate biological synapses.
  • I. INTRODUCTION: Neuromorphic approaches differ in their biological fidelity, with some using new devices and circuits to reproduce properties of synapses, neurons, and other neural structures.Analog weak-inversion or sub-threshold FET operation can produce exponential transfer relationships resembling biological dependencies.
  • I. INTRODUCTION: The survey covers approaches from conventional von Neumann neural-network implementations to hardware with co-localized memory and computation and biologically plausible neural dynamics.The reviewed systems also include realistic plasticity mechanisms that merge computation and memory storage within the same circuits.

A. Application areas for neuromorphic systems

Neuromorphic systems are applied to sensory processing, autonomous systems, and large-scale neural simulation or machine learning. Their architectures address conventional memory and throughput constraints through specialized routing, parallelism, and learning mechanisms.

  • A. Application areas for neuromorphic systems: Neuromorphic systems have been proposed for sensory processing and autonomous systems, although the field has relatively few established products and applications.The paper identifies these areas as domains where neuromorphic technology may offer advantages over conventional computers.
  • A. Application areas for neuromorphic systems: Dedicated neural hardware is also used for accelerated large-scale neural simulation and very large-scale spiking neural networks in machine learning.The paper states that dedicated implementations typically provide lower power consumption than general-purpose architectures for these applications.
  • A. Application areas for neuromorphic systems: Deep neural networks use many neuron layers and support classification tasks, while convolutional networks are explored intensively for visual processing.Figure 2 contrasts hierarchical feed-forward convolutional networks with all-to-all deep networks that can include feedback.
  • A. Application areas for neuromorphic systems: Neuromorphic processors can provide learning mechanisms that support autonomous systems adapting to environmental changes and interacting with their surroundings in real time.The cited processor example implements mechanisms intended to provide learning abilities for autonomous behavior.
  • A. Application areas for neuromorphic systems: The von Neumann bottleneck arises because processors and memory are separated and communicate through shared resources with limited throughput.The paper discusses hierarchical caches and parallel processing as conventional mitigations, while noting cache energy consumption grows linearly with cache size.
  • A. Application areas for neuromorphic systems: Custom neural hardware emerged because conventional CPUs and GPUs were not optimally suited to energy-conscious spiking-neural-network simulation.New architectures use memory-access and routing schemes specialized for neural networks.

A. Deep Networks

Deep and convolutional networks motivate specialized architectures that distribute computation, memory, and data movement across processing elements. Examples use dataflow grids, configurable memory routing, and dedicated convolutional operators to scale performance and reduce power.

  • A. Deep Networks: Deep networks contain many neuron layers and include convolutional architectures used for classification and visual processing.Convolutional networks use feed-forward layers in which neurons aggregate weighted inputs from multiple neurons in the preceding layer.
  • A. Deep Networks: Convolutional networks compute thresholded or sigmoidal outputs from weighted sums, with inter-layer connectivity patterns forming convolution kernels.They were inspired by mammalian visual-system structure and have been implemented on CPUs, GPUs, SoCs, and FPGAs.
  • A. Deep Networks: neuFlow implements scalable multilayer synthetic vision with a 2D grid of Processing Tiles containing arithmetic operators, an on-chip router, and configurable memory mapping.Its dataflow architecture is designed to process large data streams in parallel.
  • A. Deep Networks: A neuFlow implementation on a Xilinx Virtex 6 ML605 FPGA demonstrated street-scene segmentation and classification using a four-layer network at 12 frames/second.A simulated custom 45 nm SOI ASIC version was predicted to reach 320 GOPS within a 0.6 W power budget.
  • A. Deep Networks: neuFlow uses asynchronous data transfers with priority management and runtime reconfiguration through Smart DMA, a Flow-CPU, and a runtime configuration bus.A compiler extracts different levels of parallelism from sequential or flow-graph algorithm descriptions.
  • A. Deep Networks: The scalable low-power nn-X system combines a host processor, coprocessor, external memory, processing collections, a memory router, and a configuration bus.Each collection includes a convolution engine, pooling module, and nonlinear operator to address larger networks and memory-bandwidth requirements.

2) Event-Driven Solution:

Event-driven deep-network hardware uses parallel processing, local memory, and specialized routing to support scalable spike-based computation. The Minitaur system illustrates this FPGA approach, while CAVIAR demonstrates earlier multi-chip event-based vision processing.

  • Event-based convolutional systems: CAVIAR pioneered event-based convolutional network chips by combining retina, spiking convolution, winner-take-all, and learning chips.Its later digital convolution chip used a 32×32 pixel array and programmable kernels up to 32×32.
  • Minitaur: Minitaur implements a 784-500-500-10 spiking Deep Belief Network on a Spartan-6 FPGA with 32 parallel cores and 128 MB of DDR2 memory.Each core includes 2048 B of state cache, 8192 B of weight cache, and two DSPs for fixed-point computation.
  • Minitaur: 92% classification performance was reported for Minitaur on the MNIST digit database, closely matching the corresponding software network.The all-to-all layer connectivity makes weight storage and cache locality central implementation concerns.
  • Minitaur: Minitaur streams spikes through an input buffer and event queue, while a connection manager routes them to neurons or the output buffer.The architecture assigns separate DSPs to decay multiplication and input-current summation.
  • Scalability: Minitaur scales through local data processing and memory across 32 cores, but FPGA logic capacity limits the number of supported neurons.The Spartan-6 implementation supports up to 65,536 integrate-and-fire neurons.

III. LARGE SCALE MODELS OF NEURAL SYSTEMS

Large-scale neural-system platforms trade flexibility, scalability, and energy efficiency against architectural limits inherited from conventional digital hardware. FPGA and custom systems support configurable models, while SpiNNaker targets real-time simulation at very large scale but retains von Neumann memory structures.

  • Scope and flexibility: Dedicated convolutional and deep-network implementations serve domains such as visual processing and pattern recognition but cannot generally model complex biological neural systems.More flexible spiking architectures must configure connectivity, parameters, or neuron and synapse models.
  • Scope and flexibility: FPGA-based spiking systems enable relatively rapid prototyping because their conventional logic architectures are programmable.Their limited fan-in and fan-out ports restrict dramatic changes to memory organization.
  • Custom architectures: Custom ASICs can add specialized processing and asynchronous routing beyond standard FPGA designs, while preserving application-dependent trade-offs in flexibility and memory organization.The paper presents custom digital flows and asynchronous communication circuits as successive architectural choices.
  • SpiNNaker: SpiNNaker is designed to simulate up to one billion neurons in real time using 57,600 custom VLSI chips linked by asynchronous AER communication.The infrastructure supports large fan-in and fan-out and transports many small spike packets in real time.
  • SpiNNaker: SpiNNaker supports arbitrary connectivity and diverse neuron, synapse, and learning models, but increasing model complexity reduces the number of simulable elements.Its ARM cores and conventional memory hierarchies leave the von Neumann bottleneck unresolved.

B. TrueNorth

TrueNorth is presented as a fully custom, asynchronous, event-driven spiking processor that departs sharply from classical von Neumann architectures. Its chip integrates thousands of small digital neural cores with programmable synapses and routing.

  • Architecture: IBM’s TrueNorth spiking neural network ASIC represents a radical departure from classical von Neumann architectures.Its circuits use digital gates while operating asynchronously and communicating through event-driven methods.
  • Architecture: TrueNorth integrates 4096 spiking-neural-network cores on a single CMOS chip.The architecture distributes neural computation across many compact cores.
  • Core organization: Each TrueNorth core contains 256 digital leaky integrate-and-fire neurons and 256×256 binary programmable synaptic connections.Asynchronous encoding, decoding, and routing circuits manage communication among the cores.
  • Core organization: Synaptic events in TrueNorth can be assigned one of three possible strengths.This provides discrete synaptic weighting within the digital core architecture.

C. NeuroGrid

NeuroGrid combines mixed-signal sub-threshold circuits with distributed memory and asynchronous routing to model continuous-time cortical dynamics. Its shared synapses and structured connectivity support large real-time cortical networks, with flexibility constrained by routing and circuit design choices.

  • Architecture: NeuroGrid uses analog/digital mixed-signal sub-threshold circuits to model continuous-time neural processing elements.Its circuits implement functions including exponentiation, thresholding, integration, and temporal processing.
  • Architecture: A NeuroGrid board connects 16 CMOS NeuroCore chips, each containing a 256×256 array of two-compartment neurons.Shared synapse circuits linearly integrate multiple spikes over time, and asynchronous multicast routing supports multiple destinations.
  • Scale and connectivity: NeuroGrid can implement cortical models of up to one million neurons and billions of synaptic connections with sparse long-range and dense local connectivity.Local resistive networks or diffusors increase fan-out by modeling synaptic gap junctions and cortical-column organization.
  • Memory and computation: Distributed routing tables, parameters, and state variables allow circuit dynamics to serve as computational primitives that represent system state for tens to hundreds of milliseconds.Shared synapses integrate incoming spikes while memory and computation remain distributed across the network.
  • Memory and computation: NeuroGrid processes data as it arrives in real time, without time-stamping signals or storing partial processing results in external memory.Its circuit time constants therefore need to match the biological time scales of the signals being processed.

D. BrainScales

BrainScaleS pursues wafer-scale, mixed-signal simulation of large neural models with biologically realistic circuits and plastic synapses. Precision-oriented design choices improve numerical fidelity but constrain temporal dynamics, power, and compactness.

  • Architecture: BrainScaleS targets wafer-scale neural simulation with 50×10^6 plastic synapses and 200,000 biologically realistic neuron circuits per 8-inch wafer.Its mixed-signal engine aims to reproduce computational-neuroscience differential-equation results faithfully.
  • Architecture: Above-threshold circuits were chosen to improve analog precision, while small capacitors maximize the number of processing elements.The combination produces large currents relative to the modeled capacitances.
  • Memory and plasticity: BrainScaleS synapses provide temporal memory lasting hundreds of microseconds and include spike-based plasticity for learning and memory formation.These mechanisms implement several principles of brain-inspired information processing.
  • Trade-offs: The system is neither low-power nor compact because its precision-focused circuit choices prioritize reproducing numerical simulations.The paper identifies compact devices, low-power sub-threshold designs, and adaptation as routes toward compact, low-power, robust systems.
  • Plasticity: Plasticity spans structural, homeostatic, short-term, and long-term mechanisms operating across timescales from milliseconds to days.Homeostatic plasticity adapts overall network activity while controlling stability.

B. Attractor networks

Attractor networks support stable neural activity patterns that can underlie memory and nonlinear processing. Soft Winner-Take-All networks combine local cooperation with global inhibition so strongly responding neurons suppress competitors.

  • Attractor networks: Attractor networks can form short-term memories, consolidate long-term memories, and perform selective amplification for attention and decision making.They are recurrently connected networks that can settle into stable firing patterns after input removal.
  • Soft Winner-Take-All: The linear and nonlinear sWTA behaviors are plotted across neuron spatial location and response amplitude.The figure places linear behaviors in the top row and nonlinear behaviors in the bottom row.
  • Attractor networks: A two-state attractor network is computationally equivalent to a CMOS flip-flop gate.
  • Soft Winner-Take-All: Soft Winner-Take-All networks combine cooperation among nearby neuron groups with global recurrent inhibitory competition.The highest-response neurons suppress others when external inputs stimulate the network.
  • Implementations: Neuromorphic researchers have implemented physical models of sWTA networks, attractor networks, and plasticity mechanisms across temporal and spatial scales.

V. A NEUROMORPHIC PROCESSOR

The ROLLS processor combines slow sub-threshold analog circuits with asynchronous digital event communication to integrate memory, computation, plasticity, and learning. Experiments show that it can reliably form sustained attractor memories despite circuit variability and stochastic learning.

  • Architecture: ROLLS combines slow sub-threshold circuits that emulate neurons and synapses with fast asynchronous digital logic for event-based communication.
  • Architecture: Most ROLLS chip area is dedicated to nonlinear synapse circuits, which serve simultaneously as memory and computation sites.
  • Architecture: The 51.4 mm^2 chip contains 256 neurons, 133,120 synapses, and approximately 12.2 million transistors.Its synapses include linear, short-term plasticity, and long-term plasticity types.
  • Plasticity: ROLLS integrates learning signals, homeostatic scaling, and synaptic dynamics ranging from fractions of microseconds to hundreds of milliseconds.
  • Experiment: The network was trained to encode four memory patterns as separate attractor networks using recurrent excitatory LTP and inhibitory STP connections.Learning was configured to induce LTP and LTD across an intermediate firing-rate range.
  • Results: Attractors formed reliably and sustained robust population activity despite approximately 10% analog circuit variability and stochastic learning.The always-on online learning process can resume adapting when input statistics change.

VI. EMERGING NANO-TECHNOLOGIES

Emerging nanoscale devices are proposed as compact, low-power replacements for larger CMOS circuits that model synapses and neurons. Their nonvolatile and dynamic memory properties may support dense, massively parallel neuromorphic systems.

  • Motivation: Nanoscale devices are being developed to replace larger CMOS circuits used to model synapses and neurons.The goal is to build compact, low-power brain-like cognitive systems.
  • Device opportunities: Memristive devices can provide nonvolatile state retention and track past dynamics for massively parallel nanoscale synapse arrays.These devices are proposed for integration into neuromorphic VLSI systems.
  • RRAM: RRAM stores conductance through reversible resistance switching in a thin film between electrodes.Its resistance state can remain stored for a long period.
  • RRAM: RRAM-based neuromorphic approaches can potentially improve density and power consumption by at least a factor of 10 over conventional CMOS implementations.
  • Alternative devices: Candidate devices include STT-MRAMs, ferroelectric devices, and phase-change materials with distinct resistance-based behaviors.Their dynamics may provide state-dependent conductance, multilevel stability, and stochastic weight updates.

VII. DISCUSSION

The surveyed neuromorphic platforms trade flexibility, programmability, biological realism, memory organization, processing speed, bandwidth, and energy efficiency across application domains. Their design choices determine suitability for large-scale simulation, general-purpose computing, and specialized neural processing.

  • Large-scale simulation platforms: Conventional CPUs and GPUs offer flexible, rapid development for large-scale spiking-network simulation but can require very high power consumption.The cited passage gives consumption figures of up to tens of mega-Watts.
  • Large-scale simulation platforms: Dedicated simulators impose different constraints: Neurogrid and BrainScales use hard-wired neuron and synapse models, while SpiNNaker is programmable but limited by memory, resolution, and fixed-point representation.These constraints can restrict the neurons, synapses, and networks that can be simulated.
  • Large-scale simulation platforms: Large-scale simulator platforms face an I/O bottleneck because configuring neural-network parameters can take minutes to hours even when neural activity runs in real time or faster.For a network with 10^6 neurons and the stated connectivity and 8-bit weights, configuration requires at least 45 minutes at 300 Gb/s.
  • General purpose computing platforms: Neuromorphic systems also target general-purpose tasks such as pattern recognition and classification using FPGA, ASIC, TrueNorth, memristor, and RRAM-based architectures.These platforms span standard logic designs, less conventional designs, and emerging nanoscale technologies.
  • General purpose computing platforms: TrueNorth provides low-power neural computation after synaptic weights are uploaded, but its lack of learning and adaptation limits use in dynamically changing environments.Learning is typically performed on power-hungry standard or super-computers; memristive or RRAM-based mechanisms are proposed as a route toward greater adaptability.
  • Memory and information processing: The surveyed platforms illustrate alternative ways to partition memory and processing, from distributed resources to tightly co-localized memory and computation within synapse and neuron circuits.SpiNNaker distributes resources across cores and DRAM, whereas ROLLS integrates routing and parameter memory with computing circuits; these systems guide design choices for different applications.

VIII. CONCLUSIONS

The paper surveys neuromorphic systems for deep networks, cortical networks, and brain-inspired cognitive architectures, comparing their resource and biological-computation trade-offs. It concludes that current systems remain promising but do not yet deliver substantial advantages for large-scale simulations or sufficient complexity for specialized autonomous cognitive agents.

  • VIII. CONCLUSIONS: The survey evaluates state-of-the-art neuromorphic systems for deep network models, cortical network models, and brain-inspired cognitive architectures.It also discusses a mixed-signal analog/digital neuromorphic processor and its use for cognitive computing.
  • VIII. CONCLUSIONS: The systems face trade-offs involving memory requirements, processing speed, bandwidth, and implementation of computational primitives found in biological neural systems.The discussion compares the strengths and weaknesses of different architectural approaches.
  • VIII. CONCLUSIONS: Current neuromorphic systems do not yet provide substantial advantages over conventional architectures for large-scale simulations or enough complexity for specialized small-scale cognitive agents.The paper frames these limitations as motivating continued development rather than as a complete rejection of neuromorphic approaches.
  • VIII. CONCLUSIONS: Further interdisciplinary research and development are required to achieve robustness, learning abilities, and possibly cognitive abilities resembling biological systems.The paper identifies continued collaboration between neuroscience and neuromorphic computing as necessary for such breakthroughs.
Loading 1506.03264v1…