PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Neuromorphic Computing and Neural Networks in Hardware

Catherine D. Schuman, Thomas E. Potok, Robert M. Patton, J. Douglas Birdwell, Mark E. Dean, Garrett S. Rose, James S. Plank

arXiv:1705.06963v1cs.NE

TL;DR

Neuromorphic computing seeks non-von Neumann systems that address power, parallelism, real-time performance, and memory-bandwidth constraints, but model and learning capabilities remain unresolved. This paper surveys the field’s motivations, models, algorithms, hardware, materials, supporting systems, and applications, identifying directions for future development.

  • Problem

    Neuromorphic systems face the need for low-power, massively parallel, real-time non-von Neumann computation, while the full capabilities of their training and learning mechanisms remain unrealized.

  • Method

    The paper provides a broad survey of neuromorphic computing research, covering motivations, models, algorithms, hardware, devices, materials, supporting systems, and applications.

  • Results

    The survey identifies substantial prior work across neuromorphic system components and highlights multiple research directions for advancing the field.

  • Takeaways & Limitations

    Future progress requires neuromorphic-specific training and learning algorithms, alongside continued development of hardware, devices, and materials.

  • Takeaways & Limitations

    The full capabilities of spiking neuromorphic systems have not yet been realized by existing training and learning mechanisms.

Abstract

from arXiv · show

Neuromorphic computing has come to refer to a variety of brain-inspired computers, devices, and models that contrast the pervasive von Neumann computer architecture. This biologically inspired approach has created highly connected synthetic neurons and synapses that can be used to model neuroscience theories as well as solve challenging machine learning problems. The promise of the technology is to create a brain-like ability to learn and adapt, but the technical challenges are significant, starting with an accurate neuroscience model of how the brain works, to finding materials and engineering breakthroughs to build devices to support these models, to creating a programming framework so the systems can learn, to creating applications with brain-like capabilities. In this work, we provide a comprehensive survey of the research and motivations for neuromorphic computing over its history. We begin with a 35-year review of the motivations and drivers of neuromorphic computing, then look at the major research areas of the field, which we define as neuro-inspired models, algorithms and learning approaches, hardware and devices, supporting systems, and finally applications. We conclude with a broad discussion on the major research topics that need to be addressed in the coming years to see the promise of neuromorphic computing fulfilled. The goals of this work are to provide an exhaustive review of the research conducted in neuromorphic computing since the inception of the term, and to motivate further work by illuminating gaps in the field where new research is needed.

I. INTRODUCTION

Neuromorphic computing developed as a brain-inspired alternative to von Neumann systems, motivated by architectural, technological, and machine-learning considerations. This survey reviews the field’s history, research areas, and changing motivations across roughly three decades.

  • I. INTRODUCTION: The survey reviews more than 3,000 papers spanning 35 years, covering motivations, models, algorithms and learning, applications, hardware, materials, and supporting systems.Its stated goal is to provide a broad historical perspective and identify open issues for future research.
  • I. INTRODUCTION: The field emerged as a complementary non-von Neumann architecture inspired by biological neural systems and artificial neural networks.Carver Mead coined the term “neuromorphic” in 1990 for analog VLSI systems mimicking biological neural systems.
  • I. INTRODUCTION: Neuromorphic computing uses highly connected, parallel architectures that collocate memory and processing while targeting low-power operation.These properties address limitations associated with conventional von Neumann systems, including the memory-processing bottleneck.
  • I. INTRODUCTION: Machine learning motivates neuromorphic computing through potential gains in learning performance and online, real-time adaptation.The paper connects this motivation to algorithms that could learn in ways resembling biological brains.
  • II. MOTIVATION: Low-power consumption is the dominant contemporary motivation, alongside parallelism, real-time performance, speed, compactness, and reliability.The literature also links neuromorphic development to Moore’s-law and Dennard-scaling limits and to mitigating the von Neumann bottleneck.

III. MODELS

Neuromorphic systems support a wide range of neuron, synapse, and network models, from biologically inspired to computationally driven. Model selection depends on application goals, device constraints, and desired biological realism.

  • III. MODELS: Neuromorphic models define the network’s components, their operation, and their interactions, including neuron and synapse behavior.The model establishes the abstract functionality of the neuromorphic system.
  • III. MODELS: Model choice is shaped by the intended application, with biologically plausible models suited to neuroscience and convolutional networks suited to high-accuracy image recognition.The paper presents application objectives as one basis for selecting among model families.
  • III. MODELS: Device and material characteristics can constrain model selection, such as memristor properties supporting spike-timing-dependent-plasticity-like mechanisms.These mechanisms are described as especially appropriate for spiking neural network models.
  • III. MODELS: The surveyed hardware systems span models ranging from predominantly biologically inspired to predominantly computationally driven.The section organizes prior work around neuron, synapse, and network models.

A. Neuron Models

Neuromorphic hardware implements neuron models spanning biological detail, biologically inspired spiking behavior, and computationally derived artificial neurons. These models are organized into five broad categories and differ in their components and implementation prevalence.

  • Neuron models define network components, their operation, and their interactions, commonly including neurons and synapses.
  • The five categories are biologically-plausible, biologically-inspired, Neuron+Other, integrate-and-fire, and McCulloch-Pitts.Neuron+Other includes components such as axons, dendrites, or glial cells.
  • Figure 4 encodes implementation prevalence through box size and neuron-model families through color.
  • McCulloch-Pitts neurons compute an output from weighted inputs passed through an activation function.The cited formulation defines yj as output, f as activation function, wi,j as synaptic weight, and xi as input-neuron output.
  • Biologically-plausible and biologically-inspired models may include membrane, ion-channel, axonal, and dendritic dynamics.

1) Biologically-Plausible:

Biologically plausible and inspired neuron models trade biological fidelity against computational complexity. Hardware implementations range from detailed ion-channel models to simpler spiking models suited to efficient computation.

  • Biologically-Plausible: Hodgkin-Huxley is the most popular biologically-plausible model and uses four-dimensional nonlinear differential equations for ion-transfer behavior.
  • Biologically-Plausible: Morris-Lecar reduces biologically-plausible dynamics to a two-dimensional nonlinear equation.
  • Biologically-Plausible: FitzHugh-Nagumo and Hindmarsh-Rose simplify Hodgkin-Huxley behavior, reducing computation and parameters while becoming more biologically inspired than plausible.
  • Biologically-Plausible: The Izhikevich model reproduces bursting and spiking behavior with simpler computation and is popular for combining simplicity with biologically accurate behavior.
  • Biologically-Plausible: Integrate-and-fire models span basic, leaky, and nonlinear forms, with leaky integrate-and-fire among the most popular neuromorphic models.The leaky form adds a leak term that causes neuronal potential to decay over time.
  • Biologically-Plausible: Digital spiking models commonly use cellular automata rather than differential equations, while some systems support programmable neuron models.

5) McCulloch-Pitts Neurons:

McCulloch-Pitts neurons support traditional artificial neural-network hardware, while neuromorphic network implementations span feed-forward, recurrent, spiking, and other topologies. Model choice depends on biological inspiration, complexity, connectivity, learning algorithms, hardware, and application.

  • McCulloch-Pitts Neurons: McCulloch-Pitts hardware includes perceptrons, programmable activation functions, and circuits implementing sigmoid or hyperbolic-tangent functions and derivatives.
  • McCulloch-Pitts Neurons: Figure 5 qualitatively compares neuron models by biological inspiration and model complexity.
  • Network Models: Neuromorphic synapses may include plasticity mechanisms, chemical interactions, ion channels, and spike-timing-dependent potentiation or depression.
  • Network Models: Non-spiking implementations include feed-forward multilayer, winner-take-all, convolutional, recurrent, reservoir, and probabilistic networks.
  • Network Models: Hardware connectivity can restrict realizable network topologies, making topology selection dependent on available hardware.
  • Network Models: Spiking and feed-forward implementations are by far the most common; spiking activity rose in the last decade, while convolutional networks increased activity during the last five years.

D. Summary and Discussion

Neuromorphic research has explored a wide range of biological and artificial neural-network models. The appropriate model depends on the system’s intent, hardware constraints, efficiency needs, and application.

  • Neuromorphic implementations cover a wide variety of model types, from biologically inspired to computationally driven.
  • Figure 7 groups network models by overall type and sizes them according to the number of associated papers.
  • Most neural-network models have been implemented in hardware at some point.
  • Neuroscience-oriented systems generally favor biologically plausible or inspired models, whereas application-focused systems may favor artificial neural-network models with established capabilities.
  • Models may be selected or adapted to fit device characteristics, such as using STDP-compatible models with memristors, or to improve efficiency in event-driven spiking systems.

IV. ALGORITHMS AND LEARNING

Neuromorphic learning research spans supervised, evolutionary, and unsupervised approaches, but algorithm choice remains tightly coupled to models, hardware, and learning requirements. The field still lacks broadly applicable methods that fully exploit neuromorphic systems.

  • Supervised learning: Back-propagation is the most commonly used programming algorithm and supports feed-forward, recurrent, spiking, and convolutional networks.It is often run offline on a traditional host, although many on-chip implementations also exist.
  • Supervised learning: Back-propagation is restrictive for some neuron models, network topologies, and hardware implementations, making efficient hardware deployment difficult or costly.Device restrictions such as connectivity and weight resolution can require substantial adaptation and slow convergence.
  • Alternative approaches: Evolution-inspired algorithms accommodate varied models, devices, and applications and can optimize around hardware peculiarities, but may converge slowly for complex models.Their flexibility also allows off-chip methods to use hardware implementations in the loop.
  • Unsupervised learning: Online unsupervised learning, including self-organizing and Hebbian-type rules, is viewed as important for realizing neuromorphic systems’ full potential.Spiking systems commonly use STDP, but many reported systems do not specify a training rule.

Biologically-Inspired

Neuromorphic systems use biologically inspired models and span digital, analog, and mixed hardware platforms. Digital designs illustrate a central trade-off: specialized efficiency can come at the expense of flexibility.

  • Biologically-Inspired: Spiking systems are attractive because event-driven computation can improve power or energy efficiency and because they are more biologically inspired.Existing non-spiking models remain common because their training methods and applications are better established.
  • Hardware categories: Neuromorphic hardware is commonly categorized as digital, analog, or mixed analog/digital, although these categories are high-level taxonomies.Analog systems compute using physical device properties and continuous values, whereas digital systems generally use Boolean logic and discrete values.
  • 1) Digital:: TrueNorth fixes its spiking model and connectivity without on-chip learning, while SpiNNaker allows flexible neuron, synapse, learning, and network-topology choices.The comparison presents specialized optimization and broad flexibility as opposing digital design choices.
  • 1) Digital:: 25 pJ per connection for TrueNorth contrasts with 10 nJ per connection for SpiNNaker, illustrating the energy cost associated with SpiNNaker’s flexibility.Both systems are digital neuromorphic implementations, but they occupy opposite points in the flexibility–efficiency trade-off.

2) Analog:

Analog neuromorphic systems exploit physical circuit behavior and are divided into programmable and custom implementations, while mixed platforms combine analog computation with digital communication or memory.

  • 2) Analog:: Analog systems include programmable FPAAs and custom analog chips, paralleling the distinction between FPGAs and custom digital implementations.The survey treats both programmable and custom analog hardware as neuromorphic platforms.
  • 2) Analog:: Analog neuromorphic circuits use physical characteristics such as charge conservation, amplification, thresholding, and integration that also appear in biological systems.Subthreshold operation is commonly used for power efficiency and reflects the original analog meaning of neuromorphic computing.
  • 3) Mixed Analog/Digital:: Mixed analog/digital systems commonly store synaptic weights or memory components digitally because digital storage can be less noisy and more reliable.These platforms combine analog and digital components rather than belonging exclusively to one hardware category.
  • 3) Mixed Analog/Digital:: Neurogrid and BrainScaleS are major mixed analog/digital projects that use digital communication frameworks with analog components.Neurogrid operates in subthreshold mode, whereas BrainScaleS operates in superthreshold mode.
  • 3) Mixed Analog/Digital:: 10,000x speed-up is achieved by BrainScaleS relative to Neurogrid because its superthreshold operation permits a higher operating rate.The comparison links the operating regime to the reported speed difference.

1) Memristors:

Memristors and other emerging devices support neuromorphic synapses, neurons, and dense memory, but practical behavior and fabrication variability constrain their use.

  • 1) Memristors:: Memristor resistance depends on historical activity, making memristors attractive for neuromorphic circuits and synaptic implementations.They can serve as synaptic weights or as part of complete synapse designs.
  • 1) Memristors:: Memristive circuitry is studied for energy efficiency, while neural-network fault tolerance may mitigate effects of device variation.This fit has motivated extensive evaluation of memristive energy consumption in neuromorphic systems.
  • 1) Memristors:: Memristors have implemented complex spiking behavior, stochastic neuron activity, and Hodgkin-Huxley axons in hardware.Their use extends beyond synaptic weights to neuron and axon implementations.
  • 1) Memristors:: Memristor implementations face behavior-related STDP degradation, sneak paths, and geometry variations.These issues can affect the performance and physical realization of neuromorphic systems.
  • 1) Memristors:: Existing memristive neural-network theory often assumes ideal memristors and may not reflect fabricated systems.The survey identifies this realism gap across stabilization, synchronization, and passivity analyses.
  • Device alternatives: CBRAM, phase-change memory, spintronics, and floating-gate transistors have been explored for neuromorphic synapses, neurons, storage, or dense implementations.Their reported motivations include non-volatility, high density, tunable functionality, CMOS compatibility, and analog circuit use.

6) Optical:

Neuromorphic research spans optical implementations, novel materials, and nanoscale devices, with material choices shaping behavior from devices through system-level models and algorithms.

  • Optical implementations: Optical neuromorphic implementations exploit inherent parallelism, although storing information has historically been difficult in optical systems.These storage challenges contributed to reduced popularity during several decades.
  • Materials and devices: Materials science develops novel devices intended to make neuromorphic systems smaller, faster, and more efficient.The survey emphasizes this work despite focusing primarily on computing and system components.
  • Materials and devices: Different atomic-switch materials can produce different switching behaviors under different conditions.Examples include Ag2S, Cu2S, Ta2O5, and WO3–x.
  • Materials and devices: Metal-oxide memristor materials differ in resistance states, endurance, stability, and reliability, affecting stored synaptic weights.The survey lists materials including HfOx, TiOx, WOx, SiOx, TaOx, NiOx, FeOx, AlOx, and PCMO.
  • Materials and devices: Ferroelectric three-terminal synaptic devices can realize learning processes such as STDP within the device itself.This can reduce reliance on additional circuitry for the learning process.
  • Cross-layer implications: Material differences propagate through neuromorphic devices, hardware, software, models, and algorithms, requiring collaboration with materials scientists.The survey connects material selection to functionality across the full system stack.

D. Summary and Discussion

Neuromorphic systems include diverse hardware, communication frameworks, and software tools, but usability requires stronger supporting infrastructure alongside device development.

  • Hardware: Neuromorphic hardware spans programmable FPGAs and FPAAs, custom digital, analog, and mixed systems, and components such as memristors, CBRAM, phase-change memory, optical devices, and floating-gate transistors.Material properties can fundamentally affect how future neuromorphic systems operate.
  • Usability: The survey presents communication and supporting software as necessary for real utilization, while noting substantially less software work than hardware work.It identifies supporting software as an especially important area for further development.
  • Communication: Communication frameworks support both intra-chip and inter-chip communication, with AER commonly used between chips and in systems such as SpiNNaker.SpiNNaker connects chips in a toroidal mesh while using AER for inter-chip communication.
  • Communication: AER has limits for on-chip communication, motivating alternative approaches to optimize intra-chip communication.Earlier systems used buses, while later work explored removing buses from communication structures.
  • Communication: Three-dimensional integration has been used to stack memory with processing and to stack neuromorphic chips, commonly using through-silicon vias.The survey also describes early uses for pattern and object recognition tasks.
  • Supporting software: Supporting software includes hardware synthesis, mapping, programming, and simulation tools for implementing, adapting, testing, and training neuromorphic systems.Mapping tools adapt networks to hardware restrictions, while simulators help verify performance and develop training algorithms.

C. Summary

Neuromorphic systems have been applied across sensing, recognition, control, medical, language, and classification tasks, especially where real-time, low-power, or small-footprint operation matters.

  • Applications: Neuromorphic systems have broad application coverage, but the field has not yet determined which applications best showcase their capabilities.Examples include image, speech, classification, control, and anomaly-detection tasks.
  • Sensing: Vision-based sensing systems are among the most popular neuromorphic application classes and often replicate characteristics of biological visual systems.These systems are designed around particular sensory modalities and application areas.
  • Medical and biological interfaces: Neuromorphic systems support small, low-power medical interfaces and implants, including brain-machine interfaces and cardiac or retinal devices.Spike-based communication is described as a natural fit for brain-machine or brain-computer interfaces.
  • Robotics and control: Robotics and control applications commonly require real-time performance, small volume, low power, and temporal processing.These requirements favor recurrent connections or synaptic delays in neuromorphic models.
  • Image processing and classification: Image-based applications include edge detection, compression, filtering, segmentation, traffic-sign recognition, face recognition, and object analysis.The survey also reports applications involving manufacturing defects, gestures, humans, and textures.
  • Other applications: Neuromorphic systems have also been applied to speech recognition, noise filtering, natural-language processing, smart sensors, anomaly detection, and general data classification.The examples span recurrent language tasks, environmental sensing, traffic anomalies, and domain-specific classification.

VIII. DISCUSSION: NEUROMORPHIC COMPUTING MOVING FORWARD

The survey identifies future challenges across models, learning, materials, software, and applications, arguing that progress depends on coordination across these levels.

  • Research scope: The survey synthesizes neuromorphic models, algorithms, full hardware systems, device components, materials, supporting systems, and applications.It presents these areas as the basis for future research directions.
  • Models and learning: A central machine-learning question is which training and learning algorithms are appropriate for neuromorphic systems.The authors view neuromorphic platforms as a way to explore learning mechanisms at accelerated scale.
  • Materials: New materials and devices may support extremely small and ultra-fast neuromorphic computers, while their properties require coordination with models and algorithms.The survey also discusses composite materials tailored to neuromorphic computation.
  • Software: Neuromorphic computing needs more supporting software and software-engineering involvement so systems become usable by non-experts.Software engineers can both build support tools and inform device design.
  • Applications: Potential use cases include heterogeneous-computer coprocessors, IoT smart sensors and anomaly detectors, and low-power intelligent controllers.The survey frames these as application directions requiring further development and communication.
  • Coordination: The field could play a major future role in computing if research expands across levels and coordination increases across the community.The conclusion links coordination from materials through algorithms and models with continued innovation.

IX. CONCLUSION

The survey synthesizes neuromorphic computing research across models, learning, hardware, materials, supporting systems, and applications. It identifies persistent needs for low-power, massively parallel, real-time, potentially online-learning architectures while emphasizing that model diversity is likely to continue.

  • Neuromorphic systems still require low-power, massively parallel, real-time architectures with potential for online training or learning.
  • The field is unlikely to converge on one universal neuron, synapse, or network model because different models have distinct strengths and weaknesses.The expected landscape ranges from feed-forward neural networks to detailed biological neural-network emulators.
  • The survey reviews neuromorphic computing research across models, algorithms and learning, hardware and devices, supporting software, and applications.It presents an overview of past work across these areas and discusses applications to which neuromorphic systems have been applied.
  • The paper aims to provide a broad view of research across neuromorphic computing and encourage new systems and applications that address remaining gaps.
Loading 1705.06963v1…