Neuro-memristive Circuits for Edge Computing: A review

Olga Krestinskaya; Alex Pappachen James; Leon O. Chua

Source-linked AI summary

Neuro-memristive Circuits for Edge Computing: A review

Olga Krestinskaya, Alex Pappachen James, Leon O. Chua

arXiv:1807.00962v2Jul 1, 2018cs.ETcs.AIcs.ARcs.NE

TL;DR

Growing sensor data challenges the power and scalability of centralized computing, motivating more capable low-power edge devices. This review surveys CMOS-memristive neuromorphic architectures for edge integration, their reported implementations, and remaining hardware challenges.

Problem
Edge devices need greater data-processing capability at lower power as sensor-generated data increases and conventional CMOS scaling faces power limitations.
Method
The paper reviews CMOS-memristive neuron, synapse, learning, and neuromorphic architectures that can be integrated into edge-computing devices.
Results
The reviewed architectures offer lower on-chip area and power requirements and support analog dot-product computing with memristive arrays, alongside reported neural-network and HTM implementations.
Takeaways & Limitations
Neuro-memristive circuits provide a possible efficient and scalable implementation for intelligent processing on edge-computing chips.

Abstract

from arXiv · show

The volume, veracity, variability, and velocity of data produced from the ever-increasing network of sensors connected to Internet pose challenges for power management, scalability, and sustainability of cloud computing infrastructure. Increasing the data processing capability of edge computing devices at lower power requirements can reduce several overheads for cloud computing solutions. This paper provides the review of neuromorphic CMOS-memristive architectures that can be integrated into edge computing devices. We discuss why the neuromorphic architectures are useful for edge devices and show the advantages, drawbacks and open problems in the field of neuro-memristive circuits for edge computing.

I. INTRODUCTION

Edge computing shifts processing toward local devices to reduce dependence on centralized data centers, motivating low-power neuromorphic and memristive architectures. The review surveys their integration, benefits, architectures, and unresolved implementation challenges.

Neuromorphic computing is considered because conventional CMOS technology cannot linearly scale power with growing data-processing demands.
The review covers CMOS-memristive neuron cells, synapses, neural networks, HTM, LSTM, learning architectures, and memory-based computing for edge devices.
High-density crossbar development remains constrained by selector devices and device-to-device variability, although learning can compensate for variability in weights.
Edge computing processes collected data locally, reducing the need to send significant data to servers and limiting data-center complexity.
Memristive architectures can reduce processing power, extend battery life, increase available hardware modules, and lower computation costs through smaller memory requirements.
After learning, memristive neuromorphic architectures process information quickly, while analog architectures support near-sensor processing and direct sensor integration.

III. NEURON MODELS

The paper reviews neuron models inspired by biological neurons, from threshold-based linear units to nonlinear dendritic and HTM models. These models differ in computational complexity and representational capability.

A. Inspiration from biological concepts: Biological neurons use dendrites for incoming connections, a soma for the cell body, an axon for output, and synapses for signal transmission.
A. Inspiration from biological concepts: Neuron models are categorized as simple threshold-based linear units or more complex dendritic threshold nonlinear units.
A. Inspiration from biological concepts: In the threshold linear model, weighted inputs are summed and compared with threshold θ to determine neuron firing.
A. Inspiration from biological concepts: Dendritic threshold nonlinear neurons use nonlinear dendritic branches, with each dendrite supporting multiple inputs and a specific threshold function.
A. Inspiration from biological concepts: Threshold nonlinear neurons can compute linearly non-separable functions that perceptron-like threshold linear neurons cannot compute.
A. Inspiration from biological concepts: HTM neurons extend dendritic nonlinear models with feedforward, feedback, and contextual inputs corresponding to distinct biological connections.

B. Memristive circuit as a synapse

Single-memristor synapses are compact and energy-efficient, but their practical use requires additional control circuitry and faces sneak-path and weight-sign limitations.

A single memristor is the least complex synapse structure and is more efficient in on-chip area and power consumption.
1M synapses can avoid additional CMOS elements in neural-network architectures.
These synapses require control circuits, suffer from sneak paths, and need complex switching for memristor updates.
A 1M structure cannot directly represent negative synaptic weights without additional circuits.

2) Synapses with two memristors:

Two-memristor and transistor-assisted synapses extend weight representation and switching control, while bridge-based designs support zero, positive, and negative weights with greater circuit complexity.

2) Synapses with two memristors:: 2M synapses double crossbar size but enable positive and negative synaptic weights.
2) Synapses with two memristors:: PCMO-based 2M synapses associate separate memristors with LTP and LTD operations to reduce effects of asymmetric resistance changes.
2) Synapses with two memristors:: Series-connected 2M synapses combine diffusive and drift memristors to realize dynamic behavior, LTD, and LTP.
2) Synapses with two memristors:: A 2M1R synapse adds a resistor to two memristors for temporal transformations and static weights in spiking-neural-network simulations.
2) Synapses with two memristors:: Transistor-memristor synapses use transistors as switches for read and update cycles, with 1T1M proposed as a solution to sneak-path problems.
2) Synapses with two memristors:: A 4M bridge synapse uses a Wheatstone bridge-like circuit to represent zero, positive, and negative synaptic weights.

4) Memristor bridge synapses:

The review covers capacitor-, memristor-, amplifier-, comparator-, activation-function-, stochastic-, and HTM-based neuron models, each implementing different neural behaviors and circuit functions.

4) Memristor bridge synapses:: Early I&F neurons use a membrane capacitor to integrate synaptic currents while membrane resistance models leakage, but their area and power limit large-scale use.
4) Memristor bridge synapses:: A diffusive memristor neuron integrates presynaptic signals and emits a spike when its threshold is reached.
4) Memristor bridge synapses:: Summing-amplifier and comparator neurons sum input currents and generate an output pulse when the amplifier voltage exceeds a threshold.
4) Neuron models:: The neuron-cell review includes modified I&F, capacitive, sigmoid, memristor-bridge, stochastic, and HTM spatial-pooler models.

4) Neuron models for memristor bridge architecture:

Memristor-bridge neuron models convert weighted synapse voltages into currents and sum them, while broader architectures include stochastic, one-layer, and two-layer neural networks.

4) Neuron models for memristor bridge architecture:: Memristor-bridge neurons convert bridge-weighted voltages into currents, sum synaptic currents, and convert the output current back into voltage.
4) Neuron models for memristor bridge architecture:: Real memristor implementations may face switching response, switching-time, and two-memristor connection issues.
5) Stochastic neurons:: Stochastic-neuron digit recognition achieved about 60% accuracy, compared with 65% for stochastic synapses.
One layer neural network with learning:: One-layer implementations reported approximately 83% handwritten-digit accuracy and 88.08% average Yale Face Database recognition accuracy.
Two layer neural network:: A two-layer crossbar network achieved up to 100% digit-recognition accuracy without noise in simulation.

2) Two layer neural network:

Two-layer neural networks use memristive synapses and neuron circuits for pattern-recognition tasks, with implementations spanning bridge-synapse, crossbar, and cellular architectures.

Two layer neural network:: A typical two-layer network is a perceptron with a single hidden layer and can use two memristive crossbars with Ag/AgInSbTe/Ta 1M synapses.Circuit-simulation accuracy reached up to 100% for digit recognition without noise.
Two layer neural network:: A partially fabricated network with 64 input, 54 hidden, and 10 output neurons achieved 92% accuracy on rescaled MNIST images.Its 128×64 crossbar was fabricated, while activation functions and update calculations were implemented in software.
Two layer neural network:: A 4M bridge-synapse architecture with 432 inputs, 10 hidden neurons, and one output neuron produced circuit-simulation results comparable to software simulation.A related implementation reported 115ns total feedforward processing time.
Two layer neural network:: Deep networks extend the crossbar approach across many layers, with reported MNIST accuracies of 91.8% for a memristive CNN and 97.84% for a stochastic spiking CNN.The latter used 6.4 times less energy and 8 times less on-chip area than an equivalent CMOS design.
Two layer neural network:: Memristive cellular neural networks connect cells to their closest neighbors and support image-processing tasks including edge detection, filtering, noise removal, line extraction, and hole filling.Recent designs use memristive-CMOS circuits rather than only the capacitor, current-source, and resistive elements of early implementations.

5) Convolutional Neural Network:

Memristive CNNs combine convolutional feature extraction with classification, while spiking variants encode information as events and use synaptic timing or stochasticity for recognition.

Convolutional Neural Network:: Memristive CNNs divide processing into convolutional feature extraction, sub-sampling, and fully connected classification stages.Feature maps can shrink from 8×8 to 4×4 through sub-sampling while feature-map count increases through propagation.
Convolutional Neural Network:: 92% and 94% MNIST accuracy were reported for memristive CNN implementations, compared with 98.92% for software simulation.These systems use convolution filters to produce feature maps before classification.
Convolutional Neural Network:: 644.2 GOPS/W power efficiency was reported for a memristive-crossbar CNN accelerator, while another system achieved 126 TOPS/W with 98.3% MNIST and 91.4% CIFAR-10 accuracy.The latter system used 1.02mm2 area and 6.3mW power.
Convolutional Neural Network:: A 128×64 memristive crossbar reached 1.64 TOPS per reading cycle and 119.7 TOPS/W, while consuming 17 times less energy than an ASIC implementation.The image quality after convolution was worse than software-based convolution.
Spiking Neural Network:: Spiking neural networks transmit shaped spike events, commonly implement STDP with memristive crossbars, and can reach up to 99% accuracy for handwritten- and letter-recognition tasks.STDP updates synapses according to correlations between pre- and postsynaptic spikes.
Spiking Neural Network:: A 5×5 spiking position detector consumed at most 15.6µW, about 70% less than equivalent CMOS, with 6.1×10^-5cm2 on-chip area.Output spiking frequencies represented the detected object position.
Spiking Neural Network:: Stochastic spiking recognition reached 78.4% MNIST accuracy with 128 output neurons, as larger output layers captured more input patterns for majority voting.Capacitive alternatives replace resistive synapses with non-volatile pseudo-memcapacitors.

7) Recurrent Neural Network and Long Short Term Memory:

The review covers recurrent and temporal neuromorphic architectures, including RNNs, LSTMs, and HTM, with emphasis on their memristive hardware implementations and remaining challenges.

Recurrent Neural Network: RNNs use feedback so layer outputs affect subsequent outputs, but complete memristor-based RNN hardware remains an open problem.Analog implementations face feedback and architectural-complexity challenges.
Long Short Term Memory: LSTM extends RNNs with output, input, write, and forget gates that select how current inputs and previous outputs affect future outputs.Its analog implementation uses activation-function circuits and an analog multiplier.
Hierarchical Temporal Memory: HTM mimics the human neocortex through Spatial Pooler encoding and Temporal Memory prediction, with both components involving learning.Memristive crossbars represent HTM levels, while spin-neuron devices implement neurons in one mixed-signal design.
Hierarchical Temporal Memory: 95% maximum accuracy was achieved for MNIST handwritten-digit recognition using a hierarchical CMOS-memristive HTM architecture.An alternative HTM Spatial Pooler achieved 86% face-recognition accuracy and 70% speech-recognition accuracy.
Hierarchical Temporal Memory: A modified HTM system uses Spatial Pooler outputs, Temporal Memory-generated class templates, and memristive pattern matching for classification.The modification reduces hardware complexity and combines HTM with traditional supervised classification methods.

B. Neural Network learning architectures

The review examines online learning for memristive edge architectures, spanning software-assisted, digital, mixed-signal, and analog training approaches while identifying unresolved implementation challenges.

Online learning: Online training is important for large-scale memristive edge architectures, but learning is performed in software in most existing designs.One example combines partially fabricated neural-network hardware, software backpropagation, and online memristive crossbar-weight updates.
Digital and mixed-signal training: Digital training circuits combined with memristive crossbars have been proposed to accelerate learning and transfer training toward hardware.Mixed-signal designs combine analog neurons with digital error calculation and on-chip training.
Analog learning: Analog near-sensor systems avoid additional analog-to-digital and digital-to-analog conversion stages when integrated with analog sensors.This motivates analog learning circuits for neural networks and HTM.
Analog learning: Analog backpropagation can sequentially propagate output-layer errors and update memristors in different neural-network layers.Inactive layers are isolated during updates.
Open problems: Fully analog learning systems with control circuitry and no digital processing remain an open problem despite proposed implementations for MNN, BNN, DNN, LSTM, and HTM.The cited architecture includes derivative, multiplication, control-transistor, and analog weight-update circuits.
Update speed: One-at-a-time updates in 1M, 2M, 1T1M, and 2T1M synapses are slow, making memristive-weight update speed a central online-training issue.Transistor-based synapses can disconnect memristors not involved in the current update.

2) On-chip area and power dissipation:

Memristive architectures offer compact, low-power edge hardware and scalable arrays, but practical scaling is constrained by crossbar effects, device variability, resistance limits, and CMOS compatibility.

Advantages: 45% lower on-chip area and 96% lower average power consumption were reported for a memristive CAM array versus a CMOS-based design.The reported reductions are attributed to replacing conventional resistive elements with memristive devices.
Scalability: Memristive devices support scaling because they do not exhibit the leakage-current problems associated with transistors and resistors.Large crossbars nevertheless face sneak paths and variable outputs.
Scalability: Subdividing large arrays into smaller sub-arrays can address sneak-path and crossbar-output-variability problems, while selector devices increase cell area.These trade-offs constrain scalable crossbar design.
Challenges: Memristive technology is not mature enough for commercial chip design, with open issues including CMOS compatibility, unstable switching, resistance-range limits, and fabrication complexity.The review also identifies limited resistive levels and implementation challenges.
Challenges: Device-to-device and cycle-to-cycle switching variability can affect architecture accuracy, while variability-aware analog learning remains challenging.The effects of switching stochasticity on neuromorphic architectures, learning, and training speed remain insufficiently investigated.
Challenges: Neuromorphic architectures often assume resistance ranges and numbers of stable states that real devices may not provide.Realistic models incorporating finite resistive states and nonlinear switching remain open problems.

4) Endurance of the memristor:

Memristor endurance and CMOS integration constrain edge-system design, especially when online learning repeatedly updates synapses and when complex multilayer architectures require difficult fabrication and interconnection.

Endurance: 10^5 and 10^9 endurance cycles were reported for TiOx and TaOx devices, respectively, under 1µs voltage pulses.Endurance and reliability depend on process variability and degradation.
Endurance: Memristor lifetime and allowable update cycles are critical for edge architectures that include learning and training.The review calls for investigation of endurance, reliability, and lifetime limits across device types.
Endurance: A two-layer ANN trained on XOR required 5000 iterations with continuous memristive-synapse updates.Lowering the learning rate to improve accuracy can require still more update cycles.
CMOS integration: Hybrid CMOS-memristive designs require more CMOS devices as architecture size increases, which can raise power consumption.Read and write circuits for memristors are mostly based on CMOS transistors.
CMOS integration: Complex multilayer fabrication must balance deposition temperature: high temperature can damage CMOS devices, while low temperature may prevent reliable connections.BEOL integration has been used to build memristor layers above existing chips.
Large-scale systems: Complex mixed-signal and analog neuromorphic implementations remain limited, while interconnect parasitics can significantly affect analog-system performance.The review highlights scalability, datapath complexity, and application-dependent interconnection issues.

APPENDIX A SELECTION OF MEMRISTOR MODEL

Memristor model selection must balance realistic nonlinear and physical behavior with numerical stability and scalability for large neuromorphic edge-system simulations. The appendix compares model classes, their assumptions, and approaches intended to reduce simulation problems.

Simulation requirements: Large-scale SPICE simulations can suffer errors and non-convergence when memristor models are not designed for scalability.These issues motivate considering circuit side effects, nonlinearities, and drawbacks before implementing chips or integrating them with sensors.
Linear models: Ideal linear models emulate switching behavior but omit electric-field effects and other nonlinear device behavior.Pickett’s model is described as a drift-based model for TiO2 memristors and was used in earlier architecture simulations.
Nonlinear behavior: Physical ionic drift, diffusion, and thermophoresis produce nonlinear current–voltage relations and nonlinear dynamical switching in real devices.Recent work therefore focuses on models that incorporate nonlinear behavior beyond earlier ideal linear models.
Nonlinear models: Joglekar-, modified window-function, and adjustable window-function models represent nonlinear switching and allow control through parameters such as W/D, p, current i, or j.The cited models differ in how they model nonlinear boundary conditions, dopant drift, and the maximum window-function value.
Physical effects: Nonlinear window-based models still omit some device imperfections, including parasitics and leakages, whereas later models consider physical imperfections.The TEAM model provides a generalized, adjustable approach for different memristive devices, window functions, and physical effects.
Scalable models: Large-scale edge simulations require more accurate nonlinear models, while modified models can avoid window functions and reduce numerical and non-convergence problems.The reviewed large-scale models target multilayer architectures and are adapted from complex physical-phenomenological nonlinear models.