Source-linked AI summary
Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
Steven K. Esser, Paul A. Merolla, John V. Arthur, Andrew S. Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J. Berg, Jeffrey L. McKinstry, Timothy Melano, Davis R. Barch, Carmelo di Nolfo, Pallab Datta, Arnon Amir, Brian Taba, Myron D. Flickner, Dharmendra S. Modha
TL;DR
Neuromorphic computing must bridge its architectural differences from deep learning to support accurate, flexible convolutional models. This paper maps and trains deep convolutional networks on TrueNorth, showing near-state-of-the-art accuracy across eight benchmarks while preserving efficient hardware operation.
Problem
Whether neuromorphic computing’s structural and operational differences from deep learning prevent it from implementing flexible deep convolutional networks remained unresolved.
Method
The paper maps deep convolutional networks onto TrueNorth neurosynaptic cores using grouped feature representations and trains discrete synapses with backpropagation.
Results
Across 8 image and audio benchmarks, single-chip networks came within a few percent of state-of-the-art accuracy, while larger networks improved accuracy further.
Takeaways & Limitations
The results support neuromorphic processors as efficient, flexible substrates for spiking networks and embedded neural-network applications.
Takeaways & Limitations
The approach uses binary synaptic representation, although higher-precision schemes are possible and binary coding gave the best accuracy for a given energy budget.
Abstract
from arXiv · showhide
Deep networks are now able to achieve human-level performance on a broad spectrum of recognition tasks. Independently, neuromorphic computing has now demonstrated unprecedented energy-efficiency through a new chip architecture based on spiking neurons, low precision synapses, and a scalable communication network. Here, we demonstrate that neuromorphic computing, despite its novel architectural primitives, can implement deep convolution networks that i) approach state-of-the-art classification accuracy across 8 standard datasets, encompassing vision and speech, ii) perform inference while preserving the hardware's underlying energy-efficiency and high throughput, running on the aforementioned datasets at between 1200 and 2600 frames per second and using between 25 and 275 mW (effectively > 6000 frames / sec / W) and iii) can be specified and trained using backpropagation with the same ease-of-use as contemporary deep learning. For the first time, the algorithmic power of deep learning can be merged with the efficiency of neuromorphic processors, bringing the promise of embedded, intelligent, brain-inspired computing one step closer.
Approach
The approach describes how the essential elements of deep convolutional networks can be realized on the TrueNorth neuromorphic chip.
- The approach outlines relevant elements of deep convolutional networks and the TrueNorth neuromorphic chip, then explains how the former can be realized on the latter.
Deep Convolutional Networks.
Deep convolutional networks are multilayer feedforward systems that apply spatially organized filters across image-like inputs and intermediate representations. Their neurons compute weighted sums, may use batch normalization, and typically apply rectified linear activations; training uses backpropagation.
- Architecture: A deep convolutional network is a multilayer feedforward network whose layers perform convolutional filtering on image-like inputs or prior-layer outputs.Neurons are arranged across two spatial dimensions and one feature dimension for different filters.
- Architecture: Each layer is organized as rows × columns × features, with filters applied to topographically aligned support regions in the prior layer.Adjacent features shift their receptive fields by the stride, and feature groups can cover different source-feature sets.
- Neuron computation: Neurons compute summed weighted inputs using input pixels or neurons, filter weights, topographic dimensions, and feature or input-channel dimensions.The inputs are denoted x = {x_i,j,f} and weights w = {w_i,j,f}.
- Neuron computation: Batch normalization can zero-center filter responses and normalize their standard deviation to 1, after which a rectified linear unit sets negative outputs to 0.During training, normalization statistics are computed from batches; during inference, they use the entire training set.
- Training: Deep networks train with backpropagation by alternating forward evaluation, error computation, gradient calculation via the chain rule, and gradient-based weight updates.The procedure iteratively adjusts each weight to reduce error.
TrueNorth.
TrueNorth is a programmable neurosynaptic architecture in which communication occurs entirely through spikes and neurons and synapses use configurable parameters. By designing network structure, inputs, neurons, and weights during training, deep convolutional networks can be efficiently mapped to the hardware.
- Architecture: TrueNorth comprises neurosynaptic cores with programmable connectivity, synapses, and neuron parameters, using spikes for all on-chip and inter-chip communication.Each neuron can connect to an input line of any core and then to any neuron on that core through local synapses.
- Neurons: 23 configurable parameters define TrueNorth’s integrate-and-fire neurons, whose state updates each tick, typically at 1000 ticks per second.Higher update rates are possible.
- Neurons: A neuron emits a spike when V (t) reaches threshold θ, then resets using a configurable reset mode; values below a lower bound can snap to that bound.The model uses the neuron’s spiking inputs, corresponding weights, and a leak selected from {−255, −254, ..., 255}.
- Synapses: Synapses have individually configurable on/off states and strengths selected through each neuron’s 4-entry look-up table.Table values range from {−255, −254, ..., 255}, and source input types index the target neuron’s table.
- Mapping Deep Convolutional Networks to TrueNorth: Deep convolutional networks can be efficiently mapped to TrueNorth by designing their structure, neurons, network input, and weights during training.The mapping depends on appropriately adapting these network components to the neuromorphic hardware.
Structure
The network is mapped onto TrueNorth by partitioning layers into feature groups that fit each core’s input-line and neuron capacities. Overlapping spatial filters are supported through shared-core placement or replicated neuron outputs when inputs must reach multiple cores.
- Feature grouping: Each layer is partitioned into equally sized feature groups, with each group filtering a distinct, non-overlapping subset of input features.Groups are sized so total filter size per group fits available core input lines, while output features fit the core’s neuron count.
- Feature grouping: Feature grouping extends an approach originally used by AlexNet to split training across 2 parallel GPUs.The method can also adapt to hardware using other synaptic representation schemes, including higher-precision rate coding.
- Core mapping: 128 was the total filter-size limit used here to support trinary synapses.Multiple topographic locations for one group can share a core when the delivered input region contains their overlapping filter support.
- Core mapping: A 4×4×8 input region can implement overlapping 3×3×8 filters for 4 topographic locations on one core.This arrangement improves efficiency by reusing the same core for multiple spatial locations of a feature group.
- Inter-core routing: When overlapping filters span cores, identical neurons replicate outputs so the same input data reaches multiple targets.If the same core lacks enough neurons, a feature can be split across additional resources.
Neurons
The convolutional network uses binary data representations throughout to match hardware spikes. Configured TrueNorth neurons exactly reproduce the network’s activation behavior through offline normalization and specified leak, threshold, reset, and membrane-potential settings.
- Neurons: Binary representations are used for data throughout the network to match the use of spikes in hardware.The convolutional-network neurons use a binary representation scheme for data.
- Neurons: TrueNorth neurons exactly match the network’s activation behavior when their leak is configured from offline training-data normalization terms.The leak is set as L = ⌈b(σ + ϵ) −µ⌉, with the remaining variables computed from training data offline.
- Neurons: A threshold of 1, reset to 0 after spiking, and membrane-potential lower bound of 0 complete the matching configuration.These settings are specified as conditions for reproducing the activation function’s behavior.
Network input
The network avoids unnatural bit-wise spike encoding by converting high-precision, multi-bit inputs into a spiking representation using convolution filters with binary outputs.
- Network input: Multi-bit inputs, such as 8-bit RGB channels, are not encoded by converting each bit directly into a spike.Direct bit-wise encoding is unnatural because each bit represents a different value; in an 8-bit scheme, the most-significant bit carries weight 128.
- Network input: High-precision inputs are converted into a spiking representation using convolution filters with binary outputs.
Weights
TrueNorth simulates trinary weights using neuron copies and paired feature delivery, enabling synaptic weights of 0, +1, or −1.
- Weights: TrueNorth simulates trinary weights by delivering each feature’s output in pairs to target cores using neuron copies.One copy uses input type 1 for +1 and the other uses input type 2 for −1 in every neuron’s lookup table.
- Weights: Turning on neither, one, or the other paired synaptic connection creates weights of 0, +1, or −1, respectively.This implementation compensates for TrueNorth’s lack of direct trinary-weight support.
Training
Training combines constrained binary-neuron and trinary-weight representations with surrogate-gradient backpropagation and hysteretic weight updates. Standard optimization heuristics and a spike-sparsity penalty support offline GPU training and reduce deployment spike traffic.
- Surrogate-gradient training: Binary neurons use a surrogate derivative that equals 1 at zero and linearly decays toward 0 in both directions for backpropagation.The training procedure also constrains receptive-field size and features per group.
- Method novelty: The neuron derivative and hysteresis mechanism are described as unique to this work, while the training rule resembles BinaryNet.The passage states that the rule was developed independently.
- Hysteretic trinary weights: Trinary weights are produced through hysteretic rounding of clipped high-precision hidden values, with h set to 0.1.Updates apply to hidden weights bounded between −1 and 1, while discrete weights are used in forward and backward passes.
- Optimization and implementation: Standard training uses momentum (0.9), weight decay (10−7), and a learning rate reduced by 10× twice during training.Training also adds a spike-sparsity penalty with γ set to 10−4, serving as a regularizer and reducing deployment spike traffic and energy consumption; training runs offline on conventional GPUs using custom MatConvNet-based layers.
Deployment
Deployment uses reusable, composable corelets to map learned network parameters onto neuromorphic hardware. These corelets automatically compile platform-independent parameters into platform-specific configuration files that directly program TrueNorth chips.
- Deployment: Reusable, composable corelets map learned training parameters to neuromorphic hardware.The corelets provide hardware-description functions for deployment.
- Deployment: The corelets automatically compile learned network parameters independent of the neuromorphic platform.This separates learned parameters from the target hardware platform.
- Deployment: The compilation produces a platform-specific hardware configuration file that directly programs TrueNorth chips.The generated file can be used to program TrueNorth hardware directly.
Results
Across 8 image and audio benchmarks, TrueNorth networks generally approached state-of-the-art accuracy with a single chip, while scaling to 8 chips improved accuracy and surpassed state of the art on VAD. Single-chip networks were measured in hardware for accuracy, throughput, and power, whereas multi-chip networks were simulated.
- Evaluation setup: 8 image and audio benchmarks were evaluated using 5 template networks spanning 0.5 to 8 TrueNorth chips.Testing used 1 classification per hardware tick.
- Evaluation setup: Single-chip networks were run on TrueNorth hardware, while multi-chip networks were run in simulation pending infrastructure for interconnecting chips.Accuracy and throughput were measured on the NS1e development board, while power was measured on the NS1t test board at the same 1.0V supply.
- Energy and throughput: Power was estimated by combining scaled leakage power with scaled active power derived from classification measurements.Active power was computed by subtracting idle power from total classification power and scaling by throughput.
- Accuracy: Single-chip networks came within a few percent of state-of-the-art accuracy on almost all benchmarks without data augmentation or dropout.Comparisons used other works that also did not use data augmentation.
- Accuracy: Up to 8 chips improved accuracy by several percentage points and surpassed state-of-the-art performance on the VAD dataset.The comparison concerns state-of-the-art unconstrained approaches shown in Figure 6.
Discussion
The work shows that neuromorphic computing and deep learning can be combined because their structural and operational differences are not fundamental. The results validate neuromorphic computing as an efficient, flexible substrate for spiking neural networks and embedded neural-network applications.
- Implications: The work demonstrates that neuromorphic computing and deep learning have adaptable neural-network constructs and are not fundamentally different in structure or operation.The authors identify this as an important step toward embedded neural-network applications.
- Implications: TrueNorth implements convolutional networks despite being developed before the recent resurgence of convolutional networks and without being originally designed for them.This supports neuromorphic computing as an efficient, flexible substrate rather than one targeting a single application or network structure.