Source-linked AI summary
Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware
Peter U. Diehl, Guido Zarrella, Andrew Cassidy, Bruno U. Pedroni, Emre Neftci
TL;DR
Efficient processing of temporal sequences remains difficult on low-power neuromorphic hardware because such systems impose implementation constraints. The paper trains and constrains Elman RNNs before converting them to spiking RNNs for TrueNorth. The resulting hardware-constrained model achieves 74% question-classification accuracy with an estimated power consumption of ~17 uW.
Problem
Limited connectivity and synaptic-parameter resolution in neuromorphic systems complicate implementation of high-performance spike-based recurrent networks.
Method
The paper trains Elman RNNs offline, discretizes their weights and hidden states, and maps recurrent dynamics to spiking neurons using TrueNorth-compatible synaptic delays.
Results
74% accuracy on the question classification test set is achieved by the resulting TrueNorth network.
Takeaways & Limitations
15-tick synaptic delays are sufficient to support the temporal dynamics of simple recurrent neural networks in this question-classification task.
Takeaways & Limitations
The hidden-state discretization’s influence on performance for more complex tasks and on hidden-state trajectories remains to be understood.
Abstract
from arXiv · showhide
In recent years the field of neuromorphic low-power systems that consume orders of magnitude less power gained significant momentum. However, their wider use is still hindered by the lack of algorithms that can harness the strengths of such architectures. While neuromorphic adaptations of representation learning algorithms are now emerging, efficient processing of temporal sequences or variable length-inputs remain difficult. Recurrent neural networks (RNN) are widely used in machine learning to solve a variety of sequence learning tasks. In this work we present a train-and-constrain methodology that enables the mapping of machine learned (Elman) RNNs on a substrate of spiking neurons, while being compatible with the capabilities of current and near-future neuromorphic systems. This "train-and-constrain" method consists of first training RNNs using backpropagation through time, then discretizing the weights and finally converting them to spiking RNNs by matching the responses of artificial neurons with those of the spiking neurons. We demonstrate our approach by mapping a natural language processing task (question classification), where we demonstrate the entire mapping process of the recurrent layer of the network on IBM's Neurosynaptic System "TrueNorth", a spike-based digital neuromorphic hardware architecture. TrueNorth imposes specific constraints on connectivity, neural and synaptic parameters. To satisfy these constraints, it was necessary to discretize the synaptic weights and neural activities to 16 levels, and to limit fan-in to 64 inputs. We find that short synaptic delays are sufficient to implement the dynamical (temporal) aspect of the RNN in the question classification task. The hardware-constrained model achieved 74% accuracy in question classification while using less than 0.025% of the cores on one TrueNorth chip, resulting in an estimated power consumption of ~17 uW.
1. Introduction
Neuromorphic hardware offers highly power-efficient neural-network simulation, but constrained connectivity and synaptic resolution make sequence processing difficult. The paper converts machine-learning RNNs into spiking RNNs and demonstrates the approach on TrueNorth while preserving classification performance.
- Motivation: Neuromorphic systems support real-time neural-network simulation under an extremely low power budget for pattern-recognition applications.Potential applications include natural language processing, mobile devices, and robotics.
- Motivation: RNNs are essential for interpreting and generating language and have achieved state-of-the-art performance across challenging NLP tasks.Examples include translation, image caption generation, semantic similarity, and language modeling.
- Approach: The paper trains Elman RNNs conventionally, then uses their trained weights and connectivity to create spiking equivalents implemented on TrueNorth.This extends machine-learning RNNs to neuromorphic hardware while targeting high classification performance.
- Approach: 15-time-step synaptic delays, corresponding to 4-bit hidden-state discretization, do not impair the RNN’s functionality.Recurrent feedback is represented through delayed spikes, addressing a central challenge in converting RNNs.
- Motivation: Neuromorphic hardware constraints, including limited connectivity and synaptic-weight resolution, complicate implementation and have limited prior recognition systems.These constraints motivate conversion methods designed for specific hardware capabilities.
2. Material & Methods
The methods section follows the chronological construction of a spike-based RNN, from task and dataset definition through preprocessing and machine-learning architecture. It then presents the conversion process and TrueNorth implementation.
- Material & Methods: The workflow begins by specifying the task and associated dataset.This is followed by data preprocessing and the machine-learning RNN architecture.
- Material & Methods: The preprocessing and machine-learning architecture largely follow existing work, with some modifications.
- Material & Methods: The main contribution describes converting inputs to spikes and converting spikes back to rates for a TrueNorth-compatible recurrent network.The passage identifies subsection 2.4 as the main contribution besides the TrueNorth introduction.
2.1. Question Classification Task
The study classifies question sentences into six coarse answer categories using a labeled training set and a held-out evaluation set. An end-of-sentence marker is appended to every input sentence.
- Question Classification Task: The task classifies question sentences into six coarse categories: Abbreviation, Description, Number, Entity, Human, and Location.Finer-grained subcategories are described but not used in this project.
- Question Classification Task: The training dataset contains 5000 labeled sentences, while the evaluation set contains 500 sentences not used during training.
- Question Classification Task: Each input sentence receives a special End Of Sentence word represented by a vector of zeros during training and testing.
2.2. Pre-Processing and Word Vectors
The preprocessing uses pretrained word semantics to support generalization across linguistic variation. Word2vec skipgram vectors are trained on a large English Wikipedia corpus after basic text normalization.
- Pre-Processing and Word Vectors: The system uses pretrained word2vec word vectors to model semantic relationships and support generalization across linguistic variation.Semantically similar words are represented near one another in the embedding space.
- Pre-Processing and Word Vectors: The word2vec model uses skipgram with negative sampling to learn word meaning from word co-occurrences.
- Pre-Processing and Word Vectors: 64-dimensional word vectors are trained from 3.4 billion tokens of English Wikipedia text.Preprocessing includes punctuation removal and lowercasing.
2.3. Neural Network Architecture and Training
The model uses a projection layer, an Elman recurrent layer, and a softmax classifier to process question sequences. Its architecture is based on standard machine-learning techniques and is sized for TrueNorth mapping.
- The machine-learning RNN uses ReLUs and otherwise standard architecture, allowing modifications such as convolutional features or larger networks.
- The network comprises 48 projection units, 16 recurrent units, and 6 softmax classification units.
- At each time step, the recurrent layer receives projection-layer input and the previous recurrent state.The recurrent state is passed to the classification layer.
- The recurrent layer is an Elman or simple RNN, while the projection and softmax layers provide input transformation and final classification.
2.4. TrueNorth Implementation
The TrueNorth implementation converts the recurrent layer into a constrained spiking system. Synaptic delays encode recurrent state across 16-tick word windows, while discretized weights and rate-coded spikes match the hardware’s capabilities.
- TrueNorth provides 1 million programmable spiking neurons across 4096 cores, with each core containing 256 neurons and 65536 synapses.Its neuron and synaptic states update every millisecond, defined as one tick.
- 15-tick synaptic delays plus 1 tick of transmission create a 16-tick counting window for recurrent feedback.The input and output to the softmax classifier are computed offline.
- TrueNorth linear neurons replace ReLUs by integrating synaptic input, resetting negative membrane states, and subtracting threshold after each spike.This firing rule makes spike count proportional to synaptic input.
- Poisson rate coding converts projection outputs into input spikes, while recurrent outputs are represented by up to 16 spikes per neuron.The recurrent output therefore discretizes 32-bit training precision to 4-bit precision.
- The implementation uses four axons per input dimension to provide 4-bit weights, with recurrent-layer size constrained by Nin + Nhid ≤ 256/Ns.The recurrent layer was implemented on a single core as a proof of concept.
- Synaptic delays store spikes from the preceding word, so a 16-tick interval corresponds to one machine-learning RNN time step.Delayed spikes preserve firing times rather than aggregating all recurrent information before computation.
2.5. Setups for Comparison
The comparison evaluates progressively constrained versions of the recurrent model, separating the effects of weight and hidden-state discretization before full TrueNorth implementation.
- Two intermediate setups compare 4-bit weight discretization alone with simultaneous 4-bit discretization of weights and hidden state.These setups bridge the original floating-point ReLU RNN and the TrueNorth network.
3. Results
The constrained recurrent network retains substantial question-classification performance on TrueNorth. The system also supports interactive classification, including examples where semantic content matters beyond the question word.
- 85% accuracy is achieved by the floating-point ReLU network on the question-classification test set.This is the baseline setup before weight discretization.
- 72.2% accuracy results when the weights are reduced to 4-bit precision.
- 78.4% accuracy follows when the hidden state is additionally discretized to 4 bits.The reported increase was unexpected relative to the weight-discretized setup.
- 74% accuracy is obtained by the resulting TrueNorth network on the question-classification test set.The implementation replaces ReLUs with TrueNorth linear neurons and encodes 48-dimensional inputs as Poisson spike trains.
- The interactive system classifies examples such as brain-weight questions as Number and ambiguous city-of-birth questions as Location.Words absent from the Wikipedia-trained word space are substituted with the average word vector.
4. Discussion
The conversion pipeline makes recurrent networks feasible on TrueNorth, but discretization and spiking conversion create measurable accuracy losses. The proof-of-concept remains narrowly evaluated, while the framework may extend to other tasks and benefit from future training improvements.
- Results: The largest accuracy loss came from discretizing synaptic weights, although training-aware or probabilistic rounding could reduce this loss.The authors identify weight discretization as the main drop and cite alternative discretization strategies.
- Results: 4-bit hidden-state discretization increased accuracy to 78.4% rather than decreasing it, possibly by preventing state drift across irrelevant inputs.The authors caution that this improvement may not generalize beyond tasks with limited target labels.
- Results: The recurrent-to-spiking conversion caused a 4.4% performance drop, and longer integration requires delays at least as long as the desired integration time.Chains of multiple axons could provide longer delays but would consume available axons.
- Implications: The framework is compatible with future improvements in training procedures, weight initialization, and loss functions without changing the conversion method.This preserves the possibility of transferring advances in machine-learning RNNs into spiking implementations.
- Implications: The method was tested only on question classification, while its task-oblivious conversion framework could also support vision, audio, and other language tasks.The authors identify more complex tasks and hidden-state trajectory effects as future evaluation targets.