Source-linked AI summary
Long short-term memory networks in memristor crossbars
Can Li, Zhongrui Wang, Mingyi Rao, Daniel Belkin, Wenhao Song, Hao Jiang, Peng Yan, Yunning Li, Peng Lin, Miao Hu, Ning Ge, John Paul Strachan, Mark Barnell, Qing Wu, R. Stanley Williams, J. Joshua Yang, Qiangfei Xia
TL;DR
Limited bandwidth between computing and memory units remains an outstanding issue for LSTM networks, motivating alternative computing paradigms. This work experimentally implements LSTM networks with memristor hardware and demonstrates multilayer systems on regression and classification tasks.
Problem
Limited bandwidth between computing and memory units remains an outstanding issue for LSTM networks, motivating an alternative computing paradigm.
Method
The work experimentally implements multilayer recurrent neural networks with a memristor LSTM layer followed by a fully-connected layer.
Results
Successful demonstrations on both regression and classification tasks show the versatility of connecting memristor neural-network layers in different configurations.
Takeaways & Limitations
The results open a new direction for integrating multiple memristor crossbars.
Takeaways & Limitations
The approach is constrained by the relative scarcity of large memristor arrays.
Abstract
from arXiv · showhide
Recent breakthroughs in recurrent deep neural networks with long short-term memory (LSTM) units has led to major advances in artificial intelligence. State-of-the-art LSTM models with significantly increased complexity and a large number of parameters, however, have a bottleneck in computing power resulting from limited memory capacity and data communication bandwidth. Here we demonstrate experimentally that LSTM can be implemented with a memristor crossbar, which has a small circuit footprint to store a large number of parameters and in-memory computing capability that circumvents the 'von Neumann bottleneck'. We illustrate the capability of our system by solving real-world problems in regression and classification, which shows that memristor LSTM is a promising low-power and low-latency hardware platform for edge inference.
Introduction.
LSTM networks address temporal data processing but conventional implementations face latency, power, bandwidth, and memory challenges. This work experimentally implements LSTM networks in memristor crossbars and demonstrates regression and classification tasks.
- Conventional digital LSTM implementations have complicated structures with drawbacks for inference latency and power consumption.
- Limited parallelism and bandwidth between computing and memory units remain outstanding issues for LSTM networks.
- Memristors compute where information is stored, eliminating data transfer between memory and computation; crossbars provide this in-memory-computing architecture.
- The memristor crossbar implementation of an LSTM had not previously been reported, primarily because large memristor arrays were relatively scarce.
- The authors experimentally implemented multilayer LSTM-based RNNs using memristor crossbars, including monolithic 1T1R cells and connected fully-connected and recurrent layers.
- The memristor LSTM predicted airline passenger numbers and identified an individual from gait, supporting its use across regression and classification problems.
Results
The memristor LSTM stores model parameters in crossbar conductances and performs matrix multiplication in situ, reducing data transfers during recurrent computation. Experiments demonstrated regression and classification learning, including adaptation to hardware imperfections.
- Memristor LSTM architecture: Memristor conductances store the large number of parameters required by an LSTM-RNN.The crossbar stores weights and performs the associated analog matrix multiplications.
- Memristor LSTM architecture: In-situ crossbar matrix multiplication removes the need to transfer weight values back and forth during computation.The architecture combines parameter storage and analog matrix multiplication in the same crossbar.
- Memristor LSTM architecture: The LSTM layer was connected to a fully-connected layer, while gated and nonlinear units were implemented in software for these experiments.The authors state that these units could instead be implemented by analog circuits.
- Hardware implementation and training: The crossbar used stable multilevel conductance to enable analog-domain matrix multiplication, with conductance updates supported by a two-pulse training scheme.The network was trained in situ to compensate for device variation, noise, and other hardware imperfections.
- Regression experiment: The airline-passenger regression network learned to predict both training and unseen testing data after 800 epochs.The experiment used a two-layer recurrent network implemented in a 128×64 1T1R memristor crossbar.
- Classification experiment: 79.1% maximum classification accuracy closely matched defect-free simulation, confirming adaptation to hardware imperfections without hand-tuned parameters.Accuracy increased steadily during 50 training epochs.
Discussion
The study built multilayer recurrent neural networks combining memristor LSTM and fully connected layers, demonstrating successful regression and classification. The results support connecting memristor layers with different configurations and integrating multiple crossbars on one chip.
- The authors built multilayer recurrent neural networks with a memristor LSTM layer and a memristor fully-connected layer.
- Successful demonstrations on both regression and classification tasks show the approach can support different problem types.
- The results exhibited versatility in connecting memristor neural-network layers with different configurations.
- The results open a direction for integrating multiple memristor crossbars with different configurations on the same chip.
- Such integration could minimize data transfer and significantly reduce inference latency and power consumption in a deep recurrent neural network.
Methods
The methods implement a two-layer LSTM-RNN with memristor crossbars for matrix operations and software nonlinearities, using in-situ training and task-specific optimization. The hardware integrates 1T1R cells and low-resistance wiring to support accurate computation.
- Memristors are integrated with transistors as one-transistor one-memristor cells, with selector devices mitigating sneak paths in the crossbar.
- Low wire resistance, about 0.3 Ω between cells, was identified as a key factor supporting accurate matrix multiplication.
- The memristors were fabricated above a transistor array using palladium, hafnia, and tantalum electrode and switching layers.
- The network uses two layers: an LSTM layer followed by a fully-connected layer, with a cascaded structure extendable to more layers.
- The LSTM output becomes the hidden-layer output of the two-layer RNN, while a fully-connected layer reads out the final RNN output.
- Training used back-propagation through time, with mean square error for airline prediction and cross-entropy on the last time step for gait identification.
- Computationally expensive operations and weight-gradient calculations were performed in the crossbar, while gradients were accumulated before updates.
- The airline task used SGDM, whereas the gait task used RMSprop; hyperparameters and physical crossbar parameters were listed in Table 1.
Supplementary Figures
The supplementary figures document trained conductance and synaptic-weight maps, gait-data preprocessing, and classification inference outputs. Together they show how array states, input samples, and output probabilities are represented.
- Supplementary Figure 1 shows conductance maps and corresponding synaptic weights for the trained regression LSTM and fully-connected layers.
- Supplementary Figure 2 traces gait preprocessing from raw video frames to silhouettes, width-profile vectors, gait-cycle detection, and segmented samples.
- Supplementary Figure 3 shows trained conductance and synaptic-weight maps for the classification LSTM and fully-connected layers.
- In the classification LSTM, synaptic weights include input-connected weights and recurrent weights connected to outputs from the previous time step.
- Supplementary Figure 4 presents raw current outputs across eight columns and Bayesian probabilities computed from them using softmax.