Source-linked AI summary
Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices
Tayfun Gokmen, Yurii Vlasov
TL;DR
Large DNN training demands datacenter-scale resources and long runtimes. The paper proposes locally storing and updating weights in massively parallel RPU devices, with peripheral and system specifications for CMOS-compatible accelerator chips. Its analysis reports hours rather than days for some large-network workloads and substantial acceleration and power-efficiency potential, subject to system-level integration constraints.
Problem
Large DNN training is computationally intensive and time-consuming, requiring datacenter-scale resources and many days for demanding workloads.
Method
The paper proposes RPU devices that locally store and update weights while exploiting training locality and parallelism, and analyzes device and system specifications for CMOS-compatible accelerators.
Results
Problems the size of ImageNet classification that require days on a datacenter cluster can take a few hours on one RPU accelerator chip.
Takeaways & Limitations
A single RPU accelerator chip can potentially train large DNNs with about a billion weights while reducing required training time and hardware resources.
Takeaways & Limitations
Power and area efficiency degrade when multiple tiles are integrated, with throughput eventually limited by power, area, communication bandwidth, or compute resources.
Abstract
from arXiv · showhide
In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We identify the RPU device and system specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30,000X compared to state-of-the-art microprocessors while providing power efficiency of 84,000 GigaOps/s/W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisted of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration and analysis of multimodal sensory data flows from massive number of IoT (Internet of Things) sensors.
1. Introduction
DNNs have achieved commercial success, but training remains computationally intensive and time-consuming. Existing hardware approaches exploit algorithmic locality and parallelism with varying success, while further acceleration requires minimizing data movement.
- DNNs have exceeded prior methods in speech and object recognition, demonstrating significant commercial success.
- Training large DNNs requires massive computational resources and substantial time, hindering further application.A cited example achieved 70% relative improvement using 1 billion connections trained on 1,000 machines for three days.
- Backpropagation is intrinsically local and parallel, motivating hardware designs that exploit these properties.
- Existing accelerators include GPUs, FPGAs, specially designed ASICs, and non-volatile-memory devices, with reported acceleration factors from 27X to 2,140X.
- For fully connected layers, local storage and processing can reduce data movement and further exploit training parallelism.
2. Definition of the RPU device concept
The RPU concept uses cross-point devices to store and process weights locally, while stochastic computing enables local, parallel weight updates. The approach maps forward and backward matrix operations onto crossbar arrays and implements updates through pulse coincidences.
- Forward and backward backpropagation cycles compute vector-matrix multiplications that can be performed on two-terminal resistive-device crossbar arrays.
- Weight updates require a locally computed vector-vector outer product and incremental update at every cross-point, making them challenging to parallelize.
- The RPU device is designed to provide both weight storage and processing for local, parallel updates.
- Stochastic computing reduces multiplication of two stochastic streams to a simple AND operation.
- Neuron activities and output errors are encoded as stochastic bit streams whose coincidence events drive incremental weight changes.
- A pulsing scheme sends positive and negative voltage pulses from row and column stochastic translators to distinguish coincidence events at each cross-point.
3. Network training with RPU array using stochastic update rule
The study evaluates stochastic RPU-array training against a baseline on a fully connected MNIST network, examining stream length and device non-linearity. It finds that probabilistic updates can preserve baseline-like classification while enabling array-size-independent update time.
- The evaluation trains a 784-256-128-10 fully connected network on 60,000 MNIST examples using cross-entropy and backpropagation.
- The stochastic model is calibrated to match the baseline learning rate through stream length, single-event weight change, and gain parameters.
- 10 stochastic bits are sufficient for the stochastic model to become indistinguishable from the baseline model.
- The algorithm fails to converge with linear device response, whereas a non-linearity factor below 0.1 achieves classification errors comparable to the baseline.
- Probabilistic updates can yield classification errors indistinguishable from baseline results.
- The stochastic update time is independent of array size and proportional to the stochastic bit-stream length, achieving O(1) time complexity.
4. Derivation of RPU device specifications
The paper derives RPU specifications by stress-testing stochastic training against device nonidealities, noise, variation, and update asymmetry. These tests identify tolerances that preserve low error penalties while exposing interactions among parameters and the restrictive role of asymmetry.
- Device resolution: 600 stored states are required as a lower estimate for an RPU device to represent incremental weight updates.The estimate follows from the largest acceptable conductance change per coincidence event.
- Update asymmetry: Up and down conductance changes must be balanced within 5% of the mean for the stochastic model to maintain an acceptable 0.3% error penalty.Global asymmetry is varied across the array, while device-to-device asymmetry is analyzed separately.
- Device variation: Device-to-device variation in incremental conductance change must remain below 6% of the mean to achieve an acceptable 0.3% error penalty.The conductance-change variation is sampled per device and held fixed throughout training.
- Noise tolerance: 10% normalized activation-function-temperature noise reaches an acceptable 0.3% error penalty during forward and backward vector-matrix multiplication.Update-cycle noise is also included in the stress tests.
- Combined tolerances: 5.0% test error results when all threshold parameters are combined, producing a 3.0% penalty above the baseline model.The combined penalty exceeds the individual 0.3% threshold because at least some parameters interact.
- Tolerance trade-offs: At most 0.3% error penalty is achieved by co-optimizing device mismatch and noise, with C, D, and E set to 30%, H to 2%, and I to 5%.Global asymmetry can be compensated by peripheral-circuit design, whereas device-to-device asymmetry is more restrictive.
5. Circuit and system level design considerations
The design translates RPU array assumptions into tile-level circuits and system estimates, while identifying noise, communication, power, and scaling constraints.
- 5.1 RPU array design: 4096 × 4096 RPU arrays use paired conductances for weights and can complete a full update cycle in 20 N.The paired arrays encode positive and negative weights, with each device pair occupying 2.68 OO.
- 5.2 Design of peripheral circuits: Time-encoded input pulses drive summed column or row currents, which differential integrators and ADCs convert during forward and backward cycles.Positive and negative weights are supplied through identical arrays and processed by peripheral readout circuitry.
- 5.2 Design of peripheral circuits: More than 20 pulses, approximately 5-bit resolution, is sufficient to eliminate the time-quantization error penalty for the evaluated models.The quantization error decreases as the total number of pulses increases.
- 5.2 Design of peripheral circuits: Restricting both sigmoid and softmax inputs to |h| = 12 keeps the total error penalty within an acceptable range.A tighter |h| = 3 bound is acceptable for sigmoid but produces exceedingly large output-layer softmax error.
- 5.3 Noise analysis: Noise constraints remain stringent: with a 20 N integration time, acceptable input-referred noise is about 10 N0/√XY, while array thermal noise is 7.0 N0/√XY.The thermal contribution leaves about 13.4 N0/√XY for other noise sources.
- 5.4 System level design considerations: 90 GB/s unidirectional bandwidth supports a single tile, while distant-tile communication can use on-chip buses or NoCs providing multi-terabyte-per-second bandwidth.Single-tile efficiency is about 5 orders of magnitude better than state-of-the-art CPU and GPU metrics, but multi-tile integration degrades efficiency.
6. Discussion
The discussion presents RPU devices and accelerator designs as a CMOS-compatible path toward faster, more power-efficient DNN training. It also describes broader architectural flexibility and scaling to large networks.
- 6. Discussion: RPU devices store and process data locally and in parallel, and the analysis evaluates tolerance to device, system, technological, and noise imperfections.The resulting specifications guide the search for physical mechanisms, materials, and device designs compatible with CMOS technology.
- 6. Discussion: Different RPU accelerator designs trade power against acceleration factor depending on network size.The system-design analysis reports many-orders-of-magnitude potential acceleration while reducing required power and compute hardware resources.
- 6. Discussion: Convolutional layers can be mapped to RPU arrays analogously to fully connected layers by performing matrix-matrix rather than vector-matrix multiplication.The proposed chip is intended to accommodate DNN architectures beyond fully connected layers.
- 6. Discussion: A single RPU accelerator chip can train billion-parameter DNNs, reducing ImageNet-scale training from days on thousands of machines to a few hours.The claim applies to networks with parameter counts on the order of a billion.
Figures and Tables
The figures and tables define the stochastic-update mechanism, evaluate training sensitivity to device and system imperfections, and summarize accelerator architecture and RPU specifications.
- Update mechanism: Eq. 1 and Eq. 2 contrast conventional cross-point weight updates with stochastic updates implemented through local AND operations and conductance-changing pulses.Figure 1 shows separate pulsing schemes for up and down conductance changes.
- Training stress tests: The MNIST experiments compare stochastic training against a conventional-update baseline while varying stochastic bit length, device non-linearity, and numerous imperfections.Figure 3 varies update increments, weight bounds, variation, asymmetry, and vector-matrix multiplication noise.
- Combined tolerances: Figure 4 converts the stress-test results into parameter thresholds for a 0.3% error penalty and evaluates combined imperfections using radar and training-error plots.The gray region represents combinations producing at most the stated error penalty.
- Accelerator architecture: Figure 5 combines analog current integration and ADC readout, bounded nonlinear-function circuits, tiled RPU arrays, on-chip communication, and noise constraints.The architecture places RPU tiles below digital NLF circuits and uses a bus or NoC for on-chip communication.
- System and device summaries: Table 1 summarizes comparisons among RPU system designs and state-of-the-art CPUs and GPUs, while Table 2 summarizes RPU device specifications.The specifications guide implementation choices for the proposed accelerator concept.
- Device compensation: Averaged up/down conductance asymmetry can be compensated substantially by adjusting pulse widths and/or pulse amplitudes.The stated compensation applies to the averaged asymmetry across devices.