PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices

Tayfun Gokmen, O. Murat Onen, Wilfried Haensch

arXiv:1705.08014v1cs.LGcs.NEstat.ML

TL;DR

The paper addresses whether resistive-processing-unit specifications for fully connected networks extend to CNN training despite analog noise, bounds, and device variability. It maps convolutional layers onto RPU arrays and combines digital management techniques, achieving CNN classification error near the floating-point baseline.

  • Problem

    It is unclear whether resistive-device specifications developed for fully connected networks support broader architectures such as CNNs, whose analog computations introduce training-accuracy challenges.

  • Method

    The paper maps convolutional layers onto RPU arrays and uses digitally programmable noise, bound, update, and device-variability management techniques.

  • Results

    The final RPU model reaches 0.8% test error, compared with 0.8% for the floating-point baseline, after combining the management techniques and 13-device mapping.

  • Takeaways & Limitations

    The combined techniques enable successful CNN training with RPU devices, yielding classification error indistinguishable from high-precision floating-point training.

  • Takeaways & Limitations

    The simulations assume substantial device variability, including 30% cycle-to-cycle and device-to-device variation in incremental conductance changes.

Abstract

from arXiv · show

In a previous work we have detailed the requirements to obtain a maximal performance benefit by implementing fully connected deep neural networks (DNN) in form of arrays of resistive devices for deep learning. This concept of Resistive Processing Unit (RPU) devices we extend here towards convolutional neural networks (CNNs). We show how to map the convolutional layers to RPU arrays such that the parallelism of the hardware can be fully utilized in all three cycles of the backpropagation algorithm. We find that the noise and bound limitations imposed due to analog nature of the computations performed on the arrays effect the training accuracy of the CNNs. Noise and bound management techniques are presented that mitigate these problems without introducing any additional complexity in the analog circuits and can be addressed by the digital circuits. In addition, we discuss digitally programmable update management and device variability reduction techniques that can be used selectively for some of the layers in a CNN. We show that combination of all those techniques enables a successful application of the RPU concept for training CNNs. The techniques discussed here are more general and can be applied beyond CNN architectures and therefore enables applicability of RPU approach for large class of neural network architectures.

INTRODUCTION

Large DNN training is computationally intensive, motivating specialized hardware and resistive-device accelerators beyond conventional CMOS. However, specifications established for fully connected networks require validation on broader architectures such as CNNs.

  • Training large DNNs can take weeks even on distributed systems, motivating specialized hardware accelerators.
  • Conventional GPU, FPGA, and ASIC approaches optimize digital multiply-and-add units but remain constrained by shared CMOS scaling limitations.
  • Resistive-device approaches target greater acceleration and lower power and area than conventional CPU/GPU systems.
  • Prior resistive cross-point studies evaluated device and system requirements mainly with fully connected networks, including symmetric conductance updates.
  • Because resistive arrays compute multiply-and-add operations in the analog domain, it remains unclear whether fully connected-network specifications generalize to broader architectures.
  • Backpropagation repeatedly uses forward, backward, and weight-update cycles involving matrix operations.

Mapping Fully Connected Layers to Resistive Device Arrays

Resistive cross-point arrays implement the matrix operations of backpropagation locally and in parallel. This allows RPU arrays to remain active across forward, backward, and update cycles with constant-time operation independent of array size.

  • Resistive cross-point arrays implement forward and backward matrix products and weight updates using stored conductances and voltage or current signals.
  • RPU arrays support all three backpropagation cycles, enabling local parallel operations with O(1) time complexity independent of array size.
  • The RPU concept is extended to CNNs by mapping convolutional layers so array parallelism is used throughout forward, backward, and update cycles.

Convolutional Layers

A convolutional layer applies spatially local kernels across an input volume to produce an output volume. The layer can be represented schematically by its input volume, kernels, output volume, and corresponding RPU-array mapping.

  • A convolutional layer receives an input volume with width, height, and channel depth, while each kernel is spatially smaller and spans the full input depth.
  • Figure 1 contrasts the standard convolutional-layer volumes with an RPU mapping in which input and output matrices propagate through the kernel matrix across all three cycles.

Mapping Convolutional Layers to Resistive Device Arrays

Convolutional layers are mapped to RPU arrays by flattening kernels into a parameter matrix and rearranging input and output volumes. The resulting forward, backward, and update computations become repeated vector or matrix operations that exploit array parallelism.

  • Flattening each kernel into a column vector and stacking kernels forms a parameter matrix that stores the convolutional trainable parameters.
  • After rearrangement, forward convolution uses repeated vector-matrix products over local input regions, equivalent to a matrix-matrix multiplication for the output volume.
  • Using the transposed parameter matrix expresses the backward cycle as a matrix-matrix multiplication over error signals, while the update cycle computes parameter gradients by matrix multiplication.
  • The mapped parameter matrix is placed on an RPU array so its matrix operations execute as a series of parallel vector operations.
  • The mapping preserves the fundamental RPU operations and generalizes to padding, larger strides, dilated convolutions, and non-square inputs or kernels.

RESULTS

The study evaluates CNN training simulations on MNIST using a LeNet-5-like architecture mapped to four RPU arrays. The floating-point baseline achieves 0.8% test error.

  • The CNN uses two 5 × 5 convolutional layers with 16 and 32 kernels, each followed by 2 × 2 max pooling.
  • 0.8% test error is achieved by the floating-point baseline on the MNIST test data.The baseline stores parameters in four arrays with dimensions 16 × 26, 32 × 401, 128 × 513, and 10 × 129.

RPU Baseline Model

The RPU-baseline model transfers previously established device specifications and stochastic updates to CNN simulations. Its analog non-idealities produce substantially higher error than the floating-point and fully connected baselines.

  • The baseline uses device specifications previously found acceptable for fully connected-network training and a stochastic update scheme based on coincidence detection.
  • The stochastic update parameters use a bit-stream length of 10, unit gain factors, and a single-coincidence weight change of 0.001.
  • Device variability is modeled with 30% cycle-to-cycle and device-to-device variation in conductance changes, plus 2% variation in the positive-to-negative update ratio.Weight bounds average 0.6 with 30% device-to-device variation.
  • Figure 2 depicts backward and update array operations, while forward operation reverses the input and output directions relative to backward operation.
  • The RPU-baseline CNN reaches 10%–20% test error, compared with 0.8% for the FP baseline and 2.3% for the same RPU model on a fully connected network.The degradation is attributed to simultaneous analog backward-cycle noise and forward-cycle signal bounds on the final RPU array.

Noise and Bound Management Techniques

CNN training is sensitive to backward-cycle noise and output-layer signal bounds because analog signals can be overwhelmed or saturated. Digital noise and bound management mitigate both problems without adding analog-circuit complexity.

  • Backward-cycle noise dominates small error signals, causing learning to halt near 10% error when the signal does not generate sufficient output voltage.
  • Noise management divides digital error signals by their maximum magnitude before array multiplication and rescales the result afterward, preserving signal-to-noise ratio across signal ranges.
  • Output-layer bounds can make distinct class scores appear equal, so bound management repeats multiplication with halved input strength after saturation and digitally rescales the recovered signal.
  • The management techniques amplify noise together with the signal, but the signal-to-noise ratio remains fixed for the largest activation-contributing values.
  • 1.7% test error is achieved when both noise management and bound management are enabled, whereas either technique alone leaves error near 10%.The combined result is similar to eliminating analog noise and using infinite bounds.
  • Noise and bound management add only slight digital overhead because their computations can be incorporated into existing digital processing and triggered selectively.

Sensitivity to Device Variations

Device variations, especially imbalance in convolutional layers, materially degrade CNN training, while multi-device mapping offers a practical mitigation without eliminating variability.

  • 1.05% test error is achieved when device variations are removed across all four layers, compared with 1.7% for the RPU baseline with noise and bound management and 0.8% for the floating-point baseline.
  • Convolutional-layer variation is more damaging than fully connected-layer variation, with the first convolutional layer producing 1.2% error versus 1.4% for the second when variation is eliminated selectively.
  • Even a few percent of device imbalance can harm training, and eliminating imbalance produces a similar improvement trend to eliminating broader device variations.
  • Using 4 or 13 devices per weight in the most influential convolutional layer reduces test error to 1.45% and 1.35%, respectively.
  • Thirteen-device mapping reduces variation by a factor of 3.6, although it increases the array dimensions to 416 × 401.
  • Multi-device mapping can flexibly mitigate device variability without operational or circuit overhead when the mapping fits within a square array.

Update Management

Digitally controlled update parameters reshape stochastic pulse generation without changing the average learning rate, improving training when shorter update streams and asymmetric amplification are used.

  • The same average learning rate can be maintained while varying EF, JK, and JL, whose digital control is easier than controlling the device conductance increment.
  • 1.3% test error is achieved at EF = 1, while increasing EF to 40 does not improve performance despite longer update streams.
  • At EF = 1, larger amplification factors make pulse generation more likely and updates more deterministic, but cause earlier clipping.
  • Convolutional layers receive hundreds of stochastic updates per image through weight reuse, whereas fully connected layers receive only one, influencing the update trade-offs.
  • The update scheme redistributes amplification factors between rows and columns to reduce row-wise correlation when their activation ranges differ late in training.

Results Summary

Progressive digital management techniques close the gap between resistive-array and floating-point CNN training, culminating in near-floating-point accuracy with multi-device mapping.

  • 0.8% test error is reached by combining noise and bound management, reduced-bit update management, and 13-device mapping on the second convolutional layer, nearly matching the floating-point baseline.
  • Noise and bound management reduce test error from above 10% to about 1.7% by addressing backward-cycle noise and forward-cycle signal saturation.
  • Adding update management with reduced update bit length lowers the model's test error to 1.1%.
  • The management methods can be selectively enabled through digital-circuit programming and applied beyond CNN architectures without changing the analog-array design considerations.

DISCUSSION AND CONCLUSIONS

RPU accelerators extend resistive-array training beyond fully connected networks to CNNs, where performance depends on weight reuse rather than total MAC count. Digital noise, bound, update, and variability management can enable CNN training without changing the array or analog peripheral design.

  • CNN mapping: Convolutional kernels must be rearranged across RPU arrays so repeated vector operations exploit array parallelism during all three backpropagation cycles.Weight sharing causes repeated array accesses, creating both implementation challenges and opportunities for CNN training.
  • Performance implications: RPU-based CNN processing is dominated by the largest weight reuse factor rather than total MACs, enabling architectures with more or larger kernels without increasing training time.For AlexNet, the first convolutional layer dominates because its weight reuse factor is C₁ = 3025 despite the layer having the smallest array and only 10% of total MACs.
  • Hardware design: Smaller RPU arrays can reduce vector-matrix measurement time, although array dimensions are constrained by parasitic resistance and capacitance.The discussed limit for the largest array size is 4096 × 4096, with the measurement time determined under an acceptable noise threshold.
  • Architecture design: Allocating multiple arrays to the first convolutional layer can reduce its weight reuse factor by directing image regions to separate arrays and learning features independently.This change targets the layer that otherwise determines the processing time on RPU hardware.
  • Management techniques: Combining digitally programmable noise and bound management with selective update management and device-variability reduction enables successful CNN training without modifying analog array circuitry.The paper places these management techniques in the digital domain while retaining the array and supporting analog peripheral design.
Loading 1705.08014v1…