Source-linked AI summary
Adaptive Learning Rule for Hardware-based Deep Neural Networks Using Electronic Synapse Devices
Suhwan Lim, Jong-Ho Bae, Jai-Ho Eum, Sungtae Lee, Chul-Heung Kim, Dongseok Kwon, Byung-Gook Park, Jong-Ho Lee
TL;DR
Hardware DNNs need learning rules that accommodate electronic synapses with discrete, limited conductance while implementing propagation and weight updates in hardware. The paper proposes an adaptive BP rule and evaluates it in simulations; results vary with device conductance and update method, and the proposed learning rule is reported robust to device-to-device variation.
Problem
Electronic synapses have discrete, limited conductance values that can prevent precise weight updates and degrade HW-DNN learning accuracy.
Source: §1 · p. 2Method
The paper proposes an adaptive BP learning rule for HW-DNNs and evaluates it through simulations using a three-layer perceptron and electronic synapse device models.
Source: §1 · p. 3Results
Linear conductance responses yield higher and more stable accuracy, higher dynamic range improves learning performance, and the proposed learning rule is reported robust to device-to-device variations.
Source: §4 · pp. 12–13Takeaways & Limitations
The proposed unidirectional weight-updating method can mitigate accuracy decline from non-ideal synapse-device characteristics, supporting on-chip learning in the studied HW-DNN setting.
Source: §5 · p. 13, §4 · p. 13The conductance-response evaluation considers an asymmetric nonlinear shape, and device-to-device variation is modeled with Gaussian random values having mean 1 and standard deviation 0.25.
Source: §3 · p. 9, §3.2 · p. 11
Abstract
from arXiv · showhide
In this paper, we propose a learning rule based on a back-propagation (BP) algorithm that can be applied to a hardware-based deep neural network (HW-DNN) using electronic devices that exhibit discrete and limited conductance characteristics. This adaptive learning rule, which enables forward, backward propagation, as well as weight updates in hardware, is helpful during the implementation of power-efficient and high-speed deep neural networks. In simulations using a three-layer perceptron network, we evaluate the learning performance according to various conductance responses of electronic synapse devices and weight-updating methods. It is shown that the learning accuracy is comparable to that obtained when using a software-based BP algorithm when the electronic synapse device has a linear conductance response with a high dynamic range. Furthermore, the proposed unidirectional weight-updating method is suitable for electronic synapse devices which have nonlinear and finite conductance responses. Because this weight-updating method can compensate the demerit of asymmetric weight updates, we can obtain better accuracy compared to other methods. This adaptive learning rule, which can be applied to full hardware implementation, can also compensate the degradation of learning accuracy due to the probable device-to-device variation in an actual electronic synapse device.
1 Introduction
The paper motivates hardware-based DNNs as a way to reduce the power and speed costs of software computation, while noting that implementing the full learning process in hardware is difficult. It proposes an adaptive BP learning rule for synapse devices whose conductances are discrete, limited, and non-ideal.
1 Introduction
- Full hardware DNNs must implement forward propagation, backward propagation, and weight updates, since software communication with the synapse array can become a dominant bottleneck.The paper identifies activation functions and their derivatives, as well as non-ideal device conductances, as implementation challenges.Source: §1 · p. 2
- Nonlinear, asymmetric conductance responses can disrupt learning, while device variation can impair accurate weight updates.Linear, symmetric, repeatable responses are described as desirable, but emerging devices may instead have nonlinear, finite conductance and between-device variation.Source: §1 · pp. 2–3
- Discrete, limited conductance values prevent precise weight updates and can degrade learning accuracy.The learning rule must minimize this degradation while retaining the advantages of the electronic synapse array.Source: §1 · p. 3
- The paper proposes an adaptive BP learning rule for HW-DNNs and evaluates simulated learning accuracy across device characteristics and weight-updating methods.The reported results are simulations based on the proposed learning rule.Source: §1 · p. 3
2 Materials and Methods
The methods adapt BP computations and weight updates to hardware built around electronic synapse arrays, including discrete, limited conductance states. The paper describes hardware-oriented propagation, update methods, and a three-layer perceptron used to assess learning feasibility.
2.1 Software- and Hardware-based DNNs
- A pair of electronic devices represents each signed synaptic weight through a conductance difference, whose currents contribute to the neuron’s weighted sum.The array can perform forward propagation through current summation, with an activation function implemented by electronic circuits.Source: §2.1 · pp. 3–4
- A hard-sigmoid activation provides simple hardware outputs and derivatives, while transposable arrays support backward propagation.The paper describes replacing the derivative value 1/(2c) with 1 without affecting the sign of the delta value, allowing simple gradient calculations in hardware.Source: §2.1 · pp. 4–5
- Because hardware weights change in discrete conductance steps, the proposed BP approach uses delta signs to guide updates and favors online learning over batch learning.The paper notes that batch learning requires an additional vector-by-matrix multiplication that the synapse array cannot perform.Source: §2.1 · pp. 4–5
2.2 Weight-updating methods
- The proposed weight-updating methods control discrete, limited conductance changes, and the paper compares alternative strategies for changing paired-device conductances.The methods include initializing and increasing conductance when an update reaches a device’s maximum; the proposed method is described for the same example case.Source: §2.2 · pp. 5–7
- Two update methods are described for unidirectional devices that can be reset, with lower conductance favored when equal conductances represent a zero weight.The paper notes that resetting and increasing conductance iteratively is not frequent, and that lower conductance can reduce power consumption.Source: §2.2 · p. 6
2.3 Multi-layer perceptron networks
- Learning feasibility is assessed using a simulated three-layer perceptron with 784 input, 200 hidden, and ten output neurons on MNIST.The evaluation uses 60,000 training examples and 10,000 test examples, with hard-sigmoid hidden-neuron activations and a softmax output activation.Source: §2.3 · p. 7
- The described array reverses source and drain roles between forward and backward propagation, and changes device conductance to update weights.The example uses separate electrodes for forward and backward propagation and applies pulses to control-gate nodes in a three-terminal device.Source: §2.3 · pp. 7–8
3 Results
The simulations evaluate hardware back-propagation across conductance nonlinearity, dynamic range, mini-batch size, and device variation. Unidirectional method b performs well with non-ideal conductance responses, while on-chip learning retains accuracy as device variation increases.
3 Results
- The evaluation uses MNIST in a three-layer perceptron and tests finite resolution, nonlinear conductance, and device variability; its conductance model relates pulse count and current conductance to conductance change.The study varies β across 0, 1, 2, and 3 and dynamic range across 32, 64, and 128; the online procedure uses array circuits for shaded operations and software support for input/output processing.Source: §3 · pp. 8–9
3.1 Evaluation of the learning rule
- At 60,000 iterations, HW-based BP accuracy was slightly higher than SW-based BP, though the software result could improve with additional training epochs.The comparison used one epoch; the SW-based BP result was slightly smaller at that point.Source: §3.1 · p. 9
- Method b averaged 95.36%, 95.59%, 94.80%, and 93.71% accuracy at β=0, 1, 2, and 3, with less degradation than methods a and c as nonlinearity increased.The authors identify method b as suitable for devices with nonlinear, asymmetric conductance responses.Source: §3.1 · pp. 9–10
- For method b, average accuracy was 92.96%, 94.80%, and 94.71% at dynamic ranges of 32, 64, and 128, respectively; it also fluctuated less than methods a and c.The reported comparison fixes β=2 and mini-batch size at 1.Source: §3.1 · pp. 9–10
- With method b, accuracy was 94.80%, 95.64%, 95.70%, and 80.42% for mini-batch sizes 1, 2, 5, and 10, respectively.The paper reports that accuracy decreases and becomes unstable as mini-batch size increases.Source: §3.1 · p. 10
- Method b keeps weight changes closest to symmetry by changing conductance one step from its current state, unlike the larger up/down asymmetries described for methods a and c.For method c, the authors also note that decreasing conductance can provide effectively denser steps and better accuracy than method a.Source: §3.1 · p. 10
- HW-based BP was comparable to SW-based BP across networks with one, two, and three hidden layers; the authors report that accuracy improves for the three-layer network with more training epochs.For SW-based BP, average accuracies were 93.83%, 96.19%, and 91.84% for one, two, and three hidden layers; with three epochs, the three-layer result rose to 95.10%.Source: §3.1 · p. 11
3.2 On-chip learning robust to device variations
- On-chip accuracy was 94.92%, 94.81%, and 94.01% at σ=0, 0.5, and 1, while off-chip accuracy was 94.67%, 79.24%, and 57.34%, respectively.The authors report that on-chip accuracy is not affected by device variation, whereas off-chip accuracy is significantly reduced as variation increases.Source: §3.2 · p. 12
4 Discussion
The discussion presents a single-step, sign-based hardware learning rule for non-ideal synapse devices. It highlights the proposed unidirectional update method’s accuracy benefits and the rule’s robustness to device variation.
4 Discussion
- Each iteration uses one conductance step selected by the sign of δ, avoiding magnitude-based multi-pulse updates and a separate learning rate.The step size is determined by the device’s minimum conductance step.Source: §4 · p. 12
- The proposed unidirectional method mitigates accuracy decline under nonlinear, asymmetric, finite-range conductance responses and larger mini-batches.It compensates asymmetric updates, reducing variation in |ΔW| at each update; online learning is also reported to suit learning performance.Source: §4 · p. 12
- On-chip learning is described as robust to unavoidable device-to-device variation, so training and inference should not be separated.The authors contrast this with off-chip learning, where device variation can affect learning accuracy.Source: §4 · p. 13
- The learning rule supports full-hardware DNN computation with power efficiency, high speed, and robustness to device variations.The authors identify these as outcomes of applying the rule to HW-DNNs.Source: §4 · p. 13
5 Conclusion
The conclusion reports better learning performance with more linear conductance responses and higher dynamic range. It identifies the proposed unidirectional update method as suitable for non-ideal devices and summarizes the rule’s hardware scope.
5 Conclusion
- More linear conductance responses and higher dynamic range yield better learning performance; the proposed unidirectional method mitigates accuracy decline from non-ideal devices.The authors say this method is appropriate for devices with nonlinear, asymmetric, finite-range conductance responses.Source: §5 · p. 13
- The learning rule enables full-hardware implementation of deep neural network computations and is described as robust to device variations.The conclusion notes that device-to-device variation is unavoidable in massive arrays.Source: §5 · p. 13
- The authors expect the rule to suit HW-DNNs designed for power efficiency and high speed, with immunity to device variations.Source: §5 · p. 13
6 Author Contributions
SHL and JB conceived the idea and carried out learning and recognition simulations. Other authors supported the simulations and manuscript preparation; all authors discussed the results, and JL supervised the research.
6 Author Contributions
- SHL and JB conceived the idea and undertook the learning and recognition simulations.Source: §6 · p. 13
- JE, STL, CK, and DK supported the simulation and helped prepare the manuscript.Source: §6 · p. 13
- All authors discussed the results, and JL supervised the research.Source: §6 · p. 13