Source-linked AI summary
Learning in Log-Domain: Subthreshold Analog AI Accelerator Based on Stochastic Gradient Descent
Momen K Tageldeen, Yacine Belgaid, Vivek Mohan, Zhou Wang, Emmanuel M Drakakis
TL;DR
AI training’s substantial computational and energy demands motivate more efficient hardware. The paper develops a continuous-time SGDr analog accelerator using subthreshold MOS circuits and volatile memory; simulations report close agreement with ideal behavior, including 8-bit model-fitting precision.
Problem
AI training and inference require substantial computational and energy resources, creating challenges for access and edge deployment.
Source: §1 · p. 1Method
Results
Circuit-versus-ideal maximum absolute errors were 0.71% for estimated weights and 0.87% for MSE; model-fitting prediction error was 0.527%, corresponding to 8-bit resolution.
Source: §5.1 · p. 7, §5.2 · p. 11Takeaways & Limitations
The results validate the feasibility and accuracy of the weight-learning node, rather than a complete accelerator realization.
Source: §6 · p. 12The design does not address realization of the bias term, which the paper says is essential for training machine-learning models.
Source: §6.1 · p. 12
Abstract
from arXiv · showhide
The rapid proliferation of AI models, coupled with growing demand for edge deployment, necessitates the development of AI hardware that is both high-performance and energy-efficient. In this paper, we propose a novel analog accelerator architecture designed for AI/ML training workloads using stochastic gradient descent with L2 regularization (SGDr). The architecture leverages log-domain circuits in subthreshold MOS and incorporates volatile memory. We establish a mathematical framework for solving SGDr in the continuous time domain and detail the mapping of SGDr learning equations to log-domain circuits. By operating in the analog domain and utilizing weak inversion, the proposed design achieves significant reductions in transistor area and power consumption compared to digital implementations. Experimental results demonstrate that the architecture closely approximates ideal behavior, with a mean square error below 0.87% and precision as low as 8 bits. Furthermore, the architecture supports a wide range of hyperparameters. This work paves the way for energy-efficient analog AI hardware with on-chip training capabilities.
1 Introduction
AI training’s computational and energy demands make efficient deployment difficult, while digital accelerators remain costly in power and existing analog options face device or training-workload constraints. The paper proposes a subthreshold-MOS analog accelerator for SGDr training using continuous-time computation and volatile memory.
1 Introduction
- Application-specific ASIC architectures are presented as a route to efficiency gains over scaling general-purpose multicore processors, with digital and analog designs both used for neural networks.The passage attributes this motivation to Amdahl’s law.Source: §1 · p. 1
- Training and inference demand substantial computational and energy resources, while server-class GPUs are energy-inefficient and costly and digital accelerators remain power-intensive.The passage estimates GPT-3 training at approximately 1,287 MWh and notes that server-class digital accelerators are unsuitable for energy-constrained environments such as edge devices.Source: §1 · pp. 1–2
- Existing analog accelerators can reduce energy costs through in-memory computation, but memristive-device non-idealities complicate integration and PUMA-like hybrid approaches remain limited to inference.The cited challenges include non-linear characteristics and difficulties with modeling and characterization.Source: §1 · p. 2
- The accelerator implements continuous-time SGDr training with ultra-low-power subthreshold MOS circuits and volatile memory, using mature CMOS rather than emerging memristive devices.Memory access is required only at the conclusion of training.Source: §1 · p. 2
2 Background
The background describes memristive crossbars as in-memory analog compute structures, then identifies their implementation overheads and device non-idealities. The paper proposes a crossbar-based alternative using subthreshold MOS circuits.
2 Background
- A memristive crossbar performs parallel in-memory computation, with inputs applied at rows, weights encoded as conductances, and column currents representing neuron pre-activation outputs.During training, error signals drive column currents for matrix-vector multiplication in back-propagation.Source: §2 · pp. 2–3, Figure 1 · p. 2
- Conventional crossbar systems require digital activation and gradient calculations, DACs and ADCs, and sometimes repeaters to compensate for voltage drops in long interconnects.These overheads accompany the analog crossbar operations described for forward and backward passes.Source: §2 · p. 3
- Memristive-device drift and cycle-to-cycle variability can significantly affect computation accuracy and reliability, complicating integration into analog accelerators.The passage identifies these as non-idealities of memristive devices.Source: §2 · p. 3
- The proposed accelerator retains the standard crossbar architecture but uses subthreshold MOS circuits to address challenges associated with non-volatile memory.The paper states that this approach improves overall performance and energy efficiency.Source: §2 · p. 3
3 Theory
The theory recasts stochastic gradient descent with L2 regularization as a continuous-time process, then reformulates it for differential signaling to support log-domain circuit realization.
3 Theory
- The discrete SGDr update uses the weight, input feature, error term, learning rate, and regularization coefficient; one epoch processes m samples.The variables are defined in the passage, and one epoch completes after processing m samples.Source: §3 · p. 3
- The input feature x[n] is represented as a continuous-time step function x(t), with each sample held for ∆s and a DAC-like rise time set to 0.5% of ∆s.Figure 2 illustrates the discrete-to-continuous input representation.Source: §3 · p. 3, Figure 2 · p. 4
- The discrete SGDr update is approximated by an ordinary differential equation, and weights at an epoch’s end are sampled using a unit impulse function.The epoch duration is m × ∆s, where m is the sample count and ∆s is the hold time per sample.Source: §3 · p. 4
- Differential signaling represents bidirectional weights as differences of positive signals, enabling the SGDr-CT equation to be reformulated for analog circuit realization.The derivation assumes strictly positive inputs and bidirectional weights.Source: §3 · p. 4
- The differential formulation separates into positive and negative learning equations by setting c(t) to zero, and can extend to bidirectional inputs through four learning equations.The two equations are identified as the positive and negative learning equations.Source: §3 · p. 4
4 Analog Circuits Realisation
The accelerator realizes continuous-time SGDr-CT training in a crossbar of weight-learning nodes built from subthreshold MOS log-domain circuits. Each node combines paired learning cells with other analog blocks, while multiple circuit-parameter choices can implement the same learning hyperparameters.
4 Analog Circuits Realisation
- Each weight-learning node uses two identical unidirectional cells to implement the positive and negative learning equations.The node is the fundamental building block of the crossbar.Source: §4 · p. 5
- The same learning rate can be implemented with different circuit parameters, including smaller capacitance by scaling down ∆s.Learning rate α can be adjusted through Iq and/or ∆s, while λ can be tuned through Iq and/or u.Source: §4 · p. 5
- Translinear log-domain multipliers compute weight-input products in the forward pass and error-weight products during back-propagation.A geometric mean splitter generates differential current-mode error signals for back-propagation.Source: §4 · p. 5
- Figure 3 shows a crossbar of weight-learning nodes and a circuit-level node realization integrating volatile memory with translinear MOS circuits.Each node updates one model weight in memory; the crossbar connects layer inputs to outputs.Source: §4 · p. 5, Figure 3 · p. 6
5 Results
The results compare the circuit implementation with an ideal SGDr model on five synthetic datasets and on Boston Housing, finding close agreement in estimated weights and loss. The experiments also examine circuit behavior across hyperparameter settings.
5 Results
- The evaluation comprised a single-feature comparison on five random datasets and a linear-regression fit on Boston Housing.The circuit was implemented in AMS 0.35 µm CMOS and simulated in Cadence Spectre, with an ideal SGDr model developed in Python as the baseline.Source: §5 · p. 7
5.1 Single Neuron
- 0.71% maximum absolute error in estimated weights and 0.87% in MSE separated the circuit from the ideal model across five random datasets.Figure 4 shows weight estimates and loss curves over 200 epochs; Table 3 reports the comparison after epoch 200.Source: §5.1 · p. 7, Figure 4 · p. 8, Table 3 · p. 9
- The single-neuron experiment tested learning rates 1 × 10−2, 1 × 10−3, and 1 × 10−4 with λ values 0.2, 0.1, and 0.05.The learning rate was varied through ∆s, and Figure 5 compares ideal and circuit accuracy across these settings.Source: §5.1 · p. 7, Figure 5 · p. 10
- For λ = 0.2 and λ = 0.05, slight deviations were corrected by tuning mapped currents to 0.27 and 0.03, respectively.The transistor dimensions were optimized for λ = 0.1.Source: §5.1 · pp. 7–8
5.2 Model Fitting
- The Boston Housing experiment used 506 samples and 13 features, with max-normalized features shifted to mean 0.8 and an 80/20 train-test split.The dataset was selected for historical benchmarking, and the passage notes critiques of its bias.Source: §5.2 · p. 11
- 0.527% maximum error between circuit and ideal predictions on Boston Housing corresponds to 8-bit weight resolution over [−1, 1].Training and test loss curves had percentage errors below 1%, despite slight differences in parameter values.Source: §5.2 · pp. 11–12, Table 4 · p. 12
6 Discussion and Future Work
The paper validates the weight-learning node and discusses design considerations for realizing the complete accelerator. These include bias integration, smaller regularization coefficients, ReLU implementation, and process-node scaling.
6.1 Inclusion of Bias Term
- The weight-learning node is validated, but the bias term is not implemented; a Seevinck integrator is proposed for its update.Adding bias is described as a future step toward evaluating tasks with complex decision boundaries.Source: §6.1 · pp. 12–13
6.2 Regularization Coefficient
- The circuit supports λ as small as 0.05; stacking two transistors on each side of the translinear loop can extend support to coefficients two orders of magnitude smaller.The added transistors keep the circuit in the subthreshold region; the proposed bias currents are Ir0 = Ir1 = 10 × Il0 = 10 × Il1.Source: §6.2 · p. 13
6.3 ReLU Activation Function
- A current mirror can implement ReLU, while a Winner-Take-All current comparator can implement its derivative.The current mirror passes only positive currents.Source: §6.3 · p. 13
6.4 Process Nodes
- Reducing ∆s permits smaller capacitance, which decreases circuit area, increases throughput, and reduces training time.The paper identifies this scaling as effective with faster process nodes and leaves fabrication in advanced technologies for future work.Source: §6.4 · p. 14
A.1 SGDr-CT Formulation
The appendix derives the discrete-time SGDr update from its continuous-time differential equation by integrating over each sample interval and applying Euler’s method.
A.1 SGDr-CT Formulation
- Integrating the first-order differential equation over [n∆s, n∆s + ∆s] and applying Euler’s method yields an approximate update.The approximation assumes δ(t) and w(t) vary minimally during ∆s.Source: §A.1 · p. 14
- The resulting discrete-time equation is the SGD difference equation, establishing equivalence between continuous-time SGDr-CT and discrete-time SGDr.The conversion uses x(n∆s) as the discrete-time input x[n].Source: §A.1 · p. 14
A.2 Positive Learning Cell Expression
The appendix derives the positive learning cell’s governing expression by analyzing its Bernoulli Cell and translinear loop, including the subthreshold transistor behavior.
A.2 Positive Learning Cell Expression
- The subthreshold drain-current relation is transformed into a linear differential equation using T(t) = 1/ID.ID0 depends on technology and transistor aspect ratio.Source: §A.2 · p. 15
- The derivation expresses gate voltage in terms of input currents and uses the translinear principle to relate ω to output current.The text defines the capacitor current iC(t) as the current through a capacitor of capacitance C.Source: §A.2 · pp. 15–16
- The drain-current expression for the positive learning cell’s Bernoulli Cell is derived and identified as the cell’s governing expression.The derivation begins by analyzing the Bernoulli Cell and its translinear loop.Source: §A.2 · pp. 14, 16