Source-linked AI summary
Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators
Malte J. Rasch, Charles Mackin, Manuel Le Gallo, An Chen, Andrea Fasoli, Frederic Odermatt, Ning Li, S. R. Nandakumar, Pritish Narayanan, Hsinyu Tsai, Geoffrey W. Burr, Abu Sebastian, Vijay Narayanan
TL;DR
AIMC promises efficient DNN acceleration but its approximate, non-deterministic MVMs can reduce inference accuracy, while prior retraining studies used narrow workloads and simplified hardware models. The paper systematically applies HWA training with a realistic crossbar model across DNN topologies and workloads, finding that many models can approach FP accuracy, although large CNNs remain challenging.
Problem
The robustness of realistically sized and diverse DNNs to AIMC nonidealities, and the extent to which retraining can recover accuracy, remained unclear because prior studies used limited topologies and simplified, inconsistent hardware models.
Method
The paper retrains 11 DNNs with HWA noise injection and evaluates them using a standard AIMC crossbar model containing device, circuit, dynamic-range, quantization, and drift nonidealities.
Results
Five of 11 models achieve software-equivalent accuracy, defined as 99% of original-model accuracy beyond random guessing, while HWA reduces the worst-case raw-accuracy gap from 21.81% to 2.65%.
Takeaways & Limitations
CNNs are generally more difficult to deploy robustly than RNNs, and output noise and ADC S-shaped nonlinearity have the largest reported impacts on accuracy.
Takeaways & Limitations
Larger CNNs, particularly ImageNet models, can remain below iso-accuracy even after HWA training, motivating improved MVM fidelity or costly hardware remedies.
Abstract
from arXiv · showhide
Analog in-memory computing (AIMC) -- a promising approach for energy-efficient acceleration of deep learning workloads -- computes matrix-vector multiplications (MVMs) but only approximately, due to nonidealities that often are non-deterministic or nonlinear. This can adversely impact the achievable deep neural network (DNN) inference accuracy as compared to a conventional floating point (FP) implementation. While retraining has previously been suggested to improve robustness, prior work has explored only a few DNN topologies, using disparate and overly simplified AIMC hardware models. Here, we use hardware-aware (HWA) training to systematically examine the accuracy of AIMC for multiple common artificial intelligence (AI) workloads across multiple DNN topologies, and investigate sensitivity and robustness to a broad set of nonidealities. By introducing a new and highly realistic AIMC crossbar-model, we improve significantly on earlier retraining approaches. We show that many large-scale DNNs of various topologies, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers, can in fact be successfully retrained to show iso-accuracy on AIMC. Our results further suggest that AIMC nonidealities that add noise to the inputs or outputs, not the weights, have the largest impact on DNN accuracy, and that RNNs are particularly robust to all nonidealities.
1 Introduction
AIMC offers fast, energy-efficient in-memory MVMs but introduces nonidealities that complicate accurate deployment across DNN topologies. This paper addresses the need for systematic evaluation and robust HWA training across workloads using a comprehensive hardware model.
- Motivation: AIMC accelerates DNN MVMs in memory with high parallelism, speed, and energy efficiency, but computes them only approximately.Weights are stored in tunable conductive elements, while activations are applied as voltages or pulse durations and analog currents or charge are read out.
- Hardware nonidealities: AIMC conductances introduce non-deterministic errors through noise, programming variability, drift, and analog-circuit nonlinearities.These effects can affect both MVM readout and conductance programming, while programmed conductances may evolve over time.
- Research gap: Prior noise-aware studies typically covered only one or two DNNs and limited nonidealities, making their results difficult to generalize or compare.Output noise, saturation, and circuit nonlinearities were often neglected, while hardware and device choices varied across studies.
- Approach: The paper develops a robust HWA framework and standard AIMC inference model to evaluate diverse DNN topologies and estimate the relative impact of individual nonidealities.The framework extends training to previously neglected effects and supports future studies through open-source AIHWKit functions.
- Findings: Five of eleven tested AI workloads achieve inference within 1% of the FP reference after HWA training, even after at least one hour of PCM conductance drift.The workloads span CNN image classification, RNN text prediction and speech-to-text, and transformer natural-language processing.
- Findings: CNNs are generally less robust than RNNs, while additive output system noise is the most critical tested nonideality for accuracy.RNNs appear particularly tolerant of AIMC's finite signal-to-noise ratio.
2 Results
Hardware-aware training substantially improves AIMC inference across diverse DNNs, but robustness varies by architecture and nonideality. RNNs and transformers are generally more tolerant than CNNs, while input- and output-noise mechanisms are especially damaging.
- Baseline AIMC behavior: The standard AIMC model’s error, including PCM drift, roughly matches 4-bit fixed-point quantization of weights or inputs.The error is measured by the relative l2-norm deviation from the ideal MVM output.
- Baseline AIMC behavior: Direct AIMC mapping fails the normalized iso-accuracy target for almost all tested DNNs.The target requires normalized accuracy above 99%, corresponding to less than a 1% drop relative to the FP32-to-chance range.
- HWA-training results: After HWA training, all benchmark DNNs exceed 96% normalized accuracy one hour after programming, and five of eleven reach the above-99% iso-accuracy target.The iso-accuracy workloads include BERT and all evaluated LSTM workloads, plus WideResNet-32 on CIFAR100.
- HWA-training results: RNNs and transformers remain near iso-accuracy over time, whereas larger CNNs with higher-resolution ImageNet inputs show the largest long-term accuracy declines.DenseNet-121 and WideResNet-50 are identified as especially challenging, while HWA training improves long-term drift resilience over direct deployment.
- Nonideality sensitivity: Input- and output-noise nonidealities, including ADC/DAC resolution, additive output noise, and ADC nonlinearity, have the largest accuracy impact; weight-related nonidealities are less severe.CNNs are most sensitive overall, while RNNs are least sensitive; weight disturbances affect CNNs slightly more than RNNs or transformers.
3 Discussion
The discussion presents a realistic, generalizable HWA approach for AIMC while clarifying model limitations, training tradeoffs, and deployment implications. It also identifies distillation and hardware-model choices that affect robustness and system-level decisions.
- Realistic hardware modeling: The standard crossbar model combines calibrated device nonidealities with circuit effects and fixed dynamic-range constraints for realistic AIMC inference evaluation.It includes read noise, programming errors, conductance drift, system noise, DAC/ADC quantization, dynamically computed IR-drop, and input, weight, and output range limits.
- Model scope and limitations: The model supports functional verification of larger DNN workloads but does not replace detailed circuit simulations needed for hardware verification.The study uses abstractions such as Gaussian noise and a rapid input-dependent IR-drop approximation, while omitting static effects that programming can compensate.
- Training strategy: HWA training is designed to generalize across AIMC chip families rather than require costly individualized on-chip or chip-in-the-loop preparation.The approach can also be combined with online compensation, chip-in-the-loop training, or multiple device pairs per weight.
- Deployment implications: HWA training has no first-order effect on AIMC latency, throughput, or energy-efficiency because it occurs in software before deployment.Its value is in exposing accuracy–system-performance tradeoffs that inform model selection and hardware design.
- Training strategy: Distillation helps when HWA training is limited and remains significantly beneficial for the HMM Speech LSTM, but usually disappears as training continues.Most reported results therefore omit distillation.
- Training strategy: PCM-measured expected weight-noise distributions and programming-noise treatment are crucial for AIMC robustness, although once-per-mini-batch injection limits the approach.The method uses measured rather than simple Gaussian weight noise.
4 Conclusion
The conclusion finds that comprehensive HWA training improves robustness across CNNs, RNNs, and transformers, while sensitivity analysis identifies input/output noise as especially important. The resulting open-source model supports comparison and hardware–accuracy tradeoff analysis.
- Conclusion: Five of 11 studied models achieve software-equivalent accuracy, defined as at least 99% of the original DNN’s normalized accuracy-performance beyond random guessing.This is the paper’s stated software-equivalent accuracy criterion.
- Conclusion: HWA training reduces the worst-case raw model-accuracy gap from 21.81% to 2.65% across all models.The comparison spans the studied CNN, RNN, and transformer workloads.
- Sensitivity analysis: ADC and DAC resolution, additive output noise, and ADC S-shaped nonlinearity have the largest impacts on DNN accuracy.These nonidealities effectively inject noise at the inputs or outputs rather than in the weights.
- Sensitivity analysis: RNNs and shallower CNNs tolerate more AIMC nonidealities than other evaluated DNN topologies.The conclusion identifies topology-dependent robustness patterns rather than a uniform hardware sensitivity.
- Practical implications: The open-source AIHWKit crossbar model enables future HWA methods to be compared against these results and supports optimization of accuracy, latency, throughput, and energy-efficiency tradeoffs.The tools also help identify which hardware measures are needed to compensate for imperfect AIMC hardware.
Methods
The study models AIMC inference with calibrated analog, device, circuit, quantization, dynamic-range, and IR-drop effects, then uses HWA training to optimize robustness across DNN workloads.
- AIMC crossbar model: Per-crossbar input scaling α is optimized during HWA training and fixed during inference to determine the usable input range.The model primarily assumes static input ranges, while dynamic ranges are examined separately.
- AIMC crossbar model: The crossbar model represents analog MVMs with quantized, clipped, scaled inputs, conductance-based weights, nonlinearities, and additive output noise.Input and output ranges are finite, while weight programming errors, drift, and short-term read noise perturb the computation.
- Weight mapping: Weights are mapped from matrix columns to target conductances using column-specific scales γi, which HWA training further optimizes with the mapped conductances.The mapping is initialized from maximum column weights before stochastic optimization.
- Device nonidealities: PCM programming errors are modeled as conductance-dependent Gaussian deviations, while drift coefficients are statistically fitted from hardware measurements.Long-term effects are applied before evaluation at a chosen delay and then held fixed during test-set inference.
- IR-drop: IR-drop is approximated from time-averaged current, but the approximation underestimates deviations for small inputs and correlated workloads.For normally distributed weights and random correlated inputs, deviations are underestimated by roughly 5×; scaling the approximation above 10× does not significantly affect accuracy.
- HWA training: HWA training starts from an FP-trained DNN, injects calibrated nonideality noise, and optimizes training hyperparameters and hardware-mapping parameters with SGD.The approach applies expected nonidealities and hardware choices in both retraining and evaluation; on prior-study models, it reduced AIMC test error by more than 10% on average for long delays.
A.1 Static input range learning versus dynamic scaling
The model uses statically clipped and scaled input ranges that are learned during HWA training rather than dynamically adjusted for each input. Dynamic scaling provided less than 0.1% improvement for non-transformers and about 1% for transformers, suggesting static ranges were generally sufficient.
- Static input range learning: Perfect simultaneous utilization of input, weight, and output ranges is difficult across inputs and network tiles.
- Static input range learning: Static input ranges are clipped and scaled by a learnable α that is fixed and re-determined during inference.The range is not dynamically adjusted to each input vector.
- Dynamic scaling results: Less than 0.1% improvement was observed for all non-transformer DNNs when dynamic noise management was enabled.Transformer improvements were larger but still only about 1%.
- Dynamic scaling results: About 1% improvement was observed for transformers with dynamic input scaling and noise management.
- Conclusion: The authors conclude that HWA training learned the static input range well, making dynamic input scaling unnecessary for good inference accuracy.Dynamic scaling may still help models receiving less or no HWA training.
A.2.1 LSTM on Penn Treebank Dataset
The Penn Treebank experiment evaluates a two-layer LSTM with a 650-unit hidden size and a 10K-class AIMC decoder. HWA training uses an FP starting model and tuned regularization, noise injection, optimization, and stuck-device robustness.
- Model and dataset: The benchmark is a two-layer, 650-hidden-unit LSTM for word prediction on Penn Treebank using cross-entropy loss.Its encoder is digital, while the 10K-class decoder is assumed to use AIMC.
- Model and dataset: The decoder contains 10K classes, each corresponding to a dictionary word, and is the portion deployed on AIMC.
- HWA training: HWA training runs for 60 epochs from a conventionally FP-trained model.
- HWA training: The selected training setup uses a 0.01 base learning rate, 0.95 epoch-wise decay, 0.5 dropout, SGD momentum of 0.9, and gradient norm 10.It also injects PCM programming error at 5× nominal scaling.
- HWA training: Randomly fixing 1% of devices at ĝ_min = 0 during training adds drop-connect robustness.
A.2.2 Speech-to-text LSTM with HMM
The speech-to-text experiment uses a bidirectional LSTM acoustic model with HMM decoding on Switchboard data. HWA training starts from an 8-bit FP model and combines tuned optimization, PCM-error injection, knowledge distillation, and periodic weight remapping.
- Model and dataset: The acoustic model is a four-layer bidirectional LSTM trained for speech recognition on the Switchboard 300 dataset.The test set is Hub5 2000, containing Switchboard and call-home speech.
- HWA training: HWA training runs for 20 epochs while monitoring validation loss, starting from an 8-bit FP-trained model.
- HWA training: The training setup uses SGD momentum 0.9, batch size 256, gradient clipping 10, dropout 0.1, and a learning rate of 0.005.The learning rate drops by 10 at epochs 9 and 18.
- Accuracy improvements: Knowledge distillation significantly improves the HWA network's accuracy.
- Evaluation pipeline: Weights are remapped to the full conductance range every 2000 mini-batches before AIMC-simulated testing and digital HMM decoding.
A.2.3 BERT base: GLUE
The BERT-base experiment evaluates eight GLUE tasks with all linear layers of the 12-layer transformer assumed to be deployed on AIMC. WNLI is excluded because of its unusual dataset construction and small test set.
- Benchmark scope: BERT base is evaluated on eight GLUE tasks: RTE, STS-B, MRPC, CoLA, SST-2, QNLI, QQP, and MNLI.WNLI is excluded because of its unusual dataset construction and small test set.
- AIMC deployment: All linear layers of the 12-layer transformer are assumed to be deployed on AIMC.
A.2.4 Albert base: GLUE
The Albert base model matches BERT base structurally while sharing weights across its 12 layers, substantially reducing total parameters. It is evaluated on the same eight GLUE tasks through fine-tuning and hyperparameter scanning.
- Model and evaluation: Albert base is structurally identical to BERT base but shares weights across 12 layers, reducing its total parameter count.The model is evaluated using fine-tuning of the pretrained Albert model.
- Model and evaluation: The evaluation covers the same 8 GLUE tasks used for BERT, with hyperparameters scanned for highest accuracy.The learning-rate search ranges from 5 × 10^-6 to 1.5 × 10^-4; batch size is 10.
A.2.5 ResNet-32: CIFAR10
The study uses the paper's ResNet-32 architecture with a slightly non-standard channel configuration and performs additional floating-point training from an ONNX-exported pretrained model.
- Model preparation: The experiment uses the same ResNet-32 DNN as prior work, with a slightly non-standard channel setting.The architecture is defined in the cited paper [36].
- Model preparation: Additional floating-point training starts from the ONNX-exported trained model and achieves slightly improved accuracy.Training uses a 50-example batch size, a two-step learning-rate schedule, warmup, 600 epochs, cutout, random cropping, and mirroring.
A.2.6 WideResNet-16: CIFAR100
The experiments cover WideResNet-16 on CIFAR-100 and multiple ImageNet architectures, using standard floating-point training followed by hardware-aware retraining. The supplied implementation details also describe HWA training for an audio RNN model.
- WideResNet-16: CIFAR-100: WideResNet-16 is trained on CIFAR-100 with standard floating-point training followed by HWA training assuming all layers use AIMC.Floating-point training uses random resizing, flipping, and cutout augmentation; HWA training lasts 80 epochs with warmup and two learning-rate reductions.
- WideResNet-16: CIFAR-100: HWA training remaps weights roughly once per epoch and ramps injected weight noise during the initial training period.These procedures are specified for the WideResNet-16 experiment.
- ImageNet architectures: The ImageNet experiments use ResNet-18, ResNet-50, WideResNet-50, and DenseNet, differing primarily in layer depth and width.The starting models are fully trained floating-point networks provided by torchvision, followed by HWA retraining.
- Audio RNN implementation: The audio model comprises an acoustic encoder, prediction decoder, and joint network built from bidirectional and unidirectional LSTM layers.The encoder has 6 bidirectional LSTM layers and a fully connected layer; the prediction network has an embedding layer, one unidirectional LSTM, and a fully connected layer.
- Audio RNN implementation: The pretrained audio model is trained for 20 epochs on 262 hours of Switchboard audio and reaches 11.8% average WER on the Hub5-2000 test sets.Training includes speed and tempo perturbation, SpecAugment, and Sequence Noise Injection.
- Audio RNN implementation: For the audio RNN, HWA fine-tuning runs for 1–5 epochs on sequences capped at 500 frames using SGD with momentum and learning rate 1e-3.Weight-noise scale is 1; down-scaling and auto-scaling are disabled, and IR-drop is set to 0 during training.