Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey

Giorgos Armeniakos; Georgios Zervakis; Dimitrios Soudris; Jörg Henkel

Source-linked AI summary

Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey

Giorgos Armeniakos, Georgios Zervakis, Dimitrios Soudris, Jörg Henkel

arXiv:2203.08737v1Mar 16, 2022cs.ARcs.LG

TL;DR

DNN inference offers high accuracy but imposes substantial computational and energy demands, motivating hardware approximation. This survey classifies approximation techniques, analyzes evaluation complexity and error metrics, and reviews accuracy-recovery approaches. It finds promising energy savings, while emphasizing that challenging benchmarks and mobile-oriented architectures remain insufficiently evaluated.

Problem
DNNs deliver high accuracy at substantial computational cost, creating a need to understand how hardware approximation affects inference efficiency and accuracy.
Method
The survey classifies hardware approximation techniques, analyzes datasets and DNN complexity, and reviews error metrics and accuracy-mitigation approaches.
Results
Precision scaling commonly achieves minimal accuracy loss at 4-bit or 2-bit inference, while computation reduction and approximate units show promising energy gains but need more challenging evaluations.
Takeaways & Limitations
Hardware approximation can support energy-efficient DNN inference and may also address reliability and security concerns.
Takeaways & Limitations
Evaluations remain limited on challenging benchmarks such as ImageNet and on mobile-oriented architectures, restricting conclusive conclusions for those settings.

Abstract

from arXiv · show

Deep Neural Networks (DNNs) are very popular because of their high performance in various cognitive tasks in Machine Learning (ML). Recent advancements in DNNs have brought beyond human accuracy in many tasks, but at the cost of high computational complexity. To enable efficient execution of DNN inference, more and more research works, therefore, exploit the inherent error resilience of DNNs and employ Approximate Computing (AC) principles to address the elevated energy demands of DNN accelerators. This article provides a comprehensive survey and analysis of hardware approximation techniques for DNN accelerators. First, we analyze the state of the art and by identifying approximation families, we cluster the respective works with respect to the approximation type. Next, we analyze the complexity of the performed evaluations (with respect to the dataset and DNN size) to assess the efficiency, the potential, and limitations of approximate DNN accelerators. Moreover, a broad discussion is provided, regarding error metrics that are more suitable for designing approximate units for DNN accelerators as well as accuracy recovery approaches that are tailored to DNN inference. Finally, we present how Approximate Computing for DNN accelerators can go beyond energy efficiency and address reliability and security issues, as well.

1 INTRODUCTION

DNNs achieve strong task performance but require substantial computation and energy, motivating approximate hardware accelerators that exploit neural-network error resilience.

1 INTRODUCTION: DNN accuracy improvements have increased computational demands, driving the development of customized hardware accelerators.The paper cites sophisticated services including machine translation, object detection, and healthcare as motivating applications.
1 INTRODUCTION: Tens of billions of MAC operations and thousands of parallel MAC units create high energy demands, especially for edge AI.Examples include Google TPU with 64K MACs and Samsung’s NPU with 6K MACs.
1 INTRODUCTION: Approximate Computing reduces energy and performance costs by tolerating errors in error-resilient computations, including neural-network inference.Research on approximate neural-network implementations has grown rapidly as DNN error tolerance is exploited.
1 INTRODUCTION: The survey examines hardware approximation’s effects on DNN inference efficiency and accuracy amid demanding edge-AI workloads.It also notes that DRAM often consumes 30%–80% of system energy, while processing units remain significant consumers.
1 INTRODUCTION: Prior surveys cover approximate arithmetic or broader approximation techniques, but differ in their treatment of DNN-specific hardware approximation.The paper positions its survey against related work summarized in Table 1.

2 BRIEF BACKGROUND ON DEEP NEURAL NETWORKS

DNNs are layered networks of artificial neurons that transform inputs through weighted sums, biases, and activation functions, with architectures such as MLPs, CNNs, RNNs, and Transformers.

2 BRIEF BACKGROUND ON DEEP NEURAL NETWORKS: Each neuron computes a weighted sum of inputs, adds a bias, and applies an activation function to produce its output.Neurons form graph nodes organized into layers that pass outputs between successive stages.
2 BRIEF BACKGROUND ON DEEP NEURAL NETWORKS: DNNs include MLPs, CNNs, RNNs, and Transformers, which differ in connectivity, weight sharing, recurrence, and attention mechanisms.CNNs use shared weights for spatial feature hierarchies, RNNs reuse weights across time, and Transformers use self-attention for parallel processing.
2 BRIEF BACKGROUND ON DEEP NEURAL NETWORKS: The survey covers hardware approximation techniques without restricting the DNN type, although most reviewed evaluations focus on CNNs.This distinction defines the survey’s scope and an observed concentration in the literature.

2.1 Layers

DNN layers perform complementary computations: convolution carries much of the workload, pooling reduces later computation, and activations and normalization transform intermediate values.

2.1 Layers: Fully connected layers compute weighted sums for every input-output neuron pair and are typically memory intensive classifiers.They usually appear near the final stage of a DNN, unlike compute-intensive convolution layers.
2.1 Layers: Convolution layers perform matrix dot products between input feature maps and kernels, producing output feature maps and carrying the main computational load.Their high computational intensity makes them a primary approximation target.
2.1 Layers: Pooling layers down-sample convolution outputs by selecting maxima or averages, reducing later activations, memory demands, and computations.Max pooling selects the largest nearby value, whereas average pooling computes their mean.
2.1 Layers: Activation functions apply nonlinear transformations, with ReLU setting negative values to zero and preserving positive values.Approximation can focus on positive-value accuracy when ReLU is used.
2.1 Layers: Batch normalization transforms inputs using learned scale and shift parameters based on the input tensor’s mean and standard deviation.The normalization keeps values in a common range and can facilitate training.

2.2 Training & Inference

Training learns network parameters from labeled data through forward and backward propagation, whereas inference performs only the forward pass and faces tight latency and energy constraints.

2.2 Training & Inference: Training uses labeled data and iterative optimization to learn weights through forward propagation, backpropagation, and weight updates.The forward pass evaluates layers successively, while backpropagation computes information used for updates.
2.2 Training & Inference: Overfitting occurs when training fits data too closely, producing overly complex decision boundaries and poor performance on unseen inputs.The passage identifies dropout, early stopping, and weight decay as techniques that help make models simpler.
2.2 Training & Inference: Approximation-aware retraining can mitigate hardware-induced accuracy loss but may be time consuming or infeasible for proprietary models and datasets.Fine-tuning and statistical methods are discussed as alternatives, while quantization-aware training supports low-bitwidth inference.
2.2 Training & Inference: Inference applies an already trained network to unseen data using only a forward pass, typically on a single cloud, mobile, edge, or IoT device.Inference settings can impose tight latency and energy constraints.

2.3 Models and Datasets

DNN research has progressed from early CNNs to increasingly deep and complex models, while dataset choice strongly affects model complexity and operation count.

Models: CNN architectures evolved from ConvNet and LeNet toward deeper, more complex models driven by demands for higher accuracy.Fig. 4 situates notable CNN models across this progression.
Datasets: Table 2 summarizes characteristics of commonly used datasets in the reviewed DNN evaluation studies.
Datasets: Dataset difficulty influences model complexity, with harder datasets requiring more weights and consequently more MAC operations.Different datasets for the same task can vary substantially in difficulty and are therefore difficult to compare directly.

3 HARDWARE APPROXIMATIONS FOR DNNS

The survey organizes hardware approximation techniques for DNN inference into groups based on recurring approximation patterns, while treating retraining as an accuracy-recovery mechanism rather than the approximation target.

Hardware DNN approximation techniques are clustered into groups according to the type of approximation applied.The resulting organization is illustrated in Fig. 5.
Retraining may mitigate approximation-induced accuracy loss, but it is used only to improve approximate inference accuracy, not as the approximation target itself.

3.1 Precision Scaling

Precision scaling reduces the bitwidth of DNN dataflow and arithmetic, lowering hardware cost while requiring quantization strategies to limit accuracy loss.

Reduced-bitwidth dataflow and arithmetic can provide energy, latency, and bandwidth gains over 32-bit floating-point implementations.Integer-only inference quantizes weights and activations to low-bitwidth integers such as 8-bit values.
Quantization reduces model size linearly, enabling storage savings and lower memory-transfer demands.
Quantized implementations are discussed for completeness but are not further analyzed, although later approximation techniques can be compatible or orthogonal to them.
An 8-bit fixed-point multiplier consumes 15.5x less energy and 12.4x less area than a 32-bit fixed-point multiplier.
Post-training quantization methods such as uniform symmetric, asymmetric min/max, and ACIQ target high accuracy at 8 bits.ACIQ clips activation ranges to reduce rounding errors while retaining most unquantized information.
Quantization-aware training simulates quantized inference during the forward pass while retaining floating-point weights and biases during backpropagation.The approach also supports trainable quantization parameters and extremely low bitwidths in some methods.

3.2 Computation Reduction

Computation reduction approximates DNN inference by avoiding selected operations, either dynamically through skipping or by reusing outputs through memoization.

Computation reduction avoids selected multiplications or convolutions, lowering the executed workload and targeting latency and energy costs.The category includes both memoization and skipping approaches.
Pruning: Pruning removes connections, filters, or channels offline and produces a compressed network variant before inference.Pruning can be structured or fine-grained according to the granularity of removed elements.
Skipping: Skipping uses a conditional prediction at runtime to determine whether a more complex computation can be eliminated.Its efficiency depends on skip frequency, predictor complexity, and the complexity of the skipped operation.
Memoization: Memoization replaces a computation with the output of a previously performed similar computation.Its efficiency depends on input similarity and the complexity of the eliminated computation.

3.3 Approximate Units

Approximate Units replace accurate MAC components with approximate circuits to reduce accelerator energy or latency. The surveyed approaches include approximate multipliers/adders, multiplierless designs, and log-multipliers, with variations in optimization, retraining, and runtime correction.

Approximate Units: Approximate Units replace accurate MAC units in DNN accelerators, targeting energy consumption and/or latency through three families: approximate multipliers/adders, multiplierless designs, and approximate log-multipliers.Approximate multipliers/adders modify circuit implementations; multiplierless designs eliminate multiplication circuits; log-multipliers convert multiplication into approximate logarithmic addition.
Approximate Multipliers/Adders: Approximate MAC research mainly targets multipliers because they are more complex and power-consuming than adders, while repeated MAC operations amplify per-operation energy gains.The surveyed work focuses on multiplier approximation rather than equally approximating both MAC components.
Approximate Multipliers/Adders: CGP-generated approximate multipliers can enforce worst-case error constraints and exact multiplication by zero, then iteratively select constraints that satisfy a target inference-accuracy loss.The optimization replaces accurate multipliers and evaluates the resulting network during constraint selection.
Approximate Multipliers/Adders: Approximation can be specialized or generic, combined with retraining or not, and implemented statically or with runtime error compensation.The surveyed works also include heterogeneous, layer-wise approximation and curable techniques that estimate and compensate errors during execution.
Multiplierless and Log-Multipliers: Multiplierless and log-multiplier designs reduce hardware complexity by replacing multiplication with simpler operations, including adders/shifters, trigonometric functions, or approximate logarithmic transformations.Multiplierless methods exploit weight or activation properties, whereas log-multipliers approximate binary logarithm and antilogarithm operations.

4 ERROR COMPENSATION TECHNIQUES

Approximation can severely reduce DNN accuracy, so surveyed methods recover accuracy through retraining, weight or bias adjustment, and runtime error compensation. Error-aware design further targets metrics and compensation schemes that reflect DNN inference behavior.

4 ERROR COMPENSATION TECHNIQUES: Complex networks can be sensitive to slight approximation, so approximation strength must be balanced against accuracy constraints and the efficiency gains sought.The section frames compensation as a way to preserve accuracy while enabling more aggressive approximation.
4 ERROR COMPENSATION TECHNIQUES: Approximation-aware retraining can recover severe accuracy loss, restoring MNIST accuracy from almost 10% to more than 90% after five epochs.The method uses approximate multipliers during the forward pass of backpropagation.
4.1 Retraining and Weight Adaptation: When retraining is impractical, weight tuning minimizes the mean error distance of approximate multiplications, while active training teaches robustness to a stochastic hardware-error distribution.ALWANN adapts weights without preprocessing or inference evaluation; AxTrain distinguishes passive and active retraining methods.
4.2 Statistical Error Compensation: Bias updating nullifies mean convolution error but cannot remove its non-zero variance; under a 0.5% accuracy-loss constraint, it achieved 1.4x higher energy reduction than no bias update.The correction replaces the bias using the mean approximate-multiplication error.
4.2 Statistical Error Compensation: Control-variate compensation improved average inference accuracy by 0.86%–21% on CIFAR10 and 3.6%–21% on CIFAR100 versus the same approximation without compensation.The improvement range depends on the perforated-multiplier configuration m and was measured across VGG, ResNet, ShuffleNet, and GoogleNet models.
4.3 Error Metric Optimization: MRED may be suboptimal for DNN inference, motivating metrics and designs that account for input or weight distributions, worst-case error, and error variance.Reported analyses emphasize accurate multiplication by zero, weighted error distance, and variance-based compensation behavior.

5 ENERGY-ACCURACY EVALUATION

The survey evaluates approximate DNN hardware by relating energy-accuracy outcomes to benchmark complexity, including dataset, network size, precision, and approximation family. Results show substantial savings, but gains generally decline on more complex scenarios and remain unevenly validated across modern mobile-oriented networks.

5.1 Assessing the Complexity of the Evaluation Scenarios: Dataset difficulty, DNN size, and precision materially shape evaluation outcomes, so energy-accuracy claims require complexity-aware benchmarking.MNIST is comparatively easy, while larger networks and harder datasets provide more demanding tests.
5.1 Assessing the Complexity of the Evaluation Scenarios: 44% of approximate works evaluated networks below 100M MACs, whereas more than 24% examined networks exceeding 2G MACs.The evaluation literature spans both relatively small and highly computationally intensive DNNs.
5.2 Performance Analysis: 77% energy reduction with 0.53% average accuracy loss was reported for Memoization, but its evaluations used shallow networks and only 32-bit precision.Skipping handled deeper networks but delivered lower savings in harder settings, including around 10% at 8-bit precision.
5.2 Performance Analysis: 53% energy reduction with 0.37% accuracy loss was achieved for ResNet-164 on CIFAR100 using 8-bit approximate multipliers against a 12-bit baseline.This result demonstrates that substantial savings can persist for a challenging dataset and deep network.
5.2 Performance Analysis: All three approximation families appear among optimal solutions, with Log-Multipliers favored at high precision and Multipliers/Adders or Multiplierless methods prevailing in challenging low-bitwidth cases.The survey also reports balanced use of retraining and concludes that more work is needed on challenging benchmarks and efficient approximation-aware recovery.

6 NOT JUST ENERGY EFFICIENCY

Approximate computing extends beyond energy efficiency by addressing thermal reliability, transistor aging, and DNN security weaknesses. The surveyed techniques use runtime approximation, aging-aware methods, and approximation-induced robustness to target these concerns.

6.1 Reliability-Aware Approximation: At 85℃, runtime precision scaling increased DNN accelerator efficiency by 1.5x while reducing power and temperature.The approach trades accuracy loss for reduced switching activity and can also exchange power savings for higher frequency or cooling cost.
6.1 Reliability-Aware Approximation: Approximate computing can mitigate accelerator reliability degradation caused by sustained MAC utilization and elevated operating temperatures.Continuous transistor stress accelerates aging, while temperature-dependent aging mechanisms exacerbate the problem; aging-aware approximation addresses this setting.
6.2 Defensive Approximations: Approximate computing is investigated as a defense against adversarial attacks that exploit DNN security weaknesses.Vision attacks introduce input noise to create misclassified samples, motivating approximation-based robustness methods.
6.2 Defensive Approximations: Low-precision models generally show higher adversarial accuracy than identical full-precision models, potentially because quantization increases non-linearity.The surveyed explanation is that stronger non-linearity can prevent small input changes from producing successful attacks.
6.2 Defensive Approximations: An approximate floating-point multiplier made LeNet-5 87.5% more robust against Black-Box attacks with 50% lower power and 67% lower area.The result was reported for MNIST with negligible accuracy loss, using data-dependent noise injected into convolution operations.

7 CONCLUSIONS, CHALLENGES, AND PERSPECTIVE

The survey classifies hardware approximation into precision scaling, computation reduction, and approximate units, finding substantial efficiency potential but uneven evaluation coverage. It highlights retraining, challenging benchmarks, mobile-oriented networks, and safety certification as unresolved adoption barriers.

7 CONCLUSIONS, CHALLENGES, AND PERSPECTIVE: The survey groups hardware DNN approximation techniques into Precision Scaling, Computation Reduction, and Approximate Units.It also reviews error metrics, accuracy-mitigation approaches, and applications beyond energy reduction.
7 CONCLUSIONS, CHALLENGES, AND PERSPECTIVE: Precision scaling is the most widely used approach, with quantization-aware training enabling minimal accuracy loss at 4-bit or 2-bit inference.Post-training quantization is efficient for mainstream 8-bit inference and may enable 4-bit inference with limitations, whereas quantization-aware training can be time-consuming.
7 CONCLUSIONS, CHALLENGES, AND PERSPECTIVE: Computation reduction delivers very high energy reduction with minimal accuracy loss, but gains drop substantially for 8-bit inference or ImageNet evaluations.The survey calls for deeper analysis using more challenging scenarios or networks requiring high-precision inference.
7 CONCLUSIONS, CHALLENGES, AND PERSPECTIVE: Approximate units attract the second-highest research interest and can provide immense energy reduction with negligible accuracy loss for small DNNs.ImageNet evaluations remain limited, low-precision combinations are incompletely analyzed, and many techniques require retraining to recover accuracy.
7 CONCLUSIONS, CHALLENGES, AND PERSPECTIVE: Approximate DNN accelerators may face certification challenges in safety-critical systems because inference no longer exactly matches the trained computation.ML safety standards are not yet formalized, leaving the effect of approximation on system certification unclear.
7 CONCLUSIONS, CHALLENGES, AND PERSPECTIVE: Approximate computing may enhance accelerator reliability and robustness, but the survey concludes that deeper investigation is required.This extends the potential scope beyond energy efficiency while preserving an open research agenda.