Source-linked AI summary
ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining
Vojtech Mrazek, Zdenek Vasicek, Lukas Sekanina, Muhammad Abdullah Hanif, Muhammad Shafique
TL;DR
Retraining approximate DNNs can be computationally expensive, limiting the practicality of approximate arithmetic in accelerator computational paths. ALWANN searches multiplier and layer assignments and tunes weights without retraining; for ResNet-50, it reports more than 30% energy savings with accuracy falling from 89.15% to 88.1%.
Problem
Retraining to recover accuracy after approximation can be prohibitively slow for complex DNNs and may be impossible when training data is unavailable.
Source: §I · p. 1Method
ALWANN uses NSGA-II to search approximate-multiplier and layer assignments while tuning weights to adapt a trained network without retraining.
Source: §I · p. 2, A. Overview · p. 4, B. Weight tuning · p. 4Results
More than 30% energy savings are reported for ResNet-50 when accuracy falls from 89.15% to 88.1%.
Source: C. Overall results · p. 7Takeaways & Limitations
ALWANN produces approximate convolutional networks with energy savings and accuracy comparable to other automated approximation methods, without retraining.
Source: §I · p. 2, §VI · p. 8Approximate-multiplier simulation is a bottleneck: the single-core implementation runs at 10x lower performance than vectorized accurate multiplication.
Source: B. Quality of the generated AxNNs · p. 7
Abstract
from arXiv · showhide
The state-of-the-art approaches employ approximate computing to reduce the energy consumption of DNN hardware. Approximate DNNs then require extensive retraining afterwards to recover from the accuracy loss caused by the use of approximate operations. However, retraining of complex DNNs does not scale well. In this paper, we demonstrate that efficient approximations can be introduced into the computational path of DNN accelerators while retraining can completely be avoided. ALWANN provides highly optimized implementations of DNNs for custom low-power accelerators in which the number of computing units is lower than the number of DNN layers. First, a fully trained DNN is converted to operate with 8-bit weights and 8-bit multipliers in convolutional layers. A suitable approximate multiplier is then selected for each computing element from a library of approximate multipliers in such a way that (i) one approximate multiplier serves several layers, and (ii) the overall classification error and energy consumption are minimized. The optimizations including the multiplier selection problem are solved by means of a multiobjective optimization NSGA-II algorithm. In order to completely avoid the computationally expensive retraining of DNN, which is usually employed to improve the classification accuracy, we propose a simple weight updating scheme that compensates the inaccuracy introduced by employing approximate multipliers. The proposed approach is evaluated for two architectures of DNN accelerators with approximate multipliers from the open-source "EvoApprox" library. We report that the proposed approach saves 30% of energy needed for multiplication in convolutional layers of ResNet-50 while the accuracy is degraded by only 0.6%. The proposed technique and approximate layers are available as an open-source extension of TensorFlow at https://github.com/ehw-fit/tf-approximate.
I. INTRODUCTION
ALWANN targets energy savings from approximate arithmetic in DNN accelerator computational paths without time-consuming retraining. It searches layer-to-hardware assignments and tunes weights to balance accuracy and energy under accelerator constraints.
I. INTRODUCTION
- Retraining approximate networks can scale poorly: simulating approximate hardware slows full-network simulation, and one cited 50-layer case would take 89 days.The paper notes that retraining may be unavailable when proprietary-network training data cannot be accessed.Source: §I · p. 1
- The approach searches for suitable approximations separately across convolutional layers because those layers differ in error resilience.Prior approximate convolutional networks used uniform components across layers.Source: §I · p. 1
- ALWANN takes a trained network and approximate multipliers as inputs, then generates Pareto-optimal approximate networks trading off accuracy and energy.Its design must satisfy accelerator constraints such as limits on approximate execution units.Source: §I · p. 2
- A fast weight-tuning algorithm adapts layer weights to the employed multipliers and improves network accuracy by 4% on average without retraining.The method extends approximation to networks with up to 120M multiplications, compared with 200k in LeNet-6.Source: §I · p. 2
- ALWANN reports comparable results to other automated approximation methods on more complex networks without retraining, and greater energy efficiency than layer removal followed by training from scratch.An open-source TensorFlow extension provides approximate convolutional layers and an example of layer replacement.Source: §I · p. 2
II. BACKGROUND AND RELATED WORK
Prior work applies precision changes, pruning, or approximate components to improve DNN computational efficiency, but retraining and limits on supported networks or components remain concerns. The paper positions its approach against these methods and compares their complexity and properties.
A. Approximate neural networks
- Earlier work applied approximate operations and pruning to MLPs, including neuron selection based on gradients or sensitivity, and retrained the resulting networks.The cited sensitivity-based approach was evaluated only on MLPs.Source: A. Approximate neural networks · p. 2
- For CNNs, prior work used approximate multipliers with weight adaptation in LeNet and multiplierless multiplication in deeper networks, but the latter restricted which multipliers could be used.The comparison figure covers approximation-method complexity and properties; its component count is reported for 8-bit inputs.Source: A. Approximate neural networks · p. 2, Figure 2 · p. 2
- Approximate networks can also serve as stages in progressive chain classifiers, which stop evaluating later models when an earlier stage has sufficiently high confidence.The chain's models progressively increase in complexity and accuracy.Source: A. Approximate neural networks · p. 2
B. Neural network accelerators
DNN accelerators target dot products in computationally intensive layers, with designs ranging from layer-wise computation to pipelined networks. The paper considers approximate units in two DaDianNao-style hardware scenarios and assumes one approximation type per layer.
B. Neural network accelerators
- Accelerators commonly target dot products, the fundamental operation in convolutional and fully connected layers, which are among the most computationally and memory intensive.TPU and Flexflow perform layer-wise computations, while DaDianNao uses a pipelined network implementation.Source: B. Neural network accelerators · pp. 2–3
- DaDianNao places memory near Neural Functional Units, whose pipeline multiplies weights by activations, adds products, and applies an activation function.Four eDRAM banks and an NFU form a processing element; PEs form chips, which interconnect into a system.Source: B. Neural network accelerators · p. 3, Figure 3 · p. 3
- The paper considers either multiple approximate NFU variants per PE with all but one power-gated, or one approximation type per chip, which may vary across chips.A layer may map to multiple chips, but all chips in its tile are assumed to use the same approximation; configurable approximate units are excluded.Source: B. Neural network accelerators · p. 3, Figure 4 · p. 3
III. PROPOSED METHODOLOGY
ALWANN takes a trained network, a library of approximate components, and accelerator architecture details to construct approximate networks with Pareto-optimal energy and accuracy. It jointly assigns multipliers to hardware tiles and layers to tiles, tuning weights for each chosen multiplier.
A. Overview
- The user supplies a trained network, approximate-component library, accelerator architecture, and tile count; the typical setting has fewer tiles than convolutional layers and uses 8-bit multipliers.Each tile contains an array of identical MAC units, and each layer executes on one tile.Source: A. Overview · p. 3
- The method jointly maps approximate multipliers to tiles and convolutional layers to tiles, then updates each layer's weights for its assigned multiplier.The output is a set of AxNNs and corresponding accelerator configurations that are Pareto optimal for energy and classification accuracy.Source: A. Overview · p. 3
- The framework quantizes the network if needed, replaces convolutional layers with approximate ones, and uses NSGA-II to search candidate multiplier and layer assignments.Candidates are evaluated for accuracy and inference energy, initially on a subset of training data and then on the full training set before dominated solutions are filtered out.Source: A. Overview · pp. 3–4, Figure 5 · p. 4
B. Weight tuning
Weight tuning replaces each layer’s weights with values selected for the approximate multiplier assigned to that layer. The mapping minimizes multiplication error over possible inputs, avoiding retraining while remaining practical for quantized weights up to about 12 bits.
B. Weight tuning
- The mapping is precomputed offline for each approximate multiplier, using the weight and data-value ranges as its domain.The mapping is based on the multiplier’s properties rather than the NN structure or data distribution.Source: B. Weight tuning · p. 4
- For each weight, the mapping selects a replacement that minimizes the sum of absolute output differences between approximate and accurate multiplication across all inputs.Because the input range has constant size, this also minimizes mean error distance (MED); for an accurate multiplier, the replacement equals the original weight.Source: B. Weight tuning · p. 4
- During updating, each layer’s weight is replaced using the mapping for the multiplier assigned to that layer, and the approach is applicable approximately up to k ≤12 because of memory requirements.The selected multiplier is determined by the tile-to-multiplier and layer-to-tile mappings.Source: B. Weight tuning · p. 4
C. Representation of candidate AxNNs
Candidate AxNNs encode both which multiplier each accelerator tile uses and which tile computes each convolutional layer. NSGA-II explores this constrained design space for energy and accuracy trade-offs.
C. Representation of candidate AxNNs
- Each candidate AxNN is encoded by tile-to-multiplier and layer-to-tile mappings, with one integer per tile and one per layer, respectively.For pipelined accelerators, each chunk of |T| layer assignments must be a permutation so every tile has workload; power-gated accelerators impose no such mapping restrictions.Source: C. Representation of candidate AxNNs · p. 4
D. Design space exploration
- With 50 layers, 20 multipliers, and four tiles, the design space exceeds 1.2·10^22 pipelined and 2.0·10^35 power-gated AxNNs, motivating heuristic exploration.The method uses a multiobjective genetic algorithm to optimize energy and accuracy.Source: D. Design space exploration · p. 4
- The search favors nondominated AxNNs in the Pareto set, so solutions are retained when neither energy nor accuracy dominates the other.The passages define the Pareto set as containing nondominated solutions but do not supply the full dominance condition.Source: D. Design space exploration · p. 4
- The example compares execution plans for seven approximate convolution layers on three-tile pipelined and power-gated accelerators.The power-gated plan can reuse a tile for multiple layers, while pipelined tiles are active except at the last step.Source: Figure 6 · p. 5
- NSGA-II creates offspring by uniformly choosing each integer from either of two parents, then randomly mutating one integer with probability pmut while respecting constraints.The new candidates combine portions of two existing encodings.Source: D. Design space exploration · p. 5, Figure 7 · p. 5
IV. EXPERIMENTAL SETUP
The evaluation applies ALWANN to quantized ResNet models trained for CIFAR-10 and explores approximate implementations under pipelined and power-gated accelerator configurations. Search uses a subset of validation images, followed by inference on the full validation set.
IV. EXPERIMENTAL SETUP
- Three ResNet instances were trained for 105 epochs on CIFAR-10, then frozen, quantized, and converted to approximate convolutional layers for evaluation.The study uses ResNet variants to examine scalability, accuracy loss, and approximation through reducing convolution-layer counts.Source: §IV · p. 5
- Table I reports each ResNet’s convolution-layer count, inference multiplication count, floating-point accuracy, and post-quantization accuracy.The toolflow replaces standard QuantizedConv2D layers with AxConv2D layers.Source: §IV · p. 5, Table I · p. 5
- The evaluated approximate-multiplier library contains all 36 eight-bit multipliers from EvoApproxLib, with energy computed using Synopsys DC at 45 nm under uniform input distribution.Source: §IV · p. 5
- Experiments used pipelined and power-gated architectures, with tile counts of 2–4 for AxResNet-8 and AxResNet-14 and 3–6 for AxResNet-50.Search evaluated 1,000 of 10,000 validation images for 30 iterations, then ran inference on the full validation set.Source: §IV · p. 6
V. RESULTS
Weight mapping reduces multiplier mean error distance and improves classification accuracy for many evaluated AxResNet-8 configurations. The reported figures illustrate mapped multiplier behavior and compare the proposed algorithm’s AxResNet-8 designs with regular structures.
A. Impact of weight tuning
- 39 mul8u_7C1 weights changed by ±1, reducing MED from 87.3 to 69.7 (-20%); mul8u_L40 MED fell from 1011.3 to 647.7 (-36%).Computing the optimal mapping took around 0.10 seconds on average; Fig. 9 shows multiplier outputs and differences before and after mapping.Source: A. Impact of weight tuning · p. 6, Figure 9 · p. 6
- 8.2% average classification-accuracy improvement occurred in 370 of 510 mapped AxResNet-8 configurations; the remaining cases dropped by 1.4% on average.Across all configurations, the average accuracy change was +5.0%.Source: A. Impact of weight tuning · p. 6
- Fig. 10 compares AxResNet-8 networks constructed by the proposed algorithm with networks having a regular structure.Source: Figure 10 · p. 6
B. Quality of the generated AxNNs
ALWANN produces better accuracy–energy trade-offs than manually constructed and uniform AxNNs, while hardware constraints shape the available designs. The search converges early, and selected approximate multipliers exhibit low post-tuning error.
B. Quality of the generated AxNNs
- ALWANN provides better accuracy–energy trade-offs than manually created AxNNs and uniform NN architectures reported in prior work.Uniform approximation performs comparably to AxNNs with all but one layer approximated, while using one approximate layer yields significantly worse results because energy savings are small.Source: B. Quality of the generated AxNNs · p. 6
- Power-gated architectures permit more energy-efficient AxNNs than pipelined architectures; with three tiles, they offer 103× more options for AxResNet-14 and 1011× more for AxResNet-51.More tiles create more multiplier-selection options; the search converges toward better solutions, with its most important changes in the first iterations.Source: B. Quality of the generated AxNNs · p. 6, Figure 11 · p. 6, Figure 12 · p. 7
- For networks with classification accuracy above 80%, four multipliers appear in 78% of AxResNet-50 layers, 71% of AxResNet-14 layers, and 64% of AxResNet-8 layers.The four most frequent multipliers differ for AxResNet-8 and for the two larger networks.Source: B. Quality of the generated AxNNs · p. 6
- After tuning, multiplier MED ranges from 0.015–0.9%, while mean relative error ranges from 0.5–4.5%.Worst-case error distance is 0.05–1.2% (0.5% average) for the smallest network and 0.06–0.6% (0.29% average) for the larger network.Source: B. Quality of the generated AxNNs · p. 7
- CPU simulation of approximate multipliers is an algorithm bottleneck, and almost all execution time is spent on inference.The single-core application runs 10× slower than vectorized accurate multiplication; the authors estimate that an accelerator exceeding 1400× speedup could reduce ResNet-50 approximation to 10 minutes.Source: B. Quality of the generated AxNNs · p. 7
C. Overall results
For ResNet-50, the reported energy saving exceeds 30% when a 1% accuracy drop is acceptable. ALWANN also produces many design points beyond those obtained by reducing network depth, while energy–accuracy trade-offs depend on architecture.
C. Overall results
- More than 30% energy can be saved for ResNet-50 when accuracy drops from 89.15% to 88.1%.The authors report that AxResNet-50 is not advantageous beyond 4% accuracy degradation (2% for AxResNet-14), because AxResNet-14 or AxResNet-8 can provide the same quality at lower energy.Source: C. Overall results · p. 7, Table III · p. 7
- ALWANN produces more than 40 design points from a single ResNet, most unreachable by simply reducing the number of layers.Figure 13 covers seven trained ResNet architectures and AxNNs with accuracy above 65%.Source: C. Overall results · p. 7, Figure 13 · p. 7
- Each separately trained ResNet instance can require days or weeks of cluster time for complex structures.The paper notes that these instances are obtained by reducing layer counts at block boundaries.Source: C. Overall results · pp. 7–8
D. Comparison with SoA
On CIFAR-10, ALWANN offers high-granularity NN design points without retraining, including low-energy designs with accuracy below 80%. Its fast weight-mapping algorithm adapts networks to approximation errors without processing input data.
D. Comparison with SoA
- ALWANN finds high-granularity design points without retraining, unlike multiplier-less multiplication work that proposed only four architectures.The comparison covers energy reduction, accuracy degradation, retraining, architectural uniformity, and network complexity on CIFAR-10.Source: D. Comparison with SoA · p. 8
- ALWANN can produce low-energy AxNNs with accuracy below 80%, which may serve as initial stages of a Progressive Chain Classifier.The authors estimate that evaluating one ResNet-50 AxNN by retraining would take 89 days rather than the 322 seconds needed by ALWANN; the search evaluated 1,500 candidates.Source: D. Comparison with SoA · p. 8
VI. CONCLUSION
- The fast weight-mapping algorithm adapts the network to approximation errors without processing input data.The authors report better energy savings at the same accuracy than approaches that retrain, which typically approximate smaller networks or consider a limited set of components.Source: §VI · p. 8