Source-linked AI summary
AdderNet: Do We Really Need Multiplications in Deep Learning?
Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, Chang Xu
TL;DR
Deep neural networks rely heavily on costly floating-point multiplications, limiting computational efficiency and mobile deployment. The paper introduces AdderNets, which use ℓ1-distance and addition-based operations with specialized optimization, achieving competitive ImageNet accuracy without convolutional multiplications.
Problem
Multiplication-heavy convolutions impose high computation and power costs, while existing compressed networks still retain massive multiplications.
Method
AdderNets replace convolutional multiplication with ℓ1-distance between filters and features, using regularized full-precision gradients and adaptive learning-rate scaling.
Results
74.9% Top-1 accuracy and 91.7% Top-5 accuracy are achieved with ResNet-50 on ImageNet without multiplication in the convolution layer.
Takeaways & Limitations
AdderNets can approximate conventional CNN performance while using addition-based similarity measures to build deep networks with lower computational costs.
Takeaways & Limitations
Binary-network comparisons indicate that reducing computation can fail to preserve recognition accuracy and may produce unstable, slower training.
Abstract
from arXiv · showhide
Compared with cheap addition operation, multiplication operation is of much higher computation complexity. The widely-used convolutions in deep neural networks are exactly cross-correlation to measure the similarity between input feature and convolution filters, which involves massive multiplications between float values. In this paper, we present adder networks (AdderNets) to trade these massive multiplications in deep neural networks, especially convolutional neural networks (CNNs), for much cheaper additions to reduce computation costs. In AdderNets, we take the $\ell_1$-norm distance between filters and input feature as the output response. The influence of this new similarity measure on the optimization of neural network have been thoroughly analyzed. To achieve a better performance, we develop a special back-propagation approach for AdderNets by investigating the full-precision gradient. We then propose an adaptive learning rate strategy to enhance the training procedure of AdderNets according to the magnitude of each neuron's gradient. As a result, the proposed AdderNets can achieve 74.9% Top-1 accuracy 91.7% Top-5 accuracy using ResNet-50 on the ImageNet dataset without any multiplication in convolution layer. The codes are publicly available at: https://github.com/huaweinoah/AdderNet.
1. Introduction
The paper targets the high computational and power costs of multiplication-heavy CNNs, especially for mobile deployment. It proposes AdderNets, which replace convolutional multiplications with addition-based ℓ1-distance while aiming to preserve recognition accuracy.
- Motivation: 250W+ GPU power consumption illustrates why multiplication-heavy deep learning systems are difficult to deploy on mobile devices.The paper cites GeForce RTX 2080 Ti power consumption as an example and notes the additional hardware required beyond the GPU.
- Motivation: Multiplications dominate float-valued weight–activation computations, while additions have much lower computational complexity.The paper motivates replacing multiplication with addition to reduce computation costs.
- Related limitation: Binary networks reduce computation cost but may lose recognition accuracy and train unstably with slower convergence.The cited limitation concerns both accuracy preservation and training behavior.
- Proposed approach: AdderNets replace convolution operations with ℓ1-distance between input features and filters, using additions as the main operation.Subtraction can be implemented through addition using complement code, making the measure hardware-friendly.
- Proposed approach: The paper introduces an improved back-propagation scheme with regularized gradients and evaluates AdderNets on benchmark datasets and models.The stated goal is better convergence and comparable recognition accuracy to conventional CNNs.
2. Related works
Prior efficiency methods prune, decompose, redesign, or distill neural networks, but their compressed models still retain massive numbers of multiplications. The paper therefore investigates replacing multiplication with cheaper additions in convolutional networks.
- Compression and pruning: Pruning methods remove redundant weights or transform weight matrices and convolution filters to reduce calculations.Examples include singular-value decomposition, discarding subtle weights, and frequency-domain filtering.
- Efficient architectures: Efficient architectures replace conventional convolutions with bottlenecks, depthwise and pointwise filters, group convolutions, channel shuffle, or channel-interdependency blocks.These methods reduce computations or improve performance with limited additional cost.
- Knowledge distillation: Knowledge distillation transfers information from a heavy teacher network to a portable student by matching outputs or intermediate features.The related work describes both output distillation and hint-layer feature distillation.
- Remaining gap: Existing compressed networks still contain massive multiplications, motivating multiplication reduction specifically in widely used convolutional networks.The paper contrasts these operations with lower-complexity subtractions and additions.
3. Networks without Multiplication
AdderNets replace convolutional similarity with ℓ1 distance, enabling feature computation dominated by additions. Their optimization uses full-precision gradients, gradient clipping, batch normalization, and layer-adaptive learning rates to address the resulting training dynamics.
- Adder Networks: ℓ1 distance replaces cross-correlation by measuring summed absolute differences between filters and input features without multiplication.Subtraction can be implemented through addition using complement code, making the measure hardware-friendly.
- Adder Networks: Adder-layer outputs are always negative, so batch normalization rescales them before conventional activation functions are applied.This allows activation functions used in conventional CNNs to remain applicable in AdderNets.
- Optimization: The AdderNet filter gradient uses the full-precision difference X − F instead of a sign gradient restricted to +1, 0, or −1.The sign gradient generally does not follow steepest descent, with its direction worsening as dimensionality grows.
- Optimization: Full-precision input gradients are clipped to [−1, 1] because their larger magnitudes can amplify discrepancies and cause gradients to explode across preceding layers.The input-feature derivative uses the HardTanh function.
- Optimization: AdderNets introduce batch normalization after each adder layer to manage the larger output variance caused by addition-based computation.For normally distributed filters and inputs, the output variance includes their summed variances, and the small weight variance in CNNs makes this increase substantial.
- Adaptive Learning Rate Scaling: AdderNet filter gradients are smaller and vary across layers, motivating adaptive local learning rates that produce nearly equal update steps.The scaling uses each layer’s gradient magnitude, the number of filter elements, and a hyper-parameter η.
4. Experiment
Experiments across MNIST, CIFAR, and ImageNet evaluate AdderNets as multiplication-free alternatives to CNNs, including their accuracy, latency, feature representations, and optimization. AdderNets generally retain competitive recognition performance while using additions and benefit from full-precision gradients with adaptive learning rates.
- 4.1. Experiments on MNIST: 99.4% accuracy matches CNNs on MNIST while replacing approximately 435K multiplications with approximately 870K additions.The reported CPU latency is approximately 1.7M for AdderNet versus approximately 2.6M for CNN on VIA Nano 2000 series.
- 4.2. Experiments on CIFAR: 93.72% CIFAR-10 and 72.64% CIFAR-100 accuracy are nearly identical to CNNs without multiplication.For ResNet-20, AdderNets also achieve 91.84% on CIFAR-10 and 67.60% on CIFAR-100 without multiplications, compared with CNN results of 92.25% and 68.14%.
- 4.3. Experiments on ImageNet: 74.9% Top-1 and 91.7% Top-5 accuracy are achieved by AdderNets with ResNet-50 on ImageNet, close to CNN results of 76.2% and 92.9%.On ResNet-18, AdderNet achieves 66.8% Top-1 and 87.4% Top-5 accuracy versus CNN results of 69.8% and 89.1%, while CNN uses 1.8G multiplications.
- 4.4. Visualization Results: AdderNet and CNN filters share similar visual patterns, and both can extract useful information from input images despite using different distance metrics.The feature visualizations indicate similar discrimination ability for image classification.
- 4.4. Visualization Results: AdderNet weights are close to Laplace-distributed, whereas CNN weights are more Gaussian-distributed, matching their ℓ1- and ℓ2-based formulations.The paper relates the Laplace prior to ℓ1-norm and the Gaussian prior to ℓ2-norm, with cross-correlation corresponding to the latter under the stated analysis.
- 4.5. Ablation Study: 99.40% accuracy is achieved with adaptive learning-rate scaling and full-precision gradients, outperforming sign-gradient variants and matching CNN accuracy.The adaptive learning-rate method addresses the smaller filter-gradient norms observed in AdderNets.
5. Conclusions
The paper explores replacing multiplication-based convolutions with addition-based similarity measures in deep neural networks. AdderNets use the ℓ1-norm distance between features and filters to reduce computational costs.
- 5. Conclusions: AdderNets replace multiplication-heavy convolutions with addition-based similarity measures to build deep neural networks with lower computational costs.The paper frames classical convolutions as similarity measurements between features and filters and investigates replacing multiplications with additions.
- 5. Conclusions: AdderNets calculate the ℓ1-norm distance between features and filters as their similarity measure.
- 5. Conclusions: The paper investigates the feasibility of using additions instead of multiplications in deep neural networks.
A. Convergence of Sign and Full-precision Gradient
The analysis shows that sign-gradient updates generally fail to converge to the optimal ℓ1-distance solution under a fixed learning rate, whereas full-precision gradients can converge when α < 1. This motivates precise filter updates in AdderNets.
- Sign-gradient updates use only +1, 0, or -1 and generally cannot converge to the optimal value with a fixed learning rate.The convergence constraint becomes stricter as filter dimensionality grows, making convergence essentially unattainable for large neural-network filters.
- The sign gradient rarely follows the steepest-descent direction, with directional quality worsening as dimensionality increases.
- Full-precision gradients retain the difference between each input feature and filter value rather than reducing it to a sign.The proposed gradient is X(m + i, n + j, k) − F(i, j, k, t).
- For α < 1, full-precision gradient descent can converge to the optimal value under a fixed learning rate.
- Using full-precision gradients allows AdderNet filters to be updated precisely.
B. Relationship Between ℓ2-norm and Cross-correlation
The paper relates ℓ2-AdderNets to conventional CNN cross-correlation through their algebraic form. When an input-patch norm is constant, the two outputs differ by a linear transformation, and ImageNet results are similar.
- ℓ2-AdderNets compute squared distances between filters and input features and use an adaptive learning rate because gradient magnitudes can be small.
- When each input patch has the same norm, the ℓ2-AdderNet output can be viewed as a linear transformation of CNN cross-correlation output.
- ImageNet results using ResNet-18 compare ℓ2-AdderNet classification with CNN classification, and reported performance is similar.