Source-linked AI summary
Efficient and Accurate Approximations of Nonlinear Convolutional Networks
Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, Jian Sun
TL;DR
The paper addresses the cost of test-time CNN computation and the limitations of approximations that ignore nonlinear units and accumulated multilayer error. It minimizes nonlinear-response reconstruction error under a low-rank constraint, solves the resulting optimization, and introduces asymmetric reconstruction for multilayer acceleration. On an ImageNet model, it demonstrates 4× whole-model speedup with only a 0.9% increase in top-5 error.
Problem
Accurate deep CNNs have high test-time cost, while prior approximations reconstruct linear filters or responses without modeling nonlinear units or accumulated multilayer errors.
Method
The method minimizes nonlinear-response reconstruction error under a low-rank constraint, solves it with alternating optimization, and uses asymmetric reconstruction across sequentially approximated layers.
Results
4× whole-model speedup increases top-5 error by only 0.9% on a seven-convolutional-layer ImageNet model.
Takeaways & Limitations
The approach accelerates an entire deep CNN while preserving accuracy better than the cited prior method at the same speedup ratios.
Takeaways & Limitations
The method focuses on ReLU and is evaluated on one seven-convolutional-layer ImageNet model, limiting the demonstrated scope.
Abstract
from arXiv · showhide
This paper aims to accelerate the test-time computation of deep convolutional neural networks (CNNs). Unlike existing methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We minimize the reconstruction error of the nonlinear responses, subject to a low-rank constraint which helps to reduce the complexity of filters. We develop an effective solution to this constrained nonlinear optimization problem. An algorithm is also presented for reducing the accumulated error when multiple layers are approximated. A whole-model speedup ratio of 4x is demonstrated on a large network trained for ImageNet, while the top-5 error rate is only increased by 0.9%. Our accelerated model has a comparably fast speed as the "AlexNet", but is 4.7% more accurate.
1. Introduction
The paper targets the rising test-time cost of accurate deep CNNs and addresses limitations in prior low-rank approximations that ignore nonlinear responses and whole-network error accumulation.
- Motivation: Deep CNNs become increasingly expensive to evaluate, limiting practical use in high-throughput services, resource-constrained devices, and time-consuming recognition tasks.The motivation includes cloud services processing thousands of requests per second and detection on high-end GPUs.
- Prior Work: Prior acceleration methods approximate low-rank filters or linear responses, reducing complexity but sometimes degrading accuracy.These methods demonstrated speedups on single or few layers rather than complete deep networks.
- Research Gap: Ignoring ReLU makes approximation quality sensitive to errors in positive responses, because negative responses are subsequently truncated.The paper therefore treats nonlinear responses directly rather than optimizing only linear filters or responses.
- Research Gap: Approximating multiple layers is challenging because errors accumulate, while prior studies mainly accelerated one layer or a few layers.The paper identifies whole-network acceleration as necessary for practical use of deeper CNNs.
- Contribution: The proposed method minimizes nonlinear-response reconstruction error under a low-rank constraint and uses an asymmetric error to reduce accumulation across layers.The constrained optimization is decomposed into feasible subproblems solved iteratively.
- Results: 4× whole-model speedup increases top-5 error by only 0.9%, while AlexNet-comparable speed yields accuracy 4.7% higher.The evaluation uses a seven-convolutional-layer ImageNet model and reports better accuracy than Jaderberg et al. at matched speedup ratios.
2. Approaches
The approach compresses convolutional responses with low-rank structure, extends the objective to nonlinear units, and sequentially manages accumulated error while selecting ranks under a whole-model complexity budget.
- 2.1. Low-rank Approximation of Responses: Convolutional responses are approximated in a low-rank subspace by minimizing reconstruction error, reducing the number of filters needed.The method treats low-rank structure in responses as the target of optimization.
- 2.1. Low-rank Approximation of Responses: A convolutional layer is represented using d′ filters followed by a 1×1 convolution that restores the d-dimensional output.The original complexity O(dk2c) becomes O(d′k2c) + O(dd′), approximately reducing computation to d′/d when O(dd′) is small.
- 2.1. Low-rank Approximation of Responses: PCA estimates the response subspace from centered training responses by retaining the first d′ eigenvectors.With covariance decomposition YY⊤ = USU⊤, the projection is M = Ud′Ud′⊤ and P = Q = Ud′.
- 2.1. Low-rank Approximation of Responses: The response spectra support the low-rank assumption: 128 eigenvectors retain over 99.9% energy in Conv2, while 256 retain over 95% in Conv7.These results come from responses sampled from 3,000 ImageNet training images.
- 2.2. The Nonlinear Case: The nonlinear formulation minimizes reconstruction error after applying ReLU, with a low-rank matrix and optimized bias.The nonlinear response is represented as r(Wx), while the approximation uses r(MWx + b).
- 2.2. The Nonlinear Case: The constrained nonlinear problem is relaxed with auxiliary variables and solved by alternating optimization over the variables and the low-rank transformation.The transformation subproblem is reduced to reduced-rank regression and solved using generalized singular value decomposition.
- 2.3. Asymmetric Reconstruction for Multi-Layer: Whole-network approximation proceeds from shallow to deep layers, using asymmetric reconstruction to reduce errors accumulated from earlier approximations.The objective compares original-network responses with approximated-layer responses under approximate inputs.
- 2.4. Rank Selection for Whole-Model Acceleration: Rank selection chooses each layer’s d′ to maximize accumulated accuracy under a total complexity constraint for the desired model speedup.The rank d′ is the parameter determining an accelerated layer’s complexity.
3. Experiments
Experiments on a seven-convolutional-layer ImageNet model evaluate single-layer, multi-layer, whole-model, and prior-work comparisons. The nonlinear and asymmetric variants reduce approximation error, achieving substantial speedups with limited degradation.
- Experimental Setup: The experiments use SPPnet (Overfeat-7), a seven-convolutional-layer model trained on the 1000-class ImageNet 2012 dataset.Approximations are trained using responses from 3,000 randomly sampled ImageNet training images.
- Experimental Setup: Single-layer speedup ratios are reported using theoretical complexity, while error is top-5 single-view increase relative to the unapproximated model.The reported theoretical speedups are nearly the actual CPU/GPU speedups.
- 3.1. Single-Layer: Linear vs. Nonlinear: The nonlinear solution consistently outperforms the linear solution, especially in layers with substantial ReLU sparsity, including Conv7 at 95% sparsity.ReLU sparsity exceeds 60% for Conv2-7, helping explain the difference between linear and nonlinear approximations.
- 3.1. Single-Layer: Linear vs. Nonlinear: 2× single-layer acceleration increases error by < 0.1% for Conv2 and < 0.2% for Conv3-7.For Conv1, degradation is ignorable at or below 2× speedup, but grows quickly at larger ratios.
- 3.2. Multi-Layer: Symmetric vs. Asymmetric: Asymmetric reconstruction improves over symmetric reconstruction by over 1.0% at 4× speedup when three layers are approximated simultaneously.Without the asymmetric solution, error rates increase more drastically as more or all layers are approximated.
- 3.3. Whole-Model: with/without Rank Selection: Rank selection consistently outperforms using no rank selection in whole-model acceleration for both linear and nonlinear solutions.Rank selection often assigns higher ranks to Conv5-7 because their energy is less concentrated; at 3× speedup, Conv7 receives d′ = 167.
- 3.4. Comparisons with Previous Work: At 4× whole-model speedup, the asymmetric method increases error by 4.2%, compared with 6.0% for Jaderberg et al.’s method.The asymmetric method degrades more slowly as the speedup ratio increases; both methods achieve similar actual CPU speedups for a given theoretical ratio.
- 3.4. Comparisons with Previous Work: A whole-model 4× speedup produces only a 0.9% increase in 10-view top-5 error, while matching AlexNet’s speed yields 4.7% higher accuracy.The accelerated model has 12.0% 10-view error versus 11.1% for the original model, and its 3d variant has 14.1% absolute error.
4. Conclusion and Future Work
The low-rank constraint reduces computation while also serving as a potential regularizer on convolutional filters. The authors identify this regularization perspective as future work, alongside reporting AlexNet’s reference error rates.
- The low-rank constraint is designed for speedup but can also be viewed as a regularizer on convolutional filters.The authors plan to investigate this regularization perspective in future work.
- AlexNet’s 10-view reference errors are 18.2% top-5 and 40.7% top-1.