Source-linked AI summary
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Remi Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, Rob Fergus
TL;DR
Large CNNs are expensive to evaluate at test time, limiting deployment on mobile platforms and at Internet scale. The paper compresses convolutional filters using linear and low-rank structure, then fine-tunes the network. It reports 2−3× convolutional speedups and substantial parameter reductions while keeping accuracy within 1% of the original model.
Problem
Large CNNs require substantial computation and resources at test time, making deployment on mobile platforms and Internet-scale systems problematic.
Method
The paper compresses convolutional layers with low-rank tensor decompositions and filter clustering, then fine-tunes upper layers to restore prediction performance.
Results
2−3× convolutional-layer speedups and 5−10× fully connected-layer parameter reductions are reported on state-of-the-art ImageNet CNNs, with accuracy kept within 1% of the original model.
Takeaways & Limitations
The techniques reduce computation and memory for large CNN evaluation and are presented as potentially compatible with quantization and Fourier-domain methods.
Takeaways & Limitations
Practical speedups can fall short of theoretical arithmetic-operation gains, and effective implementations depend on computer and CNN architectures.
Abstract
from arXiv · showhide
We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks. These models deliver impressive accuracy but each image evaluation requires millions of floating point operations, making their deployment on smartphones and Internet-scale clusters problematic. The computation is dominated by the convolution operations in the lower layers of the model. We exploit the linear structure present within the convolutional filters to derive approximations that significantly reduce the required computation. Using large state-of-the-art models, we demonstrate we demonstrate speedups of convolutional layers on both CPU and GPU by a factor of 2x, while keeping the accuracy within 1% of the original model.
1 Introduction
Large CNNs achieve strong vision performance but are costly to deploy because of their size and computation. The paper exploits convolutional redundancy through compression and fine-tuning to accelerate evaluation while preserving performance.
- Motivation: Test-time deployment is difficult because large neural networks strain mobile CPU, memory, and battery resources and require many servers at Internet scale.Training speed is also discussed, but the paper focuses on improving test-time performance.
- Motivation: CNNs can contain approximately 10^8 parameters, and their overparameterization wastes computing resources despite supporting optimization.The paper targets this redundancy as a source of computational savings.
- Approach: The paper compresses each convolutional layer with low-rank approximations, fine-tunes upper layers, and also considers SVD-based tensor decompositions and filter clustering.These methods exploit similarities and linear structure in learned features.
- Results: 2−3× convolutional-layer speedups and 5−10× fully connected-layer parameter reductions are reported on state-of-the-art ImageNet CNNs.These are the paper’s headline contribution figures.
- Notation: Convolution weights are represented as a 4-dimensional tensor W ∈ R^C×X×Y×F, while the input signal is represented across channels and spatial dimensions.The notation defines the objects being approximated in a convolutional layer.
2 Related Work
Prior work accelerates CNN computation through CPU-specific optimization, Fourier-domain convolution, and low-rank approximations that exploit parameter redundancy. Related studies also apply low-rank convolutional decompositions across architectures and recognition tasks.
- CPU optimization: CPU-focused methods improve matrix multiplication through memory alignment and SIMD operations, while also using 8-bit linear quantization of weights and inputs.Some techniques target Intel and AMD processors, whereas others generalize to broader processor types.
- Fourier methods: Fourier-domain convolution converts convolution into element-wise multiplication, eliminating dependence on receptive-field size and yielding around 2× forward-pass speedup.The cited work targets the expensive convolutions in the first few CNN layers, where receptive fields are relatively large.
- Low-rank approximations: Low-rank approximation methods are motivated by evidence that roughly 5% of a layer’s weights can accurately predict the rest, indicating substantial neural-network over-parameterization.The approaches exploit linear structure within this over-parameterization.
- Related low-rank work: A contemporaneous study applies low-rank convolutional tensor decompositions to scene-text character recognition, supporting applicability across architectures and tasks.The present work differs by considering a significantly larger model, making efficient approximation more challenging.
3 Convolutional Tensor Compression
The paper compresses convolutional tensors by using task-aware distance metrics, low-rank decompositions, and structure-specific approximations for first and higher convolutional layers. These methods replace costly representations with efficient approximations while seeking to preserve prediction performance.
- Approximation criteria: The compression goal is to find an efficient approximation of each convolutional tensor while maintaining network prediction performance.The paper begins with Frobenius-norm approximation, then improves the criterion to reflect prediction sensitivity.
- Approximation criteria: The Mahalanobis metric emphasizes weight coordinates more likely to produce prediction errors, using back-propagated differences involving the most dangerous mistakes.The full covariance version is prohibitively expensive, so the paper uses a diagonal approximation implemented through element-wise reweighting.
- Low-rank decompositions: SVD compresses matrices and tensors by retaining dominant singular directions, reducing computation when singular values decay rapidly.For matrix multiplication, the approximate computation costs O(nmt + nt^2 + ntk) instead of O(nmk).
- Low-rank decompositions: Outer product decomposition represents a tensor as a sum of rank 1 tensors, finding factors by alternating least squares and extending the approximation greedily to rank K.Each iteration subtracts one fitted outer product from the residual tensor.
- Layer-specific approximations: Higher convolutional layers use biclustering to partition input and output features, then approximate every resulting sub-tensor with SVD or outer-product decompositions.Spatial dimensions are concatenated before decomposition; further spatial separability was found to provide minimal additional gain.
- Layer-specific approximations: The first convolutional layer uses monochromatic approximation: each filter’s color dimension is projected onto a one-dimensional subspace, with color bases optionally shared through clustering.The method applies rank-1 SVD per output feature and clusters the resulting color components into shared bases.
4 Experiments
Experiments evaluate convolutional approximations on CPU and GPU, measuring accuracy, speedups, and parameter reduction. Compressing the first two layers achieved roughly 2–2.5× empirical speedups while keeping accuracy within 1% in the reported configurations.
- Experimental setup: The experiments evaluate prediction accuracy, speedup gains, and memory overhead for the proposed approximations.Fine-tuning results used less than two passes through the ImageNet12 training dataset unless otherwise stated.
- Experimental setup: The first two convolutional layers receive most forward-propagation time, motivating the speedup experiments’ focus on those layers.The authors state that the approximations could also be applied to upper layers.
- First layer: 2–3× fewer floating-point operations and 2–2.5× empirical CPU and GPU speedups were achieved for the first layer, with less than a 1% classification-performance drop.The larger arithmetic reduction occurs for smaller C′, the number of intermediate color channels.
- Second layer: 2–2.5× empirical CPU and GPU speedups were achieved for the second layer, with less than a 1% performance drop.CPU results used biclustering with SVD, while GPU results used biclustering with outer-product decomposition.
- Combining approximations: Sequentially compressing the first two layers with fine-tuning kept accuracy within 1% of the original model using six colors in layer one and G = 48, H = 2, K = 8 in layer two.The procedure used a single pass through the training set and alternated compression with fine-tuning of upper layers.
- Memory overhead: The parameter-reduction analysis reports empirical reductions and corresponding network performance for approximation methods across their hyperparameter settings.Fully connected layers were included because they contain most of the model’s parameters, and were compressed with standard SVD.
5 Discussion
The techniques accelerate bottleneck convolutions and compress weights with little performance loss, while also motivating future investigation of regularization effects.
- 2–3× speedups are achieved for bottleneck convolution operations with negligible performance loss.The methods also reduce weight memory in the first two layers by 2–3×.
- 5–13× compression of fully connected layers can substantially reduce memory because most weights reside there.The paper connects this reduction to facilitating mobile deployment.
- The techniques are orthogonal to quantization and Fourier-domain evaluation, so they can potentially be combined for further gains.
- Future work could investigate whether low-rank projections aid regularization during or after training.The motivation includes fewer learnable parameters, cleaned-up first-layer filters, and occasional better test error with conservative approximations.
Supplement to “Exploiting Linear Structure Within Convolutional Networks
The supplement is identified as part of the NIPS 2014 material for the paper.
- The material is identified as a supplement.
- The associated paper title concerns exploiting linear structure within convolutional networks.
- The material is labeled NIPS2014.
A Forward propagation time breakdown
Forward-propagation time is measured per layer on CPU and GPU using batch size 128, with results averaged over eight runs; convolutional layers dominate runtime, especially the first two.
- Close to 90% of forward-propagation time is spent on convolutional layers.Within convolutional computation, the first two layers consume most of the time.
- Evaluation time is reported in seconds per layer for CPU and GPU runs with batch size 128, averaged over eight runs.
B Theoretical speedups
The paper uses floating-point operation counts as an upper-bound estimate for approximation speedups and examines the trade-off between theoretical speedup and classification-performance loss.
- Floating-point operation counts provide an informative upper bound on achievable speedups, although implementations are unlikely to reach it.
- The monochromatic approximation’s theoretical speedup decreases only slightly as the number of color components increases.Color transformation requires negligible operations compared with convolution.
- For biclustering with outer product decomposition, classification performance tends to degrade as approximation rank decreases.The figure compares theoretical speedup against the drop in classification performance across cluster configurations.
C Combined results
The combined results examine cascading and monochromatic approximations, relating classification error to theoretical speedup and fine-tuned outcomes. They show how approximation choices affect this accuracy–efficiency trade-off.
- C Combined results: Table 5 reports results after fine-tuning the network for one pass through ImageNet12 using several second-layer approximations.The first layer used a monochromatic approximation with 6 colors.
- C Combined results: Fewer colors make the monochromatic first-layer approximation harsher and tend to increase ImageNet12 classification error.The corresponding theoretical speedup varies only slightly as the number of colors changes.
- C Combined results: Figure 5 compares theoretically achievable speedups with classification error across biclustering approximations.
- C Combined results: Table 5 presents cascading approximations.