Source-linked AI summary
Coordinating Filters for Faster Deep Neural Networks
Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li
TL;DR
Large DNNs are challenging to deploy on resource-limited systems because of their computation demands. Force Regularization coordinates filters during training so LRA can use lower ranks; the paper reports reduced ranks and faster AlexNet inference with small accuracy loss.
Problem
High computation demands make large DNNs challenging to deploy on resource-limited systems.
Source: §1 · p. 1Method
Force Regularization adds attractive-force gradients during training to coordinate filters toward a more correlated state for lower-rank approximation.
Source: §1 · p. 1, §4.1 · p. 3Results
Force Regularization reduces required reconstruction rank from 11 to 5 at ≤20% error and yields 4.05× AlexNet speedup with small accuracy loss.
Source: §1 · p. 2Takeaways & Limitations
The resulting low-rank DNNs can initialize LRA retraining for faster convergence to better accuracy and can be further sparsified.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Very large-scale Deep Neural Networks (DNNs) have achieved remarkable successes in a large variety of computer vision tasks. However, the high computation intensity of DNNs makes it challenging to deploy these models on resource-limited systems. Some studies used low-rank approaches that approximate the filters by low-rank basis to accelerate the testing. Those works directly decomposed the pre-trained DNNs by Low-Rank Approximations (LRA). How to train DNNs toward lower-rank space for more efficient DNNs, however, remains as an open area. To solve the issue, in this work, we propose Force Regularization, which uses attractive forces to enforce filters so as to coordinate more weight information into lower-rank space. We mathematically and empirically verify that after applying our technique, standard LRA methods can reconstruct filters using much lower basis and thus result in faster DNNs. The effectiveness of our approach is comprehensively evaluated in ResNets, AlexNet, and GoogLeNet. In AlexNet, for example, Force Regularization gains 2x speedup on modern GPU without accuracy loss and 4.05x speedup on CPU by paying small accuracy degradation. Moreover, Force Regularization better initializes the low-rank DNNs such that the fine-tuning can converge faster toward higher accuracy. The obtained lower-rank DNNs can be further sparsified, proving that Force Regularization can be integrated with state-of-the-art sparsity-based acceleration methods. Source code is available in https://github.com/wenwei202/caffe
1. Introduction
The paper addresses the deployment cost of large DNNs by shifting low-rank approximation toward training: Force Regularization coordinates filters into more correlated, lower-rank representations.
1. Introduction
- Large DNNs are difficult to deploy on resource-limited platforms because they demand substantial computation and energy.The paper names mobile devices and autonomous cars as examples of constrained platforms.Source: §1 · p. 1
- Force Regularization adds attractive-force gradients during training to coordinate filters, unlike prior LRA methods that decompose pretrained weights.The goal is to improve filter correlation so LRA can produce more lightweight DNNs.Source: §1 · pp. 1–2
- 5 vs. 11 required ranks at ≤20% reconstruction error; AlexNet also reaches 4.05× speedup with small accuracy loss.The rank comparison uses the same cross-filter PCA method.Source: Figure 1 · p. 1, §1 · p. 2
- The resulting low-rank DNNs provide better initialization for LRA retraining, which converges faster toward better accuracy, and can be further sparsified.The paper presents Force Regularization as compatible with state-of-the-art sparsity methods.Source: §1 · p. 2
- Force Regularization can be generalized to Discrimination Regularization to learn more discriminative filters and improve classification accuracy.The paper also reports an open-source implementation for CPUs and GPUs.Source: §1 · p. 2
2. Related work
Prior work accelerates DNNs through low-rank decomposition or sparsity, while this paper positions filter coordination as complementary to those approaches.
2. Related work
- Earlier low-rank methods used filter or data reconstruction, clustering, and tensor decompositions; some reported 2× speedup in the first two layers with 1% accuracy loss.The related-work discussion also notes evaluations of newer decompositions and mobile-device implementations.Source: §2 · p. 2
- Non-structured sparsity prunes individual connections, while structured methods prune groups into regular patterns that are more hardware-friendly.The passage notes that non-structured sparsity often does not yield sufficient practical speedup on modern hardware.Source: §2 · p. 2
- Force Regularization is presented as orthogonal to sparsity: its lower-rank networks can also be further sparsified by structured or non-structured methods.The paper describes the additional computation gain as potential rather than guaranteed.Source: §2 · p. 2
3. Correlated Filters and Their Approximation
The section links correlations among trained filters to low-rank approximation, then describes cross-filter factorization as a way to replace a convolution with lighter layers.
3. Correlated Filters and Their Approximation
- Filters in AlexNet and GoogLeNet are normalized, grouped into four k-means clusters, and projected by LDA; the figure indicates high within-cluster correlation.The first convolutional-layer filters are shown for both networks.Source: §3 · p. 2, Figure 2 · p. 2
- Coordinating filters within and across clusters during training is proposed to enable more accurate approximation at lower rank.The paper proposes Force Regularization to achieve this coordination.Source: §3 · p. 3
- Cross-filter LRA approximates each convolutional filter as a linear combination of M shared basis filters, with M much smaller than the number of filters N.The basis-filter convolution is followed by a 1×1 convolution that combines the resulting feature maps.Source: §3 · p. 3
- This factorization reduces computation complexity, and the section selects cross-filter approximation and the method in as baselines.The chosen cross-filter methods are cited as and [30].Source: §3 · p. 3
4. Force Regularization
Force Regularization adds attractive-force gradients to data-loss gradients to coordinate filters toward lower-rank structure. Its mathematical analysis connects the update to a normalized-filter regularizer and examines how step-size scaling affects the resulting ranks.
4.1. Regularization by Attractive Forces
- Force Regularization adds attractive-force gradients to data-loss gradients, adjusting their lengths and directions to coordinate filters toward a more correlated state.Its regularization gradient is perpendicular to the filter vector and can be computed with additions and multiplications; λs trades rank against accuracy and is selected by cross-validation.Source: §4.1 · p. 3
- Attraction moves filter clusters toward typical directions, while data-loss gradients resist their collapse and preserve discriminative feature extraction.The paper models normalized filter vectors as spinning sticks with endpoint particles under pairwise attraction.Source: §4.1 · p. 4
- The method optimizes normalized filter orientations and studies ℓ2- and ℓ1-norm forces, whose strengths decrease with distance or remain constant, respectively.Scaling the gradient by filter length preserves angular velocity when updating the filter.Source: §4.1 · p. 4
4.2. Mathematical Implications
- Table 1 compares ranks across regularization-gradient step-size scalers; using ||Wi|| works better than its reciprocal for coordinating filters toward lower rank.The comparison uses PCA ranks at ≤5% reconstruction error for a ConvNet trained on CIFAR-10 without data augmentation.Source: Table 1 · p. 4, §4.2 · p. 5
- The ℓ2-force gradient shares the direction of the related regularizer’s gradient but uses filter-length scaling rather than reciprocal-length scaling.The paper relates Force Regularization to, but distinguishes it from, minimizing the sum of pairwise distances between normalized filters.Source: §4.2 · pp. 4–5
5. Experiments
Experiments evaluate Force Regularization across CIFAR-10 and ImageNet models, measuring rank reduction, accuracy, speed, fine-tuning, and compatibility with sparsification. The results show lower ranks and faster testing, including 2.50× total convolution speedup on GPU and 4.05× on CPU with small accuracy loss.
5.2. Rank Analysis of Coordinated DNNs
- Force Regularization reduced AlexNet's average layer-wise rank ratio by about 50% without accuracy loss, while retaining low ranks in early layers and reducing ranks in deeper layers.Table 2 summarizes ConvNet and AlexNet ranks; the paper defines rank M using a ≤5% PCA reconstruction-error threshold.Source: §5.2 · p. 5, Table 2 · p. 5
- Across ResNets-20 and GoogLeNet, five and six layers, respectively, reached rank M = 1 with acceptable accuracy loss.Figure 5 reports accuracy losses of 0.75% for ResNets-20 and 2.46% top-5 for GoogLeNet.Source: Figure 5 · p. 5, §5.2 · p. 6
- On AlexNet, ℓ2-norm and ℓ1-norm forces reduced average rank ratios to 28.59% and 28.72%, with accuracy losses of 1.71% and 1.80%, respectively.ℓ2-norm force better balances ranks across layers; ℓ1-norm force can reduce conv4 group ranks to one.Source: §5.2 · p. 6, Figure 6 · p. 6
- AlexNet decomposed to the same ranks converged faster to lower error after Force Regularization than when trained from scratch.Figure 7 plots training data loss and top-1 validation error; training from scratch had significant accuracy loss.Source: §5.2 · p. 6, Figure 7 · p. 6
5.3. Acceleration of DNN Testing
- The evaluation trained models with Force Regularization, applied LRA, and fine-tuned them; a shared error percentage set the rank threshold across layers.The shared threshold reduces the number of rank-selection hyperparameters, though layer-wise selection may offer a better trade-off.Source: §5.3 · pp. 6–7
- 2.50× total convolution speedup on GPU and 4.05× on CPU were achieved with small accuracy loss; CPU testing was faster than SSL at almost the same top-5 error.The SSL comparison reports 4.05× versus 3.13× speedup and 21.68% versus 21.63% top-5 error.Source: §5.3 · p. 7, Table 5 · p. 7, Table 6 · p. 8
5.4. Lower-rank and Sparse DNNs
- Force-regularized lightweight models could be further compressed and accelerated with structured or non-structured sparsity methods.Against deep-compression, the model had comparable compression rates and 2.69× faster testing; its conv3–5 shape-wise sparsity was slightly lower after aggressive LRA.Source: §5.4 · p. 8, Figure 8 · p. 8
5.5. Generalization of Force Regularization
- Using the opposite gradient of Force Regularization (λs < 0) as a repulsive force can enhance filter discrimination and improve accuracy.The paper relates filters to discriminative features, from orientation-selective patterns to higher-level features.Source: §5.5 · p. 8