Accelerating Very Deep Convolutional Networks for Classification and Detection

Motivation

• Very deep VGG models are substantially slower than earlier CNNs, creating deployment problems for high-throughput, portable, and high-resolution recognition systems.The motivation includes cloud services, phones and tablets, object detection, and semantic segmentation.Source: §1 · p. 1

Research gap

• Prior acceleration methods mainly decompose one or a few layers, with limited evidence for very deep models and complex ImageNet tasks.The passage specifically identifies very deep models as those with roughly 10 or more layers under approximation.Source: §1 · p. 1

Method

• The proposed method models nonlinear neurons, solves the resulting low-rank response reconstruction with GSVD without SGD, and uses asymmetric reconstruction across sequentially approximated layers.The asymmetric formulation accounts for errors from previous approximated layers, while rank selection adapts acceleration across layers.Source: §1 · pp. 1–2

Results

• 4× speedup increases VGG-16 ImageNet top-5 center-view error by only 0.3%.This result concerns whole-model acceleration of the publicly available VGG-16 model.Source: §1 · p. 2
• 4× convolutional speedup in Fast R-CNN produces a 0.8% mAP degradation on PASCAL VOC 2007, from 66.9% to 66.1%.The accelerated VGG-16 model is evaluated as a feature extractor for object detection.Source: §1 · p. 2

Contributions

• The extended study reports acceleration of very deep VGG models, evaluates transfer-learning-based object detection, and finds scratch-trained counterparts inferior to accelerated models with the same structure.The authors interpret the latter finding as evidence that the acceleration optimization algorithm digests information from the original model.Source: §1 · p. 2

General framework

• CNN acceleration methods generally combine a decomposition that reduces time complexity with an optimization scheme for fitting that decomposition.The optimization component matters because different decompositions may have different local optima.Source: §2 · p. 2

Prior decompositions

• Denton et al. use low-rank filter decompositions but do not explicitly minimize post-nonlinearity activation error, and report only a single OverFeat layer.No whole-model results are available in the cited work.Source: §2 · p. 2
• Jaderberg et al. separate k × k filters into k × 1 and 1 × k filters, while optimizing either filter weights or responses.Their response reconstruction uses stochastic optimization methods, whereas filter reconstruction uses conjugate gradient descent.Source: §2 · p. 2

Optimization challenges

• Lebedev et al. apply CP-decomposition to one AlexNet layer on ImageNet and report difficulty finding a suitable SGD learning rate for fine-tuning.Their report indicates optimization challenges even for a single layer in ImageNet models.Source: §2 · p. 2

Open problem

• Whole-model acceleration of very deep networks for ImageNet remained an open problem, and their use as generic feature extractors for other tasks was unclear.Other related directions include FFT-based computation, thin deep networks, and memory conservation.Source: §2 · p. 2

3.1 Low-rank Approximation of Responses

• The method uses SVD for linear neurons and GSVD for nonlinear neurons, enabling an asymmetric reconstruction that reduces accumulated error in very deep models.The nonlinear solver incorporates the response nonlinearity without SGD, while asymmetric reconstruction accounts for errors from previously approximated layers.Source: §3 · p. 2, §3.2 · p. 4, §3.3 · p. 5
• Low-rank response structure permits replacing a layer with two lower-complexity layers whose intermediate rank is d′.The response vector is projected onto a rank-d′ subspace, then reconstructed through a d′-filter layer and a 1×1 convolution.Source: §3.1 · p. 3
• O(d′k2c) + O(dd′) replaces O(dk2c), reducing computation to about d′/d when O(dd′) is much smaller than O(d′k2c).The original layer has d filters, while the approximation uses d′ intermediate filters and a d-by-d′ projection.Source: §3.1 · p. 3, Figure 1 · p. 3
• PCA measurements support the low-rank assumption: Conv2’s first 128 eigenvectors retain over 99.9% energy, while Conv7’s first 256 retain over 95%.The response energy is measured from sampled ImageNet training responses, and the concentration indicates that fractions of the filters can approximate the original filters.Source: §3.1 · p. 3, Figure 2 · p. 4

3.4 Rank Selection for Whole-Model Acceleration

• Uniform layer speedups are not optimal because layers are not equally redundant, so whole-model acceleration requires selecting d′ separately for each layer.The method relates PCA energy to classification accuracy and uses rank allocation to avoid overly aggressive compression in less redundant layers.Source: §3.4 · p. 5
• The method can be combined with spatial decomposition because operating on multiple dimensions may otherwise require a smaller d′ for a target speedup, limiting accuracy.The combined strategy is intended to avoid ranks becoming too small.Source: §3.5 · p. 6

• SPP-10 contains 7 convolutional and 3 fully connected layers, while VGG-16 contains 13 convolutional and 3 fully connected layers.Source: §4 · p. 7
• Single-view top-5 error references are 12.51% for SPP-10 and 10.09% for VGG-16 on ImageNet validation/testing.These values are used as references for increased error rates in approximated models.Source: §4 · p. 7

Single-Layer: Linear vs. Nonlinear

• Nonlinear reconstruction consistently performs better than linear reconstruction for single-layer approximation.The comparison plots increased error rates against each layer's theoretical speedup ratio.Source: Single-Layer: Linear vs. Nonlinear · p. 7
• ReLU sparsity exceeds 60% in Conv2-7, reaching 95% in Conv7, where the nonlinear solution's advantage is more obvious.The substantial truncation caused by ReLU helps explain the discrepancy between linear and nonlinear solutions.Source: Single-Layer: Linear vs. Nonlinear · p. 8
• At 2× single-layer acceleration, Conv2 increases error by < 0.1%, while Conv3-7 increase error by ≈0.2%.Source: Single-Layer: Linear vs. Nonlinear · p. 8
• Conv1 degradation is negligible near 2× speedup; the experiments therefore use d′ = 32 for Conv1 in whole-model SPP-10 acceleration.A 1.8× speedup corresponds to d′ = 32, while larger speedups require more drastic reduction because Conv1 has only 3 input channels.Source: Single-Layer: Linear vs. Nonlinear · p. 8

Multi-Layer: Symmetric vs. Asymmetric

• The experiments approximate Conv6-7, Conv2-4, and Conv5-7, sequentially processing the involved layers from shallower to deeper ones.Source: Multi-Layer: Symmetric vs. Asymmetric · p. 8
• Asymmetric reconstruction uses the previous approximated output, whereas the symmetric version uses the previous non-approximated output for both terms.A second symmetric variant using approximated outputs for both terms performed even worse.Source: Multi-Layer: Symmetric vs. Asymmetric · p. 8
• Without asymmetric reconstruction, error rates increase more drastically when more or all layers are approximated simultaneously.Source: Multi-Layer: Symmetric vs. Asymmetric · p. 8

• Rank selection consistently outperforms no rank selection in whole-model acceleration, for both linear and nonlinear solutions.Source: Whole-Model: with/without Rank Selection · p. 8
• Higher ranks in Conv5-7 reflect their less concentrated energy and the need for stronger approximations in those layers.At 3× speedup, Conv7 receives d′ = 167 with rank selection versus d′ = 153 for the corresponding single-layer speedup.Source: Whole-Model: with/without Rank Selection · pp. 8–9
• At 4× whole-model speedup, the proposed method increases error by 4.2%, versus 6.0% for Jaderberg et al.’s method.The comparison uses theoretical speedup ratios over all convolutional layers and top-5 single-view error increases.Source: Figure 6 · p. 9, Comparisons with Jaderberg et al.’s… · p. 10
• At 5× speedup, asymmetric 3d decomposition increases error by only 2.5%.The asymmetric solver is reported to control accumulated error when multiple layers are extensively decomposed.Source: Comparisons with Jaderberg et al.’s… · p. 10
• At 4.5× speedup, asymmetric 3d reconstruction drops character-classification accuracy by 0.7%, compared with 1% for the same speedup reported by Jaderberg et al.Source: Comparisons with Jaderberg et al.’s… · p. 10

• The decomposed architecture is trained for 100 epochs on ImageNet using an initialization method designed for the deeper replacement structure.Each original layer is replaced by three layers, making the decomposed model much deeper than the original.Source: Comparisons with Training from Scratch · p. 10
• The model trained from scratch has 2.8% worse accuracy than the accelerated model.The comparison uses the same decomposed architecture.Source: Comparisons with Training from Scratch · p. 10, Table 4 · p. 10
• These results indicate that acceleration algorithms can digest information from trained models and suggest redundancy in models trained from scratch.Source: Comparisons with Training from Scratch · p. 10

• The accelerated asymmetric 3d models achieve 14.1% and 13.8% top-5 error without and with fine-tuning, respectively.These are absolute top-5 error rates for accelerated models.Source: Comparisons of Absolute Performance · p. 10
• The reimplemented AlexNet has 18.8% top-5 single-view error, with 16.0% 10-view top-5 error and 37.6% top-1 error.Source: Comparisons of Absolute Performance · p. 10
• The accelerated model has 5.0% lower error than AlexNet while operating at nearly the same speed.The comparison uses the reimplemented AlexNet baseline.Source: Comparisons of Absolute Performance · pp. 10–11
• On CPU, actual speedup is 3.5× versus 4.0× theoretical speedup; on GPU, actual speedup is about 3.3×.The GPU ratio is lower because the accelerated model is less amenable to parallelism.Source: Comparisons of Absolute Performance · p. 11

• Very deep VGG models have substantially improved a wide range of visual-recognition tasks, including detection, segmentation, captioning, and video recognition.Source: §4.2 · p. 11
• Their broad impact and slow speed motivate accelerating VGG-16 for practical use.Source: §4.2 · p. 11

ImageNet classification

• Whole-model rank selection reduces the increased error at 4× speedup from 6.38% to 3.84%.The selected ranks adapt to VGG-16’s diverse feature-map sizes and layer time complexities.Source: Accelerating VGG-16 for ImageNet… · pp. 11–12
• 4× speedup with fine-tuning increases VGG-16 1-view top-5 error by 0.3%.Without fine-tuning, the increase is 0.9% at the same speedup ratio.Source: Accelerating VGG-16 for ImageNet… · p. 12

Implementation

• CPU speedup ratios are very close to theoretical ratios, whereas GPU ratios show a gap under generic Caffe kernels.The gap is attributed to specialized implementation sensitivity and unoptimized kernels for some decomposed layers.Source: Accelerating VGG-16 for ImageNet… · p. 12

ImageNet classification

• The method is substantially more accurate than a prior whole-model VGG-16 acceleration method at comparable actual CPU speedups.The prior method reports 3.4% and 7.1% increased top-5 1-view error at 3× and 4× actual CPU speedups, respectively.Source: Accelerating VGG-16 for ImageNet… · pp. 12–13

Evaluation setup

• The evaluation follows Fast R-CNN’s default setting on PASCAL VOC 2007, using trainval images for training and test images for evaluation.Accuracy is measured by mean Average Precision (mAP).Source: Accelerating VGG-16 for Object Detection · p. 13
• The experiment first approximates VGG-16 for ImageNet classification, then uses it as Fast R-CNN’s pre-trained model.The asymmetric 3d version with fine-tuning is used.Source: Accelerating VGG-16 for Object Detection · p. 13
• Fast R-CNN’s convolutional layers consume about 70% of actual running time.Reported speedup ratios concern convolutional layers only.Source: Accelerating VGG-16 for Object Detection · p. 13

Conclusion

• Nonlinear asymmetric reconstruction effectively reduces accumulated error from approximating multiple layers.The conclusion frames this as the basis for whole-model speedup evaluation.Source: §5 · p. 13
• Competitive speedups and accuracy are demonstrated on ImageNet classification and PASCAL VOC object detection.These are the two complex recognition tasks named in the conclusion.Source: §5 · p. 13