PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerating Very Deep Convolutional Networks for Classification and Detection

Xiangyu Zhang, Jianhua Zou, Kaiming He, Jian Sun

arXiv:1505.06798v2cs.CVcs.LGcs.NE

TL;DR

Very deep CNNs offer strong recognition performance but impose substantial computational costs, motivating faster inference. This paper models nonlinear units with GSVD-based reconstruction and asymmetric multi-layer error handling, achieving competitive whole-model acceleration across classification and detection, including 4× VGG-16 speedup with a 0.3% top-5 error increase.

  • Problem

    Very deep CNNs substantially increase computational cost, limiting their use in real-world services, portable devices, and high-resolution recognition tasks.

  • Method

    The method reconstructs nonlinear layer responses under a low-rank constraint, solves the optimization with GSVD without SGD, and uses asymmetric reconstruction to reduce accumulated multi-layer error.

  • Results

    The method demonstrates competitive speedups and accuracy on ImageNet classification and PASCAL VOC object detection, including 4× VGG-16 speedup with a 0.3% increase of top-5 center-view error.

  • Takeaways & Limitations

    Whole-model acceleration remains effective across very deep CNN classification and object detection evaluations.

  • Takeaways & Limitations

    Compared with spatial-and-channel decomposition methods, the approach requires a smaller d′ for a given speedup, which might limit accuracy.

Abstract

from arXiv · show

This paper aims to accelerate the test-time computation of convolutional neural networks (CNNs), especially very deep CNNs that have substantially impacted the computer vision community. Unlike previous methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We develop an effective solution to the resulting nonlinear optimization problem without the need of stochastic gradient descent (SGD). More importantly, while previous methods mainly focus on optimizing one or two layers, our nonlinear method enables an asymmetric reconstruction that reduces the rapidly accumulated error when multiple (e.g., >=10) layers are approximated. For the widely used very deep VGG-16 model, our method achieves a whole-model speedup of 4x with merely a 0.3% increase of top-5 error in ImageNet classification. Our 4x accelerated VGG-16 model also shows a graceful accuracy degradation for object detection when plugged into the Fast R-CNN detector.

1 INTRODUCTION

Very deep CNNs improve recognition accuracy but impose substantial computational costs, while prior acceleration results largely focus on shallow models or one or two layers. The paper addresses this gap with nonlinear, asymmetric reconstruction and demonstrates whole-model acceleration for VGG-16 across classification and detection.

  • Motivation: Very deep VGG models are substantially slower than earlier CNNs, creating deployment problems for high-throughput, portable, and high-resolution recognition systems.The motivation includes cloud services, phones and tablets, object detection, and semantic segmentation.
  • Research gap: Prior acceleration methods mainly decompose one or a few layers, with limited evidence for very deep models and complex ImageNet tasks.The passage specifically identifies very deep models as those with roughly 10 or more layers under approximation.
  • Method: The proposed method models nonlinear neurons, solves the resulting low-rank response reconstruction with GSVD without SGD, and uses asymmetric reconstruction across sequentially approximated layers.The asymmetric formulation accounts for errors from previous approximated layers, while rank selection adapts acceleration across layers.
  • Results: 4× speedup increases VGG-16 ImageNet top-5 center-view error by only 0.3%.This result concerns whole-model acceleration of the publicly available VGG-16 model.
  • Results: 4× convolutional speedup in Fast R-CNN produces a 0.8% mAP degradation on PASCAL VOC 2007, from 66.9% to 66.1%.The accelerated VGG-16 model is evaluated as a feature extractor for object detection.
  • Contributions: The extended study reports acceleration of very deep VGG models, evaluates transfer-learning-based object detection, and finds scratch-trained counterparts inferior to accelerated models with the same structure.The authors interpret the latter finding as evidence that the acceleration optimization algorithm digests information from the original model.

2 RELATED WORK

Earlier CNN acceleration work combines layer decomposition with an optimization scheme, but reported results emphasize individual layers or shallower models. Consequently, whole-model acceleration of very deep networks on ImageNet remained an open problem, alongside questions about transfer to other recognition tasks.

  • General framework: CNN acceleration methods generally combine a decomposition that reduces time complexity with an optimization scheme for fitting that decomposition.The optimization component matters because different decompositions may have different local optima.
  • Prior decompositions: Denton et al. use low-rank filter decompositions but do not explicitly minimize post-nonlinearity activation error, and report only a single OverFeat layer.No whole-model results are available in the cited work.
  • Prior decompositions: Jaderberg et al. separate k × k filters into k × 1 and 1 × k filters, while optimizing either filter weights or responses.Their response reconstruction uses stochastic optimization methods, whereas filter reconstruction uses conjugate gradient descent.
  • Optimization challenges: Lebedev et al. apply CP-decomposition to one AlexNet layer on ImageNet and report difficulty finding a suitable SGD learning rate for fine-tuning.Their report indicates optimization challenges even for a single layer in ImageNet models.
  • Open problem: Whole-model acceleration of very deep networks for ImageNet remained an open problem, and their use as generic feature extractors for other tasks was unclear.Other related directions include FFT-based computation, thin deep networks, and memory conservation.

3 APPROACHES

The approach accelerates convolutional layers through low-rank response decomposition, explicitly models nonlinear units, and uses asymmetric reconstruction to limit error accumulation across approximated layers. Rank selection allocates compression unevenly because layers differ in redundancy, while spatial decomposition can complement the method when ranks become too small.

  • 3.1 Low-rank Approximation of Responses: The method uses SVD for linear neurons and GSVD for nonlinear neurons, enabling an asymmetric reconstruction that reduces accumulated error in very deep models.The nonlinear solver incorporates the response nonlinearity without SGD, while asymmetric reconstruction accounts for errors from previously approximated layers.
  • 3.1 Low-rank Approximation of Responses: Low-rank response structure permits replacing a layer with two lower-complexity layers whose intermediate rank is d′.The response vector is projected onto a rank-d′ subspace, then reconstructed through a d′-filter layer and a 1×1 convolution.
  • 3.1 Low-rank Approximation of Responses: O(d′k2c) + O(dd′) replaces O(dk2c), reducing computation to about d′/d when O(dd′) is much smaller than O(d′k2c).The original layer has d filters, while the approximation uses d′ intermediate filters and a d-by-d′ projection.
  • 3.1 Low-rank Approximation of Responses: PCA measurements support the low-rank assumption: Conv2’s first 128 eigenvectors retain over 99.9% energy, while Conv7’s first 256 retain over 95%.The response energy is measured from sampled ImageNet training responses, and the concentration indicates that fractions of the filters can approximate the original filters.
  • 3.4 Rank Selection for Whole-Model Acceleration: Uniform layer speedups are not optimal because layers are not equally redundant, so whole-model acceleration requires selecting d′ separately for each layer.The method relates PCA energy to classification accuracy and uses rank allocation to avoid overly aggressive compression in less redundant layers.
  • 3.4 Rank Selection for Whole-Model Acceleration: The method can be combined with spatial decomposition because operating on multiple dimensions may otherwise require a smaller d′ for a target speedup, limiting accuracy.The combined strategy is intended to avoid ranks becoming too small.

4 EXPERIMENTS

Experiments on SPP-10 evaluate single-layer acceleration and compare linear with nonlinear reconstruction. The results show that nonlinear reconstruction generally performs better, while modest speedups cause small error increases.

  • SPP-10 contains 7 convolutional and 3 fully connected layers, while VGG-16 contains 13 convolutional and 3 fully connected layers.
  • Single-view top-5 error references are 12.51% for SPP-10 and 10.09% for VGG-16 on ImageNet validation/testing.These values are used as references for increased error rates in approximated models.
  • Single-Layer: Linear vs. Nonlinear: Nonlinear reconstruction consistently performs better than linear reconstruction for single-layer approximation.The comparison plots increased error rates against each layer's theoretical speedup ratio.
  • Single-Layer: Linear vs. Nonlinear: ReLU sparsity exceeds 60% in Conv2-7, reaching 95% in Conv7, where the nonlinear solution's advantage is more obvious.The substantial truncation caused by ReLU helps explain the discrepancy between linear and nonlinear solutions.
  • Single-Layer: Linear vs. Nonlinear: At 2× single-layer acceleration, Conv2 increases error by < 0.1%, while Conv3-7 increase error by ≈0.2%.
  • Single-Layer: Linear vs. Nonlinear: Conv1 degradation is negligible near 2× speedup; the experiments therefore use d′ = 32 for Conv1 in whole-model SPP-10 acceleration.A 1.8× speedup corresponds to d′ = 32, while larger speedups require more drastic reduction because Conv1 has only 3 input channels.

Multi-Layer: Symmetric vs. Asymmetric

The multi-layer experiments compare symmetric and asymmetric reconstruction across two- and three-layer approximations. Asymmetric reconstruction substantially improves results by accounting for errors from previously approximated layers.

  • Multi-Layer: Symmetric vs. Asymmetric: The experiments approximate Conv6-7, Conv2-4, and Conv5-7, sequentially processing the involved layers from shallower to deeper ones.
  • Multi-Layer: Symmetric vs. Asymmetric: Asymmetric reconstruction uses the previous approximated output, whereas the symmetric version uses the previous non-approximated output for both terms.A second symmetric variant using approximated outputs for both terms performed even worse.
  • Multi-Layer: Symmetric vs. Asymmetric: Without asymmetric reconstruction, error rates increase more drastically when more or all layers are approximated simultaneously.

Whole-Model: with/without Rank Selection

Whole-model experiments show that rank selection improves accelerated SPP-10 models, while asymmetric reconstruction controls error as layers are extensively approximated.

  • Rank selection consistently outperforms no rank selection in whole-model acceleration, for both linear and nonlinear solutions.
  • Higher ranks in Conv5-7 reflect their less concentrated energy and the need for stronger approximations in those layers.At 3× speedup, Conv7 receives d′ = 167 with rank selection versus d′ = 153 for the corresponding single-layer speedup.
  • At 4× whole-model speedup, the proposed method increases error by 4.2%, versus 6.0% for Jaderberg et al.’s method.The comparison uses theoretical speedup ratios over all convolutional layers and top-5 single-view error increases.
  • At 5× speedup, asymmetric 3d decomposition increases error by only 2.5%.The asymmetric solver is reported to control accumulated error when multiple layers are extensively decomposed.
  • At 4.5× speedup, asymmetric 3d reconstruction drops character-classification accuracy by 0.7%, compared with 1% for the same speedup reported by Jaderberg et al.

Comparisons with Training from Scratch

The paper compares its accelerated architecture with an identically structured model trained from scratch, finding that the accelerated model retains substantially better accuracy.

  • The decomposed architecture is trained for 100 epochs on ImageNet using an initialization method designed for the deeper replacement structure.Each original layer is replaced by three layers, making the decomposed model much deeper than the original.
  • The model trained from scratch has 2.8% worse accuracy than the accelerated model.The comparison uses the same decomposed architecture.
  • These results indicate that acceleration algorithms can digest information from trained models and suggest redundancy in models trained from scratch.

Comparisons of Absolute Performance

Absolute-performance comparisons show that accelerated SPP-10 models can approach or exceed similarly fast AlexNet accuracy, with actual speedups below theoretical GPU ratios.

  • The accelerated asymmetric 3d models achieve 14.1% and 13.8% top-5 error without and with fine-tuning, respectively.These are absolute top-5 error rates for accelerated models.
  • The reimplemented AlexNet has 18.8% top-5 single-view error, with 16.0% 10-view top-5 error and 37.6% top-1 error.
  • The accelerated model has 5.0% lower error than AlexNet while operating at nearly the same speed.The comparison uses the reimplemented AlexNet baseline.
  • On CPU, actual speedup is 3.5× versus 4.0× theoretical speedup; on GPU, actual speedup is about 3.3×.The GPU ratio is lower because the accelerated model is less amenable to parallelism.

4.2 Experiments with VGG-16

VGG models have improved many visual-recognition tasks but remain substantially slower than earlier models, motivating practical acceleration of VGG-16.

  • Very deep VGG models have substantially improved a wide range of visual-recognition tasks, including detection, segmentation, captioning, and video recognition.
  • Their broad impact and slow speed motivate accelerating VGG-16 for practical use.

Accelerating VGG-16 for ImageNet Classification

Whole-model rank selection and nonlinear asymmetric reconstruction make VGG-16 acceleration effective despite errors accumulating across many approximated layers. At 4× speedup, fine-tuned acceleration increases 1-view top-5 error by 0.3%, while actual CPU speedups closely track theoretical ratios.

  • ImageNet classification: Whole-model rank selection reduces the increased error at 4× speedup from 6.38% to 3.84%.The selected ranks adapt to VGG-16’s diverse feature-map sizes and layer time complexities.
  • ImageNet classification: 4× speedup with fine-tuning increases VGG-16 1-view top-5 error by 0.3%.Without fine-tuning, the increase is 0.9% at the same speedup ratio.
  • Implementation: CPU speedup ratios are very close to theoretical ratios, whereas GPU ratios show a gap under generic Caffe kernels.The gap is attributed to specialized implementation sensitivity and unoptimized kernels for some decomposed layers.
  • ImageNet classification: The method is substantially more accurate than a prior whole-model VGG-16 acceleration method at comparable actual CPU speedups.The prior method reports 3.4% and 7.1% increased top-5 1-view error at 3× and 4× actual CPU speedups, respectively.

Accelerating VGG-16 for Object Detection

The accelerated VGG-16 model is used as the pre-trained feature extractor in Fast R-CNN object detection. A 4× convolution speedup produces a 0.8% mAP degradation on PASCAL VOC 2007.

  • Evaluation setup: The evaluation follows Fast R-CNN’s default setting on PASCAL VOC 2007, using trainval images for training and test images for evaluation.Accuracy is measured by mean Average Precision (mAP).
  • Evaluation setup: The experiment first approximates VGG-16 for ImageNet classification, then uses it as Fast R-CNN’s pre-trained model.The asymmetric 3d version with fine-tuning is used.
  • Evaluation setup: Fast R-CNN’s convolutional layers consume about 70% of actual running time.Reported speedup ratios concern convolutional layers only.

5 CONCLUSION

The paper presents whole-model acceleration for very deep networks using nonlinear asymmetric reconstruction to reduce accumulated error across multiple layers. It demonstrates competitive speedups and accuracy on ImageNet classification and PASCAL VOC object detection.

  • Conclusion: Nonlinear asymmetric reconstruction effectively reduces accumulated error from approximating multiple layers.The conclusion frames this as the basis for whole-model speedup evaluation.
  • Conclusion: Competitive speedups and accuracy are demonstrated on ImageNet classification and PASCAL VOC object detection.These are the two complex recognition tasks named in the conclusion.
Loading 1505.06798v2…