Source-linked AI summary
Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, Dongjun Shin
TL;DR
Mobile deployment of deep CNNs is challenged by tight computing, battery, and memory constraints. The paper proposes one-shot whole-network compression using VBMF rank selection, Tucker decomposition, and fine-tuning, and reports substantial reductions in model size, runtime, and energy with a small accuracy loss across four CNNs.
Problem
Deep CNN inference on mobile devices must operate under strict computing, battery, and memory constraints.
Method
The paper compresses entire CNNs through VBMF-based rank selection, Tucker decomposition of kernel tensors, and fine-tuning.
Results
Across AlexNet, VGG-S, GoogLeNet, and VGG-16, compression reduces model size, runtime, and energy consumption at a small accuracy cost.
Takeaways & Limitations
On a smartphone, the four CNNs obtain average runtime improvements ranging from ×1.42 to ×3.68 and energy-consumption improvements ranging from ×1.60 to ×4.26.
Takeaways & Limitations
The optimality of the one-shot selected rank remains insufficiently investigated, and 1 × 1 convolutions lack cache efficiency.
Abstract
from arXiv · showhide
Although the latest high-end smartphone has powerful CPU and GPU, running deeper convolutional neural networks (CNNs) for complex tasks such as ImageNet classification on mobile devices is challenging. To deploy deep CNNs on mobile devices, we present a simple and effective scheme to compress the entire CNN, which we call one-shot whole network compression. The proposed scheme consists of three steps: (1) rank selection with variational Bayesian matrix factorization, (2) Tucker decomposition on kernel tensor, and (3) fine-tuning to recover accumulated loss of accuracy, and each step can be easily implemented using publicly available tools. We demonstrate the effectiveness of the proposed scheme by testing the performance of various compressed CNNs (AlexNet, VGGS, GoogLeNet, and VGG-16) on the smartphone. Significant reductions in model size, runtime, and energy consumption are obtained, at the cost of small loss in accuracy. In addition, we address the important implementation level issue on 1?1 convolution, which is a key operation of inception module of GoogLeNet as well as CNNs compressed by our proposed scheme.
1 INTRODUCTION
The paper targets mobile CNN deployment under strict computing, battery, and memory constraints by introducing a simpler one-shot scheme for compressing entire networks. The scheme combines global rank selection, tensor decomposition, and fine-tuning, producing resource reductions with small accuracy loss.
- Mobile CNN inference is constrained by computing power, battery capacity, and memory, motivating networks tailored to mobile resources.
- The proposed one-shot whole-network compression covers entire convolutional and fully connected layers through a simpler multi-step design.The approach addresses whole-network compression rather than focusing only on selected convolutional layers.
- The scheme consists of rank selection, low-rank tensor decomposition, and fine-tuning.
- Tucker decomposition uses ranks selected by global analytic VBMF, while fine-tuning on ImageNet recovers accumulated accuracy loss.
- Each step can be implemented with publicly available tools for VBMF, Tucker decomposition, and Caffe fine-tuning.
- Compressed AlexNet, VGG-S, GoogLeNet, and VGG-16 achieve significant reductions in model size, runtime, and energy consumption at small accuracy cost.
- 1 × 1 convolution is a key operation in the compressed model and GoogLeNet's inception module, but its cache inefficiency creates a gap between theoretical and practical speedups.
2 RELATED WORK
Prior work compresses CNNs through low-rank approximations, pruning, and implementation-level optimizations. This paper uses Tucker tensor decomposition for whole-network compression, applying different Tucker variants across layers.
- Prior CNN compression: CNN compression methods exploit redundancy in convolutional and fully connected layers using low-rank approximations, vector quantization, and pruning.
- Prior CNN compression: Asymmetric 3D decomposition accelerates entire convolutional layers, while related methods use PCA-based rank selection and reconstruction-error optimization.
- Prior CNN compression: Pruning can reduce parameters and computation but may not achieve runtime speedup with conventional GPU implementations.
- Implementation-level approaches: FFT-based convolution and CPU code optimization represent implementation-level approaches that are distinct from model-level compression.
- Tensor decomposition: Tucker decomposition generalizes matrix SVD to tensors by analyzing mode-n matricizations and combining them with a core tensor.
- Tensor decomposition: The scheme applies Tucker-2 from the second convolutional layer through the first fully connected layers, and Tucker-1 to the remaining layers.
3 PROPOSED METHOD
The proposed one-shot whole-network compression scheme selects ranks with global analytic VBMF, applies Tucker decomposition to kernel tensors, and fine-tunes the entire network. Tucker variants are assigned across layers to reduce computation while preserving accuracy through fine-tuning.
- Overall scheme: The pipeline performs rank selection, Tucker decomposition, and whole-network fine-tuning in three sequential steps.Ranks are selected from kernel-tensor matricizations using global analytic VBMF before decomposition and standard back-propagation fine-tuning.
- Overall scheme: Tucker-2 decomposition is applied from the second convolutional layer through the first fully connected layers, while Tucker-1 is used for the remaining layers.Tucker-1 is equivalent to SVD in this setting.
- Tucker decomposition: Tucker-2 leaves spatial modes undecomposed because their typical kernel dimension D is already small, usually 3 or 5.The resulting core tensor has dimensions D × D × R3 × R4.
- Tucker decomposition: The Tucker-2 convolution is evaluated through consecutive 1 × 1, D × D, and 1 × 1 convolutions using intermediate tensors Z and Z′.The first 1 × 1 convolution performs pixel-wise linear recombination before the spatial convolution, and the final 1 × 1 convolution maps to output channels.
- Rank selection: The ranks R3 and R4 control the trade-off between memory, speed, energy improvements, and accuracy loss, and are selected with global analytic VBMF rather than trial-and-error.VBMF analyzes mode-3 and mode-4 matricizations of each kernel tensor.
- Fine-tuning: Fine-tuning rapidly recovers accuracy lost by minimizing linear-kernel reconstruction error, although recovering original accuracy requires more than 10 epochs in the reported AlexNet example.Accuracy recovered quickly within one epoch, but more than 10 epochs were needed to recover the original accuracy.
4 EXPERIMENTS
Experiments evaluate compressed AlexNet, VGG-S, GoogLeNet, and VGG-16 on Titan X and Samsung Galaxy S6, measuring accuracy, model size, runtime, and energy. Compression reduces computational and memory demands, with especially notable smartphone gains and power-consumption effects linked to 1 × 1 convolutions.
- Experimental setup: Experiments use four CNNs on Titan X and Samsung Galaxy S6, with smartphone runtime and energy averaged over 50 runs.Top-5 single-view accuracy is measured on 50,000 ImageNet2012 validation images; models run through the mobile S-Caffe implementation.
- Overall results: ×5.46/×2.67, ×7.40/×4.80, ×1.28/×2.06, and ×1.09/×4.93 reductions in total weights/FLOPs are obtained for AlexNet, VGG-S, GoogLeNet, and VGG-16, respectively.The corresponding smartphone/Titan X runtime improvements are ×1.42∼×3.68 and ×1.23∼×2.33.
- Overall results: 89.40% accuracy is reported for compressed VGG-16 at theoretical ×4.93 speed-up, comparable to 89.6% and 88.9% at theoretical ×4 and ×5 speed-ups.The comparison is against Zhang et al. (2015a)'s method.
- Layerwise analysis: The smartphone tends to show larger performance gains than Titan X because its mobile GPU has 24 times fewer maximum threads, making cache and memory effects more consequential.Compression reduces weights, cache conflicts, and memory latency.
- Layerwise analysis: At equal compression rates, fully connected layers yield larger smartphone gains than convolutional layers because their dead-on-arrival weights create more harmful cache behavior.Convolution kernel weights are reused multiple times, whereas fully connected weights are utilized only once.
- Energy consumption analysis: Compressed networks reduce power consumption and runtime, while frequent power fluctuations reflect extensive 1 × 1 convolution usage and reduced GPU core power.Uncompressed AlexNet and VGG-S show more stable GPU power during convolutional layers, whereas GoogLeNet fluctuates because it has many small layers and heavily uses 1 × 1 convolutions.
5 DISCUSSION
The proposed one-shot whole-network compression reduces design complexity and improves smartphone runtime and energy consumption across four CNNs, while leaving rank optimality and 1 × 1 convolution efficiency for further study.
- The optimality of the one-shot selected rank has not been fully investigated.
- Although 1 × 1 convolutions reduce channels and computation, their cache inefficiency requires further investigation for best use.
- The scheme addresses whole-network compression’s large design space and long design time with a single general low-rank method and global rank selection.It enables fast design and implementation with publicly available tools.
- Average smartphone runtime improvements were ×2.72 for AlexNet, ×3.68 for VGG-S, ×1.42 for GoogLeNet, and ×3.34 for VGG-16.
- Average smartphone energy-consumption improvements were ×3.41 for AlexNet, ×4.26 for VGG-S, ×1.60 for GoogLeNet, and ×3.53 for VGG-16.
A EXPERIMENTAL SETUP
This section introduces the experimental setup for measuring power consumption and presents measured data examples.
- The experimental setup includes a power-consumption measurement system and examples of its measured data.
A.1 MEASUREMENT SYSTEM
The power measurement system uses a probe board, monitor board, and probes connected to smartphone processor and memory power pins.
- The system consists of a probe board containing a Samsung Galaxy S6 smartphone and power probes, plus a monitor board.
- The probe board provides eight probes connected to application-processor power pins.
- The monitor samples electric current for each probe every 0.1ms and outputs time-stamped power-consumption data.
- The measurement system probes the mobile GPU and LPDDR4 DRAM main-memory power pins.
A.2 MEASURED DATA EXAMPLE: GoogLeNet CASE
The uncompressed GoogLeNet power profile identifies layer periods and shows that Inception modules consume most of the total energy, with more frequent fluctuations than convolutional layers.
- The power profile identifies the periods of individual layers, including Conv 1, Conv 2, and the first Inception module, i3a.
- Inception modules show more frequent power-consumption fluctuations than convolutional layers.
- About 1/4 of total energy consumption comes from Conv 1 and Conv 2, while Inception modules consume about 3/4.
B LAYERWISE ANALYSIS
The section reports detailed layerwise compression analyses for VGG-S, GoogLeNet, and VGG-16. For VGG-16, the first convolutional and fully connected layers remain uncompressed, while the theoretical speed-up reaches ×5.03 for convolutional layers and ×4.93 overall.
- Detailed layerwise comparison results are reported for VGG-S, GoogLeNet, and VGG-16.
- The VGG-S and GoogLeNet analyses report input-channel dimension, output-channel dimension, and Tucker-2 ranks for each layer.Here, S denotes input channel dimension, T output channel dimension, and (R3, R4) the Tucker-2 rank.
- VGG-16 leaves the first convolutional and fully connected layers uncompressed, following Zhang et al. (2015a).
- The theoretical speed-up for VGG-16 is ×5.03 on convolutional layers and ×4.93 across whole layers.