Source-linked AI summary
Compression-aware Training of Deep Networks
Jose M. Alvarez, Mathieu Salzmann
TL;DR
Deep networks are costly because they are heavily over-parameterized, while conventional compression usually starts after standard training. This paper trains networks with a low-rank regularizer, achieving substantially higher compression with little or no accuracy loss, though low-rank post-processing does not necessarily reduce inference time on modern hardware.
Problem
Deep networks are over-parameterized and expensive in computation and memory, while existing compression methods typically do not account for future compression during training.
Method
The method adds a low-rank regularizer to layer parameters during training and can combine it with group sparsity before SVD-based post-processing.
Results
More than 90% compression was achieved on ImageNet and ICDAR, with models more compact than baselines at similar or higher accuracies.
Takeaways & Limitations
Accounting for compression during training can produce compact networks with at least comparable prediction accuracy and higher compression rates than existing methods.
Takeaways & Limitations
Low-rank post-processing does not significantly reduce inference time on modern hardware, despite reducing the operation count.
Abstract
from arXiv · showhide
In recent years, great progress has been made in a variety of application domains thanks to the development of increasingly deeper neural networks. Unfortunately, the huge number of units of these networks makes them expensive both computationally and memory-wise. To overcome this, exploiting the fact that deep networks are over-parametrized, several compression strategies have been proposed. These methods, however, typically start from a network that has been trained in a standard manner, without considering such a future compression. In this paper, we propose to explicitly account for compression in the training process. To this end, we introduce a regularizer that encourages the parameter matrix of each layer to have low rank during training. We show that accounting for compression during training allows us to learn much more compact, yet at least as effective, models than state-of-the-art compression techniques.
1 Introduction
Deep networks are powerful but over-parameterized, making them costly to deploy. The paper proposes training networks with compression in mind to obtain compact models with little accuracy loss.
- Motivation: Over-parameterized deep networks can match equivalent results with more compact models, benefiting runtime- and memory-sensitive deployment.Embedded systems are given as an example of hardware with limited resources.
- Motivation: Existing compression methods commonly begin by compressing networks trained in the standard way.The introduction contrasts this post-training workflow with compression-aware training.
- Contribution: The proposed method adds a low-rank regularizer for each layer during training and optimizes it with stochastic proximal gradient descent.The regularizer encourages correlated units that can later be pruned.
- Contribution: The approach explicitly adapts the learned network to a subsequent compression stage, unlike methods that learn uncorrelated units to prevent overfitting.This design seeks correlated units that are easier to remove after training.
- Results: More than 90% compression was achieved on ImageNet and ICDAR, reducing the number of operations required at inference.The experiments used the 8-layer DecomposeMe network and the 50-layer ResNet.
2 Related Work
Prior work reduces deep-network cost mainly through post-training pruning, low-rank approximation, sparsity, or quantization. This paper instead belongs to the smaller group that explicitly learns compact networks during training.
- Motivation: Deep networks create both overfitting concerns and substantial computational and memory costs, motivating compression research.The related-work discussion distinguishes these two drawbacks.
- Pruning: Early compression methods removed individual parameters or entire units according to their output influence, but this analysis becomes difficult for very deep networks.Later methods therefore analyze larger parameter groups.
- Low-rank compression: Low-rank approximations became a popular alternative, applied to individual units or complete layer parameter matrices and tensors.Some methods fine-tune the resulting representation, while others learn it from scratch given known ranks.
- Training-time compression: Several methods explicitly learn compact networks during training by inducing sparsity in individual parameters or units.These approaches differ from methods that compress a conventionally trained network afterward.
- Quantization: Quantization is presented as orthogonal to compact-network design, since a compact network could still be quantized afterward.The paper emphasizes non-quantized compact networks in the context of efficient floating-point hardware.
3 Compression-aware Training of Deep Networks
The method encourages low-rank layer parameters during training, then combines low-rank compression with group sparsity and SVD-based post-processing. This reduces parameters and can also remove units for more compact inference models.
- Low-rank training: The approach regularizes each layer’s parameter matrix toward low rank while training, preparing the network for later compression.The method is introduced for convolutional networks but also applies to fully connected layers.
- Low-rank training: Convolutional parameters are reshaped into matrices so singular-value decomposition can produce basis kernels and activation-mixing matrices.Truncating small singular values yields a compact approximation with rank r_l.
- Optimization: Because direct rank minimization is NP-hard, the method uses the nuclear norm as a convex relaxation in the training regularizer.The regularizer’s influence is controlled by the hyper-parameter τ.
- Optimization: Proximal stochastic gradient descent updates the supervised loss for one epoch before applying the regularizer’s proximity operator independently to each layer.The proximity solution is obtained by soft-thresholding the layer’s singular values.
- Low-rank and group-sparse layers: Combining low-rank regularization with sparse group Lasso removes units while preserving low-rank structure among remaining units.This addresses the limitation that low-rank compression alone does not change the original input and output channel counts.
- Post-processing: After training, SVD decomposes each layer and retains singular values selected by an energy threshold, trading prediction accuracy against runtime and memory savings.The same retained-energy percentage is used across layers in the experiments.
- Post-processing: Removing zeroed units reduces parameters and can lower inference cost, while low-rank post-processing may not reduce inference time on modern hardware.The latter limitation is attributed to hardware optimized for convolutions with fewer operations than the naive MAC count suggests.
4 Experimental Settings
Experiments evaluate compression-aware training on decomposed DecomposeMe and ResNet architectures using ImageNet and ICDAR, with accuracy, MACs, and training configurations as principal measures. The setup combines low-rank and sparse-group regularization with post-processing to reduce computational cost while preserving classification performance.
- Datasets: Experiments use ImageNet and ICDAR, including 1.2 million ImageNet training images and 185,639 ICDAR training samples across 1000 and 36 categories, respectively.ImageNet also includes 50,000 validation images, while ICDAR includes 5,198 test samples.
- Network architectures: The evaluated architectures use convolutional kernels decomposed into vertical and horizontal 1D kernels, with batch normalization and nonlinearities after each convolution.The study includes DecomposeMe and a decomposed ResNet-50, and reports no significant classification-accuracy impact from using 1D kernels in the ResNet.
- Training protocol: Models and baselines are trained from scratch with the same random seed, framework, and computer setup to support fair comparisons.The implementation uses the torch-7 multi-GPU framework.
- Regularization settings: The experiments vary the low-rank regularization strength τ over {0,1,5,10} and configure sparse-group Lasso penalties across layers, including weaker penalties for the first two layers.The sparse-group setting uses α = 0.2 to favor group-level sparsity.
- Evaluation metrics: Computational cost is measured by multiply-accumulate operations, with layer cost determined by output-map size, channel counts, and kernel dimensions.Reducing the number of units in early convolutional layers has greater impact when the output feature map contains many pixels.
- Evaluation design: Parameter-sensitivity experiments compare accuracy and MACs before and after post-processing, using the standard model as the no-regularization baseline.The comparison includes configurations with and without sparse-group Lasso and post-processing energy percentages of 100% and 80%.
5 Experimental Results
Experiments show that compression-aware regularization preserves or improves accuracy while reducing operations, parameters, and training time across ImageNet and ICDAR models. The low-rank regularizer can also eliminate entire units before post-processing, supporting substantial compactness gains.
- Accuracy remains stable across a wide range of low-rank and sparsity regularization strengths, with small improvements under moderate regularization.
- High compression rates can reduce computational cost with virtually no accuracy loss; using el = 80% increases MAC savings but typically causes a small accuracy drop.
- The low-rank regularizer alone significantly reduces operations before post-processing by automatically zeroing out complete units and singular values.
- More than 20% and 50% parameter reductions were achieved on ImageNet with el = 100% and el = 80%, respectively, while ICDAR reductions were around 90% in both cases.
- 75.0% top-1 accuracy with 20.6% compression and 75.2% with 27% compression outperformed the 74.7% baseline accuracy and 4% baseline compression on ResNet-50.
- Training time fell from 1.69 to 0.77 hours, a 54.5% speedup, while accuracy improved by 2% and parameters dropped from 3.7M to 210K.
6 Conclusion
The paper trains deep networks with compression in mind by encouraging low-rank parameter matrices, and combines this with sparsity to pursue higher compression.
- The training loss includes a regularizer that encourages each layer’s parameter matrix to have low rank.
- The approach also combines low-rank regularization with a sparsity-inducing regularizer for higher compression.
- Experiments report higher compression rates than state-of-the-art methods.