Convolutional neural networks with low-rank regularization

Cheng Tai; Tong Xiao; Yi Zhang; Xiaogang Wang; Weinan E

Source-linked AI summary

Convolutional neural networks with low-rank regularization

Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, Weinan E

arXiv:1511.06067v3Nov 19, 2015cs.LGcs.CVstat.ML

TL;DR

Large CNNs are difficult to deploy on mobile devices because of their computational and storage demands. The paper develops an exact low-rank tensor-decomposition algorithm and a from-scratch training method for constrained CNNs, achieving significant speedups with minimal performance loss and sometimes higher accuracy. Its low-rank NIN model reaches 91.31% CIFAR-10 accuracy without data augmentation.

Problem
Large CNNs require substantial computation and storage, motivating methods to reduce deployment costs while preserving performance.
Method
The paper introduces an exact global-optimizer algorithm for low-rank tensor decomposition and a parameterization for training low-rank constrained CNNs from scratch.
Results
Significant speedup is achieved with minimal performance loss, while some constrained models improve accuracy; low-rank NIN reaches 91.31% CIFAR-10 accuracy without data augmentation.
Takeaways & Limitations
Low-rank tensor decomposition provides evidence of being a generic tool for speeding up large CNNs across modern architectures.
Takeaways & Limitations
Fully connected layers still occupy a large fraction of parameters, limiting the overall compression ability of the low-rank constraint.

Abstract

from arXiv · show

Large CNNs have delivered impressive performance in various computer vision applications. But the storage and computation requirements make it problematic for deploying these models on mobile devices. Recently, tensor decompositions have been used for speeding up CNNs. In this paper, we further develop the tensor decomposition technique. We propose a new algorithm for computing the low-rank tensor decomposition for removing the redundancy in the convolution kernels. The algorithm finds the exact global optimizer of the decomposition and is more effective than iterative methods. Based on the decomposition, we further propose a new method for training low-rank constrained CNNs from scratch. Interestingly, while achieving a significant speedup, sometimes the low-rank constrained CNNs delivers significantly better performance than their non-constrained counterparts. On the CIFAR-10 dataset, the proposed low-rank NIN model achieves $91.31\%$ accuracy (without data augmentation), which also improves upon state-of-the-art result. We evaluated the proposed method on CIFAR-10 and ILSVRC12 datasets for a variety of modern CNNs, including AlexNet, NIN, VGG and GoogleNet with success. For example, the forward time of VGG-16 is reduced by half while the performance is still comparable. Empirical success suggests that low-rank tensor decompositions can be a very useful tool for speeding up large CNNs.

1 INTRODUCTION

The paper targets the computational and storage burden of large CNNs by developing low-rank tensor-decomposition methods for convolutional kernels and training constrained networks from scratch. Experiments across modern architectures report substantial speedups with minimal performance loss, and sometimes higher accuracy.

Motivation: Large CNNs often require 10^7∼10^9 parameters and substantial computation, complicating deployment on low-end mobile devices.The paper notes that training may take weeks even with high-end GPUs, while mobile inference can be problematic for real-time operation.
Motivation: Because convolution operations constitute most CNN computation, reducing redundancy in their 4D tensor kernels can directly improve overall speed.The paper identifies tensor decomposition as a promising way to simplify convolution layers.
Contributions: The paper introduces an exact global-optimization algorithm for a low-rank tensor decomposition instead of relying on approximate iterative solutions.The method is presented as a best data-independent approximation and as more effective than iterative schemes.
Contributions: It also proposes training low-rank constrained CNNs from scratch by parameterizing convolutional kernels to enforce the low-rank structure.Batch normalization enables training very deep constrained CNNs with more than 30 layers.
Evaluation: The method is evaluated on CIFAR-10 and ILSVRC12 using NIN, AlexNet, VGG, and GoogleNet models.The experiments address large and varied CNN architectures rather than only isolated convolutional layers.
Results: 91.31% classification accuracy is achieved by the low-rank NIN model on CIFAR-10 without data augmentation, improving upon the original NIN and state-of-the-art results.The broader experiments report significant speedup with minimal performance loss, while some constrained models increase classification accuracy.

2 RELATED WORK

The related work covers low-rank filtering, tensor decompositions, and CNN acceleration, while distinguishing the paper's exact decomposition and broader evaluation from prior approaches.

Comparison with prior work: Jaderberg et al. approximated networks layer by layer, fixed each approximated layer, and fine-tuned higher layers during the approximation process.The paper identifies this layerwise procedure as one difference from its own approximation stage.
Comparison with prior work: Lebedev et al. used CP decomposition with nonlinear least squares, whereas the proposed scheme always exists and has an exact closed-form solution.The paper also notes that best low-rank CP approximations may not exist in general.
Scope of evaluation: The paper evaluates more and larger CNN models than the cited prior studies, supporting applicability across state-of-the-art architectures.The authors frame this broader scope as more challenging and as evidence for applicability to varied models.

3 METHOD

The method replaces convolutional kernels with rank-K factorizations, computes an exact global low-rank approximation, and trains constrained CNNs from scratch while reducing computation.

Low-rank parametrization: The proposed scheme replaces each 4D convolutional kernel with two consecutive one-dimensional convolutional filters of rank K.The factorization uses horizontal filters H and vertical filters V as learnable parameters.
Low-rank parametrization: The factorization exploits redundancy across both spatial dimensions and channels, reducing computation from O(d^2NCXY) to O(dK(N + C)XY).Acceleration requires K < dNC.
Computational efficiency: When C ≪ N, typical of the first CNN layer, the method can achieve approximately d-fold acceleration.The first-layer illustration reports K = 8 and 3.67× speedup for AlexNet.
Approximation and fine-tuning: The approximation procedure minimizes Frobenius reconstruction error layer by layer, then fine-tunes the whole CNN to restore classification accuracy.Layerwise approximation can be performed in parallel because there is no inter-layer dependence during that step.
Exact decomposition algorithm: A tensor-to-matrix transformation followed by SVD provides a closed-form solution that is the exact global optimizer of the decomposition.The resulting solution is not unique because reciprocal rescaling of H and V gives equivalent solutions.
Training constrained CNNs: The algorithm avoids iterative local minima and completes in under 1 second for most tested modern CNNs, while low-rank constrained networks can be trained from scratch.The constrained parametrization introduces additional layers and makes optimization more challenging because of exploding and vanishing gradients.

4 EXPERIMENTS

Experiments on CIFAR-10 and ILSVRC12 show that low-rank constrained CNNs can reduce computation and convolutional parameters while maintaining or improving classification performance.

4.1 CIFAR-10: CIFAR-10 models were trained from scratch, comparing customized CNN and NIN baselines with corresponding low-rank constrained versions.
4.1 CIFAR-10: 91.31% accuracy without data augmentation was achieved by the low-rank NIN model, improving on the baseline NIN and previously published results.
4.1 CIFAR-10: Low-rank constrained versions outperformed both baseline networks with and without data augmentation, including an improvement exceeding 1% for low-rank NIN.
4.1 CIFAR-10: Up to 7× layer speedup and 2-3× whole-network speedup were achieved, although practical gains depend on convolution software and hardware optimization.
4.1 CIFAR-10: With K1 = 12, K2 = 16 and K3 = 32, convolution-kernel parameters were reduced by 91% while relative performance changed by +0.25%.
4.2 ILSVRC12: On ILSVRC12, the method was evaluated with AlexNet, VGG-16 and GoogLeNet, using layer-specific rank choices to balance theoretical speedup and classification performance.

5 DISCUSSION

The proposed tensor-decomposition approach successfully speeds up several modern CNNs and sometimes improves performance over unconstrained models, while overall compression remains limited by fully connected layers.

The method successfully speeds up AlexNet, NIN, VGG, and GoogleNet, supporting tensor decomposition as a generic tool for accelerating large CNNs.The paper reports success across a variety of modern CNN architectures.
Theoretical speedup and weights reduction in Table 5 concern only decomposed convolutional layers, whereas actual speedup measures whole-network forward time.This distinction affects how the reported efficiency measures should be interpreted.
Low-rank constrained CNNs sometimes outperform their non-constrained counterparts despite having strictly smaller expressive power.The authors associate this observation with local-minima and over-fitting issues.

APPENDIX

The appendix establishes equivalence between two low-rank approximation formulations and shows that the data-independent problem has an exact solution, while the data-dependent weighted problem is NP-hard.

The appendix formulates the low-rank approximation as minimizing an objective subject to a rank constraint.The construction maps a solution of one formulation into a feasible solution of the other.
The two formulations attain equal objective values, establishing equivalence between their optimal solutions.The stated equalities connect E1(H*, V*) and E2(W*) through the constructed factors.
The data-dependent criterion becomes a weighted low-rank approximation problem involving patch sampling and a Hadamard product.The formulation uses sampled image patches and elementwise weighting.
The data-dependent weighted problem is NP-hard except for a few special cases, unlike the closed-form data-independent problem.Identity or rank-one weight matrices are cited as examples of special cases.