Source-linked AI summary
Ultimate tensorization: compressing convolutional and FC layers alike
Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, Dmitry Vetrov
TL;DR
CNNs are computationally and memory intensive, motivating compression beyond earlier fully-connected-layer methods. The paper reshapes convolutional kernels into higher-order tensors and factorizes them with matrix TT-format, combining this with fully-connected compression to obtain 80× network compression with a 1.1% accuracy drop on CIFAR-10.
Problem
CNNs require substantial computation, storage, memory bandwidth, and energy, while prior tensor-factorization work addressed fully-connected rather than convolutional layers.
Method
The paper reshapes each convolutional kernel into a matrix and then a higher-order tensor before applying matrix TT-format, combining this with fully-connected-layer compression.
Results
80× network compression with a 1.1% accuracy drop was achieved by compressing both convolutional and fully-connected layers on CIFAR-10.
Takeaways & Limitations
The combined approach moves toward embedding compressed neural networks in smartphones for continuous surrounding-sensing applications.
Takeaways & Limitations
The paper identifies future evaluation on the ILSVRC-2012 dataset and state-of-the-art neural architectures as remaining work.
Abstract
from arXiv · showhide
Convolutional neural networks excel in image recognition tasks, but this comes at the cost of high computational and memory complexity. To tackle this problem, [1] developed a tensor factorization framework to compress fully-connected layers. In this paper, we focus on compressing convolutional layers. We show that while the direct application of the tensor framework [1] to the 4-dimensional kernel of convolution does compress the layer, we can do better. We reshape the convolutional kernel into a tensor of higher order and factorize it. We combine the proposed approach with the previous work to compress both convolutional and fully-connected layers of a network and achieve 80x network compression rate with 1.1% accuracy drop on the CIFAR-10 dataset.
1 Introduction
CNNs deliver strong performance but impose substantial computation, memory, and energy demands. This paper proposes tensor-factorization methods for compressing convolutional layers alongside previously compressed fully-connected layers.
- Motivation: Millions of operations, parameters, and hundreds of megabytes of storage and bandwidth make CNN deployment on mobile devices difficult.Using RAM rather than processor cache further increases energy consumption.
- Prior work: Tensor decomposition previously compressed fully-connected layers, shifting VGG-16’s storage bottleneck toward convolutional layers.The earlier approach represented fully-connected parameters in Tensor Train format and learned the network from scratch in that representation.
- Contributions: The paper proposes tensor factorization for convolutional layers and reshapes their kernels into higher-order tensors to use Tensor Train compression more effectively.Direct Tensor Train decomposition of a convolutional tensor produces poor results according to the paper’s experiments.
- Contributions: 4× compression of a convolution-only network incurred a 2% accuracy decrease.This result concerns the proposed approach applied to a network consisting only of convolutional layers.
- Contributions: 82× network compression with a 1% accuracy drop resulted from combining convolutional compression with fully-connected-layer compression.The combined result is reported as an experimental contribution of the paper.
2 Convolutional Layer
A convolutional layer maps an input tensor to an output tensor by applying a kernel across spatial patches and channels. The paper rewrites this operation as matrix multiplication by reshaping outputs, input patches, and the kernel.
- Convolutional Layer: A convolutional layer transforms X ∈ R^(W×H×C) into Y ∈ R^((W−ℓ+1)×(H−ℓ+1)×S) using K ∈ R^(ℓ×ℓ×C×S).The basic convolution is used for clarity, although the compression approach also works with padding, larger strides, and rectangular filters.
- Matrix formulation: The output tensor is reshaped into a matrix with W′H′ rows and S columns, where W′ = W − ℓ + 1 and H′ = H − ℓ + 1.Each output-matrix row corresponds to one spatial output location.
- Matrix formulation: The input is reshaped into a W′H′ × ℓ^2C matrix whose k-th row contains the input patch used to compute the k-th output row.The patch contains ℓ × ℓ spatial positions across C channels.
- Matrix formulation: The kernel is reshaped into an ℓ^2C × S matrix by flattening spatial and channel indices.This aligns each flattened input patch with the kernel weights producing the S output channels.
- Matrix formulation: With these reshaped matrices, convolution becomes the matrix product Y = XK.The transformation exposes convolution as a matrix-by-matrix multiplication suitable for subsequent factorization.
3 Tensor Train Decomposition
Tensor Train decomposition represents a tensor through interconnected matrix-valued cores and TT-ranks. Its rank choices control the memory–accuracy trade-off, while matrix TT-format reshapes matrices into higher-order tensors before decomposition.
- TT representation: A tensor’s TT-representation consists of matrix-valued cores G_k[j_k] whose products represent individual tensor elements.The endpoint ranks are fixed at r_0 = r_d = 1.
- TT representation: The intermediate values r_k are TT-ranks, while the collections of matrices indexed by j_k are TT-cores.These components define the compact tensor representation used by the method.
- Compression trade-off: The TT-format requires a number of elements determined by the tensor dimensions and can perform linear algebra on TT-cores without materializing the full tensor.This supports memory-efficient tensor operations.
- Compression trade-off: Smaller TT-ranks reduce memory usage but trade off representation accuracy.The TT-format therefore exposes an explicit parameter-count versus accuracy trade-off.
- Matrix TT-format: For matrices, matrix TT-format reshapes row and column indices through bijective mappings into a higher-order tensor before applying TT-representation.The matrix dimensions are factorized into products of mode sizes, allowing the matrix TT-format to be more compact than ordinary matrix low-rank format.
4 TT-convolutional Layer
The paper develops TT-convolution by reshaping convolutional kernels into higher-order tensors before applying matrix TT-format, preserving the fully-connected-layer case for 1 × 1 convolutions.
- Kernel representation: Direct TT-decomposition of a 4-dimensional convolutional kernel performs poorly, motivating a reshaping strategy that better uses TT compression.The paper illustrates this limitation with 1 × 1 convolutions, where direct decomposition becomes ordinary matrix low-rank decomposition.
- Kernel representation: The 4-dimensional kernel is reshaped into a matrix of size ℓ2C × S and then into a tensor whose dimensions factorize the channel dimensions.The resulting tensor has dimension lengths C_kS_k for channel-output factors and ℓ2 for the spatial factor.
- Kernel representation: The reshaped kernel is represented as a product of TT cores, with the spatial indices assigned to the first core and channel indices assigned to subsequent cores.The decomposition is expressed through G0[x, y]G1[c1, s1]...Gd[cd, sd].
- Layer pipeline: A TT-convolution layer reshapes the input into a tensor with factored channel dimensions, applies the TT transformation, and produces an output tensor with factored output channels.The TT-core elements are trained directly using stochastic gradient descent with momentum and automatic differentiation.
- Kernel representation: For 1 × 1 convolutions, the spatial indices vanish, and the decomposition coincides with the Tensor Train format previously proposed for fully-connected layers.This provides a unified representation across convolutional and fully-connected layers.
5 Related Work
Prior work compressed fully-connected layers and convolutional layers using pruning, quantization, coding, and tensor or matrix decompositions; the proposed approach is compatible with some of these techniques.
- Compression motivation: Fully-connected layers have traditionally been treated as memory bottlenecks, but compression can shift the bottleneck to convolutional layers.This shift motivates methods specifically targeting convolutional computation and storage.
- Weight-based methods: Pruning removes less important weights, while quantization restricts weight variation; both are established approaches for compressing convolutional kernels.The proposed method can quantize TT-core elements and can be combined with Huffman coding.
- Decomposition methods: CP-decomposition and Kronecker product factorization compress convolutional networks while also enabling faster inference.These methods use tensor or matrix decompositions as alternatives to weight pruning and quantization.
6 Experiments
Experiments on CIFAR-10 compare TT-convolution with a naive kernel decomposition and evaluate combined convolutional and fully-connected compression under different network bottlenecks.
- Experimental setup: The experiments use CIFAR-10 with 50,000 training images and 10,000 test images, training for 100 epochs with stochastic gradient descent and momentum.The learning rate starts at 0.1 and decreases 10× after each 30 epochs.
- Experimental setup: The reference architectures isolate convolution-dominated and fully-connected-dominated parameter distributions, with 99.54% and 95.98% of parameters in the respective layer types.This setup tests compression under different locations of the network memory bottleneck.
- Convolutional compression: 0.8% versus 2.4% accuracy loss at 2× compression: proposed TT-convolution outperforms direct TT-decomposition of the 4-dimensional kernel.The comparison uses a convolution-dominated baseline and evaluates the proposed TT-conv against TT-conv (naive).
- Results organization: Table 1 reports CIFAR-10 compression results across rows representing different choices of TT-ranks.The second baseline compares compressing only fully-connected layers with compressing both convolutional and fully-connected parts.
- Combined compression: 80× network compression with 1.1% accuracy loss is achieved by additionally factorizing convolutions after fully-connected compression reaches about 21×.Compressing only fully-connected layers initially gives 10.72× compression with 0.2% accuracy loss.
7 Conclusion
The paper proposes tensor decomposition for convolutional-layer compression and combines it with fully-connected-layer compression to obtain 80× network compression; broader evaluation is left for future work.
- Conclusion: 80× network compression is obtained by combining the proposed convolutional approach with prior fully-connected-layer compression.The authors connect this result to embedding compressed models into smartphones for continuous sensing applications.
- Conclusion: Future work will evaluate the approach on the ILSVRC-2012 dataset and state-of-the-art neural architectures.The stated future evaluation extends beyond the CIFAR-10 experiments reported in this paper.