Source-linked AI summary
Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network
Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavsky, Valeriy Glukhov, Ivan Oseledets, Andrzej Cichocki
TL;DR
The paper addresses the computational cost of overparameterized CNNs and the instability caused by degeneracy when convolutional kernels are fitted with CPD. It introduces a stable low-rank decomposition procedure that corrects diverging components before fine-tuning. Experiments report better accuracy behavior and substantial compression across CNN architectures.
Problem
Overparameterized CNNs are expensive to deploy, while degeneracy in CPD can make decomposed networks difficult to fine-tune stably.
Method
The method approximates kernels with CPD or TKD-CPD, corrects diverging CP components using an error-preserving method, and fine-tunes the resulting network.
Results
The approach yields compression ratios of (×1.10, ×5.26) for VGG-16, (×3.82, ×3.09) for ResNet-18, and (×2.51, ×2.64) for ResNet-50 in weights and FLOPs, respectively.
Takeaways & Limitations
Across ILSVRC-12 and CIFAR-100 experiments, the paper reports stable decompositions with minimal sensitivity for CPD and hybrid TKD-CPD CNN compression.
Takeaways & Limitations
Stable fine-tuning for architectures containing higher-order CP convolutional layers is identified as a direction for future research.
Abstract
from arXiv · showhide
Most state of the art deep neural networks are overparameterized and exhibit a high computational cost. A straightforward approach to this problem is to replace convolutional kernels with its low-rank tensor approximations, whereas the Canonical Polyadic tensor Decomposition is one of the most suited models. However, fitting the convolutional tensors by numerical optimization algorithms often encounters diverging components, i.e., extremely large rank-one tensors but canceling each other. Such degeneracy often causes the non-interpretable result and numerical instability for the neural network fine-tuning. This paper is the first study on degeneracy in the tensor decomposition of convolutional kernels. We present a novel method, which can stabilize the low-rank approximation of convolutional kernels and ensure efficient compression while preserving the high-quality performance of the neural networks. We evaluate our approach on popular CNN architectures for image classification and show that our method results in much lower accuracy degradation and provides consistent performance.
1 Introduction
CNNs are costly because their convolutional and fully connected layers are heavily parameterized, motivating low-rank tensor compression. CPD is compact and effective for convolutional kernels, but standard numerical fitting can produce degeneracy that destabilizes fine-tuning; the paper proposes stable CPD and TKD-CPD alternatives.
- Modern CNN architectures contain many parameters and therefore demand high computational costs, complicating deployment on resource-limited devices.
- Low-rank tensor formats replace dense layers with sequences of smaller-kernel layers, reducing parameters and computational cost.
- CPD represents convolutional kernels as sums of rank-1 tensors and can achieve high compression because its required rank is often small.
- Standard CPD fitting can create diverging components whose large rank-1 tensors cancel each other, producing unstable tensor representations.
- For CNN compression, CPD degeneracy makes fine-tuning difficult and complicates parameter selection and stability across the network.
- The proposed method corrects degeneracy with minimal sensitivity and intensity, while TKD-CPD improves the accuracy/compression trade-off over CPD.
2 Stable Tensor Decomposition Method
The method decomposes convolutional kernels with CPD or Tucker-based CPD, then corrects degeneracy by minimizing sensitivity while preserving approximation error. Tucker factors are estimated under a bounded-error formulation using a closed-form core and eigenvalue-based factor updates.
- CP decomposition: Convolutional kernels are reshaped as order-3 tensors and represented by sums of R rank-1 tensors with factor matrices A, B, and C.The CP representation uses (D^2 + S + T) × R parameters.
- Degeneracy and sensitivity: Sensitivity measures the expected normalized squared Frobenius change caused by perturbations to individual factor matrices, and high-sensitivity CPDs are usually unusable.Its expression uses Gram matrices of the factor matrices and Hadamard products.
- Degeneracy correction: The proposed EPC variant minimizes decomposition sensitivity subject to a bounded approximation error, producing a lower-sensitivity tensor for continued CPD optimization.The error bound can represent the approximation error associated with diverging components.
- CPD optimization: Alternating updates hold two factor matrices fixed while updating the third through a bounded regression formulation solvable by quadratic programming over a sphere.When B and C have unit-length columns, the update seeks a minimum-norm weight matrix A.
- Tucker decomposition: The Tucker-2 model minimizes parameter count under an approximation-error bound, with a closed-form core and factor matrices estimated sequentially by eigenvalue decomposition.The core is G⋆ = K ×2 U^T ×3 V^T, while U and V have orthonormal columns.
3 Implementation
The compression pipeline decomposes each convolutional kernel, corrects CPD degeneracy, replaces layers with factorized blocks, and fine-tunes the complete network. Rank is selected heuristically to limit the accuracy drop after single-layer fine-tuning.
- Compression pipeline: Each convolutional kernel is approximated using CPD, TKD-CPD, or SVD according to convolution type and a specified rank.SVD is used for 1 × 1 convolutions, while CPD or TKD-CPD is used for ordinary convolutions.
- Compression pipeline: Diverging CPD components are corrected with an error-preserving method to obtain a new CP model with minimal sensitivity.This correction precedes replacement of the original convolutional layer.
- Factorized blocks: Factorized layers replace original convolutions with sequences of smaller-kernel layers containing fewer parameters, after which the entire network is fine-tuned by backpropagation.The CPD block uses 1 × 1 convolutions around a depthwise D × D group convolution.
- Factorized blocks: TKD-CPD blocks use four 1 × 1 convolutional layers when the CP rank exceeds the multilinear ranks, while smaller ranks permit merging into a CPD block.The TKD-CPD structure can additionally reduce parameters and floating-point operations.
- Rank selection: Rank is chosen by heuristic binary search for the smallest value whose single-layer fine-tuning accuracy drop stays below a predefined threshold EPS.The procedure reflects the NP-hardness of determining CP rank.
4 Experiments
Experiments evaluate CPD, CPD-EPC, and TKD-CPD-EPC across CNN architectures, datasets, layers, and full-model compression settings. Sensitivity correction and hybrid decomposition generally improve accuracy recovery while reducing computational cost or preserving compression.
- Experimental setup: The study evaluates VGG-16, ResNet-18, and ResNet-50 on ILSVRC-12 and CIFAR-100 using CPD, CPD-EPC, and TKD-CPD-EPC.Evaluation uses top-1 and top-5 accuracy after fine-tuning, including single-layer and full-network compression.
- CPD-EPC vs CPD: CPD-EPC regularly achieves considerably higher top-1 and top-5 accuracy than standard CPD across decomposition ranks.For layer4.1.conv1, CPD-EPC reaches original accuracy at rank 450, while rank 250 incurs less than 1% accuracy loss.
- CPD-EPC vs CPD: CPD-EPC suppresses model sensitivity and improves compressed-network performance, with the largest recovery on deeper CNN layers.Some deep layers show an approximately 2% top-1 accuracy difference between methods.
- CPD-EPC vs TKD-CPD-EPC: TKD-CPD-EPC quickly attains original top-1 accuracy and exceeds original top-5 accuracy when RCP ≥110 in the ResNet-18 layer4.0.conv1 experiment.At small ranks, the hybrid method performs worse than CPD-based compression, but it recovers accuracy rapidly as rank increases.
- CPD-EPC vs TKD-CPD-EPC: Across ResNet-18 CIFAR-100 layer compressions, TKD-CPD-EPC achieves higher accuracy with fewer parameters and FLOPs than CPD-EPC.The comparison covers multiple layers and represents each model by its accuracy and FLOP count.
- Full Model Compression: Full-model compression reduces weights and FLOPs by (×1.10, ×5.26) for VGG-16, (×3.82, ×3.09) for ResNet-18, and (×2.51, ×2.64) for ResNet-50.The reported compression ratios are accompanied by a moderate accuracy drop on ILSVRC-12.
5 Discussion and Conclusions
The paper develops stable CP-based compression to control sensitivity during CNN fine-tuning, extending this approach to CPD and hybrid TKD-CPD models. Experiments support the method’s reliability across ILSVRC-12 and CIFAR-100, while higher-order CP convolutional layers remain future work.
- Stable CP-format weight construction provides a direct way to control sensitivity during CNN compression and fine-tuning.The paper presents this as a stable decomposition method with minimal sensitivity for CPD and hybrid TKD-CPD.
- Experiments on ILSVRC-12 and CIFAR-100 demonstrate the proposed method’s validity and reliability for CPD and hybrid TKD-CPD compression.
- CP-format representations are used in modern architectures with 1 × 1 convolutions followed by depth-wise separable convolutions.The paper connects this structure to mapping features into a more separable higher-dimensional subspace.
- The method’s stability and sensitivity control may matter for incremental learning and multimodal tasks using shared weight factors.
- Higher-order CP convolutional architectures are identified as a promising direction for future research rather than an evaluated scope.The paper states that CPD-EPC may enable more stable fine-tuning for these architectures and leaves them for further research.