Source-linked AI summary
Low-Rank+Sparse Tensor Compression for Neural Networks
Cole Hawkins, Haichuan Yang, Meng Li, Liangzhen Lai, Vikas Chandra
TL;DR
Modern vision networks already build in factorized weights, complicating the use of low-rank tensor compression to reduce the memory cost of neural networks. The paper combines low-rank tensor decomposition with sparse pruning and finds the combination competitive with sparse pruning across the tested models, without significant gains over that baseline.
Problem
Modern vision backbones already use factorized weights, leaving it an open question how low-rank tensor compression can contribute to these architectures.
Source: §1 · p. 1Method
The paper combines low-rank tensor weights with sparse pruning, reconstructing weights from a low-rank component and a masked sparse residual.
Source: §3.1 · p. 5Results
Across MobileNet-v3, EfficientNet-B0, and DEIT, low-rank-plus-sparse decomposition is competitive with sparse pruning but does not significantly outperform it.
Source: §4.3 · p. 7, §4.2 · p. 7Takeaways & Limitations
Many layers in the tested pretrained models are not approximately low-rank, making direct tensor decomposition challenging for these efficient architectures.
Source: §4.4 · p. 7Many fully connected layers and 1x1 convolutions are not approximately low-rank, even after reshaping from matrix to tensor format.
Source: §4.4 · p. 7
Abstract
from arXiv · showhide
Low-rank tensor compression has been proposed as a promising approach to reduce the memory and compute requirements of neural networks for their deployment on edge devices. Tensor compression reduces the number of parameters required to represent a neural network weight by assuming network weights possess a coarse higher-order structure. This coarse structure assumption has been applied to compress large neural networks such as VGG and ResNet. However modern state-of-the-art neural networks for computer vision tasks (i.e. MobileNet, EfficientNet) already assume a coarse factorized structure through depthwise separable convolutions, making pure tensor decomposition a less attractive approach. We propose to combine low-rank tensor decomposition with sparse pruning in order to take advantage of both coarse and fine structure for compression. We compress weights in SOTA architectures (MobileNetv3, EfficientNet, Vision Transformer) and compare this approach to sparse pruning and tensor decomposition alone.
1 INTRODUCTION
The paper targets memory costs from storing multiple task-specific models on edge devices, and asks whether low-rank compression can help modern vision architectures that already use factorized weights. It studies combining low-rank tensor structure with sparse pruning and examines remaining low-rank structure in these networks.
1 INTRODUCTION
- Multiple task-specific models for edge applications make memory reduction important because off-chip reads incur high latency and power costs.The paper therefore targets memory cost reduction.Source: §1 · p. 1
- Modern vision backbones already use factorized weights, motivating the question of how tensor compression can contribute to these architectures.The paper investigates combining low-rank tensor compression and sparse pruning to capture coarse and fine structure, respectively.Source: §1 · p. 1
- The proposed alternatives are additive low-rank-plus-sparse weights or weights represented by either a sparse pruned form or a low-rank form.The introduction identifies the two forms of decomposition but does not describe their implementation in detail.Source: §1 · pp. 1–2
1.1 RELATED WORK
- Prior work combined low-rank matrices and sparse representations for VGG-style models and ResNets, while low-rank compression was also studied for EfficientNet on CIFAR-10/100.Whether low-rank structure remains in efficient modern networks trained on ImageNet was an open question.Source: §1.1 · p. 2
1.2 CONTRIBUTIONS
- The paper studies low-rank structure in recent efficient vision networks and, to the authors’ knowledge, is the first to combine low-rank tensor compression with sparse pruning for these architectures.The authors report that architecture trends have removed low-rank structure by building in factorized weights, making direct low-rank compression challenging.Source: §1.2 · p. 2
2 BACKGROUND
The background introduces tensor decompositions, reshaping weights for tensor compression, global magnitude pruning, and distillation-based training. Together, these sections describe the representations and training components used in the compression pipeline.
2.1 TENSOR FORMATS
- A tensor is a multi-way data array, and the paper uses CP, Tucker, TT, and TTM decompositions to reduce neural-network parameters.The motivation is that representing a tensor directly requires a number of entries that grows exponentially with its order.Source: §2.1 · p. 2
- CP represents a tensor as a sum of rank-1 outer products, collected into factor matrices.The minimum number of such terms is the CP rank.Source: §2.1 · pp. 2–3
- Tucker represents a tensor through mode-wise products of a small core and factor matrices, allowing richer cross-dimension interactions than CP.Its parameter reductions may be limited by exponential dependence on tensor order.Source: §2.1 · p. 3
- TT and TTM support more expressive interactions without Tucker’s exponential dependence on tensor order; TTM applies the train structure to a reshaped matrix.TT factorizes a tensor through products of slices from its cores; the TTM form uses matrix products for an order-2d tensor.Source: §2.1 · pp. 3–4
2.2 FOLDING TO HIGH-ORDER TENSORS.
- The paper folds matrix weights and reshapes convolution filters into higher-order tensors so CP and Tucker can be applied to matrix-format weights.Reshaping imposes additional higher-order and hierarchical tensor structure.Source: §2.2 · p. 4
2.3 SPARSE PRUNING
- Sparse pruning applies a binary mask to weights and gradually zeros low-magnitude entries under a global magnitude-pruning schedule.The schedule increases sparsity until the specified final ratio is reached.Source: §2.3 · p. 4
2.4 DISTILLATION
- Distillation trains the compressed student against a pretrained ImageNet teacher using a standard loss plus a temperature-scaled KL-divergence term.The paper uses cross-entropy, α = 0.9, and T = 3; the KL term penalizes differences between student and teacher output distributions.Source: §2.4 · pp. 4–5
3 METHOD: LOW-RANK + SPARSE TENSOR DECOMPOSITION
The method compresses a trained network through low-rank-plus-sparse tensor decomposition followed by iterative pruning and fine-tuning. It combines additive or masking reconstruction with initialization strategies that guide low-rank factors toward structure suited to pruning.
3 METHOD: LOW-RANK + SPARSE TENSOR DECOMPOSITION
- The pipeline decomposes each layer, then iteratively prunes sparse residuals and fine-tunes the network.It takes a trained network as input and uses global magnitude pruning with a target sparsity during training.Source: §3 · p. 5
3.1 RECONSTRUCTION
- Additive reconstruction forms each weight tensor from a low-rank component and a masked sparse residual, with backpropagation updating both.Low-rank weights are stored in factorized form; the sparse residual retains unpruned entries.Source: §3.1 · p. 5
- Masking reconstruction assigns each weight entry to either the low-rank tensor or sparse tensor, avoiding overlap between their contributions.The authors propose it because overlapping contributions waste representational capacity; low-rank factors can instead fit entries where the pruned sparse tensor cannot contribute.Source: §3.1 · p. 5
3.2 INITIALIZATION
- Residual initialization fits the low-rank tensor to the original weights, exactly reconstructing them before pruning begins.The approach does not account for entries that will later be pruned; robust-PCA-style alternatives were tried without clear gains, so residual initialization is used with additive reconstruction.Source: §3.2 · p. 6
- Masking initialization fits the low-rank decomposition only to the bottom-magnitude entries, excluding high-magnitude weights intended to remain unpruned.The method masks the top-K% of entries and pairs this initialization with masking reconstruction.Source: §3.2 · p. 6
4 EXPERIMENTS
On ImageNet-1K, LR+S is competitive with sparse pruning across the tested model families but does not significantly outperform it; weight analyses show uneven low-rank structure, and tensor-only decomposition struggles to match accuracy.
4 EXPERIMENTS
- Experiments used ImageNet-1K validation accuracy, 50 training epochs, and tensor ranks requiring low-rank factors with 10% as many elements as the original tensor.Each method used batch size 256 and distillation settings α = 0.9, T = 3.Source: §4 · p. 6
4.1 RECONSTRUCTION AND INITIALIZATION
- Residual initialization with additive reconstruction showed no clear gain over masking initialization with masking reconstruction, so the simpler residual strategy was used thereafter.Figure 3 compares the two strategies on MobileNet-v3-Small and MobileNet-v3-Large.Source: §4.1 · p. 7, Figure 3 · p. 7
4.2 MOBILENETV3 RESULTS
- Across MobileNet-v3, EfficientNet-B0, and DEIT-Small, LR+S was competitive with sparse pruning but did not significantly outperform it.The comparison reports ImageNet top-5 accuracy across CP, Tucker, and TT formats.Source: §4.3 · p. 7, §4.2 · p. 7, Figure 4 · p. 8
4.4 EFFECT ON MODEL WEIGHTS
- Many MobileNet-v3-Small and EfficientNet-B0 layers had high low-rank relative error, especially pointwise convolutions, while several layers fit almost exactly.Figure 5 reports layer-wise relative error for tensor-train factorization; other tensor types are described as similar.Source: §4.4 · pp. 7, 9, Figure 5 · p. 9
- Factorization made spatial-convolution weights easier to prune, but captured less pointwise-convolution structure and did not make those weights more prune-friendly.The authors describe this as removing coarse structure before sparse pruning.Source: §4.4 · p. 9, Figure 6 · p. 10
4.5 LOW-RANK ONLY
- Post-training tensor-only decomposition was difficult to fine-tune to similar accuracy; longer training (>50 epochs) or low-rank-format training from scratch might resolve this.Figure 7 compares ImageNet top-5 accuracy for sparse pruning and low-rank decomposition on MobileNet-v3-Small and MobileNet-v3-Large.Source: §4.5 · p. 9, Figure 7 · p. 11