Source-linked AI summary
Wide Compression: Tensor Ring Nets
Wenqi Wang, Yifan Sun, Brian Eriksson, Wenlin Wang, Vaneet Aggarwal
TL;DR
Growing neural networks demand substantial memory, storage, and computation, limiting deployment on resource-constrained devices. The paper introduces Tensor Ring Networks to compress fully connected and convolutional layers. TRN achieves large compression factors with limited accuracy degradation, while repeated reshaping creates computational overhead.
Problem
Increasingly large neural networks require substantial memory, storage, and computation, limiting their use on resource-constrained devices.
Method
Tensor Ring Networks compress fully connected weight matrices and convolutional filters using tensor ring factorization while preserving convolutional spatial dimensions.
Results
243× compression of Wide-ResNet on Cifar10 incurred only 2.3% performance decay, while LeNet-5 was compressed by 11× without degradation.
Takeaways & Limitations
The compression scheme shows promise for deep learning on resource-constrained devices and achieves substantial compressibility with little or no benchmark image-classification performance loss.
Takeaways & Limitations
Repeated reshaping in fully connected and convolutional layers creates computational overhead, especially when the tensor-ring rank is moderately large.
Abstract
from arXiv · showhide
Deep neural networks have demonstrated state-of-the-art performance in a variety of real-world applications. In order to obtain performance gains, these networks have grown larger and deeper, containing millions or even billions of parameters and over a thousand layers. The trade-off is that these large architectures require an enormous amount of memory, storage, and computation, thus limiting their usability. Inspired by the recent tensor ring factorization, we introduce Tensor Ring Networks (TR-Nets), which significantly compress both the fully connected layers and the convolutional layers of deep neural networks. Our results show that our TR-Nets approach {is able to compress LeNet-5 by $11\times$ without losing accuracy}, and can compress the state-of-the-art Wide ResNet by $243\times$ with only 2.3\% degradation in {Cifar10 image classification}. Overall, this compression scheme shows promise in scientific computing and deep learning, especially for emerging resource-constrained devices such as smartphones, wearables, and IoT devices.
1. Introduction
Deep neural networks are becoming harder to deploy as their parameter counts and resource demands grow, especially on constrained devices. Tensor Ring Networks address this challenge by factorizing both fully connected and convolutional layers, achieving substantial compression with limited accuracy loss.
- Motivation: Model growth has increased parameter counts, while mobile, wearable, and IoT deployments face tight storage, memory, and runtime limits.These constraints make network compression important for resource-limited applications.
- Motivation: Most prior compression focused on fully connected layers, although convolutional layers account for over 99% of Wide ResNet parameters.This motivates methods that compress convolutional layers as well as fully connected layers.
- Approach: Tensor Ring factorization generalizes CP decomposition by replacing outer vector products with matrix products and connecting the first and last matrices into a ring.The formulation is also a generalization of Tensor Train factorization.
- Approach: TRN compresses fully connected weight matrices and convolutional filters while preserving spatial mask dimensions and compressing input and output channel dimensions.For fully connected layers, merge and reshape orders are investigated to reduce computation and memory needs.
- Results: 243× compression of 28-layer Wide-ResNet on Cifar10 incurred only 2.3% performance decay, yielding 92.7% accuracy.The experiments also evaluate compressed LeNet models and report strong compression with little or no accuracy degradation.
- Results: The paper reports experimental improvements in compressibility without significant performance degradation and identifies convolutional-layer compression as a key novelty.The results target state-of-the-art deep learning models for resource-constrained applications.
2. Related Work
Prior neural-network compression research includes postprocessing methods, low-rank tensor approximations, and tensor-network representations. Tensor Ring methods extend this landscape by offering a more expressive alternative to Tensor Train representations and by targeting both fully connected and convolutional layers.
- Neural-network compression: Earlier compression techniques largely targeted fully connected layers, whereas modern architectures increasingly rely on wide convolutional layers.This shift motivates compression schemes that address both layer types.
- Neural-network compression: Hashing and quantization are postprocessing methods that can be combined orthogonally with other compression schemes.Circulant projections and truncated SVD are more similar representation-based approaches.
- Tensor methods: Low-rank tensor approximation has been studied for model compression, low generative error, and fast prediction, while tensor networks support multidimensional representation and deep learning.These methods form the broader tensor-based compression context.
- Tensor methods: Tucker factorization has shown good performance in data representation and compression of fully connected and convolutional neural-network layers.Prior work applies Tucker decomposition to both layer types.
- Tensor methods: Tensor Train factorization decomposes tensors into two boundary matrices and a set of third-order tensors.For multidimensional data completion, Tensor Ring can be more expressive than Tensor Train at the same intermediate rank.
3. Tensor Ring Nets (TRN)
Tensor Ring Nets factorize fully connected and convolutional layers into tensor-ring components, reducing parameters and computation while preserving a structured representation. The method uses merge-order choices and layer-specific decompositions to balance compression, memory, and runtime costs.
- Tensor ring factorization: A tensor ring decomposition factors a tensor into d connected 3-mode tensors with shared ring rank R.The construction operator merges the factors and sums over the two outer ring modes to recover the original tensor.
- Tensor ring factorization: R2 Σ_i I_i parameters replace the original tensor's Π_i I_i entries under the low-rank factorization.The reduction is substantial when the tensor dimensions are large and the ring rank is small.
- Merge ordering: Hierarchical merging is used because different merge orders change flop counts, while the worst choice is at most 2x more expensive than the best.When the number of modes is a power of 2, the hierarchical order achieves the minimum flop count.
- Fully connected layer compression: Fully connected TRNs decompose the weight tensor and separate the resulting computation into low-rank feed-forward operations without constructing the full weight matrix.The separated operations reduce computation when the ring rank R is smaller than the input and output dimensions.
- Convolutional layer compression: Convolutional TRNs factorize kernel channel dimensions while preserving spatial dimensions, then implement the convolution through tensor multiplications and a central spatial convolution.The channel factors can be further decomposed for greater storage compression, but this introduces a storage–computation trade-off.
4. Experiments
Experiments evaluate TRN compression on fully connected and convolutional networks across MNIST, Cifar10, and Cifar100. TRNs achieve substantial compression with small accuracy or performance losses, including strong results on WideResNet.
- Fully connected layer compression: 13× compression on LeNet-300-100 incurs only 0.14% accuracy loss on MNIST.
- Convolutional layer compression: 11× compression on LeNet-5 produces a lower error rate than the original model and Tucker factorization.The compression uses tensor ring rank r = 20.
- ResNet and WideResNet compression: TRNs significantly compress both ResNet and WideResNet on Cifar10 and Cifar100.For ResNet with r = 10, compression matches Tucker’s ratio while improving performance by almost 3% on Cifar10 and almost 10% on Cifar100.
- ResNet and WideResNet compression: 243× compression on WideResNet gives only a 2% performance decay on Cifar10 relative to uncompressed WideResNet.At 243× compression, TRN WideResNet also outperforms uncompressed ResNet on Cifar10 with fewer parameters.
- ResNet and WideResNet compression: TRN compression trades training speed for memory savings: Tucker trains about 2 or 3 times faster than TRNs on WideResNet.The paper notes that this tradeoff may remain desirable for memory-constrained devices.
5. Conclusion
The paper introduces tensor ring factorization as a compression approach for deep neural networks on resource-limited devices. It achieves substantial compression with little or no performance loss on benchmark image-classification tasks, while computational complexity remains a future-work concern.
- Tensor ring factorization compresses deep neural networks for resource-limited devices.
- Significant compressibility is achieved with little or no performance loss on benchmark image-classification tasks.
- Repeated reshaping in fully connected and convolutional layers creates computational overhead, especially when r is moderately large.
- Reducing computational complexity is identified as an area for future work.
Appendix A. Merge ordering
The appendix analyzes how tensor merge order affects computation and storage. Hierarchical merging reduces flop cost relative to sequential merging, while total costs remain bounded under the stated assumptions.
- Merge ordering determines the total flop count and memory needs for constructing merged tensors.
- Hierarchical merging uses fewer flops than sequential merging for the 4-factor example.For d = 4, sequential merging costs 2I1I2R3 + 2I1I2I3R3 + 2I1I2I3I4R2 flops, whereas hierarchical merging costs 2I1I2R3 + 2I3I4R3 + 2I1I2I3I4R2.
- The merge-ordering analysis represents each merging strategy as a binary tree whose leaves correspond to the individual tensors.
- The total computational cost lies between 2R3I and 4R3I.
- Storage cost is bounded between R2I and 2R2I when the reshaped dimension satisfies the stated condition.
- For d that is a power of 2, a fully balanced hierarchical merge is optimal in flop count.The induction argument establishes that sequential merging always costs more than hierarchical merging under the stated assumptions.
Appendix B. Initialization
The initialization analysis examines how variance changes when tensor-ring factors are multiplied and merged. Products of independent zero-mean factors remain zero-mean, while their variance grows with tensor-ring rank and the number of merged factors.
- A product of two independent zero-mean symmetric variables has variance σ4 and is not Gaussian distributed.
- After merging d tensor-ring factors, each tensor element has mean 0 and variance Rdσ2d.