Source-linked AI summary
Tensorizing Neural Networks
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, Dmitry Vetrov
TL;DR
Fully-connected layers make neural networks resource-intensive because their dense weights can consume substantial memory. The paper replaces those matrices with Tensor Train representations, reporting large parameter reductions while retaining counterpart performance. For Very Deep VGG networks, the reported whole-network compression factor reaches up to 7.
Problem
Fully-connected-layer weights can occupy most of modern neural networks' memory, making compact representations important for reducing resource demands.
Method
The paper stores fully-connected-layer weights as Tensor Train cores and trains the compressed layers directly with respect to those cores.
Results
TT-layers match uncompressed counterparts with up to 200 000 times fewer parameters, while whole-network compression reaches up to 7.
Takeaways & Limitations
The compact representations support much wider layers and can capture complex data dependencies with fewer parameters.
Takeaways & Limitations
The ImageNet results use a slightly outdated VGG-16 version and the up-to-date VGG-19 version.
Abstract
from arXiv · showhide
Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size. In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved. In particular, for the Very Deep VGG networks we report the compression factor of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression factor of the whole network up to 7 times.
1 Introduction
Deep neural networks achieve state-of-the-art performance but require substantial computational resources, especially for fully-connected layers. The paper targets these layers by compressing their dense weight matrices while preserving performance.
- Fully-connected layers apply linear transformations between high-dimensional inputs and outputs using large dense matrices.
- The paper converts fully-connected-layer weight matrices to the Tensor Train format to reduce parameters while preserving expressive power.
- TT-layers match uncompressed counterparts while using up to 200 000 times fewer parameters.
- The whole-network compression factor reaches up to 7.
2 Related work
Related work addresses neural-network resource demands through compact weight representations and reduced numerical precision. These approaches exploit redundancy or constrain how weights are stored.
- 89% or 100% of memory can be occupied by fully-connected-layer weights in modern networks.
- Low-rank representations reduce parameters by restricting the rank of fully-connected-layer weight matrices without significant predictive-accuracy drops.
- Hashing methods tie random subsets of weights and reportedly achieve a compression factor of 8 on a two-layer MNIST network without accuracy loss.
- Other compression strategies reduce memory through lower numerical precision or fewer carefully chosen numerical values.
3 TT-format
The Tensor Train format represents multidimensional arrays through structured core matrices and TT-ranks. Specialized vector and matrix mappings extend this representation to compact neural-network parameters and efficient operations.
- 3 TT-format: The paper distinguishes vectors, matrices, tensors, and their elements according to array dimensionality and notation.
- 3 TT-format: A tensor is in TT-format when each element is computed as a product of indexed core matrices.
- 3 TT-format: All core matrices associated with dimension k share size r_k−1 × r_k, with boundary ranks r_0 and r_d equal to 1.
- 3 TT-format: TT-ranks determine representation structure, while the core collections are three-dimensional arrays indexed by tensor dimensions.
- 3 TT-format: Explicit storage enumerates all tensor elements, whereas TT-storage uses Σ_k n_k r_k−1 r_k numbers and is memory-efficient when ranks are small.
- 3.1 TT-representations for vectors and matrices: A TT-vector reshapes vector coordinates into multidimensional tensor indices through a bijection and stores the resulting tensor in TT-format.
- 3.1 TT-representations for vectors and matrices: A TT-matrix reshapes row and column indices into paired tensor dimensions, then represents the resulting tensor through indexed TT-cores.
- 3.1 TT-representations for vectors and matrices: For an explicit vector, TT-matrix multiplication has complexity O(d r^2 m max{M, N}), but efficiency depends on the chosen index mappings.
4 TT-layer
The TT-layer stores a fully connected layer’s weight matrix in Tensor Train format, expressing its linear transformation through tensorized inputs, TT-cores, and an output bias. This representation controls parameter count through tensor dimensions and TT-ranks while retaining an efficient forward computation.
- TT-layer construction: The TT-layer is a fully connected layer whose weight matrix is stored in TT-format.Networks containing one or more such layers are called TensorNets.
- TT-layer construction: A TT-layer stores the fully connected layer’s weight matrix W in TT-format and transforms tensorized input X into output tensor Y with bias B.The transformation uses TT cores Gk[ik, jk] instead of explicitly storing W.
- Parameter control: The number of parameters can be controlled by varying the number of hidden units and the TT-ranks of the weight matrix.The input and output vectors are reshaped into d-dimensional tensors before applying the layer transformation.
5 Learning
The learning procedure derives back-propagation for TT-layers, including gradients with respect to inputs, biases, and TT-cores. It replaces explicit weight-gradient storage with direct core-gradient computation and provides complexity bounds for forward and backward operations.
- Back-propagation: Back-propagation computes gradients sequentially through reversed network layers, starting from the loss gradient with respect to the final output.The procedure computes gradients with respect to both network parameters and intermediate quantities.
- Back-propagation: Gradients with respect to the bias and input use the standard fully connected formulas, with the input gradient evaluated through a TT-matrix-by-vector product.The input-gradient operation has complexity O(dr^2n max{m, n}^d) = O(dr^2n max{M, N}).
- Core learning: Directly computing gradients with respect to TT-cores avoids the O(MN) memory required to form the full weight-matrix gradient.An alternative procedure converts the full gradient to TT-format using TT-SVD before updating the weights.
- Core learning: The gradient of the loss with respect to a TT-core is obtained using the chain rule and partial products surrounding that core.The derivation considers the dependence of the output on one core slice at a time.
- Core learning: For a selected core index, terms with mismatched input or output indices have zero dependence and can be omitted from the gradient calculation.This restriction reduces the summation to terms whose relevant indices match the selected core slice.
- Core learning: The required partial products can be computed by dynamic programming, and the overall backward-pass complexity is O(d^2r^4m max{M, N}).The resulting algorithm consists of matrix-by-matrix products and dimension permutations, which can be accelerated on a GPU.
6 Experiments
Experiments evaluate TT-layers on MNIST, CIFAR-10, ImageNet, and wide shallow networks, showing substantial parameter reductions with competitive or improved error in several settings.
- MNIST: 1.6% test error is achieved with 12 602 parameters, compared with HashedNet’s 2.79% error using 12 720 parameters.
- CIFAR-10: 23.13% CIFAR-10 test error slightly improves on the 23.25% baseline with 3 125 hidden units and 4 160 TT-layer parameters.
- CIFAR-10: 24.39% CIFAR-10 test error results when both fully-connected layers are replaced by TT-layers, reducing fully-connected-layer parameters 11.9 times and total parameters 1.7 times.
- Wide and shallow network: 31.47% test error is reported for a wide and shallow TensorNet, described as the best result achieved by a non-convolutional neural network.
- Inference: 392MB versus 0.766MB of memory is used for one-image feedforward by the fully-connected layer and corresponding TT-layer, respectively.
- ImageNet: 194 622-fold reduction in the largest VGG fully-connected weight matrix increases top 5 error from 11.2 to 11.5; compressing another layer raises whole-network compression to 7.4.
7 Discussion and future work
The discussion presents TT-layers as compact alternatives that can preserve performance, support wider networks, and potentially improve inference efficiency for real-time and mobile use.
- TT-decomposition reduces fully-connected-layer parameters by up to 200 000× without significant error increase.
- TensorNets can use much wider layers than previously available, with promising preliminary CIFAR-10 results for wide and shallow models.
- The TT-layer is reported to have faster inference time than the corresponding fully-connected layer.