PaperlayerView exact revision on arXiv

Source-linked AI summary

Ultimate tensorization: compressing convolutional and FC layers alike

Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, Dmitry Vetrov

arXiv:1611.03214v1cs.LG

TL;DR

CNNs are computationally and memory intensive, motivating compression beyond earlier fully-connected-layer methods. The paper reshapes convolutional kernels into higher-order tensors and factorizes them with matrix TT-format, combining this with fully-connected compression to obtain 80× network compression with a 1.1% accuracy drop on CIFAR-10.

  • Problem

    CNNs require substantial computation, storage, memory bandwidth, and energy, while prior tensor-factorization work addressed fully-connected rather than convolutional layers.

    Source: §1 · p. 1

  • Method

    The paper reshapes each convolutional kernel into a matrix and then a higher-order tensor before applying matrix TT-format, combining this with fully-connected-layer compression.

    Source: §4 · p. 3

  • Results

    80× network compression with a 1.1% accuracy drop was achieved by compressing both convolutional and fully-connected layers on CIFAR-10.

    Source: §6 · p. 5, §7 · p. 5

  • Takeaways & Limitations

    The combined approach moves toward embedding compressed neural networks in smartphones for continuous surrounding-sensing applications.

    Source: §7 · p. 5

    The paper identifies future evaluation on the ILSVRC-2012 dataset and state-of-the-art neural architectures as remaining work.

    Source: §7 · p. 5

Abstract

from arXiv · show

Convolutional neural networks excel in image recognition tasks, but this comes at the cost of high computational and memory complexity. To tackle this problem, [1] developed a tensor factorization framework to compress fully-connected layers. In this paper, we focus on compressing convolutional layers. We show that while the direct application of the tensor framework [1] to the 4-dimensional kernel of convolution does compress the layer, we can do better. We reshape the convolutional kernel into a tensor of higher order and factorize it. We combine the proposed approach with the previous work to compress both convolutional and fully-connected layers of a network and achieve 80x network compression rate with 1.1% accuracy drop on the CIFAR-10 dataset.

1 Introduction

CNNs deliver strong performance but impose substantial computation, memory, and energy demands. This paper proposes tensor-factorization methods for compressing convolutional layers alongside previously compressed fully-connected layers.

Motivation

  • Millions of operations, parameters, and hundreds of megabytes of storage and bandwidth make CNN deployment on mobile devices difficult.Using RAM rather than processor cache further increases energy consumption.Source: §1 · p. 1

Prior work

  • Tensor decomposition previously compressed fully-connected layers, shifting VGG-16’s storage bottleneck toward convolutional layers.The earlier approach represented fully-connected parameters in Tensor Train format and learned the network from scratch in that representation.Source: §1 · p. 1

Contributions

  • The paper proposes tensor factorization for convolutional layers and reshapes their kernels into higher-order tensors to use Tensor Train compression more effectively.Direct Tensor Train decomposition of a convolutional tensor produces poor results according to the paper’s experiments.Source: §1 · p. 1
  • 4× compression of a convolution-only network incurred a 2% accuracy decrease.This result concerns the proposed approach applied to a network consisting only of convolutional layers.Source: §1 · p. 1
  • 82× network compression with a 1% accuracy drop resulted from combining convolutional compression with fully-connected-layer compression.The combined result is reported as an experimental contribution of the paper.Source: §1 · p. 2

2 Convolutional Layer

A convolutional layer maps an input tensor to an output tensor by applying a kernel across spatial patches and channels. The paper rewrites this operation as matrix multiplication by reshaping outputs, input patches, and the kernel.

Convolutional Layer

  • A convolutional layer transforms X ∈ R^(W×H×C) into Y ∈ R^((W−ℓ+1)×(H−ℓ+1)×S) using K ∈ R^(ℓ×ℓ×C×S).The basic convolution is used for clarity, although the compression approach also works with padding, larger strides, and rectangular filters.Source: §2 · p. 2

Matrix formulation

  • The output tensor is reshaped into a matrix with W′H′ rows and S columns, where W′ = W − ℓ + 1 and H′ = H − ℓ + 1.Each output-matrix row corresponds to one spatial output location.Source: §2 · p. 2
  • The input is reshaped into a W′H′ × ℓ^2C matrix whose k-th row contains the input patch used to compute the k-th output row.The patch contains ℓ × ℓ spatial positions across C channels.Source: §2 · p. 2
  • The kernel is reshaped into an ℓ^2C × S matrix by flattening spatial and channel indices.This aligns each flattened input patch with the kernel weights producing the S output channels.Source: §2 · p. 2
  • With these reshaped matrices, convolution becomes the matrix product Y = XK.The transformation exposes convolution as a matrix-by-matrix multiplication suitable for subsequent factorization.Source: §2 · p. 2, Figure 1 · p. 2

3 Tensor Train Decomposition

Tensor Train decomposition represents a tensor through interconnected matrix-valued cores and TT-ranks. Its rank choices control the memory–accuracy trade-off, while matrix TT-format reshapes matrices into higher-order tensors before decomposition.

TT representation

  • A tensor’s TT-representation consists of matrix-valued cores G_k[j_k] whose products represent individual tensor elements.The endpoint ranks are fixed at r_0 = r_d = 1.Source: §3 · pp. 2–3
  • The intermediate values r_k are TT-ranks, while the collections of matrices indexed by j_k are TT-cores.These components define the compact tensor representation used by the method.Source: §3 · p. 3

Compression trade-off

  • The TT-format requires a number of elements determined by the tensor dimensions and can perform linear algebra on TT-cores without materializing the full tensor.This supports memory-efficient tensor operations.Source: §3 · p. 3
  • Smaller TT-ranks reduce memory usage but trade off representation accuracy.The TT-format therefore exposes an explicit parameter-count versus accuracy trade-off.Source: §3 · p. 3

Matrix TT-format

  • For matrices, matrix TT-format reshapes row and column indices through bijective mappings into a higher-order tensor before applying TT-representation.The matrix dimensions are factorized into products of mode sizes, allowing the matrix TT-format to be more compact than ordinary matrix low-rank format.Source: §3 · p. 3

4 TT-convolutional Layer

The paper develops TT-convolution by reshaping convolutional kernels into higher-order tensors before applying matrix TT-format, preserving the fully-connected-layer case for 1 × 1 convolutions.

Kernel representation

  • Direct TT-decomposition of a 4-dimensional convolutional kernel performs poorly, motivating a reshaping strategy that better uses TT compression.The paper illustrates this limitation with 1 × 1 convolutions, where direct decomposition becomes ordinary matrix low-rank decomposition.Source: §4 · p. 3
  • The 4-dimensional kernel is reshaped into a matrix of size ℓ2C × S and then into a tensor whose dimensions factorize the channel dimensions.The resulting tensor has dimension lengths C_kS_k for channel-output factors and ℓ2 for the spatial factor.Source: §4 · p. 3
  • The reshaped kernel is represented as a product of TT cores, with the spatial indices assigned to the first core and channel indices assigned to subsequent cores.The decomposition is expressed through G0[x, y]G1[c1, s1]...Gd[cd, sd].Source: §4 · p. 3

Layer pipeline

  • A TT-convolution layer reshapes the input into a tensor with factored channel dimensions, applies the TT transformation, and produces an output tensor with factored output channels.The TT-core elements are trained directly using stochastic gradient descent with momentum and automatic differentiation.Source: §4 · pp. 3–4

Kernel representation

  • For 1 × 1 convolutions, the spatial indices vanish, and the decomposition coincides with the Tensor Train format previously proposed for fully-connected layers.This provides a unified representation across convolutional and fully-connected layers.Source: §4 · p. 4

5 Related Work

Prior work compressed fully-connected layers and convolutional layers using pruning, quantization, coding, and tensor or matrix decompositions; the proposed approach is compatible with some of these techniques.

Compression motivation

  • Fully-connected layers have traditionally been treated as memory bottlenecks, but compression can shift the bottleneck to convolutional layers.This shift motivates methods specifically targeting convolutional computation and storage.Source: §5 · p. 4

Weight-based methods

  • Pruning removes less important weights, while quantization restricts weight variation; both are established approaches for compressing convolutional kernels.The proposed method can quantize TT-core elements and can be combined with Huffman coding.Source: §5 · p. 4

Decomposition methods

  • CP-decomposition and Kronecker product factorization compress convolutional networks while also enabling faster inference.These methods use tensor or matrix decompositions as alternatives to weight pruning and quantization.Source: §5 · p. 4

6 Experiments

Experiments on CIFAR-10 compare TT-convolution with a naive kernel decomposition and evaluate combined convolutional and fully-connected compression under different network bottlenecks.

Experimental setup

  • The experiments use CIFAR-10 with 50,000 training images and 10,000 test images, training for 100 epochs with stochastic gradient descent and momentum.The learning rate starts at 0.1 and decreases 10× after each 30 epochs.Source: §6 · p. 4
  • The reference architectures isolate convolution-dominated and fully-connected-dominated parameter distributions, with 99.54% and 95.98% of parameters in the respective layer types.This setup tests compression under different locations of the network memory bottleneck.Source: §6 · p. 4

Convolutional compression

  • 0.8% versus 2.4% accuracy loss at 2× compression: proposed TT-convolution outperforms direct TT-decomposition of the 4-dimensional kernel.The comparison uses a convolution-dominated baseline and evaluates the proposed TT-conv against TT-conv (naive).Source: §6 · pp. 4–5

Results organization

  • Table 1 reports CIFAR-10 compression results across rows representing different choices of TT-ranks.The second baseline compares compressing only fully-connected layers with compressing both convolutional and fully-connected parts.Source: Table 1 · p. 5, §6 · p. 5

Combined compression

  • 80× network compression with 1.1% accuracy loss is achieved by additionally factorizing convolutions after fully-connected compression reaches about 21×.Compressing only fully-connected layers initially gives 10.72× compression with 0.2% accuracy loss.Source: §6 · p. 5

7 Conclusion

The paper proposes tensor decomposition for convolutional-layer compression and combines it with fully-connected-layer compression to obtain 80× network compression; broader evaluation is left for future work.

Conclusion

  • 80× network compression is obtained by combining the proposed convolutional approach with prior fully-connected-layer compression.The authors connect this result to embedding compressed models into smartphones for continuous sensing applications.Source: §7 · p. 5
  • Future work will evaluate the approach on the ILSVRC-2012 dataset and state-of-the-art neural architectures.The stated future evaluation extends beyond the CIFAR-10 experiments reported in this paper.Source: §7 · p. 5

Related papers

Loading 1611.03214v1…