PaperlayerView exact revision on arXiv

Source-linked AI summary

Compressing deep neural networks by matrix product operators

Ze-Feng Gao, Song Cheng, Rong-Qiang He, Z. Y. Xie, Hui-Hai Zhao, Zhong-Yi Lu, Tao Xiang

arXiv:1904.06194v2cs.LGcs.CVcs.NEphysics.comp-phquant-ph

TL;DR

Deep networks concentrate most of their trainable parameters in linear transformations, making parameter reduction without sacrificing prediction power a central challenge. The paper represents these transformations with MPOs and tests the approach across five network types and two datasets, reporting comparable or improved accuracy with fewer parameters.

  • Problem

    Deep networks’ linear transformations contain most trainable parameters, and reducing them without losing prediction power is challenging for training efficiency and overfitting risk.

    Source: §I · p. 1

  • Method

    The paper uses MPOs to represent neural-network linear transformations, motivated by localized correlations in image information.

    Source: §I · p. 2, §IV · pp. 5–6

  • Results

    Across five network types and two datasets, MPO representations use much fewer parameters while maintaining or slightly improving test accuracy.

    Source: §IV · p. 6

  • Takeaways & Limitations

    MPOs can represent fully connected and convolutional layers as linear mappings alongside parameter-free nonlinear activations, simplifying the network representation.

    Source: §IV · p. 6

    The approach is expected to work efficiently where locality is well defined; otherwise, a large bond dimension may reduce its efficiency and advantage.

    Source: §IV · p. 6

Abstract

from arXiv · show

A deep neural network is a parametrization of a multilayer mapping of signals in terms of many alternatively arranged linear and nonlinear transformations. The linear transformations, which are generally used in the fully connected as well as convolutional layers, contain most of the variational parameters that are trained and stored. Compressing a deep neural network to reduce its number of variational parameters but not its prediction power is an important but challenging problem toward the establishment of an optimized scheme in training efficiently these parameters and in lowering the risk of overfitting. Here we show that this problem can be effectively solved by representing linear transformations with matrix product operators (MPOs), which is a tensor network originally proposed in physics to characterize the short-range entanglement in one-dimensional quantum states. We have tested this approach in five typical neural networks, including FC2, LeNet-5, VGG, ResNet, and DenseNet on two widely used data sets, namely, MNIST and CIFAR-10, and found that this MPO representation indeed sets up a faithful and efficient mapping between input and output signals, which can keep or even improve the prediction accuracy with a dramatically reduced number of parameters. Our method greatly simplifies the representations in deep learning, and opens a possible route toward establishing a framework of modern neural networks which might be simpler and cheaper, but more efficient.

I. Introduction

Linear transformations contain most trainable parameters in deep networks, making training and storage costly and potentially increasing overfitting risk. The paper proposes MPO representations, motivated by localized correlations, and distinguishes its approach by retaining neural-network layers and activations while studying multiple architectures and datasets.

I. Introduction

  • Linear mappings contain most trainable parameters, while their large total can make training and storage challenging and may increase overfitting risk.These costs can limit generalization and applications with constrained storage, such as mobile terminals.Source: §I · p. 1
  • MPOs factor higher-order tensors into local tensors and can reduce parameter growth from exponential to linear with system size.The representation originated in physics for short-range entanglement and is described as a tensor-train approximation.Source: §I · p. 2
  • MPOs are proposed for neural-network linear transformations because input correlations are localized and convolutional features arise from small local clusters.The paper notes that fully connected weight matrices can have restricted rank because of short-range correlations.Source: §I · p. 2
  • Unlike prior tensor-network applications, this approach retains neural-network layers, activations, and neural-network training algorithms while representing fully connected and convolutional linear transformations.The paper also studies five network families across MNIST and CIFAR-10.Source: §I · p. 2

II. Method

The method replaces selected weight matrices with MPO factorizations of reshaped input and output indices. A tunable bond dimension controls expressive power, while the resulting parameters are trained with a regularized objective and standard neural-network optimization.

II. Method

  • The method reshapes weight matrices into indexed tensors and factorizes them into a product of local MPO tensors.It applies to all fully connected layers and some of the heaviest convolutional layers.Source: §II · pp. 2–3
  • Bond dimension D controls MPO expressive power, and replacing dense matrices can greatly reduce the number of variational parameters.The index decomposition is non-unique; the reported results use a convenient ordered reshaping.Source: §II · p. 3
  • Training minimizes a cost combining cross-entropy with L2 regularization over MPO tensors and any untouched convolutional kernels.The paper says MPO representations should remain valid without L2 regularization.Source: §II · p. 3
  • All parameters are updated in parallel using standard backpropagation and stochastic gradient descent with momentum until the cost stops decreasing.The parameters are randomly initialized before each training step.Source: §II · p. 3

III. Results

The experiments cover five typical network families across MNIST and CIFAR-10, and the paper defines how MPO compression and repeated-run test accuracy are assessed. The supplied passages identify the FC2 comparison baseline but do not report the experiment’s comparative outcomes.

III. Results

  • The study evaluates five typical network families across MNIST and CIFAR-10, spanning relatively shallow and deeper architectures.The supplied passage names FC2, LeNet-5, VGG, ResNet, and DenseNet.Source: §III · p. 3
  • MPO-Net denotes a network with all or some linear layers represented by MPOs, and its compression ratio compares parameter counts in replaced layers.A smaller compression ratio means fewer parameters in the MPO representation.Source: §III · p. 3
  • The FC2 figure uses normal FC2 test accuracy of 98.35% ± 0.2% as its reference line.The study reports accuracy as a mean and standard deviation over five independent training runs.Source: Figure 2 · p. 4, §III · p. 4

A. MNIST dataset

The MNIST evaluation uses handwritten-digit images divided into ten classes, with separate training and test sets.

A. MNIST dataset

  • MNIST comprises 60,000 training and 10,000 test images of 28×28 grayscale pixels, divided into ten digit classes.Source: A. MNIST dataset · p. 4

1. FC2

In FC2, MPO test accuracy rises with bond dimension and matches normal FC2 at D = 16; MPO-Net also faithfully reproduces LeNet-5 test accuracy.

1. FC2

  • FC2's two fully connected weight matrices are replaced by MPOs, with the second layer's bond dimension fixed at 4 and the first varied.The matrices have 784 × 256 and 256 × 10 elements, respectively.Source: §1 · p. 4

2. LeNet-5

  • MPO-Net faithfully reproduces LeNet-5 test accuracy.Table I reports test accuracy and compression ratios for original and MPO representations.Source: Table I · p. 4, §2 · p. 4

1. FC2

  • At D = 2, MPO-Net uses 1024 parameters—about 200 times fewer than FC2—and at D = 16 it matches normal FC2 accuracy with a compression ratio below 8%.Test accuracy increases as bond dimension D increases.Source: §1 · p. 4

2. LeNet-5

  • LeNet-5's last convolutional layer and two fully connected layers, which contain most of its parameters, are represented by three MPOs.The reported compression ratio is approximately 0.05.Source: §2 · p. 4

B. CIFAR-10 dataset

On CIFAR-10, MPO representations improve VGG accuracy at about 0.0005 compression ratio and match ResNet accuracy in the studied cases. The experiments apply MPOs to fully connected layers and selected parameter-heavy convolutional layers.

B. CIFAR-10 dataset

  • As a preliminary CIFAR-10 test, MPOs are applied only to fully connected layers and selected heavily parameter-consuming convolutional layers.CIFAR-10 images are 32 × 32 RGB pixels across ten classes.Source: B. CIFAR-10 dataset · p. 4

2. ResNet

  • At k = 4 and compression ratio ρ ∼0.11, MPO ResNet matches original test accuracy across all studied depths.Figure 3 compares test accuracy for the original and MPO ResNet on CIFAR-10.Source: Figure 3 · p. 5, §2 · p. 5

1. VGG

  • At a compression ratio of about 0.0005, MPO VGG-16 and VGG-19 achieve better prediction accuracy than the original networks.MPOs replace the two heaviest convolutional layers and all three fully connected layers.Source: §1 · p. 5

2. ResNet

  • For a 56-layer ResNet, representing the last heaviest ResUnit and the fully connected layer with MPOs preserves the original network's accuracy.The paper reports similar observations for other k values.Source: §2 · p. 5

3. DenseNet

For DenseNet, MPOs sharply compress the fully connected layer while largely preserving test accuracy. The discussion frames MPOs as locality-focused representations of linear mappings, outlines their scope and possible extensions, and notes limits tied to data locality and input ordering.

3. DenseNet

  • The DenseNet has one fully connected layer of size (n + 3km) × 10, and the authors report faithful MPO compression on Fashion-MNIST and Street View House Number as well.The network depth is L = 3m + 4; m, k, and n determine its structure and MPO representation.Source: §3 · p. 5
  • 93.52 ± 0.40 versus 93.56 ± 0.26 accuracy, with ρ ∼0.497, results when the last heaviest convolutional layer is also replaced by an MPO while retaining an MPO classifier.This variant is described for the first DenseNet structure.Source: §3 · p. 5
  • The fully connected layer is compressed by more than seven to 21 times, with compression ratios from 0.044 to 0.129; test precisions match in three cases and are slightly higher in the fourth.Table II summarizes the DenseNet results on CIFAR-10.Source: §3 · p. 5, Table II · p. 6

IV. Discussion

  • The authors motivate MPOs through localized image correlations and report slightly improved test accuracy with fewer parameters across five network types and two datasets.They suggest reduced parameter counts may aid training, constrain transformations toward local correlations, and reduce overfitting risk.Source: §IV · pp. 5–6
  • A trained MPO’s canonical form can yield entanglement entropy as a measure of network expressive ability and dataset complexity.The authors propose using it to study how input locality relates to MPO-Net performance.Source: §IV · p. 6
  • MPOs represent fully connected and convolutional layers alike as linear mappings, with particular memory and training-time benefits expected in networks dominated by fully connected layers.The discussion names recurrent networks used for video data as an example.Source: §IV · p. 6
  • Unlike prior work on selected fully connected layers or single-MPO mappings, this approach retains neural-network layers and activations while representing classifiers and convolutional layers too.The authors also distinguish their one-dimensional, local-entanglement approach from Tucker and tree tensor-network methods for intermediate layers.Source: §IV · p. 6
  • When data locality is unsuitable, MPOs may need large bond dimensions; with small dimensions, input order that better preserves locality may improve accuracy.A coarse-grained image ordering is proposed for systematic future study.Source: §IV · pp. 6–7

Supplemental Materials: Compressing deep neural networks by matrix product operators

The supplemental material provides further network details and additional analyses, while the paper’s source code is available online.

Supplemental Materials

Structure Details of the Neural Networks

This section identifies the network structures used in the paper and supplies shorthand definitions and performance summaries for selected architectures.

Structure Details of the Neural Networks

  • The paper’s FC2, LeNet5, VGG, ResNet, and DenseNet structures are summarized in Tables II–VII, with prototypes cited to their original references.Short-hands used in the supplemental material are introduced in Table I.Source: Structure Details of the Neural Networks · p. 9
  • Table I lists the short-hands used in the supplemental material.These abbreviations support the following network-structure descriptions.Source: Table I · p. 9, Structure Details of the Neural Networks · p. 9
  • The FC2 and LeNet5 structure tables define their network layouts; the LeNet5 subscript np marks convolutions without padding.The FC2 caption notes that ReLu and Softmax are element-wise operations.Source: Table II · p. 9, Table III · p. 10
  • The VGG-16 table defines ConvUnit m× as m convolutional layers separated by ReLu; VGG-19’s partially MPO label marks its last two convolutional layers in that unit..Source: Table IV · p. 10, Table V · p. 11
  • The ResNet table defines depth as L = 6m + 2 and describes ResUnits and ResBlocks; the DenseNet table gives L = 3m + 4 and defines partially MPO as an MPO-represented last convolutional layer.The VGG/DenseNet performance table reports that original and MPO results are essentially consistent given their stated standard deviations.Source: Table VI · p. 11, Table VII · p. 12, Table VIII · p. 12

A. Performance on more datasets

Across Fashion-MNIST and SVHN, MPO representations work well in VGG and ResNet, and also DenseNet on SVHN; the reported factorization tests find near-identical accuracy across MPO variants at larger bond dimensions.

A. Performance on more datasets

B. Factorization manner

  • For FC2 on MNIST, two MPO factorization schemes produce essentially the same test accuracy when D ≥ 4, and both reproduce the normal FC2 result.The normal FC2 test accuracy is 98.35% ± 0.2%.Source: B. Factorization manner · p. 14, Figure 3 · p. 14
  • For VGG-19 on CIFAR-10, two MPO factorization schemes have accuracy identical within the error bar and slightly better than the normal VGG-19 result, 93.36% ± 0.09%.The second MPO-Net was run once, and its result was reported as consistent with the first variant.Source: B. Factorization manner · p. 14, Table IX · p. 14

C. Entanglement entropy

The authors estimate dataset entanglement entropy from trained MPO spectra and compare MNIST with Fashion-MNIST. Fashion-MNIST generally has higher entropy, while the first and third bonds remain below their theoretical bounds in both datasets.

C. Entanglement entropy

  • The authors estimate classical-data entanglement entropy from the entanglement spectra of trained MPOs, using the first MPO in FC2-MPO networks trained on each dataset.The spectra are obtained from the trained MPO's canonical form or singular value decomposition.Source: C. Entanglement entropy · pp. 14–15
  • Fashion-MNIST has generally higher entanglement entropy than MNIST across all three measured bonds, with bond 2 nearly reaching its upper bound at D = 24.The reported interpretation is that MNIST correlations are more local, consistent with its better FC2 performance.Source: C. Entanglement entropy · p. 15, Figure 5 · p. 15
  • On bonds 1 and 3, entropy remains below the theoretical limits for both datasets, which the authors interpret as evidence of locality and possible applicability of MPO-Nets.The authors suggest that more layers may be needed.Source: C. Entanglement entropy · p. 15, Figure 4 · p. 15, Figure 5 · p. 15

D. About L2 regularization

MPO-Nets retain comparable performance without L2 regularization in the reported MNIST and CIFAR-10 comparisons. Their convergence advantage is conditional: bond dimension and represented input entanglement affect training behavior, so fewer parameters do not guarantee faster convergence.

D. About L2 regularization

E. Convergence of training

  • Fewer parameters do not guarantee faster convergence: the authors report that training speed also varies with the optimization landscape and, for MPO-Nets, the entanglement represented in the input and grasped by the MPO.Their examples include FC2 converging faster than smaller LeNet-5 and DenseNet requiring more epochs than VGG-19 and ResNet despite fewer parameters.Source: E. Convergence of training · pp. 16–17, Table XI · p. 17
  • On MNIST, MPO-Net with D = 16 converges faster than FC2 when input entanglement is larger (χ = 10 or 28), while D = 2 underfits and converges slowly.With χ = 2, D = 16 eventually reaches 100% training accuracy, but after more epochs than FC2.Source: E. Convergence of training · p. 17, Figure 7 · p. 17
Loading 1904.06194v2…