PaperlayerView exact revision on arXiv

Source-linked AI summary

Compressing Recurrent Neural Networks with Tensor Ring for Action Recognition

Yu Pan, Jing Xu, Maolin Wang, Jinmian Ye, Fei Wang, Kun Bai, Zenglin Xu

arXiv:1811.07503v1cs.CVcs.LG

TL;DR

High-dimensional inputs make RNN input-to-hidden transformations costly in memory and computation, complicating training. The paper proposes TR-LSTM, which uses tensor ring decomposition for a compact transformation; on UCF11 it reports 0.869 accuracy and a compression ratio over 34,000 against standard LSTM.

  • Problem

    High-dimensional inputs such as video make RNN input-to-hidden transformations large in memory and computational cost, while video-frame dimensions can yield millions of parameters.

    Source: Introduction · p. 1

  • Method

    TR-LSTM applies tensor ring decomposition to the RNN input-to-hidden transformation, forming a compact layer that can be trained end to end.

    Source: Introduction · pp. 1–2

  • Results

    0.869 accuracy on UCF11 exceeded standard LSTM (0.681), TT-LSTM (0.803), and BT-LSTM (0.856), with a compression ratio over 34,000 versus standard LSTM.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The tensor ring layer is presented as a building block for LSTM-based approaches handling large inputs, and TR-LSTM also improved accuracy while compressing LSTM with InceptionV3 features.

    Source: Conclusion · p. 7

    The paper identifies tensor train decomposition’s restricted rank settings and core order as constraints, with finding the best dimension alignment remaining challenging.

    Source: Tensor Train Decomposition · pp. 2–3

Abstract

from arXiv · show

Recurrent Neural Networks (RNNs) and their variants, such as Long-Short Term Memory (LSTM) networks, and Gated Recurrent Unit (GRU) networks, have achieved promising performance in sequential data modeling. The hidden layers in RNNs can be regarded as the memory units, which are helpful in storing information in sequential contexts. However, when dealing with high dimensional input data, such as video and text, the input-to-hidden linear transformation in RNNs brings high memory usage and huge computational cost. This makes the training of RNNs unscalable and difficult. To address this challenge, we propose a novel compact LSTM model, named as TR-LSTM, by utilizing the low-rank tensor ring decomposition (TRD) to reformulate the input-to-hidden transformation. Compared with other tensor decomposition methods, TR-LSTM is more stable. In addition, TR-LSTM can complete an end-to-end training and also provide a fundamental building block for RNNs in handling large input data. Experiments on real-world action recognition datasets have demonstrated the promising performance of the proposed TR-LSTM compared with the tensor train LSTM and other state-of-the-art competitors.

Introduction

High-dimensional video inputs make LSTM and GRU input-to-hidden matrices costly and difficult to train. The paper introduces TR-LSTM, using tensor ring decomposition for a compact input-to-hidden transformation, and reports strong action-recognition results.

Introduction

  • High-dimensional video frames create very large input-to-hidden matrices; a UCF11 frame can yield over 57,000 input dimensions and up to 14 million matrix parameters for a 256-unit hidden layer.Pre-processing with CNN feature extraction does not fully solve the over-parameterization problem.Source: Introduction · p. 1
  • Tensor train RNNs can be sensitive to parameter selection because their ranks and core order are restricted, including the need for small border ranks and larger middle ranks.The cited optimal rank pattern is described as resembling an olive.Source: Introduction · p. 1
  • TR-LSTM reshapes the input-to-hidden matrix as a tensor and factorizes it with tensor ring decomposition, which models inter-parameter correlations with fewer parameters.The authors describe tensor ring decomposition as relaxing tensor train constraints by circularly connecting the first and last cores, and propose end-to-end training.Source: Introduction · pp. 1–2
  • 0.869 accuracy on UCF11 exceeded standard LSTM (0.681), TT-LSTM (0.803), and BT-LSTM (0.856), with a compression ratio over 34,000 versus standard LSTM.The paper also reports that TR-LSTM outperformed most previous competitors when using pre-trained CNN features as LSTM inputs.Source: Introduction · p. 2

Model

The model represents RNN input-to-hidden transformations compactly with tensor ring decomposition. The section previews tensor decomposition background before presenting TR-LSTM.

Model

  • Tensor ring decomposition provides a compact representation of the input-to-hidden layer for high-dimensional RNN inputs.The model section first presents tensor decomposition preliminaries, including tensor train and tensor ring illustrations, then introduces TR-LSTM.Source: Model · p. 2
  • The section’s presentation proceeds from tensor decomposition background to the proposed TR-LSTM model.The background includes graphical illustrations of tensor train and tensor ring decompositions.Source: Model · p. 2
  • TR-LSTM is introduced as an LSTM model using the tensor ring representation of the input-to-hidden layer.The section frames this representation as a response to high-dimensional RNN inputs.Source: Model · p. 2

Preliminaries and Background

The preliminaries establish tensor notation and contraction, then describe tensor train decomposition and the tensor ring structure used to address TT constraints. TRD connects the first and last cores and can be viewed as a sum of tensor trains.

Notation

  • The paper denotes a d-order tensor with a boldface Euler script letter.Subscripts identify elements or subtensors.Source: Notation · p. 2

Tensor Contraction

Tensor Train Decomposition

  • TT decomposition factors a high-order tensor into core tensors, with endpoint ranks fixed at 1 and other ranks chosen manually.The TT ranks correspond to decomposition complexity.Source: Tensor Train Decomposition · p. 2, Figure 2 · p. 2
  • TT rank constraints limit representation flexibility, and finding the best dimension alignment for ordered core multiplication is challenging.TRD's ring structure is introduced to alleviate these limitations.Source: Tensor Train Decomposition · pp. 2–3
  • TRD connects the first and last cores into a ring, and can be interpreted as a sum of R tensor trains.Fixing the shared endpoint index yields the tensor-train terms.Source: Tensor Train Decomposition · p. 3

TR-RNN model

The model replaces input-to-hidden weight matrices with tensor-ring forms in RNNs and their variants. It reshapes inputs and weights into tensors, applies TRD, and uses the resulting tensor ring layer to compute outputs.

TR-RNN model

  • The paper applies the input-to-hidden weight transformation in tensor-ring form to RNNs and their variants to construct TR-RNN models.This is the stated core conception of the model.Source: TR-RNN model · p. 3

Tensorizing x, y and W

  • The method tensorizes the input vector, output vector, and weight matrix before decomposing the weight tensor into tensor-ring form.The input tensor is contracted with input and output core tensors sequentially to produce the output.Source: Tensorizing x, y and W · p. 3, Decomposing W · p. 4

Decomposing W

  • For n input and m output dimensions, TRD represents the weight tensor with n + m core tensors, each corresponding to an input or output dimension.The input- and output-dimension cores are grouped separately.Source: Decomposing W · p. 3

Tensor Ring Layer (TRL)

  • The tensor ring layer denotes the calculation from input vector x to output vector y when the weight matrix is factorized with TRD.The output tensor is reshaped into the final output vector.Source: Tensor Ring Layer (TRL) · p. 4

TR-RNN

  • TR-RNN replaces the vanilla RNN’s multiplication of the input-to-hidden matrix by the input vector with the tensor ring layer.The hidden state also uses the hidden-to-hidden matrix and sigmoid activation.Source: TR-RNN · p. 4

TR-LSTM

  • TR-LSTM applies the tensor ring layer to the input-to-hidden mappings for the input, forget, output, and cell-update gates.Its gate and cell-state equations retain the hidden-state transformations and LSTM updates.Source: TR-LSTM · p. 4
  • The tensor ring representation produces a hidden matrix smaller than the original input, which the paper frames as an information bottleneck balancing compression with prediction information.The authors suggest this may reduce redundant high-dimensional input while preserving good performance relative to standard LSTM.Source: TR-LSTM · p. 4

Experiments

Experiments evaluate TR-LSTM on synthetic weight recovery and on UCF11 and HMDB51, using end-to-end training and CNN-extracted features. Tensor ring performs well in weight recovery and action recognition, with reported compression in the video-classification settings.

Experiments

  • The synthetic test estimates a low-rank 81 × 81 weight matrix from 3,200 noisy examples and omits regularization for a qualitative, intuitive comparison.The experiments also cover UCF11 and HMDB51, with end-to-end and CNN-feature settings.Source: Synthetic Experiment · p. 4, Experiments · p. 4

Synthetic Experiment

Experiments on the UCF11 Dataset

  • 0.869 accuracy gives TR-LSTM the top UCF11 end-to-end result, versus 0.856 for BT-LSTM, 0.803 for TT-LSTM, and 0.681 for LSTM.The reported comparison uses 1,725 TR-LSTM parameters, about half the parameters of TT-LSTM and BT-LSTM.Source: Figure 7 · p. 5, Table 1 · p. 6, Experiments on the UCF11 Dataset · p. 6
  • 93.8 accuracy for feature-based TR-LSTM exceeds the vanilla LSTM's 0.923, with a compression ratio of 25; it can replace vanilla LSTMs in Two Stream LSTM to reduce parameters.The Two Stream LSTM with the highest UCF11 accuracy has more than 141M parameters.Source: Table 2 · p. 6, Experiments on the UCF11 Dataset · p. 6

Experiments on the HMDB51 Dataset

Related Work

Prior work reduces RNN input-to-hidden parameters through preprocessing or low-rank structure, but these approaches retain limitations that motivate tensor ring decomposition.

Related Work

  • CNN preprocessing can improve classification accuracy, but it only partially solves the over-parametric problem in RNNs.Source: Related Work · pp. 6–7
  • SVD-based matrix decomposition can reduce parameters, but its compression is limited by rank and it can reduce model performance.Source: Related Work · p. 7
  • Tensor decompositions capture higher-order correlations with fewer parameters, while CP instability, Tucker computation, and TT constraints present trade-offs.Block-Term decomposition combines CP and Tucker; TT methods have also reduced parameters in CNNs and RNNs.Source: Related Work · p. 7
  • Tensor ring decomposition is proposed to address TT rank and ordering constraints while offering greater computation efficiency than Block-Term decomposition.Source: Related Work · p. 7

Conclusion

The conclusion presents TR-LSTM as a tensor-ring replacement for the RNN input-to-hidden matrix and reports strong UCF11 results, including with InceptionV3 features.

Conclusion

  • TRD replaces the over-parametric input-to-hidden matrix, capturing feature-dimension correlations with fewer parameters.Source: Conclusion · p. 7
  • TR-LSTM achieved the best compression ratio and highest classification accuracy on UCF11 among the compared end-to-end low-rank RNNs.Source: Conclusion · p. 7
  • With InceptionV3-extracted features as input, TR-LSTM still compressed the LSTM while improving accuracy.The authors propose the models as building blocks for large-input RNNs and plan to extend them to more advanced RNN structures.Source: Conclusion · p. 7

Complexity in Forward Process

The forward computation uses a sequential core-multiplication order, with tensor contraction as its operation and stated complexity and memory costs.

Complexity in Forward Process

  • The multiplication order of the input tensor with input and output cores determines computational cost; the implementation multiplies them sequentially.Source: Complexity in Forward Process · p. 9
  • Tensor contraction matches dimensions between tensors and is described as a higher-dimensional analogue of matrix multiplication, inner product, and outer product.Source: Complexity in Forward Process · p. 9
  • Forward computation costs O(nIR^3 + mOR^3), with space complexity O(IR^2), when all model ranks are R.Table 4 compares vanilla RNN, TT-RNN, BT-RNN, and TR-RNN complexity and memory usage at the same rank.Source: Complexity in Forward Process · p. 9, Table 4 · p. 9

Complexity in Backward Process

The backward analysis uses a representative core gradient and reports computational and space complexity for the model.

Complexity in Backward Process

Loading 1811.07503v1…