PaperlayerView exact revision on arXiv

Source-linked AI summary

Tensor-Train Recurrent Neural Networks for Video Classification

Yinchong Yang, Denis Krompass, Volker Tresp

arXiv:1707.01786v1cs.CV

TL;DR

RNNs are effective for sequence modeling but become impractical on high-dimensional video inputs because their input-to-hidden matrices are too large. This paper factorizes those matrices with Tensor-Train decomposition and trains the resulting RNN models end-to-end on raw video pixels. On three large-scale video datasets, the approach achieves competitive performance with much lower complexity than related models.

  • Problem

    High-dimensional video inputs make RNNs difficult to train directly because the input-to-hidden weight matrix becomes extremely large.

  • Method

    The paper factorizes the RNN input-to-hidden weight matrix using Tensor-Train decomposition and trains the resulting models end-to-end on raw pixels.

  • Results

    On three large-scale video datasets, Tensor-Train RNNs achieved competitive state-of-the-art performance while reducing model complexity by several orders of magnitude.

  • Takeaways & Limitations

    TT-RNNs provide a building block for applying RNN architectures directly to high-dimensional sequential data such as video.

  • Takeaways & Limitations

    The paper notes that tensorized models may require less labeled data, but their training time does not decrease in proportion to their parameter reduction because GPUs parallelize larger models effectively.

Abstract

from arXiv · show

The Recurrent Neural Networks and their variants have shown promising performances in sequence modeling tasks such as Natural Language Processing. These models, however, turn out to be impractical and difficult to train when exposed to very high-dimensional inputs due to the large input-to-hidden weight matrix. This may have prevented RNNs' large-scale application in tasks that involve very high input dimensions such as video modeling; current approaches reduce the input dimensions using various feature extractors. To address this challenge, we propose a new, more general and efficient approach by factorizing the input-to-hidden weight matrix using Tensor-Train decomposition which is trained simultaneously with the weights themselves. We test our model on classification tasks using multiple real-world video datasets and achieve competitive performances with state-of-the-art models, even though our model architecture is orders of magnitude less complex. We believe that the proposed approach provides a novel and fundamental building block for modeling high-dimensional sequential data with RNN architectures and opens up many possibilities to transfer the expressive and advanced architectures from other domains such as NLP to modeling high-dimensional sequential data.

1. Introduction

RNNs perform well on sequence modeling but are difficult to train directly on video because high-dimensional frames create enormous input-to-hidden mappings. The paper proposes Tensor-Train factorization to make raw-pixel video modeling with RNNs more efficient.

  • Motivation: High-dimensional video frames make direct RNN training impractical because the input-to-hidden weight matrix becomes extremely large.A 160×120×3 RGB frame produces 57,600 input values; with 100 hidden nodes, the input-to-hidden mapping has 5,760,000 free parameters.
  • Motivation: CNN preprocessing reduces frame dimensions and produces compact representations, but end-to-end CNN-RNN training from scratch is impractical for large video datasets.Consequently, many related approaches emphasize the CNN component and reduce its size.
  • Related limitations: Earlier PCA and random-projection approaches generated image representations, but such models were often linear and could not be trained jointly with the classifier.
  • Approach: The proposed RNN consumes raw pixels, maps each frame to a lower-dimensional latent vector, and recurrently enriches it using hidden-to-hidden transitions.This design aims to capture inter-frame patterns while learning the sequence representation directly.
  • Contribution: Tensor-Train decomposition factorizes the input-to-hidden matrix, enabling simple RNN architectures to compete with state-of-the-art models at several orders of magnitude lower complexity.The experiments use three large-scale video benchmarks.

2. Related Works

Prior video models largely process frames with CNNs and aggregate their representations, while RNN-based alternatives face computational constraints. Tensor-Train methods provide a route to compressing large neural-network weight matrices and have also been studied in lower-dimensional sequence classification.

  • Video modeling: Most video-modeling approaches use deep CNNs to process frames individually and aggregate their outputs.Proposed fusion strategies include early, late, and slow fusion, while other methods use global pooling or recurrently combine frame-wise representations.
  • Video modeling: Framewise CNN-RNN systems face computational complexity that leads researchers to restrict sequence length, use pretrained CNNs, or omit large-scale video evaluation.One compact LSTM approach uses convolutional input-to-hidden and hidden-to-hidden mappings but was not evaluated on large-scale video data.
  • Tensor-Train methods: Tensor-Train was introduced as a tensor factorization model that scales to arbitrarily many dimensions.It was later used to reshape and factorize fully connected layers for end-to-end compression of large neural-network weight matrices.
  • RNN sequence classification: Earlier RNN sequence-classification studies addressed spoken words, handwriting, and sentiment using relatively low-dimensional inputs or word embeddings.These tasks illustrate sequence modeling settings with substantially smaller input representations than raw video frames.

3. Tensor-Train RNN

The proposed Tensor-Train RNN replaces the large input-to-hidden weight matrix with an end-to-end learned Tensor-Train Layer, enabling RNNs to process raw high-dimensional frames. Tensor-Train factorization represents weights through low-rank core tensors, reducing storage and computation while supporting SRNN, LSTM, and GRU mappings.

  • 3.3. Tensor-Train RNN: A Tensor-Train Layer replaces the input-to-hidden matrix in an RNN and is trained end-to-end with the rest of the model.This mapping is applied to the input-to-hidden transformation in SRNNs and to input-to-gate matrices in LSTMs and GRUs.
  • 3.1. Tensor-Train Factorization: A Tensor-Train Factorization represents each target-tensor entry through a sequence of vector-matrix-vector multiplications that yields a scalar.Its complexity is determined by the Tensor-Train ranks, with boundary ranks restricted to 1.
  • 3.2. Tensor-Train Factorization of a Feed-Forward Layer: The double-index construction reshapes tensor dimensions so a Tensor-Train representation can factorize weight matrices in fully connected layers.The full tensor is replaced by low-rank core tensors rather than stored explicitly.
  • 3.2. Tensor-Train Factorization of a Feed-Forward Layer: O(d · ˜m · ˜r^2 · ˜n^d) is the stated total forward-pass complexity for the Tensor-Train feed-forward layer, compared with O(M · N) for a fully connected layer.Here ˜m, ˜n, and ˜r are the maximum factor and rank values across dimensions.
  • 3.2. Tensor-Train Factorization of a Feed-Forward Layer: 2,976 parameters with TT-rank 4 and 4,520 with TT-rank 5 represent a 160×120×3-to-256 mapping, versus 14,745,600 fully connected parameters.The corresponding compression rates are 2.0e-4 and 3.1e-4.
  • 3.3. Tensor-Train RNN: Parallelizing all four LSTM gates or three GRU gates by enlarging the first output factor reduces parameters relative to calculating their Tensor-Train Layers successively.For the example frame and hidden dimensions, parallelized TT-LSTM requires 3,360 parameters at rank 4 and 5,000 at rank 5.

4. Experiments

Experiments evaluate TT-RNN variants on UCF11 and Hollywood2, showing that Tensor-Train input-to-hidden layers make simple recurrent models competitive with state-of-the-art methods while greatly reducing parameters and training time. The experiments also examine frame-sequence structure, multi-label evaluation, and comparisons with plain recurrent and feed-forward baselines.

  • UCF11: 14,745,600 parameters in a fully connected UCF11 input-to-hidden mapping are reduced to 3,360 in TT-GRU and 3,232 in TT-LSTM.The UCF11 input dimension is 57,600, factorized as 8 × 20 × 20 × 18, with a 256-unit hidden layer and Tensor-Train ranks [1, 4, 4, 4, 1].
  • UCF11: TT-LSTM and TT-GRU significantly improve UCF11 classification accuracy while compressing the input-to-hidden mapping from over 40 million parameters to 3 thousands.The study compares these models with TT-MLP and plain GRU/LSTM baselines using five-fold cross-validation.
  • UCF11: A factor of 4 to 5 reduction in training time is reported on UCF11: TT-GRU and TT-LSTM take approximately 2 days versus more than 8 and 10 days for plain GRU and LSTM.The comparison is reported for commodity hardware, whose GPU parallelization prevents parameter-count differences from producing an equivalent training-time gap.
  • State-of-the-art comparison: State-of-the-art comparisons report TT-LSTM as the second-best overall score and the best score among neural-network models on the evaluated task.The cited comparison attributes the strongest competing scores to improved trajectory and Histogram of Optical Flow features, while the TT-LSTM is the strongest neural-network result in that comparison.
  • Hollywood2: TT-GRU and TT-LSTM achieve accuracy very close to the best state-of-the-art model on Hollywood2, while TT-MLP performs well on simpler clips and can outperform plain GRU and LSTM.Hollywood2 is a challenging multi-label dataset with actions appearing in varied styles and backgrounds; performance is measured using Mean Average Precision across classes.

5. Conclusions and Future Work

The work integrates Tensor-Train layers into RNNs, including LSTM and GRU, enabling end-to-end training on high-dimensional sequential data. The resulting lightweight models improve over plain RNNs and are competitive with state-of-the-art approaches.

  • Tensor-Train layers integrated into LSTM and GRU models enable end-to-end training on high-dimensional sequential data.The integration is presented as the core architectural contribution.
  • The Tensor-Train integration alone significantly improves modeling performance over plain RNNs, which perform very poorly on the evaluated video datasets.
  • Despite much lower complexity than state-of-the-art solutions, the simple tensorized networks reach second-best results in all cases.
  • The lightweight architecture reduces free parameters from tens of millions to thousands, supporting training and deployment on commodity hardware and mobile devices.The paper also expects fewer free parameters to reduce labeled-data requirements, although video labels are expensive.
  • TT-RNNs are proposed as a building block for transferring successful RNN-based techniques from NLP and other domains to high-dimensional sequential data such as videos.The paper specifically discusses direct video modeling and possible extensions such as autoencoders, captioning networks, and attention-based models.
  • Source code, unit tests, simulation studies, and additional HMDB51 experiments are publicly available.
Loading 1707.01786v1…