Source-linked AI summary
Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
Jinmian Ye, Linnan Wang, Guangxi Li, Di Chen, Shandian Zhe, Xinqi Chu, Zenglin Xu
TL;DR
High-dimensional inputs make standard RNNs computationally expensive because their large parameter counts hinder efficient computer-vision modeling. The paper replaces dense input-to-hidden transformations with Block-Term decomposition, and reports stronger accuracy and convergence than standard and Tensor Train RNNs across three tasks.
Problem
LSTM input-to-hidden transformations can contain excessive parameters, making models difficult to train and susceptible to overfitting.
Method
Block-Term RNNs replace redundant dense input-to-hidden operations with a low-rank sum of Tucker decompositions that models inter-parameter correlations.
Results
BT-RNN outperforms Tensor Train RNNs and standard RNNs in prediction accuracy and convergence rate across action recognition, image captioning, and image generation tasks.
Takeaways & Limitations
BT-RNN provides a compact recurrent architecture with fewer parameters, stronger feature-dimension correlation modeling, and easier training within the evaluated tasks.
Takeaways & Limitations
Tensor Train methods face rank-setting and border-rank constraints, while BT model core-order selection involves a trade-off between parameter reduction, computation, memory, and spatial information.
Abstract
from arXiv · showhide
Recurrent Neural Networks (RNNs) are powerful sequence modeling tools. However, when dealing with high dimensional inputs, the training of RNNs becomes computational expensive due to the large number of model parameters. This hinders RNNs from solving many important computer vision tasks, such as Action Recognition in Videos and Image Captioning. To overcome this problem, we propose a compact and flexible structure, namely Block-Term tensor decomposition, which greatly reduces the parameters of RNNs and improves their training efficiency. Compared with alternative low-rank approximations, such as tensor-train RNN (TT-RNN), our method, Block-Term RNN (BT-RNN), is not only more concise (when using the same rank), but also able to attain a better approximation to the original RNNs with much fewer parameters. On three challenging tasks, including Action Recognition in Videos, Image Captioning and Image Generation, BT-RNN outperforms TT-RNN and the standard RNN in terms of both prediction accuracy and convergence rate. Specifically, BT-LSTM utilizes 17,388 times fewer parameters than the standard LSTM to achieve an accuracy improvement over 15.6\% in the Action Recognition task on the UCF11 dataset.
1. Introduction
The paper targets parameter-intensive LSTM input-to-hidden transformations, replacing redundant dense connections with Block-Term tensor decomposition to obtain compact BT-RNNs. The resulting models retain LSTM design while improving representation, convergence, and performance across computer-vision tasks.
- 1. Introduction: 4.1 × 10^6 and 1.0 × 10^8 parameters can arise in the four input-weight matrices when image features have dimensions 4096 and 14 × 14 × 512, respectively.These large matrix-vector products increase training difficulty, computation complexity, and memory usage.
- 1. Introduction: BTD replaces redundant dense input-to-hidden connections with a low-rank tensor representation while retaining similar RNN representation power with far fewer parameters.The decomposition represents weight matrices and inputs as tensors and learns inter-parameter correlations during training.
- 1. Introduction: BT-LSTM reduces model parameters by several orders and converges faster than traditional LSTM while preserving the LSTM design philosophy.The paper evaluates BT-LSTM against standard and low-rank recurrent baselines on three computer-vision tasks.
- 1. Introduction: BT representation shares weights across input dimensions through core tensors, strengthening modeling of local correlations.The approach is motivated by limitations of dense connections for spatially structured image features.
- 1. Introduction: BT models use multiple Tucker decompositions, reducing sensitivity to noisy inputs and avoiding the difficult rank-setting problem associated with Tensor Train approaches.The paper also notes that BT representation has better representation power than Tensor Train at the same parameter count.
2. Related Work
Prior work addresses inefficient dense connections with sparse CNN structures and low-rank tensor approximations. The paper positions BTD as a more flexible alternative to Tensor Train decomposition for representing correlations and setting ranks.
- 2. Related Work: Sparse convolutional connections improve extraction of latent regional image features, motivating related efforts to reduce dense connections in RNNs.The paper distinguishes these explicit architectural changes from low-rank methods that implicitly prune dense connections.
- 2. Related Work: SVD-based low-rank compression can reduce up to 30% of parameters in a weight matrix but has also been reported with up to 10% accuracy loss.This motivates tensor methods for capturing higher-order spatial correlations.
- 2. Related Work: Tensor Train methods reduce computation and parameters in neural-network matrix-vector products, but their rank constraints can limit representation ability and flexibility.In particular, border ranks must equal 1, while practical equal-rank settings may differ from the optimal pattern.
- 2. Related Work: BTD uses a core tensor to model cross-dimensional correlations, equal core ranks for balanced weight sharing, and multiple Tucker models for wider representation.These design choices are presented as alternatives to difficult Tensor Train rank selection.
3. Tensorizing Recurrent Neural Networks
BT-RNN replaces dense recurrent matrix-vector products with tensorized Block-Term representations, using contractions over decomposed weights and reshaped inputs to reduce parameters and computational complexity.
- 3. Tensorizing Recurrent Neural Networks: BT-RNN approximates W · x with lightweight tensor products while preserving the memorization mechanism of existing RNN formulations.The BTD layer learns inter-parameter correlations to implicitly prune redundant dense connections.
- 3. Tensorizing Recurrent Neural Networks: Block-Term decomposition represents a high-order tensor as a sum of Tucker decompositions, parameterized by CP-rank N, Tucker-ranks R_k, and core-order d.Each block term combines a core tensor with factor matrices through tensor products.
- 3. Tensorizing Recurrent Neural Networks: The model tensorizes inputs and weights, contracts the decomposed weight tensor with the input tensor, and reverses tensorization to produce the output.Reshaping regroupes elements without duplicating input data, while contraction proceeds along matching input dimensions.
- 3. Tensorizing Recurrent Neural Networks: For Tucker-rank R, increasing core-order d initially reduces parameters but eventually increases them because the core-tensor term grows as R^d.The parameter count therefore has a minimum at an intermediate core-order rather than decreasing monotonically.
- 3. Tensorizing Recurrent Neural Networks: Extremely high core-order can lose important spatial information, so the paper recommends choosing d ∈ [2, 5] in practice.Increasing d reduces each input-tensor dimension logarithmically but can discard spatial information.
- 3. Tensorizing Recurrent Neural Networks: BT-RNN reduces the stated computation complexity from O(NdIJR) to O(NdIJmaxR^d), with backward complexity matching the forward pass form.The comparison passage states that both TT-RNN and BT-RNN use lower complexity and memory than vanilla RNN, with comparable complexity for d ∈ [2, 5].
4. Experiments
Experiments evaluate BT-LSTM against LSTM and TT-LSTM across action recognition, image generation, image captioning, and hyper-parameter studies. BT-LSTM achieves strong parameter efficiency, faster convergence, improved action-recognition accuracy, and comparable or qualitatively improved outputs on other tasks.
- Action Recognition: 60% validation accuracy arrives at epoch 16 for BT-LSTM versus epoch 230 for LSTM, indicating a 14x convergence speedup.The comparison is based on the training losses and validation accuracies in Figure 6.
- Action Recognition: BT-LSTM uses 721, 1470, or 3387 parameters at Tucker-ranks 1, 2, and 4, corresponding to compression ratios of 81693x, 40069x, and 17388x.The vanilla LSTM has 58.9 million parameters in W.
- Action Recognition: At nearly equal parameter counts, BT-LSTM-4 decreases training loss faster than TT-LSTM-4, supporting stronger spatial-information capture.This comparison is reported over epochs 0–50.
- Action Recognition: 15.6% higher accuracy than LSTM and 7.2% higher accuracy than TT-LSTM with comparable parameters demonstrate BT-LSTM’s strongest action-recognition result.The comparison is reported for UCF11 action recognition.
- Hyper-Parameter Analysis: Increasing Tucker-rank improves accuracy while different Tucker-ranks converge to identical losses; higher core-order can instead deteriorate reconstructed weights.The hyper-parameter analysis attributes this behavior to the amount of information captured by the tensor representation.
- Image Generation: BT-LSTM and LSTM generate comparable MNIST-style images, while BT-LSTM uses 1577 times fewer parameters.The comparison is shown for the image-generation task.
- Image Captioning: BT-LSTM, TT-LSTM, and LSTM all generate proper image captions, with little improvement in BT-LSTM and qualitative evidence of improved local-correlation capture.The captions are evaluated on MSCOCO using an Inception-v3 feature representation.
5. Conclusion
BT-RNN addresses redundancy in recurrent networks by pruning input-to-hidden connections with Block-Term tensor decomposition. It uses fewer parameters while improving training and performance relative to standard LSTM and TT-LSTM.
- BT-RNN prunes input-to-hidden connections with Block-Term tensor decomposition to address redundancy in recurrent networks.
- The architecture uses several orders fewer parameters while improving performance over standard LSTM and TT-LSTM on video action recognition.
- Future work targets sparsity, temporal tensor modeling, and quantization to further reduce parameters and memory usage.