Source-linked AI summary
TEINet: Towards an Efficient Architecture for Video Recognition
Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Tong Lu
TL;DR
TEINet addresses the high parameter count and computational cost of 3D convolutions by adding an efficient temporal module to existing 2D CNNs. Its TEI module separates motion-related feature enhancement from channel-wise temporal interaction, achieving strong recognition performance across several benchmarks while retaining high efficiency.
Problem
3D CNNs improve video action recognition but introduce many parameters and high computational cost compared with 2D CNNs.
Method
TEINet inserts TEI blocks into 2D CNNs, sequentially using MEM to enhance motion-related features and TIM to capture local temporal context through channel-wise temporal convolution.
Results
TEINet achieves state-of-the-art performance on Something-Something V1&V2 and competitive or comparable performance on Kinetics, UCF101, and HMDB51 while maintaining lower computational cost or fast inference.
Takeaways & Limitations
Decoupling motion enhancement from temporal interaction provides an efficient temporal modeling module that can be plugged into existing 2D CNN architectures.
Abstract
from arXiv · showhide
Efficiency is an important issue in designing video architectures for action recognition. 3D CNNs have witnessed remarkable progress in action recognition from videos. However, compared with their 2D counterparts, 3D convolutions often introduce a large amount of parameters and cause high computational cost. To relieve this problem, we propose an efficient temporal module, termed as Temporal Enhancement-and-Interaction (TEI Module), which could be plugged into the existing 2D CNNs (denoted by TEINet). The TEI module presents a different paradigm to learn temporal features by decoupling the modeling of channel correlation and temporal interaction. First, it contains a Motion Enhanced Module (MEM) which is to enhance the motion-related features while suppress irrelevant information (e.g., background). Then, it introduces a Temporal Interaction Module (TIM) which supplements the temporal contextual information in a channel-wise manner. This two-stage modeling scheme is not only able to capture temporal structure flexibly and effectively, but also efficient for model inference. We conduct extensive experiments to verify the effectiveness of TEINet on several benchmarks (e.g., Something-Something V1&V2, Kinetics, UCF101 and HMDB51). Our proposed TEINet can achieve a good recognition accuracy on these datasets but still preserve a high efficiency.
1 Introduction
Action recognition requires effective temporal modeling, but 3D CNNs impose substantial computational costs. TEINet addresses this by inserting a two-stage TEI module into 2D CNNs, combining motion enhancement with temporal interaction while retaining efficiency.
- 3D CNNs directly learn spatiotemporal features but substantially increase computational cost when replacing 2D convolutions densely.
- TEINet decouples temporal feature learning into channel-level enhancement and local temporal interaction through the TEI module.
- MEM uses motion information and temporal differences to enhance discriminative features while suppressing irrelevant background information.
- TIM models local temporal variations with a channel-wise convolution over a local time window to control model complexity and inference cost.
- TEI blocks plug into 2D ResNets, forming TEINet architectures that improve recognition performance while preserving fast inference.
2 Related Work
Prior 2D CNN approaches aggregate or otherwise incorporate temporal information, while Figure 2 depicts TEI's separate MEM and TIM processing stages with channel-specific temporal kernels.
- 2D CNN action-recognition methods include two-stream architectures using optical flow or motion vectors and TSN using sparse-frame temporal aggregation.
- Figure 2 separates the TEI pipeline into a motion enhanced module and a temporal interaction module.
- TIM uses distinct kernel weights for different channels rather than sharing one kernel across channels.
3 Method
The TEI module factorizes temporal modeling into motion-aware channel enhancement followed by local temporal interaction, enabling efficient integration into 2D CNNs.
- 3.1 Motion Enhanced Module: MEM applies channel-wise attention to enhance motion-salient features and suppress irrelevant background information.Its motion-aware attention differs from SE attention, which calibrates channels using each feature map’s own global feature.
- 3.1 Motion Enhanced Module: MEM aggregates each frame spatially to form channel-level representations before generating channel-importance weights from adjacent-frame differences.Global average pooling produces ˆx_t ∈ R^C×1×1, while temporal differences approximate motion saliency; the channel reduction ratio is r = 8.
- 3.2 Temporal Interaction Module: TIM applies channel-wise temporal convolution independently to each channel, modeling local temporal variations with substantially fewer parameters than 3D convolution.A 3×1×1 kernel interacts with adjacent times, while deeper layers progressively enlarge the temporal receptive field; TIM uses parameters proportional to C_out × 1 × t.
- 3.2 Temporal Interaction Module: TIM generalizes TSM’s fixed temporal shifts by learning a flexible convolutional kernel for temporal interaction.TSM corresponds to fixed kernels for non-shift, backward-shift, and forward-shift operations.
- TEI module: TEI combines MEM and TIM sequentially: MEM enhances motion-related channel features, then TIM captures temporal contextual information.This two-stage block is integrated into existing 2D CNN architectures to form TEINet.
- 3.3 TEINet: TEINet inserts TEI into a 2D CNN backbone rather than replacing 2D convolutions with more expensive 3D or (2+1)D convolutions.The integration supports ImageNet-pretrained initialization and adds fewer computational FLOPs than 3D CNNs; ResNet-50 is used as the backbone.
4 Experiments
Experiments evaluate TEINet across action-recognition datasets, ablate its components and placement, and measure efficiency. The results show improvements over prior temporal models, competitive performance with more expensive methods, and a trade-off between added TEI blocks and efficiency.
- Experimental setup: Experiments use Something-Something V1 and compare TEI-module designs and ResNet50 integration under center-crop, one-clip testing.The broader evaluation also covers Something-Something V2, Kinetics-400, UCF101, and HMDB51.
- Ablation studies: 46.1% versus 33.5% shows TIM yields higher recognition accuracy than MEM on Something-Something V1.The study reports that TIM is more effective than MEM and compares TIM with efficient temporal baselines including TSN and TSM.
- Ablation studies: Later-stage TEI insertion improves performance, while res4 and res5 integration achieves a similar result despite res4 containing three more blocks.The authors associate this finding with temporal modeling on higher-level features.
- Ablation studies: Adding more TEI blocks boosts performance, but res2−5 exceeds res4−5 by 0.7% while requiring 7 extra blocks; therefore, res4−5 is recommended for efficiency.The default configuration in the remaining experiments uses TEI blocks in all stages.
- Runtime analysis: TEINet models provide acceptable latency and throughput, and the experiments exclude data-loading time from runtime measurements.Table 2 reports latency and throughput comparisons on Something-Something V1.
- Comparison with the state of the art: TEINet achieves state-of-the-art performance on Something-Something, competitive performance against expensive 3D models on Kinetics, and competitive transfer results on UCF101 and HMDB51.On Something-Something V1, the 16-frame model surpasses TSM by 5.1%, while TEINet16f+8f surpasses TSM-2Stream by 2.5%; on Kinetics, TEINet16f outperforms NLI3D32f by 1.3%.
5 Conclusion
TEINet efficiently models temporal features for video action recognition by inserting TEI blocks into a vanilla ResNet. Its MEM and TIM components support strong recognition performance while maintaining high efficiency.
- 5 Conclusion: TEINet converts a vanilla ResNet into an efficient video architecture by inserting TEI blocks for temporal feature modeling.The TEI blocks consist of a Motion Enhanced Module and a Temporal Interaction Module.
- 5 Conclusion: MEM enhances motion-related features by calculating temporal attention weights.It focuses the model on motion-related information.
- 5 Conclusion: TIM learns temporal contextual features using a channel-wise temporal convolution.This captures temporal interactions while preserving the module’s efficient design.
- 5 Conclusion: TEINet achieves state-of-the-art performance on Something-Something V1&V2 and competitive performance on Kinetics with high efficiency.These results come from a series of empirical studies on video action recognition.