TEINet: Towards an Efficient Architecture for Video Recognition

1 Introduction

• Action recognition depends on temporal information, but dense replacement of 2D convolutions with 3D convolutions raises computational cost, while some efficient 2D designs rely on hand-crafted temporal modeling.The paper seeks a flexible module that learns spatiotemporal representations while preserving 2D CNN efficiency.Source: §1 · p. 1
• TEI decouples temporal learning into channel-level enhancement followed by local temporal interaction, separating channel correlation from temporal relations while retaining flexible modeling and high inference efficiency.The two stages are plugged in sequence as a generic building block for existing 2D CNNs such as ResNets.Source: §1 · p. 2
• MEM uses temporal differences as an approximate motion map to emphasize important channel-level features, while TIM models adjacent-frame variation with temporal channel-wise convolution over a local window.Feature maps are squeezed to focus on channel-level importance, and the local window controls model complexity.Source: §1 · p. 2
• TEINet reports state-of-the-art performance on Something-Something, comparable performance to prior 3D CNN methods at lower computational cost on Kinetics, and competitive accuracy on UCF101 and HMDB51.The experiments use a 2D ResNet backbone.Source: §1 · p. 2

2D CNNs in Action Recognition

• TEINet builds on efficient 2D backbones, extending work from two-stream models, TSN, TRN, and TSM with a focus on capturing temporal clues.Two-stream methods use optical flow or motion vectors; TSN pools sparsely sampled frames, TRN models multi-scale temporal relations, and TSM shifts feature maps. (Simonyan and Zisserman 2014; Wang et al. 2016; Zhou et al. 2018; Lin, Gan, and Han 2018)Source: §2 · p. 2

3D CNNs in Action Recognition

• 3D CNNs learn spatiotemporal representations directly, but their computation is heavy; TEINet instead uses a temporal module based purely on 2D CNNs.I3D inflates 2D kernels into 3D, while ARTNet adds higher-order relation modeling to capture motion. (Carreira and Zisserman 2017; Wang et al. 2018a)Source: §2 · p. 2

Efficient Temporal Modules

• ECO combines 2D and 3D convolutions, while P3D, S3D, and R(2+1)D factor 3D convolution into spatial 2D and temporal 1D operations; TEINet adds a temporal block to a purely 2D backbone.(Zolfaghari, Singh, and Brox 2018; Qiu, Yao, and Mei 2017; Xie et al. 2018; Tran et al. 2018)Source: §2 · p. 2

Attention in Action Recognition

• Unlike prior attention methods, TEINet’s MEM derives temporal attention weights from local motion tendency, trains end-to-end without extra supervision, and boosts accuracy.(Hu, Shen, and Sun 2018; Li, Hu, and Yang 2019; Wang et al. 2018b; Girdhar and Ramanan 2017)Source: §2 · p. 2

3.1 Motion Enhanced Module

• MEM uses adjacent-frame feature differences to generate channel-wise attention that enhances motion-salient features while suppressing irrelevant background information.It pools each frame spatially before comparing features; unlike SE’s self-attention, MEM uses motion-aware attention.Source: §3.1 · p. 3

3.2 Temporal Interaction Module

• TIM applies a learnable channel-wise temporal convolution with a 3×1×1 kernel, interacting across adjacent time steps while keeping parameters to Cout × 1 × t.Its temporal receptive field grows through deeper layers, and the channel-wise operator uses fewer parameters than the described 3D and temporal 1D convolutions.Source: §3.2 · pp. 3–4

3.3 TEINet

• TEINet combines MEM followed by TIM in an efficient module that plugs into 2D CNNs; the modules complementarily model channel importance and adjacent-feature temporal variation.The paper describes the factorized module as effective on both motion-dominated Something-Something V1&V2 and appearance-dominated Kinetics-400 datasets.Source: §3.3 · p. 4
• TEINet inserts its module into a 2D CNN backbone rather than replacing 2D convolutions with 3D or (2+1)D convolutions, enabling ImageNet initialization and fewer extra FLOPs than 3D CNNs.The experiments instantiate TEINet with ResNet-50 as the backbone.Source: §3.3 · p. 4

4.1 Datasets

• The evaluation spans Something-Something V1/V2 and Kinetics-400, plus UCF101 and HMDB51 to assess transfer to smaller datasets.Something-Something focuses on motion; UCF101 and HMDB51 results average three splits, and the paper notes that Kinetics-400 and the smaller datasets are less sensitive to temporal relationships.Source: §4.1 · p. 4

4.2 Experimental Setup

• Experiments use ResNet-50, generally ImageNet-pretrained, with 8- or 16-frame inputs; training uses SGD, and inference averages scores over 10 clips and 3 spatial crops.The setup specifies 50 training epochs on Something-Something and 100 on Kinetics-400.Source: §4.2 · pp. 4–5

4.3 Ablation Studies

• TIM yields higher recognition accuracy than MEM (46.1% vs. 33.5%), and MEM+TIM exceeds SE+TIM by 1.3%; TIM also outperforms TSN and TSM baselines.The component ablations use Something-Something V1 with center-crop, one-clip testing.Source: §4.3 · p. 5
• Later-stage TEI insertion improves performance, while res4 and res5 together achieve results similar to res5 alone; adding blocks boosts performance, but res2−5 beats res4−5 by 0.7% with seven extra blocks.The authors recommend res4−5 for efficiency, although remaining experiments use TEI blocks in all stages.Source: §4.3 · pp. 5–6, Table 1 · p. 5

4.4 Comparison with the State of the Art

• 5.1% on Something-Something V1 validation separates 16-frame TEINet from TSM, while 61.3% on V2 exceeds TSM16f×10 by 1.9%.TEINetEn also surpasses existing RGB or RGB+Flow models on V1; on V2, TEINet16f+8f outperforms TSM-2Stream by 2.5%.Source: Table 3 · p. 6, Table 4 · p. 6, §4.4 · pp. 6–7

4.3 Ablation Studies

• TEINet models achieve acceptable latency and throughput on Something-Something V1, measured on a Tesla P100 with batch sizes of 1 and 16, respectively.Data-loading time is excluded from the runtime measurement.Source: Table 2 · p. 5, §4.3 · p. 6

4.4 Comparison with the State of the Art

• On Kinetics-400, TEINet improves performance among lightweight models and its 16-frame variant exceeds 32-frame NL I3D by 1.3%; on UCF101 and HMDB51, it is competitive with I3D-RGB and R(2+1)D-RGB.The smaller-dataset results use a Kinetics-400-pretrained TEINet fine-tuned with 16-frame inputs, evaluated using 10 clips and 3 crops per video.Source: Table 5 · p. 7, Table 6 · p. 7, §4.4 · p. 7

• TEINet achieves state-of-the-art performance on Something-Something V1&V2 and competitive performance on Kinetics with high efficiency.Source: §5 · p. 7
• TEINet converts vanilla ResNet into a temporal model by inserting TEI blocks composed of a motion enhanced module (MEM) and a temporal interaction module (TIM).Source: §5 · p. 7
• MEM enhances motion-related features using temporal attention weights, while TIM learns temporal contextual features with channel-wise temporal convolution.Source: §5 · p. 7