PaperlayerView exact revision on arXiv

Source-linked AI summary

TAM: Temporal Adaptive Module for Video Recognition

Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, Tong Lu

arXiv:2005.06803v3cs.CV

TL;DR

Video recognition needs temporal modeling that is both flexible to complex motion and computationally efficient. The paper introduces TAM, which combines local location-sensitive enhancement with global video-adaptive aggregation, and shows strong performance across Kinetics-400 and Something-Something, including state-of-the-art results on Something-Something.

  • Problem

    Existing temporal modules struggle to combine strong flexibility for complex video dynamics with high efficiency.

  • Method

    TAM uses local temporal convolutions for location-sensitive importance maps and global fully connected layers for video-adaptive aggregation kernels within 2D CNNs.

  • Results

    TAM outperforms several temporal modeling counterparts on Kinetics-400 and achieves state-of-the-art performance on Something-Something.

  • Takeaways & Limitations

    TAM provides an efficient and flexible temporal module for extending 2D CNNs to video recognition.

Abstract

from arXiv · show

Video data is with complex temporal dynamics due to various factors such as camera motion, speed variation, and different activities. To effectively capture this diverse motion pattern, this paper presents a new temporal adaptive module ({\bf TAM}) to generate video-specific temporal kernels based on its own feature map. TAM proposes a unique two-level adaptive modeling scheme by decoupling the dynamic kernel into a location sensitive importance map and a location invariant aggregation weight. The importance map is learned in a local temporal window to capture short-term information, while the aggregation weight is generated from a global view with a focus on long-term structure. TAM is a modular block and could be integrated into 2D CNNs to yield a powerful video architecture (TANet) with a very small extra computational cost. The extensive experiments on Kinetics-400 and Something-Something datasets demonstrate that our TAM outperforms other temporal modeling methods consistently, and achieves the state-of-the-art performance under the similar complexity. The code is available at \url{ https://github.com/liu-zhy/temporal-adaptive-module}.

1. Introduction

Video understanding requires temporal modules that capture complex dynamics flexibly while remaining computationally efficient. TAM addresses this with local and global adaptive modeling, and TANet achieves strong benchmark performance at similar complexity.

  • Effective video temporal modules must capture complex temporal structure with high flexibility and low computational cost.
  • Video data has complex temporal dynamics from camera motion, speed variation, and diverse activities.
  • TAM decomposes video-specific temporal kernels into location-sensitive importance maps and location-invariant aggregation weights.
  • The local branch models short-term information, while the global branch models long-term temporal structure.
  • TANet integrates TAM into 2D CNNs and achieves competitive accuracy with FLOPs similar to 2D CNNs, plus state-of-the-art performance on Something-Something.

2. Related Work

Prior video-recognition methods use fixed or otherwise video-invariant temporal operations, while TAM introduces video-specific adaptive temporal modeling.

  • Deep learning methods have enabled CNN-based approaches for learning spatiotemporal video representations.
  • TSN aggregates sparsely sampled frames after the final fully connected layer to capture long-range information.
  • The reviewed methods commonly use video-invariant operations and overlook temporal diversity across videos.
  • TAM instead decomposes video-specific processing into location-sensitive excitation and location-invariant convolution with adaptive kernels.

3. Method

TAM efficiently models video temporal structure through complementary local and global branches operating on spatially squeezed features. It produces location-sensitive excitation and video-adaptive temporal aggregation, and can replace fixed temporal operations in 2D CNN architectures.

  • 3.1. The Overview of Temporal Adaptive Module: TAM first applies global spatial average pooling to reduce X ∈ R^C×T×H×W to a spatially squeezed temporal signal.
  • 3.1. The Overview of Temporal Adaptive Module: TAM computes Y by convolving the locally enhanced feature map L(X) ⊙ X with global adaptive weights G(X).
  • 3.2. Local Branch in TAM: The local branch uses temporal convolutions in a kernel-size-3 window to produce location-sensitive importance weights for short-term motion.
  • 3.2. Local Branch in TAM: The local branch rescales its importance weights across spatial dimensions and applies them element-wise to X to form Z.
  • 3.3. Global Branch in TAM: The global branch uses fully connected layers and a softmax-normalized kernel to aggregate temporal features using long-term global context.
  • 3.3. Global Branch in TAM: Unlike vanilla temporal convolution, TAM generates aggregation weights from each video’s input features rather than using fixed weights.
  • 3.3. Global Branch in TAM: TAM’s location-sensitive excitation and location-invariant aggregation derive from input features while targeting short-term and long-term structures.
  • 3.4. Exemplar: TANet: TAM can generalize fixed temporal schemes: its global branch degenerates into TSN or TSM for specific kernel-weight settings.

4. Experiments

Experiments on Kinetics-400 and Something-Something evaluate TAM through ablations, cross-backbone transfer, comparisons with temporal modules, and visualization of learned kernels. TANet consistently performs strongly while retaining flexible, video-specific temporal modeling.

  • Experimental Setup: Experiments use Kinetics-400 and Something-Something V1&V2, covering large-scale and fine-grained, motion-dominated action recognition.Kinetics-400 contains approximately 300k clips across 400 categories; Something-Something focuses on interacting-object actions.
  • Experimental Setup: TANet uses 8- or 16-frame inputs with dataset-specific sampling and multi-scale spatial augmentation, while evaluation averages predictions across spatiotemporal views.Kinetics-400 uses 10 clips for 8-frame models and 4 clips for 16-frame models, with three spatial crops.
  • Ablation Studies: The selected configuration uses α = 2 and β = 4, a global temporal receptive field K = 3, TAM inserted after the first convolution, and TA-Blocks in res2−5.These choices are selected through ablations, with larger K helping when more frames are sampled but degrading performance for 8-frame inputs.
  • Ablation Studies: TAM generalizes across ShuffleNet V2, MobileNet V2, Inception V3, and I3D-ResNet-50, outperforming their corresponding C2D and I3D baselines by a large margin.The experiments include both 2D and 3D backbones without temporal downsampling before global average pooling.
  • Comparison with Temporal Modules: TANet outperforms fixed temporal alternatives and other temporal modules, exceeding C2D-TIM by 1.6%, TSM by 2.2%, TEINet by 1.4%, and NL C2D by 1.9%.C2D-TConv and C2D-Pool achieve similar accuracy, while C2D-TIM is the strongest temporal-convolution baseline in the comparison.
  • State-of-the-Art Comparisons: TANet achieves strong results across benchmarks, including 0.7% higher accuracy than SlowFast with similar FLOPs per view, 1.4% higher accuracy than 32-frame NL I3D, and 50.6% versus 49.7% top-1 accuracy against TSMEn on Something-Something V1.On Something-Something V2, the models achieve state-of-the-art performance; kernel visualizations show distributions varying across videos and actions.

5. Conclusion

The paper presents TAM as a generic temporal module for complex video motion and builds TANet around it. TAM combines local importance maps with global aggregation kernels, achieving stronger temporal modeling on Kinetics-400 and state-of-the-art performance on Something-Something V1&V2.

  • TAM generates video-specific kernels by combining a local importance map with a global aggregation kernel.The design targets complex temporal structure and adaptive temporal modeling.
  • TANet is a video architecture built from the proposed TAM module.
  • On Kinetics-400, networks equipped with TAM outperform existing temporal modules in action recognition.
  • TANet achieves state-of-the-art performance on the motion-dominated Something-Something V1&V2 datasets.

A. TAM in the different position

The supplementary material presents four insertion positions for TAM within a network block. These variants differ by whether TAM is placed before or after specific convolutions.

  • TANet-a, TANet-b, TANet-c, and TANet-d place TAM before the first, after the first, after the second, or after the last convolution.
  • The four TAM insertion styles are graphically presented in Fig. 4.

B. Visualizations of Learned Kernel

Visualizations show that TAM learns diverse temporal kernels across videos and datasets. The local importance maps are smoother than the global kernels, while learned kernels distinguish Kinetics-400 from motion-dominated Something-Something V2 more clearly than fixed I3D kernels.

  • The local importance maps are smoother than the global adaptive kernels, and the local branch attends differently to each video.
  • The global kernel performs adaptive aggregation to learn temporal diversity across videos.
  • Learned kernels show clear differences between Kinetics-400 and Something-Something V2, consistent with domain shift between the datasets.
  • The fixed I3D3×1×1 kernels show narrow distributions and only minor differences between the two datasets.
  • Kernel diversity demonstrates that videos contain temporal variation and supports adaptive spatio-temporal representation learning.
Loading 2005.06803v3…