PaperlayerView exact revision on arXiv

Source-linked AI summary

TDN: Temporal Difference Networks for Efficient Action Recognition

Limin Wang, Zhan Tong, Bin Ji, Gangshan Wu

arXiv:2012.10071v2cs.CV

TL;DR

Action recognition needs efficient temporal modeling that captures motion across both local and long-range timescales. TDN introduces temporal difference modules in a two-level framework for 2D CNNs, and reports state-of-the-art results on Something-Something and competitive performance on Kinetics with small added cost.

  • Problem

    Efficient temporal modeling remains challenging because 3D convolutions can incur higher computational cost and may lack specific temporal consideration.

  • Method

    TDN uses temporal difference modules in a two-level framework, with short-term differences for local motion and long-term differences across segments.

  • Results

    TDN achieves new state-of-the-art performance on Kinetics and Something-Something with similar backbones.

  • Takeaways & Limitations

    Temporal difference modules provide an efficient alternative to 3D convolutions for extracting fine-grained temporal information.

Abstract

from arXiv · show

Temporal modeling still remains challenging for action recognition in videos. To mitigate this issue, this paper presents a new video architecture, termed as Temporal Difference Network (TDN), with a focus on capturing multi-scale temporal information for efficient action recognition. The core of our TDN is to devise an efficient temporal module (TDM) by explicitly leveraging a temporal difference operator, and systematically assess its effect on short-term and long-term motion modeling. To fully capture temporal information over the entire video, our TDN is established with a two-level difference modeling paradigm. Specifically, for local motion modeling, temporal difference over consecutive frames is used to supply 2D CNNs with finer motion pattern, while for global motion modeling, temporal difference across segments is incorporated to capture long-range structure for motion feature excitation. TDN provides a simple and principled temporal modeling framework and could be instantiated with the existing CNNs at a small extra computational cost. Our TDN presents a new state of the art on the Something-Something V1 & V2 datasets and is on par with the best performance on the Kinetics-400 dataset. In addition, we conduct in-depth ablation studies and plot the visualization results of our TDN, hopefully providing insightful analysis on temporal difference modeling. We release the code at https://github.com/MCG-NJU/TDN.

1. Introduction

TDN addresses the challenge of efficient temporal modeling by explicitly using temporal differences within a unified 2D-CNN framework. Its two-level design captures complementary short-term and long-term motion information, achieving strong performance with low added cost.

  • 3D convolutions can increase computational cost while lacking specific temporal consideration, leaving efficient motion modeling challenging.
  • TDN generalizes RGB differences into an end-to-end temporal difference module that jointly models appearance and motion.
  • Short-term and long-term temporal information are modeled through complementary TDMs for local motion patterns and cross-segment variations.
  • TDN provides a video-level framework that can be instantiated with existing CNNs at a small extra computational cost.
  • TDN achieves new state-of-the-art performance on Kinetics and Something-Something with the same backbones.

2. Related work

Prior action-recognition methods separately model appearance and motion, use spatiotemporal convolutions, or aggregate sparse long-range information. TDN differs by explicitly computing temporal differences in a two-level video architecture for short- and long-term modeling.

  • Short-term temporal modeling: Two-stream CNNs separately process RGB and optical flow, while 3D-CNNs directly learn spatiotemporal features.
  • Long-term temporal modeling: Short-term clip-based networks struggle with long-range temporal structure, motivating recurrent, long-convolutional, and sparse-sampling approaches.
  • Temporal difference representation: Earlier temporal-difference methods used RGB or feature differences for motion extraction, often treating RGB difference as a separate modality.
  • Temporal difference representation: TDN explicitly computes motion information and integrates short-term and long-term temporal differences end to end with smaller FLOPs than 3D convolutions.

3. Temporal Difference Networks

TDN is a video-level framework that uses temporal differences at two scales: short-term local motion and long-term cross-segment structure. Its modules enhance 2D CNN representations through efficient processing and residual or attention-based fusion.

  • Overview: TDN uses sparse, holistic video sampling and combines short-term and long-term temporal difference modules in a unified framework.Frames sampled from video segments are processed by a 2D CNN, while TDMs model motion at different temporal scales.
  • Short-term temporal modeling: Short-term TDM supplies early frame-wise representations with local motion extracted from adjacent frames around each sampled frame.It stacks RGB differences in a local window and processes them with a lightweight low-resolution CNN before fusion with RGB features.
  • Short-term temporal modeling: The short-term module fuses motion information with single-frame RGB representations through lateral or residual connections to improve local spatiotemporal description.The design uses low-resolution processing because RGB differences are sparse and concentrate high responses in motion-salient regions.
  • Long-term temporal modeling: Long-term TDM enhances frame-level features with bidirectional, multi-scale temporal differences across adjacent segments.Feature compression, spatial smoothing, multi-scale receptive fields, and cross-segment differences address efficiency and spatial misalignment.
  • Long-term temporal modeling: Long-term difference representations serve as attention maps, and residual fusion combines the enhanced and original frame-level representations.The framework places short-term modules in early stages for finer motion and long-term modules in latter stages for coarser, higher-level temporal modeling.

4. Experiments

Experiments evaluate TDN through ablations, benchmark comparisons, and activation-map visualization. Results show that temporal difference modeling benefits accuracy, combines complementary short- and long-term modules, and remains effective across datasets and temporal-module comparisons.

  • Experimental setup: TDN is evaluated through ablations, state-of-the-art comparisons, and activation-map visualizations using ResNet backbones on Something-Something and Kinetics.The experiments include design studies, benchmark comparisons, and qualitative analysis.
  • Ablation studies: Temporal difference is essential: removing it or replacing it with averaging decreases recognition accuracy by around 10%.The authors attribute this to overfitting static information and failing to capture temporal variation.
  • Ablation studies: For long-term modeling, spatiotemporal attention reaches 52.3%, versus 44.1% without attention and 50.9% with channel attention.Multi-scale difference extraction improves performance from 49.7% to 52.3%, while bidirectional difference improves performance by 2.3%.
  • Ablation studies: The selected placement uses S-TDM in stages 1-2 and L-TDM in stages 3-5, achieving the best accuracy with reasonable computational cost.Adding more S-TDMs slightly increases computational cost because of temporal-difference feature extraction.
  • Ablation studies: 46.6% baseline accuracy rose to 51.5% with S-TDM, 49.9% with L-TDM, and 52.3% with both modules.The combined result suggests complementary contributions from short-term and long-term modeling.

5. Conclusion

TDN is a video-level framework that uses temporal-difference modules to capture short- and long-term temporal information efficiently. It outperforms previous state-of-the-art methods using similar backbones and offers an alternative to standard 3D convolution.

  • TDN generalizes temporal difference into efficient temporal modules designed to capture both short-term and long-term temporal information.The framework includes two customized TDM forms whose effects are assessed through ablation studies.
  • TDN achieves superior performance to previous state-of-the-art methods on Kinetics-400 and Something-Something with similar backbones.
  • Temporal difference extracts fine-grained temporal information more effectively than standard 3D convolution with more frames.The paper presents this finding as an ablation-based analysis and positions TDM as an alternative to 3D convolution.

Appendix

The appendix adds supplementary evaluations, runtime analysis, and visualization results for TDN.

  • The appendix reports results on UCF101 and HMDB51, analyzes TDN running time, and provides additional visualization results.

A. Results on the UCF101 and HMDB51

This section evaluates TDN's generalization on UCF101 and HMDB51 using transferred 16-frame Kinetics-400 models. It compares TDN with established 2D, 3D, and temporal-modeling methods under standard evaluation.

  • The comparison includes 2D baselines, 3D CNNs, and other temporal modeling approaches on both datasets.
  • 16-frame TDN models learned on Kinetics-400 are transferred to UCF101 and HMDB51 to assess generalization.The datasets are relatively small, and performance is reported as mean accuracy over three standard evaluation splits.
  • Performance is compared with TSN, I3D, C3D, R(2+1)D, and other temporal modeling methods.

B. Running time analysis

The running-time analysis measures TDN inference on a Tesla V100, including data loading and network inference. TDN is slower than previous methods but remains capable of real-time operation.

  • TDN inference is evaluated on a Tesla V100 with testing batchsize 16, including data loading and network inference.
  • ≥25 FPS: TDN remains capable of real-time operation despite being slower than previous methods.

C. Visualization analysis

The visualization analysis uses Grad-CAM to compare baseline and short-term TDM features, with results indicating greater attention to motion-relevant information when S-TDM is added.

  • Short-term motion visualization: Grad-CAM visualizations compare feature representations from a baseline model and a model equipped only with S-TDM.The comparison targets the Res2 stage and uses the center frame of 8-frame models.
  • Efficiency analysis: Table 5 presents a running-time analysis conducted on a Tesla V100.
  • Short-term motion visualization: Models equipped with S-TDM focus more on motion-relevant information than the baseline in the visualizations.Additional activation-map examples are provided in Figures 6 and 7.
Loading 2012.10071v2…