PaperlayerView exact revision on arXiv

Source-linked AI summary

TSM: Temporal Shift Module for Efficient Video Understanding

Ji Lin, Chuang Gan, Song Han

arXiv:1811.08383v3cs.CV

TL;DR

Video understanding requires both accuracy and efficiency, but 2D CNNs lack temporal modeling while 3D CNNs are computationally expensive. The paper introduces TSM, which shifts channels across time within 2D CNNs, including a uni-directional online variant. TSM maintains 2D CNN complexity while achieving state-of-the-art offline performance and low-latency online recognition.

  • Problem

    2D CNNs are computationally cheap but cannot capture temporal relationships, whereas 3D CNNs model them at high computational cost, limiting efficient deployment.

  • Method

    TSM shifts part of the channels along the temporal dimension inside 2D CNNs to exchange information among neighboring frames, with partial, residual, and uni-directional variants.

  • Results

    TSM achieves state-of-the-art offline performance, ranks first on the Something-Something leaderboard upon publication, and supports real-time online recognition and object detection.

  • Takeaways & Limitations

    TSM offers a hardware-efficient route to joint spatial-temporal video modeling and low-latency recognition on edge devices.

  • Takeaways & Limitations

    Naive shift increases CPU latency by 13.7% and GPU latency by 12.4%, motivating partial and residual shift modifications.

Abstract

from arXiv · show

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: https://github.com/mit-han-lab/temporal-shift-module.

1. Introduction

Video understanding needs temporal modeling, but efficient 2D CNNs miss temporal relationships while 3D CNNs are costly to deploy. TSM shifts channels across time to exchange neighboring-frame information without added computation, supporting accurate offline and low-latency online recognition.

  • Applications: Bi-directional TSM targets high-throughput offline recognition, while uni-directional TSM targets low-latency online recognition on edge devices.The two variants differ in whether future-frame information can be mixed with the current frame.
  • Motivation: 2D CNNs are efficient but poorly model temporal information, whereas 3D CNNs jointly model space and time at high computational cost.These trade-offs make real-time online recognition and edge deployment difficult.
  • Method: TSM shifts channels forward and backward along the temporal dimension so neighboring-frame information mingles with the current frame.For online recognition, uni-directional TSM shifts only past information because future frames are unavailable.
  • Module Design: Naive shifting can reduce efficiency and accuracy because data movement adds latency and shifting too many channels harms video-model performance.The authors therefore introduce partial shift and residual shift modifications.
  • Method: TSM provides computationally free temporal modeling on top of 2D convolution while retaining strong spatio-temporal modeling ability.The paper describes this as a design with no additional computation and no additional parameters.
  • Results: Bi-directional TSM achieves state-of-the-art offline performance and ranked first on the Something-Something leaderboard upon publication.The paper separately proposes uni-directional TSM for real-time online video recognition.

2. Related Work

Prior video-recognition methods trade temporal expressiveness against computation: 2D CNNs are efficient but limited temporally, while 3D and hybrid approaches improve modeling at greater cost. Other alternatives use fusion, recurrent, attention, decomposition, or efficient-architecture techniques.

  • 2.1. Deep Video Recognition: 2D CNN approaches are more efficient than 3D counterparts but cannot infer temporal order or complex temporal relationships.They commonly process sampled-frame features independently or aggregate them after extraction.
  • 2.1. Deep Video Recognition: 3D CNNs jointly learn spatio-temporal features but are computationally heavy, harder to deploy, and more parameterized than 2D models.The related-work discussion also notes their greater tendency to over-fit.
  • 2.2. Efficient Video Recognition: Hybrid 2D–3D architectures and factorized convolutions reduce computation by distributing or decomposing spatial and temporal processing.Examples include bottom-heavy, top-heavy, and spatial-plus-temporal convolution designs.
  • 2.1. Deep Video Recognition: Temporal relationships have also been modeled with post-hoc fusion, LSTMs, attention mechanisms, temporal relation networks, and non-local modules.These approaches provide alternatives to direct 3D convolution for temporal modeling.
  • 2.2. Efficient Video Recognition: Efficient deployment research includes efficient architecture design, neural architecture search, pruning, quantization, compression, and hardware-friendly shift operations.The paper distinguishes these general efficiency techniques from its temporal-shift approach for video recognition.

3. Temporal Shift Module (TSM)

TSM separates temporal information exchange from convolution by shifting selected channels across neighboring frames, while partial and residual shifts address data-movement overhead and spatial-feature degradation. The module supports bidirectional offline and unidirectional online designs.

  • 3.1. Intuition: TSM decouples convolution into channel shifting across time and multiply-accumulate merged into the following 2D convolution, adding no extra cost.The shift moves channels by −1, 0, or +1 along the temporal dimension.
  • 3.3. Module Design: Online TSM shifts only information from past frames toward future frames, avoiding access to unavailable future frames.The online variant is uni-directional rather than bidirectional.
  • 3.2. Naive Shift Does Not Work: Naive temporal shifting increases latency and reduces accuracy by moving too many channels, with 13.7% CPU overhead, 12.4% GPU overhead, and a 2.6% accuracy drop versus TSN.The larger video activation tensor exacerbates data-movement costs and loss of current-frame spatial information.
  • 3.2. Naive Shift Does Not Work: Partial shift limits data movement: shifting 1/8 of channels restricts latency overhead to 3%, whereas shifting all channels reaches 13.7% CPU overhead.The measurements use ResNet-50 with 8-frame input across different shift proportions.
  • 3.3. Module Design: Residual shift places TSM inside the residual branch, preserving access to the original activation through identity mapping and protecting spatial feature learning.Unlike in-place shift, residual shift retains the unshifted activation through the shortcut connection.
  • 3.3. Module Design: Residual shift outperforms in-place shift at every tested shift proportion on Kinetics, and even full-channel residual shifting exceeds the 2D baseline.The captioned design uses a 1/4-proportion residual shift as the default because it combines higher accuracy with negligible overhead.

4. TSM Video Network

TSM can be inserted into 2D CNN video models without changing their parameters or computation cost, while also supporting online recognition through cached temporal features. Its online design provides low latency, low memory use, and multi-level temporal fusion.

  • 4. TSM Video Network: Offline TSM models process sampled frames with exactly the same parameters and computation cost as 2D CNN models.Convolution layers still operate independently on frames, while TSM supplies temporal interaction.
  • 4. TSM Video Network: TSM converts off-the-shelf 2D CNNs into models handling spatial and temporal information using operations already optimized across common hardware.The framework requires support for standard 2D CNN operations rather than specialized 3D operators.
  • 4. TSM Video Network: Uni-directional online TSM replaces 1/8 of each current feature map with cached features from the previous frame and repeats this across residual blocks.The remaining 7/8 current features are combined with 1/8 old features to produce the next layer.
  • 4. TSM Video Network: Per-frame online inference has latency almost equal to the 2D CNN baseline because caching and replacement require no extra computation.Unlike methods that aggregate multiple frames for one prediction, the design avoids that additional latency.
  • 4. TSM Video Network: ResNet-50 online TSM requires only 0.9MB of memory cache for intermediate features.The low cache size follows from storing only a small portion of the features.
  • 4. TSM Video Network: TSM enables temporal fusion at all feature levels, whereas compared online approaches use late or mid-level temporal fusion.Experiments reported in Table 2 identify multi-level temporal fusion as important for complex temporal modeling.

5. Experiments

Experiments show that TSM consistently improves 2D CNN video recognition without extra computation, including on datasets requiring temporal modeling. It also achieves strong accuracy-efficiency trade-offs, low latency, and real-time online recognition performance.

  • Improving 2D CNN Baselines: TSM consistently outperforms 2D counterparts on multiple datasets at zero extra computation.The protocol uses ResNet-50 with 8-frame input, 10 clips for Kinetics, 2 for other datasets, and full-resolution evaluation.
  • Improving 2D CNN Baselines: On temporally demanding datasets, adding TSM to 2D CNNs improves accuracy by double digits.This pattern is reported for Something-Something V1, Something-Something V2, and Jester.
  • Improving 2D CNN Baselines: TSM consistently improves accuracy across MobileNet-V2, ResNet-50, ResNext-101, and ResNet-50 with a Non-local module.The gains extend even to NL R-50, which already has temporal modeling ability.
  • Comparison with State-of-the-Arts: TSM ranks first on the Something-Something-V1 leaderboard upon publication.The comparison uses center-crop testing and one clip per video unless otherwise specified.
  • Comparison with State-of-the-Arts: With 8 input frames, TSM reaches 45.6% top-1 accuracy using 33G FLOPs, 4.2% higher accuracy than ECO with 1.9× less computation.TSM also achieves 1.2% better accuracy with 10× fewer FLOPs than Non-local I3D on the validation set.
  • Latency and Throughput Speedup: TSM is faster by an order of magnitude than I3D at 1.8% higher accuracy and has 1.75× lower latency than ECO.Against ECO, TSM records 17.4ms versus 30.6ms latency, 1.7× higher throughput, and 2% better accuracy.

6. Conclusion

TSM enables joint spatial-temporal modeling in 2D CNN backbones without additional cost. Its efficiency and accuracy support low-latency video recognition on edge devices.

  • TSM shifts part of the channels along the temporal dimension to exchange information with neighboring frames.
  • TSM can be inserted into a 2D CNN backbone for joint spatial-temporal modeling at no additional cost.
  • TSM enables low-latency video recognition on edge devices while maintaining efficiency and accuracy.

A. Uni-directional TSM for Online Video Detection

Uni-directional TSM adds temporal fusion to online video object detection through the backbone, helping correct appearance-related errors without changing the detection module or using optical flow features.

  • Temporal information helps address poor object appearance caused by motion blur, occlusion, and defocus.
  • Uni-directional TSM injects temporal fusion into the detector backbone without changing the detection module or using optical flow features.
  • 4.6% mAP improvement occurs on fast-moving objects versus the 2D baseline R-FCN.
  • TSM achieves similar or higher performance than offline FGFA while using only past frames and much smaller latency per frame.
  • Less than 1ms per-frame latency overhead makes the TSM module practical for real deployment.

B. Video Demo

The paper provides additional video demonstrations of the TSM model through its project page.

  • Additional TSM video demos are available on the project page.

R-FCN TSM

The R-FCN baseline produces false positives and misses an occluded motorcycle in the illustrated clips, while TSM addresses these issues using temporal information.

  • TSM suppresses R-FCN false positives caused by car-headlight glare across frames 2/3/4.
  • TSM addresses R-FCN false positives around an occluded bus across frames 2/3/4.
  • TSM helps detect a motorcycle that R-FCN misses under occlusion on frame 4.
Loading 1811.08383v3…