PaperlayerView exact revision on arXiv

Source-linked AI summary

TSM: Temporal Shift Module for Efficient Video Understanding

Ji Lin, Chuang Gan, Song Han

arXiv:1811.08383v3cs.CV

TL;DR

Video understanding needs high accuracy and low computation, while 2D CNNs lack temporal modeling and 3D CNNs are computationally intensive. TSM shifts channels across time in 2D CNNs and ranks first on the Something-Something leaderboard upon publication.

  • Problem

    Hardware-efficient video understanding matters for real-world cloud and edge deployment, while traditional 2D CNNs operate independently across frames without temporal modeling.

    Source: §1 · p. 1

  • Method

    TSM shifts channels forward and backward along time so neighboring-frame information is mingled in a 2D CNN without additional cost.

    Source: §1 · p. 1, §6 · p. 8

  • Results

    TSM ranks first on the Something-Something leaderboard upon publication.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports low-latency video recognition on edge devices.

    Source: §6 · p. 8

    Naively shifting all or most channels increases latency, with a 13.7% increase in CPU latency and a 12.4% increase in GPU latency.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: https://github.com/mit-han-lab/temporal-shift-module.

1. Introduction

The introduction motivates accurate, efficient video understanding and presents TSM as a way to model temporal information with low computation cost. It proposes partial and residual shifts, with bi-directional TSM for offline understanding and uni-directional TSM for online use.

Introduction

  • Video understanding must be both accurate and efficient for cloud and edge deployments, and temporal order can determine recognition outcomes.The introduction cites high-volume video processing and privacy-sensitive hospital video as examples of these deployment needs.Source: §1 · p. 1
  • 2D CNNs do not model temporal information across individual frames, while 3D CNNs are costly and difficult to deploy at the edge or in real-time online recognition.Post-hoc and mid-level fusion approaches trade temporal modeling for efficiency, potentially losing useful information before fusion.Source: §1 · p. 1
  • TSM shifts channels forward and backward across time so neighboring-frame information is mingled with the current frame, enabling temporal modeling in 2D CNNs.For online understanding, where future frames cannot be shifted to the present, the paper uses uni-directional TSM.Source: §1 · pp. 1–2
  • Partial shifts limit data movement, while placing TSM inside the residual branch preserves current-frame activations and the 2D backbone’s spatial feature learning.These modifications address the latency and accuracy problems the paper associates with naively shifting too many channels.Source: §1 · p. 2
  • The paper proposes bi-directional TSM for offline video understanding, reporting that it ranked first on the Something-Something leaderboard upon publication, and uni-directional TSM for low-latency online recognition on edge devices.The introduction describes temporal shift as computationally free while providing spatio-temporal modeling ability.Source: §1 · p. 2

2. Related Work

The related work contrasts efficient but temporally limited 2D CNNs with expressive but costly 3D CNNs and other temporal methods. TSM is presented as enabling 3D-like temporal modeling at 2D CNN cost, while prior efficiency techniques do not directly solve video recognition.

2.1. Deep Video Recognition

  • 2D CNN video methods, including two-stream networks [39] and TSN, are more efficient than 3D counterparts but cannot infer temporal order or complex temporal relationships.Other cited 2D CNN video-recognition methods include [39] [8].Source: §2.1 · p. 2
  • 3D CNNs jointly learn spatio-temporal features, but are computationally heavy, use more parameters, and are more prone to overfitting; TSM is presented as matching their modeling ability at 2D CNN cost.Examples include C3D [45] and inflating 2D Inception filters into 3D [4].Source: §2.1 · p. 2
  • Prior trade-offs include motion filters [27], mixed 2D–3D networks [61], and decomposed convolutions [42], but retain temporal computation or omit modeling at some levels.The paper claims TSM removes temporal-modeling computation and offers better hardware efficiency than decomposed convolutions.Source: §2.1 · p. 2

2.2. Temporal Modeling

  • Temporal alternatives include non-local modules [49], region graphs [50], post-hoc fusion [13] [58] [7], LSTMs, attention [37] [32], and relation networks [58], but trade computation or lose low-level information.TSM is presented as enabling low- and high-level temporal modeling at 2D CNN cost, like 3D CNN methods.Source: §2.2 · p. 2

2.3. Efficient Neural Networks

  • Existing efficiency work includes architecture design, search, and compression, but image-recognition address shifts do not preserve efficiency or accuracy on video.Prior work designs efficient models [21] [20] [36], searches architectures [62] [63] [31] [44] [3], and prunes, quantizes, or compresses models [15] [18] [47]; address shift [51] [57] is hardware-friendly for compact image CNNs.Source: §2.3 · p. 3

3. Temporal Shift Module (TSM)

TSM exchanges temporal information by shifting channel subsets across frames and folding the operation into a 2D convolution without extra cost. Partial residual shifts mitigate the latency and spatial-feature losses of naive shifting.

3.1. Intuition

  • TSM shifts some channels to neighboring frames, leaves others unshifted, and merges multiply-accumulate work into the following 2D convolution at no extra cost.The shift itself requires no multiplication; the following convolution performs the multiply-accumulate work.Source: §3.1 · p. 3

3.3. Module Design

  • Residual shift preserves the original activation through identity mapping and outperforms in-place shift at every tested proportion; 1/4 shifted channels (1/8 per direction) peaks.Even shifting all channels with residual shift outperforms the 2D baseline; Figure 2 identifies 1/4 residual shift as the default, with negligible overhead.Source: Figure 2 · p. 3, §3.3 · p. 4

3.1. Intuition

  • For online recognition, TSM shifts only from past frames toward future frames because future frames are unavailable.The online variant uses a uni-directional temporal shift.Source: §3.1 · p. 3

3.2. Naive Shift Does Not Work

  • Naive shifting increased CPU latency by 13.7% and GPU latency by 12.4%, and dropped accuracy by 2.6% versus the 2D CNN baseline (TSN).Shifting all or most channels also reduces spatial modeling capacity because shifted-channel information is unavailable to the current frame.Source: §3.2 · p. 3

3.3. Module Design

  • Shifting all channels adds up to 13.7% of inference time on CPU, while shifting only 1/8 limits latency overhead to 3%, motivating the partial-shift strategy.Source: §3.3 · p. 4

4. TSM Video Network

TSM adds temporal information exchange to 2D CNNs without increasing their computation, using bidirectional shifts for offline recognition and unidirectional shifts for online recognition. The online design supports per-frame prediction with low latency and memory use while enabling multi-level temporal fusion.

4.1. Offline Models with Bi-directional TSM

  • Bi-directional TSM in each residual block gives offline 2D CNNs temporal fusion with the same parameters and computation cost as the baseline.Each inserted module enlarges the temporal receptive field by 2; the reported backbone is ResNet-50 [17] unless otherwise specified.Source: §4.1 · p. 4
  • TSM converts off-the-shelf 2D CNNs into pseudo-3D models without additional computation, using operations supported by optimized 2D CNN software and hardware.The cited software includes CuDNN [6], MKL-DNN, and TVM; hardware examples include CPU, GPU, TPU, and FPGA.Source: §4.1 · p. 4

4.2. Online Models with Uni-directional TSM

  • Uni-directional TSM shifts features only from previous frames, enabling online recognition without relying on future-frame features.For each frame, the model caches the first 1/8 of each residual block's feature maps and combines them with 7/8 current-frame maps for the next layer.Source: §4.2 · pp. 4–5
  • Per-frame prediction latency is almost the same as the 2D CNN baseline, and ResNet-50 requires only 0.9MB of feature-cache memory.The method replaces and caches 1/8 of features per frame without extra computations.Source: §4.2 · p. 5
  • TSM enables temporal fusion at all levels, which experiments identify as important for complex temporal modeling.The passage contrasts this with prior methods supporting only late or mid-level fusion [58] [61].Source: §4.2 · p. 5

5. Experiments

The experiments show that TSM improves 2D CNN video recognition without extra computation and achieves strong accuracy with efficient inference. They also evaluate TSM for online recognition, object detection, and edge deployment.

5.1. Setups

  • The experiments use multiple-clip, full-resolution evaluation for accuracy and one clip with a center 224×224 crop for efficiency comparisons.The study emphasizes temporally focused datasets such as Something-Something, while also reporting results on Kinetics, UCF101, and HMDB51.Source: §5.1 · p. 5, Table 1 · p. 5

5.3. Comparison with State-of-the-Arts

  • With 8 frames, TSM reaches 45.6% top-1 accuracy at 33G FLOPs, 4.2% above ECO with 1.9× less computation.TSM also achieves 1.2% better accuracy with 10× fewer FLOPs than Non-local I3D on the validation set, and better performance than the ECO family with 3× less computation.Source: Table 2 · p. 6, §5.3 · pp. 6–7, Figure 5 · p. 7

5.2. Improving 2D CNN Baselines

  • TSM consistently beats TSN at no extra computation, delivering double-digit gains on temporal-focused datasets and improving Kinetics accuracy across four backbones.The temporal-focused datasets include Something-Something V1 and V2 and Jester; the backbones are MobileNet-V2, ResNet-50, ResNext-101, and ResNet-50 with a Non-local module.Source: §5.2 · pp. 5–6, Table 3 · p. 6

5.3. Comparison with State-of-the-Arts

  • TSM takes first place on the Something-Something-V1 leaderboard and achieves state-of-the-art on V2 using RGB input.The V1 evaluation uses center crops for efficiency unless otherwise specified.Source: §5.3 · pp. 6–7, Table 2 · p. 6, Table 4 · p. 7

5.4. Latency and Throughput Speedup

  • Against ECO [61], TSM has 1.75× lower latency (17.4ms vs. 30.6ms), 1.7× higher throughput, and 2% better accuracy.Against I3D [50], TSM is faster by an order of magnitude at 1.8% higher accuracy; compared with two inflated-convolution alternatives, it has higher throughput and accuracy or is faster and performs better.Source: Table 5 · p. 7, §5.4 · p. 7, Figure 6 · p. 8

5.5. Online Recognition with TSM

  • Online TSM is within 1.0% of offline TSM on Something-Something and comparable or sometimes better on Kinetics, UCF101, and HMDB51.It adds less than 0.1ms latency per frame while improving accuracy by up to 25%; at 10% of UCF101 frames, it reaches 90% accuracy, 6.6% above the best ECO model.Source: §5.5 · p. 8, Table 6 · p. 8, Figure 6 · p. 8
  • On ImageNet-VID, online TSM improves R-FCN mAP by 4.6% on fast-moving objects and matches or exceeds FGFA with much smaller latency.A MobileNet-V2 online TSM reaches 69.5% accuracy on Kinetics and runs at low latency and power across the reported edge devices.Source: Table 7 · p. 8, §5.5 · p. 8, Table 8 · p. 8

6. Conclusion

The Temporal Shift Module enables joint spatial-temporal modeling in 2D CNNs by shifting channels across neighboring frames, with no additional cost. The framework is designed for efficient, accurate, low-latency video recognition on edge devices.

6. Conclusion

  • The module shifts part of the channels along the temporal dimension to exchange information with neighboring frames.Source: §6 · p. 8
  • Inserted into a 2D CNN backbone, the module enables joint spatial-temporal modeling at no additional cost.Source: §6 · p. 8
  • The framework combines efficiency and accuracy to enable low-latency video recognition on edge devices.Source: §6 · p. 8

A. Uni-directional TSM for Online Video Detection

Uni-directional TSM enables temporal fusion in an online video-detection backbone without changing the detector design or using optical flow. Experiments show improved detection over 2D R-FCN and competitive performance with FGFA using only past frames.

A. Uni-directional TSM for Online Video Detection

  • Injecting uni-directional TSM into the backbone enables online temporal fusion without changing the detection module design or using optical flow features.The paper contrasts this with existing video-detection methods that fuse information after backbone feature extraction.Source: A. Uni-directional TSM for Online… · p. 11
  • TSM increases mAP by 4.6% on fast-moving objects over 2D R-FCN [23] and achieves similar or higher performance than offline FGFA using only past frames.The TSM module adds less than 1ms of latency per frame; experiments used R-FCN with a ResNet-101 backbone on ImageNet-VID.Source: A. Uni-directional TSM for Online… · p. 11
  • TSM suppresses false positives caused by headlight glare and traffic-sign occlusion, and helps detect a motorcycle obscured by a traffic sign.These examples compare TSM with 2D baseline R-FCN across frames 2–4.Source: A. Uni-directional TSM for Online… · p. 11

B. Video Demo

The section directs readers to the project page for additional TSM video demos.

B. Video Demo

  • Additional video demos of the TSM model are available on the project page.The paper provides the project-page link for these demos.Source: B. Video Demo · p. 11

R-FCN TSM

Compared with 2D R-FCN, TSM uses temporal information to address detection errors caused by headlight glare and traffic-sign occlusion.

R-FCN TSM

  • TSM avoids the false positives that 2D R-FCN produces from car-headlight glare across frames 2–4.Source: Figure 7 · p. 12
  • Under traffic-sign occlusion, TSM addresses R-FCN’s false positives around the bus and its missed motorcycle detection on frame 4.Source: Figure 8 · p. 13

Related papers

Loading 1811.08383v3…