PaperlayerView exact revision on arXiv

Source-linked AI summary

Token Shift Transformer for Video Classification

Hao Zhang, Yanbin Hao, Chong-Wah Ngo

arXiv:2108.02432v1cs.CVcs.MM

TL;DR

Transformers handle language and images effectively, but applying them to video is computationally demanding because video sequences lengthen pair-wise attention. The paper introduces TokShift, a zero-cost temporal shift of partial [Class] token features within a 2D vision transformer, and reports comparable or better performance than leading 3D-CNNs across three benchmarks.

  • Problem

    Applying transformers to video is computationally demanding because longer spatiotemporal sequences make pair-wise self-attention costly.

    Source: §1 · p. 2

  • Method

    TokShift shifts partial [Class] token features across frames and is inserted into each encoder of a plain 2D vision transformer.

    Source: §1 · p. 2

  • Results

    Across Kinetics-400, EGTEA-Gaze+, and UCF-101, TokShift-xfmr reports SOTA top-1 accuracies comparable or better than the best 3D-CNNs, including 80.40% on Kinetics-400 (12f).

    Source: §1 · p. 2

  • Takeaways & Limitations

    TokShift provides a convolution-free video-transformer approach that maintains common 2D ViT complexity while achieving comparable or better performance than 3D-CNN SOTAs.

    Source: §5 · p. 7

    Despite TokShift's zero parameters and FLOPs, the 2D transformer still uses more computation than 3D-CNNs and retains attention burden, especially for long videos.

    Source: §4.4 · p. 7, §5 · p. 8

Abstract

from arXiv · show

Transformer achieves remarkable successes in understanding 1 and 2-dimensional signals (e.g., NLP and Image Content Understanding). As a potential alternative to convolutional neural networks, it shares merits of strong interpretability, high discriminative power on hyper-scale data, and flexibility in processing varying length inputs. However, its encoders naturally contain computational intensive operations such as pair-wise self-attention, incurring heavy computational burden when being applied on the complex 3-dimensional video signals. This paper presents Token Shift Module (i.e., TokShift), a novel, zero-parameter, zero-FLOPs operator, for modeling temporal relations within each transformer encoder. Specifically, the TokShift barely temporally shifts partial [Class] token features back-and-forth across adjacent frames. Then, we densely plug the module into each encoder of a plain 2D vision transformer for learning 3D video representation. It is worth noticing that our TokShift transformer is a pure convolutional-free video transformer pilot with computational efficiency for video understanding. Experiments on standard benchmarks verify its robustness, effectiveness, and efficiency. Particularly, with input clips of 8/12 frames, the TokShift transformer achieves SOTA precision: 79.83%/80.40% on the Kinetics-400, 66.56% on EGTEA-Gaze+, and 96.80% on UCF-101 datasets, comparable or better than existing SOTA convolutional counterparts. Our code is open-sourced in: https://github.com/VideoNetworks/TokShift-Transformer.

1 INTRODUCTION

The introduction frames video as an open extension of transformer success in language and images, with long spatiotemporal sequences making pair-wise attention computationally demanding. It proposes TokShift, which moves partial [Class] token features across frames, and reports competitive benchmark results against 3D-CNNs.

1 INTRODUCTION

  • The paper presents video understanding as an open hypothesis after convolution-free transformers had extended to image understanding.It frames the study as addressing whether transformer-like approaches can also interpret videos.Source: §1 · p. 1
  • Video transformers face heavy pair-wise self-attention computations because video sequences include an extra temporal axis and are longer than image sequences.The introduction describes the resulting computation as an exponential explosion, including T^2·(N+1)^2 pair-wise calculations.Source: §1 · p. 2
  • TokShift is a zero-parameter, zero-FLOPs operator that shifts partial [Class] token features back-and-forth across frames to model temporal relations.The module is densely inserted into each encoder of a plain 2D vision transformer to learn 3D video representations.Source: §1 · p. 2
  • TokShift shifts only the [Class] token features rather than all receptive fields, and the authors report this variant improves most among the tested shift variants.Their explanation relates partial temporal exchange to motion information while leaving remaining features for static semantics.Source: §1 · p. 2
  • 79.83%/80.40% top-1 accuracy on Kinetics-400 (8/12f), 66.56% on EGTEA-Gaze+ (8f), and 96.80% on UCF-101 (8f) are reported as SOTA, comparable or better than best 3D-CNNs.The compared 3D-CNNs named are TSM, SlowFast-R101-NL, and X3D-XXL.Source: §1 · p. 2

2 RELATED WORKS

The related-work section situates TokShift among video-classification 3D-CNNs, shift operators for convolutional efficiency, and language and vision transformers.

2 RELATED WORKS

  • The paper organizes related literature into 3D-CNN video classification, shift strategies for convolutional efficiency, and 1D/2D transformers.Source: §2 · p. 2
  • Video classification literature includes 3D-CNNs such as C3D, two-stream networks, I3D, Pseudo-3D, SlowFast, and X3D.The review notes Pseudo-3D decomposes spatiotemporal convolution into spatial and temporal components to balance computation, parameters, and performance.Source: §2 · p. 2
  • SlowFast uses dual paths with different sampling rates, while X3D searches for network hyperparameters; the review identifies them as leading 3D-CNNs.Source: §2 · p. 2
  • CNN shift methods use zero-parameter, zero-FLOPs feature aggregation; TSM applies temporal shift with spatial convolutions, while RubiksNet uses learnable spatiotemporal shifts.Source: §2 · pp. 2–3
  • Transformer work spans attention optimizations for long inputs, vision transformers for image classification, and transformer applications to detection and segmentation.The review associates Reformer [17] and Longformer [1] with attention optimization, ViT [9] with image classification, and detection and segmentation with and [35].Source: §2 · p. 3

3 METHOD

TokShift-xfmr applies a transformer pipeline to video, inserting temporal processing into its encoders by shifting partial global [Class] token features. Its method compares shift variants and encoder insertion positions.

3 METHOD

  • TokShift-xfmr uses L replicated encoders containing TokShift, Layer-Norm, multi-head self-attention, and a feed-forward network.TokShift manipulates the global [Class] token along the temporal axis before video embeddings pass through MSA or FFN.Source: §3 · p. 3

3.2 Integration into ViT Encoder

  • The encoder-integrated TokShift can be placed before the residual, layer normalization, or MSA/FFN, or after MSA/FFN.Earlier positions affect more subsequent modules in the residual blocks.Source: Figure 3 · p. 4, §3.2 · p. 5

3 METHOD

  • The video label is obtained by averaging frame-level predictions from the final encoder’s outputs.The classification layer applies a fully connected layer to those outputs.Source: §3 · p. 4

3.1 Shift Variants and the TokShift

  • TokShift splits the global token tensor by channel, shifts two splits across prior or later timestamps, and leaves the middle split unchanged.The channel-shift percentages depend on a.Source: §3.1 · p. 4
  • Temporal Shift exchanges partial channels across all patches and the [Class] token, whereas Patch Shift shifts patch embeddings but exempts the global token.Source: §3.1 · p. 4
  • TokShift performs best among the four shift variants, supporting the use of [Class] tokens for temporal modeling in video transformers.The authors describe it as making minimum modifications to z_l while granting maximum benefits.Source: §3.1 · p. 5

3.2 Integration into ViT Encoder

  • Placing TokShift before the residual block brings the best effect among the tested encoder positions.The authors contrast this with TSM on CNN, whose optimal position is post-residual.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments compare TokShift transformers with video-classification baselines and examine datasets, implementation choices, ablations, attention maps, and fine-tuning. TokShift-Large performs comparably to or better than leading 3D-CNNs on Kinetics-400, while fine-tuning results vary with the pretraining weights.

4.1 Datasets

  • Kinetics-400 is a large-scale benchmark; EGTEA Gaze+ covers 106 fine-grained first-person daily actions, and UCF-101 covers 101 action categories.The experiments use EGTEA Gaze+ and UCF-101 split 1.Source: §4.1 · p. 5

4.2 Implementations

  • Implemented atop 2D ViT in PyTorch, models are evaluated with Top-1/5 accuracy and parameters/GFLOPs; each video prediction averages 30 sampled sub-clips.The 30 sub-clips come from 10 sampled clips, each cropped into three spatial views.Source: §4.2 · p. 5, §4 · p. 5

4.3 Ablation Study

  • Among TokShift placements, prior-residual placement achieves the best top-1 accuracy while all models share the same GFLOPs; TokShift-xfmr performs better with deeper layers, with Large-16 compared to Base-16.Performance further increases when using more frames (79.83→80.40; 77.55→78.34).Source: §4.3 · pp. 5–6
  • TokShift tops the shift variants while patch shifts reduce performance; shifting one-quarter of channels each way is optimal, leaving half unchanged.The paper attributes patch-shift reductions to visual misalignment of moving objects across frames.Source: §4.3 · p. 6, Table 3 · p. 6, Table 4 · p. 6
  • 78.18 versus 78.14 with 3,152 versus 4,616 visual words favors temporal over spatial resolution, while increasing either frame or spatial-grid counts improves performance.The comparisons use middle or high resolutions and vary visual word counts.Source: Table 6 · p. 6, §4.3 · p. 6

4.4 Comparison with the State-of-the-Art

  • 80.40 for TokShift-Large exceeds SlowFast (79.80) and matches X3D-XXL (80.4) on Kinetics-400, using 12 frames versus 32 and 16+ frames, respectively.TokShift-xfmr (MR) also scores 77.68 versus Non-Local R101's 77.7 using 8 versus 128 frames at the same spatial resolution.Source: §4.4 · p. 6, Table 7 · p. 7

4.5 Attention Visualization

  • Attention maps from sampled Kinetics-400 clips highlight action-relevant objects, including kites, and sometimes all kites in the sky.The maps average the last encoder's multiple heads; half of each clip's eight frames are shown.Source: §4.5 · p. 7

4.6 Fine-tune on small-scale datasets

  • Kinetics-400 pretraining scores 64.82 versus 62.85 and 28.90 on EGTEA Gaze+, and 95.35 versus 91.65 and 91.60 on UCF-101; optical flow raises EGTEA performance from 64.82 to 65.08.Freezing layer norms helps fine-tuning from Kinetics-400 weights but hurts with ImageNet-21k weights.Source: §4.6 · p. 7, Table 8 · p. 7

5 CONCLUSION

TokShift-xfmr uses a zero-parameter, zero-FLOPs operator to model video temporal relations through global token shifts, reporting performance comparable to or better than 3D-CNN SOTAs. The 2D ViT still carries attention costs, particularly for long videos.

5 CONCLUSION

  • 80.40% performance is reported as better or comparable to 3D-CNN SOTAs.TokShift-xfmr alleviates pair-wise distance calculations while maintaining the complexity of a common 2D ViT.Source: §5 · p. 7
  • The conclusion references Kinetics-400 attention maps, t-SNE feature distributions for pretrained weights, and a UCF-101 pretrained-weight comparison.Source: Figure 4 · p. 8, Figure 5 · p. 8, Table 9 · p. 8
  • The global [Class] token aggregates spatial patches and reflects global visual content, providing a basis for temporal interaction across frames.The operator is described as zero-parameter and zero-FLOPs.Source: §5 · pp. 7–8
  • The 2D ViT retains an inherent attention-computation burden, especially for long videos, which the paper leaves for future work.Source: §5 · p. 8

Related papers

Loading 2108.02432v1…