PaperlayerView exact revision on arXiv

Source-linked AI summary

An Image is Worth 16x16 Words, What is a Video Worth?

Gilad Sharir, Asaf Noy, Lihi Zelnik-Manor

arXiv:2103.13915v2cs.CV

TL;DR

Action recognition methods using 3D convolutions commonly rely on dense, multi-clip sampling, while efficient recognition is needed for applications such as online recognition and retrieval. STAM applies spatial and temporal transformers to uniformly sampled frames. On Kinetics-400, it reports 78.8 top1-accuracy with ×40 faster inference than the leading method cited in the introduction.

  • Problem

    Dense, multi-clip 3D-convolution methods create computational bottlenecks, while efficient recognition is needed for online recognition and video retrieval.

    Source: §1 · p. 1

  • Method

    STAM applies spatial and temporal transformers to uniformly sampled video frames, using global attention to model the sequence.

    Source: §1 · p. 2

  • Results

    78.8 top1-accuracy with ×40 faster inference time is reported on Kinetics-400.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present STAM as a candidate for latency-sensitive applications such as real-time recognition and video retrieval.

    Source: §5 · p. 8

Abstract

from arXiv · show

Leading methods in the domain of action recognition try to distill information from both the spatial and temporal dimensions of an input video. Methods that reach State of the Art (SotA) accuracy, usually make use of 3D convolution layers as a way to abstract the temporal information from video frames. The use of such convolutions requires sampling short clips from the input video, where each clip is a collection of closely sampled frames. Since each short clip covers a small fraction of an input video, multiple clips are sampled at inference in order to cover the whole temporal length of the video. This leads to increased computational load and is impractical for real-world applications. We address the computational bottleneck by significantly reducing the number of frames required for inference. Our approach relies on a temporal transformer that applies global attention over video frames, and thus better exploits the salient information in each frame. Therefore our approach is very input efficient, and can achieve SotA results (on Kinetics dataset) with a fraction of the data (frames per video), computation and latency. Specifically on Kinetics-400, we reach $80.5$ top-1 accuracy with $\times 30$ less frames per video, and $\times 40$ faster inference than the current leading method. Code is available at: https://github.com/Alibaba-MIIL/STAM

1. Introduction

STAM applies spatial and temporal transformers to uniformly sampled video frames, aiming to reduce the dense-frame computation that limits efficient action recognition. The paper presents it as a fast, end-to-end trainable approach with strong Kinetics-400 accuracy.

1. Introduction

  • Dense, multi-clip 3D-convolution methods process many frames, creating computational bottlenecks that can make real-time applications impractical.The introduction notes that real-world applications can span longer intervals than the short actions targeted by multi-clip inference.Source: §1 · pp. 1–2
  • STAM uses spatial and temporal transformers to model video frames and capture global attention across the sequence.Frames are sampled uniformly, divided into patches, and processed through spatial then temporal transformer encoders.Source: §1 · pp. 1–2
  • Using 16 frames instead of 480 in the described inference scenario reduces frame-access cost 30-fold, in addition to reducing runtime.The paper highlights this input reduction as useful when retrieving frames from storage or across a communications network incurs a cost.Source: §1 · p. 2
  • STAM is end-to-end trainable, using the same sampling strategy and model for training and inference rather than adding test-time multi-clip averaging.The paper contrasts this with an inference-only temporal aggregation layer whose effects are not accounted for during training.Source: §1 · p. 2
  • 78.8 top1-accuracy with ×40 faster inference, or +8% ECO accuracy while twice as fast, is reported for Kinetics-400.The paper presents these as alternative efficiency–accuracy comparisons.Source: §1 · p. 3

2. Related Work

Related work spans transformer-based vision methods and approaches to action recognition that modify convolutional architectures, sampling, or temporal aggregation. The paper distinguishes STAM by applying transformers across spatial and temporal dimensions without relying on dense-frame 3D convolutions.

2.1. Transformers and Self Attention

  • Self-attention relates positions within a sequence to compute its representation and forms a building block of Transformer encoders.The paper describes Transformers as dominant in NLP and notes their use in vision tasks.Source: §2.1 · p. 3
  • Vision transformers use attention-based layers for image tasks, while a related action-recognition model combines transformers with 3D convolutions and retains dense-sampling requirements.The paper also notes that some vision methods use pixel-level attention and approximate global attention through downsampling or local attention.Source: §2.1 · p. 3

2.2. Action Recognition

  • X3D, SlowFast, and R3D improve convolutional action-recognition efficiency through architecture or resolution choices, but still require densely sampled frames.STAM instead models temporal information with a self-attention sequence model.Source: §2.2 · p. 3
  • Other methods propagate temporal information through local neighborhoods or reduce computation with adaptive resolution, importance sampling, or feature-map approximations, yet still use multi-clip testing.The paper identifies multi-clip inference as a remaining source of inefficiency in these approaches.Source: §2.2 · p. 3
  • Subsampled-video methods include single-frame representations, video encodings, and ECO, which uniformly samples frames before applying a long-term spatio-temporal architecture.The paper says STAM has similar motivation and input data to ECO, with spatial and temporal encoders offering an improvement.Source: §2.2 · p. 3

3. Method

STAM extends visual transformers to video with separate spatial and temporal self-attention, processing each frame before aggregating frame-level representations. The design reduces computation by applying temporal attention to frame embeddings rather than all spatiotemporal patches.

3. Method

  • STAM combines spatial attention within each frame with temporal attention over frame embeddings to produce a video-level classification.The model is convolution-free and extends the Vision Transformer approach to video.Source: §3 · pp. 3–4, Figure 3 · p. 4
  • Applying temporal attention to frame-level representations reflects the design choice to model temporal information at a more abstract level than individual patches.The paper motivates this choice by noting that frame representations convey more about a video than individual patches.Source: §3 · pp. 4, 6
  • Each video frame is divided into non-overlapping patches that are flattened, linearly projected, and combined with learnable positional embeddings and a classification token.A separate classification token is used for each frame to carry its information into temporal modeling.Source: §3 · p. 4
  • Spatial attention operates between patches of the same frame, and its outputs are combined through attention heads and MLP layers with skip connections.The spatial Transformer produces a class embedding for each frame.Source: §3 · p. 5
  • Temporal attention processes the per-frame embeddings along the frame axis and uses a classification embedding from the top layer to form the video representation.The model applies a classifier to the resulting video representation.Source: §3 · p. 6
  • Separating spatial and temporal attention reduces computation: STAM has complexity O(FN^2 + F^2), compared with O((FN)^2) for attention over all spatiotemporal patches.The temporal stage acts on F frame embeddings, while the spatial stage attends over N patches within each frame.Source: §3 · pp. 4, 6
  • The alternative of applying temporal attention to patches after spatial attention within each block produced slightly worse results and higher computational cost.The paper also reports that 3D-convolution methods degraded on the same subsampled data: X3D error increased by 23% and SlowFast error by 50%.Source: §3 · pp. 4, 6

4. Experiments

STAM combines spatial and temporal attention over uniformly sampled video frames, aiming to reduce computation while retaining action-recognition accuracy. Experiments report competitive Kinetics-400 accuracy and faster inference, with ablations examining attention, backbone, frame count, and runtime.

4. Experiments

  • STAM uses a ViT B spatial Transformer and a smaller six-layer, eight-head temporal Transformer, with uniform frame sampling for training and inference.The spatial model has 12 multi-head self-attention layers with 12 heads and ImageNet-21K pretraining; the temporal layers are trained from scratch. Training uses 8 V100 GPUs for 30 epochs.Source: §4 · p. 6
  • 43× faster inference than X3D L accompanies a reduction from 2.27 to 0.05 hours on the validation set.The paper attributes the runtime improvement partly to fewer input frames and partly to the ViT B model's efficiency relative to X3D.Source: §4 · p. 6
  • 79.3% top-1 accuracy on Kinetics-400 with 16 frames (270 GFLOPS) compares with X3D L's 77.5% using 30 clips (744 GFLOPS), a 1.8-point gain at 36% of its computation.Table 1 reports the model comparison; inference runtime was measured on the Kinetics-400 validation set using 8 GPUs, and VPS on a single GPU.Source: §4 · p. 6, Table 1 · p. 7

4.1. Ablation Experiments

  • The temporal Transformer improves over averaging frame embeddings, with a reported gain of 2.7; the improvement is also observed with a TResNet-M backbone.Table 2 compares the temporal Transformer representation with the mean of frame embeddings.Source: §4.1 · pp. 6, 8, Table 2 · p. 7
  • The full spatial-and-temporal Transformer achieves the highest accuracy among the tested backbones, while CNN backbones with temporal attention remain viable but less powerful.The comparison includes TResNet-M, TResNet-L, and ResNet50; Table 3 notes their respective pretraining datasets.Source: Table 3 · p. 7, §4.1 · p. 8
  • Adding 16 frames raises accuracy by 0.6% from 16 to 32 frames and by another 0.6% from 32 to 64, while larger inputs do not improve accuracy.The experiments use uniformly sampled frames with the ViT+Temporal model; the paper also notes that four frames can classify a 10-second video and estimates 24 frames for a one-minute video at the same sampling rate.Source: Table 4 · p. 7, §4.1 · p. 8
  • With a single 16-frame clip, X3D and SlowFast lose substantial accuracy, indicating that their runtime cannot be improved to the same extent by sparse sampling.Table 5 evaluates the methods with one clip and reports 30-view accuracy in parentheses.Source: Table 5 · p. 7, §4.1 · p. 8
  • At 20 videos per second, STAM matches ECO's runtime while exceeding its accuracy by 8%.Because ECO also samples individual frames across the video, the authors conclude that STAM's temporal Transformer better captures temporal information from separate frames.Source: §4.1 · p. 8

5. Conclusion

The paper presents an end-to-end, transformer-based method for video action recognition that models videos as paragraphs and uniformly selected frames as sentences. It reports accuracy comparable to state-of-the-art methods with faster inference, and identifies latency-sensitive recognition and video retrieval as potential applications.

5. Conclusion

  • STAM models video as a paragraph and uniformly selected frames as sentences to capture spatiotemporal dependencies with transformers.The paper describes the method as simple, end-to-end, and generic for further video-understanding tasks.Source: §5 · p. 8
  • The models make accurate predictions from a small fraction of video data, with accuracy comparable to state-of-the-art methods and inference faster by orders of magnitude.Source: §5 · p. 8
  • The authors identify latency-sensitive real-time recognition and video retrieval as candidate applications for the method.Source: §5 · p. 8

A. Additional Datasests

The paper evaluates its model on UCF101 and Charades in addition to its other datasets. It reports a runtime–accuracy trade-off on UCF101 and an efficient-model result on Charades, where it does not reach state-of-the-art accuracy.

A. Additional Datasests

Related papers

Loading 2103.13915v2…