PaperlayerView exact revision on arXiv

Source-linked AI summary

SVFormer: Semi-supervised Video Transformer for Action Recognition

Zhen Xing, Qi Dai, Han Hu, Jingjing Chen, Zuxuan Wu, Yu-Gang Jiang

arXiv:2211.13222v2cs.CV

TL;DR

Semi-supervised action recognition can use plentiful unlabeled videos, but transformer approaches to the setting had received limited exploration and image augmentations do not account for video’s temporal structure. SVFormer combines consistency training with Tube TokenMix and temporal warping, and reports state-of-the-art results across three benchmarks without increased overhead.

  • Problem

    Video annotation is labor-intensive, transformer use in semi-supervised action recognition remained limited, and image mixing does not account for video’s temporal structure.

    Source: §1 · pp. 1–2

  • Method

    SVFormer uses consistency between differently augmented views, with token-level Tube TokenMix and temporal warping to augment video clips.

    Source: §1 · p. 2

  • Results

    SVFormer outperforms state-of-the-art methods on UCF-101, HMDB-51, and Kinetics-400 without increased overhead; at 1% labels, it improves on by 15.0% on Kinetics-400.

    Source: §4.2 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    The authors present SVFormer as a benchmark for semi-supervised action recognition and encourage future work on transformer architectures.

    Source: §5 · p. 8

Abstract

from arXiv · show

Semi-supervised action recognition is a challenging but critical task due to the high cost of video annotations. Existing approaches mainly use convolutional neural networks, yet current revolutionary vision transformer models have been less explored. In this paper, we investigate the use of transformer models under the SSL setting for action recognition. To this end, we introduce SVFormer, which adopts a steady pseudo-labeling framework (ie, EMA-Teacher) to cope with unlabeled video samples. While a wide range of data augmentations have been shown effective for semi-supervised image classification, they generally produce limited results for video recognition. We therefore introduce a novel augmentation strategy, Tube TokenMix, tailored for video data where video clips are mixed via a mask with consistent masked tokens over the temporal axis. In addition, we propose a temporal warping augmentation to cover the complex temporal variation in videos, which stretches selected frames to various temporal durations in the clip. Extensive experiments on three datasets Kinetics-400, UCF-101, and HMDB-51 verify the advantage of SVFormer. In particular, SVFormer outperforms the state-of-the-art by 31.5% with fewer training epochs under the 1% labeling rate of Kinetics-400. Our method can hopefully serve as a strong benchmark and encourage future search on semi-supervised action recognition with Transformer networks.

1. Introduction

SVFormer brings transformer models to semi-supervised action recognition, a setting with limited prior exploration, and introduces video-specific augmentations. The paper reports strong comparisons with prior methods, including at low labeling rates.

1. Introduction

  • Video annotations are labor-intensive, motivating semi-supervised action recognition that uses labeled and unlabeled videos.Prior approaches use pseudo-labels to refine pretrained models, but may add training or inference costs through extra modalities or auxiliary networks.Source: §1 · p. 1
  • Transformer use in semi-supervised video tasks remained unexplored, while low-data results for vision transformers and direct SSL transfer to ViT were weak.The authors note that ImageNet-initialized TimeSformer showed promising results with limited video annotations, and that ImageNet pretraining is important for decent low-data action-recognition results.Source: §1 · pp. 1–2
  • Tube TokenMix mixes strongly augmented video clips after tokenization with a mask whose masked tokens remain consistent across time.The authors propose this token-level design as more suitable for video transformers than pixel-level mixing strategies.Source: §1 · p. 2
  • SVFormer is a transformer-based semi-supervised action-recognition method using consistency between differently augmented views; its SVFormer-B and SVFormer-S variants outperform state-of-the-art approaches by clear margins across three benchmark datasets.Source: §1 · p. 2
  • Temporal Warping Augmentation changes the duration of selected frames to cover complex temporal variation beyond simple frame-rate scaling or shifting.The paper reports significant improvements when TWAug is combined with Tube TokenMix.Source: §1 · p. 2
  • SVFormer-S outperforms prior state-of-the-art methods with similar parameters and inference cost, while SVFormer-B also performs strongly; the paper reports results across three benchmarks.At a 1% labeling rate on Kinetics-400, the supervised Transformer baseline is comparable with the 3D-ResNet state-of-the-art method.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Works

Related work covers semi-supervised learning, video action recognition, and image augmentation, while identifying limited transformer exploration in semi-supervised video recognition. The paper situates its video-specific mixing and temporal augmentation against these approaches.

Deep Semi-supervised Learning

  • Semi-supervised learning uses a few labeled and many unlabeled samples, commonly combining augmented views with consistency regularization or generating mixed data and labels.FixMatch and variants have been applied to other tasks, but may not transfer satisfactorily to video action recognition without temporal augmentation.Source: §2 · p. 2

Semi-supervised Action Recognition

  • Earlier semi-supervised action-recognition methods use contrastive learning, self-supervised tasks, extra modalities, or auxiliary networks; prior methods were convolutional and required more training epochs.The passage names VideoSSL, TCL, MvPL, LTG, and CMPL.Source: §2 · pp. 2–3

Video Transformer

  • Video transformers explored varied temporal-attention designs and convolutional inductive biases, but work in the semi-supervised setting remained limited.Examples include VTN, TimeSformer, ViViT, MViT, Video Swin, Uniformer, and Video Mobile-Former [53].Source: §2 · p. 3

Data Augmentation

  • Most surveyed augmentation methods were designed for images, whereas the paper presents TTMix coupled with TWAug as devised for video.The passage describes Cutout [12], Mixup, CutMix, AutoAugment, PixMix, and token-level TokenMix.Source: §2 · p. 3

3. Method

SVFormer trains a video transformer on labeled and unlabeled clips using an EMA teacher, confidence-filtered pseudo-labels, and consistency losses. Tube TokenMix mixes video tokens with a temporally consistent mask, while Temporal Warping varies frame durations before mixing.

3.1. Preliminaries of SSL

  • Semi-supervised learning uses labeled and unlabeled videos together, with the unlabeled set typically much larger than the labeled set.Source: §3.1 · p. 3

3.2. Pipeline

  • SVFormer uses weak and strong views for pseudo-label consistency, filters low-confidence labels, and updates an EMA teacher from student parameters.Source: §3.2 · pp. 3–4

3.3. Tube TokenMix

  • Tube TokenMix mixes strongly augmented clips after tokenization using a mask shared across time, then interpolates their teacher pseudo-labels using the mask ratio.Source: §3.3 · pp. 4–5
  • Temporal Warping selects frames and pads masked positions with neighboring visible frames, preserving frame order while varying temporal duration.Source: §3.3 · p. 5

3.4. Training Paradigm

  • The training objective combines supervised, unlabeled pseudo-label consistency, and Tube TokenMix consistency losses.Source: §3.4 · p. 5

4. Experiment

Experiments evaluate SVFormer across Kinetics-400, UCF-101, and HMDB-51, reporting benchmark results and examining the framework, augmentations, inference schemes, and hyperparameters.

4.1. Experiment Settings

  • Kinetics-400 uses 1%/10% labels, UCF-101 uses 1/10 labeled videos per category, and HMDB-51 uses 40%/50%/60%; main results report Top-1 accuracy.Unless stated otherwise, inference uses RGB on the official validation set.Source: §4.1 · p. 6, §4 · p. 6

4.2. Main Results

  • 6.3% in UCF-101 and 15.0% in Kinetics-400: SVFormer-S improves prior approach by these margins and leads RGB-only methods in accuracy with the fewest epochs.The authors also report that SVFormer-B significantly outperforms state-of-the-art methods.Source: §4.2 · p. 7
  • 69.4% on Kinetics-400 with 10% labeled data is comparable to TimeSformer’s 77.9% fully supervised result; on HMDB-51, SVFormer-S/B improve over by about 10%/15%.Table 2 compares state-of-the-art methods on HMDB-51 using Top-1 accuracy.Source: Table 2 · p. 6, §4.2 · p. 7

4.3. Ablation Studies

  • EMA-Teacher outperforms FixMatch on Kinetics-400 and UCF-101 with very few labeled samples, probably because it improves training stability.Both SSL frameworks improve the baseline approach.Source: §4.3 · p. 7, Table 3 · p. 7
  • All mixing methods improve performance; token-level Rand and Tube TokenMix outperform pixel-level methods, while Frame TokenMix performs worse than pixel-level mixing.The authors hypothesize that consistent masked tokens prevent information leakage between adjacent frames and better model spatio-temporal correlations.Source: §4.3 · pp. 7–8, Table 4 · p. 7
  • Spatial augmentation and temporal warping each improve the baseline, while using both gives the best results.Removing both strong spatial augmentation and TWAug causes performance to drop dramatically.Source: §4.3 · p. 8, Table 5 · p. 8
  • Video-based 8 × 32 sampling reduces inference cost but performs worse than clip-based sampling; 8 × 8 and 16 × 4 perform better.The authors select 8 × 8 as the default for its efficiency-accuracy trade-off.Source: §4.3 · p. 8, Table 6 · p. 8
  • On Kinetics-400 with 1% labels, the selected settings are δ = 0.3, Bu = 5, m = 0.99, and γ1 = γ2 = 2.The study varies the confidence threshold, unlabeled-to-labeled batch ratio, EMA momentum, and loss weights.Source: §4.3 · p. 8, Figure 7 · p. 8

5. Conclusion

SVFormer combines Tube TokenMix with temporal warping augmentation for semi-supervised video action recognition. The paper reports state-of-the-art results across three datasets without increased overheads and presents the method as a benchmark for future transformer-based work.

5. Conclusion

  • SVFormer outperforms state-of-the-art methods by a large margin on UCF-101, HMDB-51, and Kinetics-400 without increasing overheads.Source: §5 · p. 8
  • Tube TokenMix is designed for video transformers, and temporal warping augmentation varies frame lengths to cover complex temporal variation.Together, these augmentations improve on conventional augmentations.Source: §5 · p. 8
  • The work establishes a benchmark for semi-supervised action recognition and encourages future transformer-based research.Source: §5 · p. 8

Related papers

Loading 2211.13222v2…