PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications

Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, Krzysztof Chalupka

arXiv:2003.01455v4cs.CV

TL;DR

Video annotation is costly, and whether modern deep networks can recognize actions zero-shot remains an open question. The paper trains an end-to-end video model and evaluates it under stricter class and domain separation. It reports that the method outperforms previous work even under this stricter protocol.

  • Problem

    Video action data is expensive to annotate, and how well modern deep networks recognize actions zero-shot remains an open question.

    Source: §1 · p. 1

  • Method

    The paper proposes an end-to-end zero-shot action-recognition model and evaluates one trained model across datasets with disjoint classes and differing video domains.

    Source: §1 · p. 2

  • Results

    The method outperforms previous works under a stricter protocol than evaluations with overlapping training and test sets and shared domains.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Experiments indicate that good ZSL datasets should contain many diverse classes, and the paper’s pretraining technique boosts ZSL performance.

    Source: §6 · p. 8

    The evaluation argues that training and test video domains should ideally have disjoint video sources.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Trained on large datasets, deep learning (DL) can accurately classify videos into hundreds of diverse classes. However, video data is expensive to annotate. Zero-shot learning (ZSL) proposes one solution to this problem. ZSL trains a model once, and generalizes to new tasks whose classes are not present in the training dataset. We propose the first end-to-end algorithm for ZSL in video classification. Our training procedure builds on insights from recent video classification literature and uses a trainable 3D CNN to learn the visual features. This is in contrast to previous video ZSL methods, which use pretrained feature extractors. We also extend the current benchmarking paradigm: Previous techniques aim to make the test task unknown at training time but fall short of this goal. We encourage domain shift across training and test data and disallow tailoring a ZSL model to a specific test dataset. We outperform the state-of-the-art by a wide margin. Our code, evaluation procedure and model weights are available at github.com/bbrattoli/ZeroShotVideoClassification.

1. Introduction

Because video annotations are costly and the action space is large, the paper asks whether modern deep networks can perform video action recognition zero-shot. It introduces an end-to-end model and a more realistic evaluation protocol.

1. Introduction

  • Because human actions span a huge set of possible classes, action recognition is a strong testbed, but modern deep networks’ zero-shot performance remains an open question.Video action data is particularly expensive to source and annotate.Source: §1 · p. 1
  • The paper proposes the first end-to-end-trained model for zero-shot action recognition, adapting modern supervised video-classification practices and adding a pretraining technique for the ZSL scenario.The authors describe the method as simple and report that it outperforms previous work.Source: §1 · p. 2
  • The evaluation protocol tests one trained model on multiple datasets with disjoint training and test classes, and encourages different training and test video domains.The authors present these conditions as a more realistic ZSL setting.Source: §1 · p. 2
  • The paper analyzes the end-to-end model and a pretrained baseline, and uses guided experiments to explore characteristics of good ZSL datasets.The authors also provide the model, training code, and evaluation code.Source: §1 · p. 2

2. Related work

Prior video ZSL commonly maps pretrained visual features into semantic embeddings, while many evaluations reuse datasets or overlap training and test classes. This paper focuses on inductive ZSL and discusses differences in video-classification architectures and universal action representations.

2. Related work

  • The paper studies inductive ZSL, where test data is unknown during training, and does not cover transductive methods that use unlabeled test videos or images during training.The cited transductive literature includes [34] [55] [56] [59] [61].Source: §2 · p. 2
  • Video-classification literature distinguishes 2D networks operating on 1–5-frame snippets from 3D networks operating on 16–128 frames; later work found 16 training frames sufficient for most datasets.Increasing the training frame count from 16 to 128 improved performance only marginally.Source: §2 · p. 2
  • Common video ZSL maps features from a pretrained network into semantic embeddings, then predicts a test class by nearest-neighbor matching its embedding to the model output.Word2Vec [33] commonly supplies ground-truth word embeddings for class names.Source: §2 · pp. 2–3
  • Two recent methods extract C3D features from 52 clips of 16 frames per video, encode them with a recurrent neural network, and map the result to Word2Vec embeddings.Both methods use the same dataset for training and testing after splitting its classes.Source: §2 · p. 3
  • The paper’s universal action representation resembles Zhu et al. [65], but their ResNet200 pretrained on ImageNet cannot use temporal information.The authors adapt video-classification sampling practices to train their visual embedding end to end.Source: §2 · pp. 2–3
  • Earlier work often trains and tests on the same dataset using random class splits, and prior models may include classes that overlap the target test set.Under the paper’s ZSL definition, Zhu et al. [65] have 23 training classes overlapping the test dataset.Source: §2 · p. 3

3. Zero-shot action classification

The section defines video ZSL around unseen test classes, then proposes end-to-end training and a more realistic evaluation protocol that accounts for domain shift and multiple test datasets.

3.1. Problem setting

  • Video ZSL trains on source videos and tests on non-overlapping classes, classifying each video by its nearest test-class semantic embedding.Class-name embeddings are computed with Word2Vec.Source: §3.1 · p. 3

3.2. End-to-end training

  • End-to-end training jointly optimizes visual and semantic encoders, using 16-frame snippets and pooling multiple snippets at inference.This design allows a simple linear semantic encoder and avoids requiring visual-feature pretraining.Source: §3.2 · p. 4

3.3. Towards realistic ZSL

  • The proposed protocol uses semantic class-name distance to screen for overlap, disjoint video sources to reflect domain shift, and one model tested across multiple datasets.The experiments train on Kinetics and test on UCF, HMDB, and ActivityNet.Source: §3.3 · p. 4

3.4. Easy pretraining for video ZSL

  • The model outperforms previous work in both standard-benchmark scenarios.The scenarios differ in which training classes overlapping with test datasets are removed.Source: §3.4 · p. 5
  • Synthetic training videos made from still images boost performance, especially when available training data is small.The image-to-video scheme uses moving crops to simulate video-like motion.Source: §3.4 · p. 5

4. Experimental setup

The setup details the datasets and protocols used to make ZSL training and evaluation reproducible. Both evaluation protocols apply one model to multiple test datasets.

4.1. Datasets

  • ActivityNet contains 27,801 untrimmed videos across 200 daily-activity classes, from which only labeled frames were extracted; the newest Kinetics version has 700 classes and 541,624 YouTube videos.SUN397 has 397 scene categories and over 100K images; it was converted to simulated 16-frame videos using randomly selected start and end crops and linear interpolation.Source: §4.1 · p. 5

4.2. Training protocol

  • Training Protocol 1 creates Kinetics 664 by excluding classes similar to UCF or HMDB; Protocol 2 also excludes classes similar to ActivityNet, leaving Kinetics 605.The second protocol is described as more restrictive and closer to true ZSL; the aim is one model applicable to diverse test datasets.Source: §4.2 · pp. 5–6

4.3. Evaluation protocol

  • Evaluation Protocol 1 averages ten random half-class tests per dataset, while Protocol 2 evaluates all 101 UCF and 51 HMDB classes using the separate Kinetics-trained model.Protocol 1 selects 50 UCF and 25 HMDB classes per test.Source: §4.3 · p. 6

4.4. Implementation details

  • Visual embeddings use R(2+1)D 18 or C3D features from 16-frame clips, averaged across snippets; Word2Vec class-name embeddings average component words and feed a linear classifier with 512×300 weights.Training uses one clip per video and evaluation uses 25; optimization uses Adam at learning rate 1e−3 for 150 epochs, and τ is 0.05.Source: §4.4 · p. 6

5. Results

The experiments compare the end-to-end method with prior work and a fixed-feature baseline, then examine how training data, pretraining, and domain shift affect zero-shot performance. The method outperforms previous video-based approaches under a stricter evaluation protocol; the analyses find benefits from diverse classes and SUN-based pretraining.

5.1. Comparison to the state of the art

  • Despite removing overlapping training classes, introducing domain shift, and testing one model across datasets, the method outperforms previous video-based methods by a large margin and beats URL on UCF.URL uses a network with 200 layers versus the method’s 18, while 23 classes overlap between URL’s training and test sets.Source: §5.1 · pp. 6–7

5.2. Comparison to a baseline method

  • The end-to-end method consistently outperforms the fixed-feature baseline on both datasets, and its error keeps decreasing as training data grows while the baseline flattens near 100K datapoints.Both methods perform worse with smaller training sets.Source: §5.2 · p. 7
  • Few training samples from many classes outperform many samples from a small number of classes, with the effect more pronounced for the end-to-end model.The classwise experiments compared training on 2, 4, 10, 25, 50, 100, 200, 400, and 664 classes with randomly subsampled datapoints.Source: §5.2 · p. 7
  • As training-class semantic diversity decreases, the end-to-end model’s test error increases, even though greater class diversity makes the task harder.The study compared randomly chosen classes with classes sampled from Word2Vec clusters.Source: §5.2 · pp. 7–8, Figure 5 · p. 7

5.3. Easy pretraining with images

  • SUN synthetic-video pretraining decreases test errors by up to 10% and further improves performance under large train–test class-domain shift.The method was pretrained on a synthetic video dataset created from SUN still images.Source: Figure 6 · p. 7, §5.3 · p. 8

5.4. Generalization and domain shift

  • At distance threshold τ ∼0.57, the fixed-feature baseline’s accuracy drops to zero, while the end-to-end model remains above zero at high thresholds and SUN pretraining further increases performance.The thresholds measure distance from UCF test classes to their nearest Kinetics training classes.Source: §5.4 · p. 8, Figure 7 · p. 8

5.5. Ablation study

  • The ablation finds low performance with a fixed visual embedding, a large improvement from end-to-end training, and a further boost from class augmentation.Averaging embeddings from 25 test-time video snippets improves performance without influencing model training time.Source: §5.5 · p. 8, Table 3 · p. 8

6. Conclusion

The paper presents an end-to-end system for video zero-shot learning and evaluates it with a stricter protocol than prior work. Its experiments identify diverse training classes and simple pretraining as useful, and its code and protocols are available for others to build on.

6. Conclusion

  • The paper trains the first end-to-end system for video recognition zero-shot learning using practices from recent video-classification literature.Source: §6 · p. 8
  • Its evaluation protocol is stricter than existing work and measures zero-shot accuracy more realistically, while the method outperforms prior works evaluated with overlapping training and test sets and shared domains.Source: §6 · p. 8
  • Directed experiments indicate that good zero-shot datasets should contain many diverse classes, and the paper’s pretraining technique boosts performance.Source: §6 · p. 8
  • The authors provide code and protocols to encourage extensions and future video zero-shot learning research.Source: §6 · p. 8

SUPPLEMENTARY MATERIAL Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications

The supplementary material includes a backbone comparison table covering two Kinetics versions and evaluation on a single clip. The supplied material also gives the preprint version and date.

SUPPLEMENTARY MATERIAL Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications

  • The supplied preprint is arXiv version 2003.01455v4, dated 20 June 2020.Source: SUPPLEMENTARY MATERIAL Rethinking… · p. 12
  • Table 1 reports accuracy for different backbone architectures trained on the first (K400) and last (K700) version of Kinetics [19], evaluated using a single 16-frame clip.Source: Table 1 · p. 12
  • The backbone comparison covers the K400 and K700 versions of Kinetics.Source: Table 1 · p. 12
  • The models in the table are evaluated on a single 16-frame clip.Source: Table 1 · p. 12

1. Backbone choice

The backbone analysis tests the effect of including classes that overlap with the test set. Even a small overlapping subset is associated with large accuracy boosts, underscoring the zero-shot constraint’s importance.

1. Backbone choice

  • Adding the 6% of training classes most overlapping with the test set yields an accuracy boost of >40% for UCF and 25% on HMDB.Source: §1 · p. 12
  • The supplementary table compares three 3D convolutional backbones on two Kinetics versions using Training Protocol 1.Source: §1 · p. 12
  • The comparison also includes full Kinetics 400/700 datasets without removing classes that overlap with test classes.Source: §1 · p. 12

2. SUN pretraining: easier task or better representation?

The analysis asks whether SUN397 pretraining helps because it brings training data closer to UCF101 classes or because it improves generalization. The authors conclude that it improves performance across almost all test classes, including those not made closer to training data.

2. SUN pretraining: easier task or better representation?

  • The analysis tests whether SUN397's performance boost reflects better generalization or simply a closer source domain.Source: §2 · p. 12
  • The class-level comparison tracks UCF101 accuracy against distance to Kinetics training classes, with colored dots marking classes made over 10% closer by adding SUN.Source: Figure 1 · p. 12, §2 · p. 12
  • SUN pretraining boosts many UCF101 classes that were not made closer to the training data, supporting better generalization across almost all test classes.The comparison marks classes whose distance to training data falls by more than 10%.Source: §2 · p. 12, Figure 1 · p. 12

3. Training class diversity

The analysis finds that greater diversity among training classes is associated with lower test error and lower variability across class selections. The authors conclude that high variety in the training-class distribution can benefit performance and dataset collection.

3. Training class diversity

  • Test error decreases as training classes become more diverse.Source: §3 · p. 13, Figure 2 · p. 13
  • Standard deviation also decreases with greater diversity; for compact class sets, performance depends more on which classes are sampled, known only once the test set is available.Source: §3 · p. 13
  • Despite expecting increased class variance to make the task harder, the authors observe no performance decrease and conclude that high training-class variety can benefit dataset collection.Source: §3 · p. 13

4. Analyze the model capability action per action

The per-class analysis finds that end-to-end models correctly recognize some actions that the baseline struggles with, while rarely reversing that pattern. Confusion matrices also expose model-specific biases, including ones attenuated by SUN pretraining.

4.1. Direct comparison by sorting classes

  • End-to-end models correctly classify some actions difficult for the baseline, while the reverse is rare and baseline-easy actions remain easy for the end-to-end models.Source: §4.1 · p. 13, Figure 3 · p. 13, Figure 4 · p. 14
  • SUN-pretrained and Kinetics-trained end-to-end results are highly correlated, but SUN pretraining achieves better overall performance; baseline results are less correlated with them.The authors suggest SUN supplies complementary information and that fixed visual features should be fine-tuned.Source: §4.1 · p. 13

4.2. Confusion matrices

  • Confusion-matrix comparisons show biases in the baseline that end-to-end training removes, with pairwise L2 distances summarizing differences among the models.Source: §4.2 · pp. 13–14, Figure 5 · p. 15
  • All models confuse classes beginning with “Playing,” but the bias is less pronounced in end-to-end models; the baseline also wrongly classifies many actions as JumpingRope.The authors suggest that averaging words to embed class names may contribute to the Playing bias.Source: §4.2 · p. 14
  • The Kinetics-only model is biased toward HandStandWalking, and SUN pretraining attenuates that bias.Source: §4.2 · p. 14
Loading 2003.01455v4…