PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Video Representations using Contrastive Bidirectional Transformer

Chen Sun, Fabien Baradel, Kevin Murphy, Cordelia Schmid

arXiv:1906.05743v2cs.LGcs.CVstat.ML

TL;DR

BERT-style masked-language modeling requires discrete tokens, and vector-quantizing video can discard fine-grained information. This paper trains bidirectional transformers on continuous video features with contrastive and optional sequence-level cross-modal objectives, reporting gains across downstream video tasks.

  • Problem

    BERT’s masked language modeling requires discrete tokens, while vector quantization of video can lose fine-grained information important for downstream tasks.

    Source: §1 · p. 1

  • Method

    CBT applies noise contrastive estimation to bidirectional transformers over real-valued video features and can add ASR through sequence-level cross-modal mutual-information training.

    Source: §1 · p. 1

  • Results

    CBT reports state-of-the-art results across video classification, segmentation, and captioning; UCF101 improves from 75.7% to 79.5% and HMDB51 from 35.7% to 44.6% over previous self-supervised methods.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The authors conclude that CBT learns more useful features than existing self-supervised methods across video tasks and suggest its modularity may support scaling to larger unlabeled datasets.

    Source: §5 · p. 8

    Cross-modal training assumes video and text have something in common, making instructional videos a suitable setting but not arbitrary videos with unrelated speech.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.

1 INTRODUCTION

The paper adapts BERT-style self-supervised learning to continuous video features and optionally ASR, then reports strong results across downstream video tasks. Its cross-modal approach matches video and text at the sequence level, rather than requiring frame-level alignment.

1 INTRODUCTION

  • BERT’s masked language modeling requires discrete tokens, while vector quantization of video can lose fine-grained information important for downstream tasks.The paper addresses this by training bidirectional transformers on real-valued video vectors with noise contrastive estimation, without pre-trained visual encoders.Source: §1 · p. 1
  • The cross-modal transformer maximizes mutual information between video and ASR sequences, making it robust to small misalignments between them.Source: §1 · p. 1
  • For visual representations, the paper encodes windows of K frames with S3D and pretrains CBT on unlabeled Kinetics videos before classification fine-tuning.Source: §1 · pp. 1–2
  • UCF101 improves from 75.7% to 79.5% and HMDB51 from 35.7% to 44.6% over previous state-of-the-art self-supervised methods.The reported evaluation fine-tunes a linear classifier on the self-supervised visual representations.Source: §1 · p. 2
  • The approach also reports large gains on temporal video classification, segmentation, and captioning, especially with cross-modal pretraining.For temporal representations, the paper uses frozen S3D features pretrained on Kinetics and CBT pretraining on unlabeled HowTo100M.Source: §1 · p. 2

2 RELATED WORK

Related work spans video representations, self-supervised context modeling, mutual-information objectives, and cross-modal learning. CBT adapts bidirectional BERT-style modeling to continuous video, while distinguishing itself from visual-linguistic approaches that assume fixed, supervised visual encoders.

Video representations

  • Many video representation methods capture only a few seconds; existing long-term context methods are supervised, and self-supervised pretext-task methods also capture only a few seconds.The cited approaches include recurrent networks, graph convolutional networks, long-term feature banks, and tasks such as ordering, rotation, temporal cycle consistency, and colorization.Source: §2 · pp. 2–3

Self-supervised context modeling

  • BERT jointly conditions on left and right context, and CBT adapts this bidirectional representation approach to continuous video using a contrastive loss.Source: §2 · p. 3

Mutual information estimation and maximization

  • CPC uses noise contrastive estimation to maximize a mutual-information lower bound with autoregressive context, whereas CBT uses BERT for bidirectional context within and across modalities.Source: §2 · p. 3

Cross-modal learning

  • Prior cross-modal work uses audio-video supervision or aligns video with language from ASR or textual descriptions.Source: §2 · p. 3
  • Concurrent BERT-style visual-linguistic methods assume fixed visual representations from supervised encoders; the authors characterize CBT as the first fully self-supervised BERT-style video pretraining demonstration.Source: §2 · p. 3

3 METHOD

The method adapts BERT-style masked-context learning to continuous visual features with NCE, then adds sequence-level cross-modal training for video and ASR. Its overall model combines separate text and visual transformers with a cross-modal transformer.

3.1 THE BERT MODEL

  • BERT embeds discrete tokens and uses a transformer to predict each masked token from its bidirectional context, minimizing pseudo negative log likelihood.The context sequence contains all words except the token at the prediction position.Source: §3.1 · pp. 3–4

3.2 THE CBT MODEL

  • For continuous visual features, CBT replaces BERT’s discrete-vocabulary objective with softmax NCE, using a visual transformer to identify the correct frame against negative examples.The negatives are other frames from the same minibatch; the loss encourages identification of the frame from its context and maximizes a lower bound on mutual information.Source: §3.2 · p. 4

3.3 THE CROSS-MODAL CBT MODEL

  • Because speech and visible content may be temporally misaligned, the cross-modal model maximizes mutual information between video and ASR sequences rather than between frame-level pairs.The approach assumes the modalities share content, as in instructional videos; arbitrary videos may contain unrelated speech.Source: §3.3 · p. 4
  • For cross-modal training, separate CBT and BERT encodings are concatenated and passed to a shallow cross-modal transformer and an MLP that computes an MI-like score.Training uses NCE with ASR sequences not associated with the video as negatives.Source: §3.3 · p. 4

3.4 OVERALL MODEL

  • The overall model combines an ASR-token BERT, a continuous-video transformer, and a third transformer that estimates mutual information between modalities.The BERT loss weight is zero for the pretrained, frozen ASR model; the visual loss weight is one, while the cross-modal loss is enabled or disabled by its weight.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate self-supervised visual representations for action recognition and temporal representations on action anticipation, captioning, and segmentation, alongside ablations of data, cross-modal training, and model size.

4.1 LEARNING SELF-SUPERVISED VISUAL REPRESENTATIONS

  • 3.8 points on UCF101 and 8.9 points on HMDB51: CBT beats DPC; CBT visual-loss pretraining also surpasses Shuffle&Learn, 3DRotNet, and random initialization.The evaluations compare action-recognition accuracy on UCF101 and HMDB51.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 LEARNING SELF-SUPERVISED TEMPORAL REPRESENTATIONS

  • Temporal pretraining uses HowTo100M narrated videos, frozen S3D features, and optional BERT-encoded ASR combined through a cross-modal transformer.The learned representations are evaluated on action anticipation, video captioning, and action segmentation.Source: §4.2 · p. 6
  • CBT outperforms prior self-supervised and end-to-end methods on Breakfast, 50Salads, and ActivityNet for action anticipation.Table 2 (Left) compares methods across the three datasets.Source: Table 2 · p. 7, §4.2 · p. 7
  • CBT outperforms AvgPool and LSTM on all three datasets, and its performance rises with longer observation windows while theirs plateaus or decreases.The baselines use the same S3D features; LSTM hidden size is adjusted to make its parameter count comparable to CBT.Source: Table 2 · p. 7, §4.2 · pp. 7–8
  • More HowTo100M pretraining data improves performance on all three benchmarks, while cross-modal training helps significantly, especially on Breakfast and Salads.Table 3 varies the percentage of HowTo100M videos and whether the cross-modal objective is used.Source: Table 3 · p. 8, §4.2 · p. 8
  • Visual-transformer performance initially improves and then declines as layers or attention heads increase; the authors conjecture the smaller pretraining set may explain the contrast with NLP-BERT.The visual transformer has 15M parameters, compared with 110M for the BERT NLP transformer.Source: Table 3 · p. 8, §4.2 · p. 8
  • The learned temporal representations are also evaluated for YouCook2 video captioning and COIN action segmentation.Table 4 reports captioning and segmentation results; the segmentation evaluation uses frame accuracy.Source: Table 4 · p. 8, §4.2 · p. 8

5 CONCLUSION

The paper extends BERT-style self-supervised learning to video without vector quantization or pre-trained visual features, and to cross-modal training when ASR is available. It reports more useful features than existing self-supervised methods across video classification, captioning, and segmentation.

5 CONCLUSION

  • CBT extends BERT-style self-supervised learning to video without vector quantization or pre-trained visual features, and supports cross-modal learning when ASR is available.The authors report useful features across downstream video tasks including classification, captioning, and segmentation.Source: §5 · p. 8

6 SUPPLEMENTARY MATERIALS

The section evaluates HowTo100M-pretrained video features for captioning on YouCook2 and temporal action segmentation on COIN. CBT outperforms the compared methods in both tasks, including a reported 19.6-point margin for segmentation.

6.1 VIDEO CAPTIONING

  • Captioning uses HowTo100M-pretrained features with a supervised transformer trained on YouCook2’s ground-truth video segments and captions.YouCook2 contains 2,000 videos, with no overlap with the pretraining datasets.Source: §6.1 · p. 12
  • On YouCook2 captioning, CBT beats average-pooled S3D features, Zhou et al. (2018c), and VideoBERT on every reported metric.The CBT–VideoBERT gap is smaller than on Breakfast and 50Salads action anticipation, possibly because YouCook2 is more similar to VideoBERT’s cooking-video pretraining data; the authors suggest removing video-feature quantization is important for fine-grained representations.Source: §6.1 · p. 12

6.2 ACTION SEGMENTATION

  • The COIN evaluation uses 11,827 instructional videos annotated with segment boundaries and labels across 779 classes.Videos average 2.36 minutes, with 3.91 segments per video averaging 14.9 seconds each.Source: §6.2 · p. 12
  • For COIN segmentation, frozen S3D and visual-transformer features feed a linear classifier; videos are processed in 72-second windows, with padding for shorter sequences.At test time, long videos use a sliding 72-second window.Source: §6.2 · p. 12
  • In comparison with various state-of-the-art approaches, CBT outperforms them by 19.6 points using frame accuracy as the evaluation metric.Compared approaches include Tang et al. (2019), Richard et al. (2018), and Ding & Xu (2018).Source: §6.2 · p. 12
Loading 1906.05743v2…