PaperlayerView exact revision on arXiv

Source-linked AI summary

Broaden Your Views for Self-Supervised Video Learning

Adrià Recasens, Pauline Luc, Jean-Baptiste Alayrac, Luyu Wang, Ross Hemsley, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patraucean, Florent Altché, Michal Valko, Jean-Bastien Grill, Aäron van den Oord, Andrew Zisserman

arXiv:2103.16559v3cs.CV

TL;DR

Video self-supervised methods often adapt image techniques and use views with matching temporal extent, leaving different time scales underused. BraVe predicts a broad view from a narrow clip using dedicated view backbones, and reports state-of-the-art results across video and audio classification benchmarks.

  • Problem

    Many video self-supervised methods use views with the same temporal extent, so they do not exploit information across different time scales.

    Source: §1 · p. 1

  • Method

    BraVe learns by predicting a broad, longer-context view from a narrow video clip, using dedicated backbones and potentially different modalities for the broad view.

    Source: §1 · p. 2

  • Results

    BraVe reports state-of-the-art results on UCF101, HMDB51, Kinetics, ESC-50 and AudioSet; with a larger backbone, it is competitive with supervised pre-trained techniques.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Breaking view symmetry permits stronger augmentations and different broad-view modalities, which improve the quality of the final representations.

    Source: §1 · p. 2

Abstract

from arXiv · show

Most successful self-supervised learning methods are trained to align the representations of two independent views from the data. State-of-the-art methods in video are inspired by image techniques, where these two views are similarly extracted by cropping and augmenting the resulting crop. However, these methods miss a crucial element in the video domain: time. We introduce BraVe, a self-supervised learning framework for video. In BraVe, one of the views has access to a narrow temporal window of the video while the other view has a broad access to the video content. Our models learn to generalise from the narrow view to the general content of the video. Furthermore, BraVe processes the views with different backbones, enabling the use of alternative augmentations or modalities into the broad view such as optical flow, randomly convolved RGB frames, audio or their combinations. We demonstrate that BraVe achieves state-of-the-art results in self-supervised representation learning on standard video and audio classification benchmarks including UCF101, HMDB51, Kinetics, ESC-50 and AudioSet.

1. Introduction

Video self-supervision often borrows image methods that treat views symmetrically in time, missing the opportunity to learn across different temporal scales. BraVe predicts a longer-context view from a short clip and reports state-of-the-art results on video and audio classification benchmarks.

1. Introduction

  • Video is a natural self-supervised learning domain because data is abundant but difficult to annotate at scale due to temporal complexity.Self-supervision learns from data without manually defined labels and supports training on large amounts of uncurated data.Source: §1 · p. 1
  • BraVe predicts a broad view spanning longer video context from a narrow view lasting a few seconds, breaking the temporal symmetry of conventional video views.The task asks representations to extrapolate the event context before and after the narrow clip.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Under a fixed computational budget, supervision from broad views performs better than alternatives using symmetric augmentation procedures.BraVe uses direct regression between views and does not require explicit negatives.Source: §1 · p. 2
  • Dedicated processing for the broad view permits stronger augmentations and modalities such as audio, flow or randomly convolved RGB frames.The paper reports that these options improve the quality of the final representations.Source: §1 · p. 2
  • BraVe reports state-of-the-art results on video and audio classification benchmarks including UCF101, HMDB51, Kinetics, ESC-50 and AudioSet.The framework is evaluated with and without audio as an auxiliary supervisory signal.Source: §1 · p. 2

2. Related work

Prior self-supervised work spans contrastive, clustering and regression approaches, while video methods also use temporal prediction and audio-video signals. BraVe uses dedicated backbones and mutual regression across views, including different modalities and augmentations.

Image-based self-supervised learning

  • Image-based self-supervision includes contrastive, clustering and regression methods; BraVe is inspired by BYOL [30] but uses dedicated backbones and views from different modalities or augmentations that regress each other.Contrastive approaches require careful positive and negative pair choices and often many negatives, which can raise computational cost.Source: §2 · p. 2

Video-based self-supervised learning

  • Video methods predict pixels, frame order or the arrow of time, while contrastive approaches use temporal cues but incur significant training costs and hard-negative mining.Pixel-space future predictions are usually blurred and do not extend beyond short clips.Source: §2 · p. 2
  • ρBYOL [22] applies BYOL to video without negatives but requires an EMA target network, whereas BraVe avoids that computational overhead.BraVe learns predictive tasks in feature space, including possible prediction of other video crops or optical flow [77] [86].Source: §2 · p. 2

Audio-video self-supervised learning

  • Audio-video self-supervision includes matching clips with corresponding audio and using speech-recognition language; BraVe needs no modality-specific losses and is simpler to train than a flow-and-audio distillation approach [68].The paper reports better performance across the board than that approach.Source: §2 · pp. 2–3

3. Broaden Your Views for Self-Supervised Video Learning

BraVe learns by regressing between a short, high-resolution narrow view and a broader, lower-resolution view that spans more of a video. Different backbones allow the broad view to use visual transformations or modalities such as flow and audio.

3. Broaden Your Views for Self-Supervised Video Learning

  • BraVe regresses from a high-resolution narrow clip to a lower-resolution broad view with greater temporal context, using RGB, flow, audio, or combined views.The framework is evaluated with visual signals alone and with visual and audio modalities.Source: §3 · p. 3, §3.1 · p. 3

3.1. The BraVe learning framework

  • The loss combines narrow-to-broad prediction with complementary broad-to-narrow regression; the predictor and stop-gradient operator help avoid collapse, and EMA is not necessary in this setting.The broad-view network receives training signal from the broad-to-narrow loss.Source: §3.1 · pp. 3–4
  • The framework uses separate narrow- and broad-view backbones, a shared projector, and a predictor, with simple regression losses across views.A linear layer matches backbone output dimensions when needed; the shared projector is reported to improve performance.Source: §3.1 · pp. 3–4
  • For same-modality views, independent temporal sampling makes it difficult for the broad backbone to predict the narrow view, while multiple broad views can share a backbone and average their projections.For multiple modalities, each broad view has its own backbone, while the projector and predictor are shared and regression losses are aggregated.Source: §3.1 · p. 4

3.2. Broad views from visual and audio modalities

  • For visual broad views, BraVe trades spatial resolution for more frames to keep pixel count and computational cost fixed.Random convolutions can augment only the broad view, and the narrow- and broad-view transformations need not match.Source: §3.2 · p. 5
  • Optical flow can provide motion-focused supervision, while audio broad views use log-mel spectrograms and can be combined with visual views.The visual narrow view and audio broad view are synchronized at their starting point; syncing is reported to help slightly.Source: §3.1 · p. 4, §3.2 · p. 5

4. Experiments

BraVe is evaluated through video-only and audio-video pretraining, linear evaluation and fine-tuning, ablations, and comparisons with prior methods. Results vary with view timing, broad-view inputs, and weight sharing; the final method is competitive with state-of-the-art methods across video and audio benchmarks.

4.1. Experimental setting

  • Experiments mainly use Kinetics-600 for video-only pretraining and AudioSet for audio-video pretraining, sampling visual views independently in time.AudioSet experiments train for 620k steps, compared with 300k for Kinetics-600.Source: §4.1 · pp. 5–6, §4.4 · p. 9

4.2. Downstream tasks

  • The study evaluates narrow-backbone representations with frozen-feature linear evaluation and end-to-end fine-tuning, and evaluates broad-backbone audio on ESC-50 and AudioSet.Video evaluation includes HMDB51, UCF101, and Kinetics-600; top-1 accuracy is reported for video and ESC-50, and mean average precision for AudioSet.Source: §4.2 · p. 6

4.3. Ablation study

  • Syncing narrow and broad visual views lowers performance; with a 1.3s narrow view, extending the RGB+RC broad view from 1.3s to 10s raises HMDB51 from 59.8 to 65.2.Using 10s for both views degrades performance; extending the audio broad view from 5s to 10s brings no additional improvement.Source: Table 1 · p. 6, §4.3 · p. 7, Table 2 · p. 7
  • Random convolutions on broad-view RGB improve performance over standard RGB, while flow further improves HMDB51 and UCF101 but reduces Kinetics-600 performance.The authors attribute the Kinetics-600 decrease to the dataset’s semantic demands and the richer semantic information not provided by flow.Source: §4.3 · p. 7, Table 3 · p. 7
  • Adding broad views of the same modality improves UCF-101 and Kinetics performance.The authors suggest that regressing to the average of multiple projections may provide a more representative video representation.Source: Table 4 · p. 7, §4.3 · p. 8
  • Sharing more model parameters improves performance on various benchmarks, but the authors find it important to retain at least one module with separate weights.Their final configuration uses separate narrow- and broad-view backbones with a single predictor and projector.Source: Table 5 · p. 7, §4.3 · p. 8

4.4. Comparison with the state-of-the-art

  • BraVe outperforms matched state-of-the-art methods on AudioSet and establishes HMDB51 and UCF101 fine-tuning records with TSM-50x2.With flow and matched R3D/K400, it is −0.4% on UCF101 and +1% on HMDB51 compared with ρ-BYOL [22].Source: Table 6 · p. 8, §4.4 · p. 9

5. Conclusion

BraVe learns video representations by supervising a temporally narrow view with a broader view that can use RGB, flow, or audio. The conclusion reports state-of-the-art performance on Kinetics and AudioSet, and competitiveness with supervised pre-training when using a larger backbone.

5. Conclusion

  • BraVe achieves state-of-the-art performance when trained on datasets such as Kinetics or AudioSet.With a larger backbone, it is competitive with supervised pre-trained techniques.Source: §5 · p. 9
  • BraVe predicts a general broad view from a temporally narrow view, with the broad view computed from RGB, flow, or audio.Source: §5 · p. 9

Appendix

  • The appendix covers training-pipeline details, downstream evaluation procedures, and experiments on syncing views, image-based narrow backbones, broad visual backbones, and joint RGB-flow broad views.Source: Appendix · p. 13

A.1. Architecture and model hyperparameters

BraVe uses two-layer MLPs for its predictor and projector and trains with LARS, cosine learning-rate decay, and linear warm-up. Training duration and hardware vary for specified model configurations.

A.1. Architecture and model hyperparameters

  • The predictor and projector are two-layer MLPs with 4096 hidden dimensions, batch normalization after each hidden layer, and an additional batch-normalization layer after the projector output.Source: §A.1 · p. 13
  • Training uses LARS with cosine learning-rate decay, 5000 warm-up steps, batch size 512, and weight decay 0.01 except for R(2+1)d-18, which uses 10^-7.The initial learning rate is 1.0 for audio models and 4.8 otherwise; predictor-MLP learning rates are multiplied by 10.Source: §A.1 · p. 13
  • Models train for 300k steps, except audio models reported in Table 5, which train for 620k steps; most use 16 Cloud TPUs, while TSM-50x2 and R(2+1)D-50 use 32.Source: §A.1 · p. 13

A.2. Data augmentation and feature extraction

The training pipeline independently samples and augments views, with modality-specific frame sampling and preprocessing for RGB, flow, random convolutions, and audio.

A.2. Data augmentation and feature extraction

  • RGB videos are generally subsampled at 12.5 FPS, with 6.25 FPS used for broad views in visual-only models, the 10-second narrow-view ablation, and R3D.Source: §A.2 · p. 13
  • RGB preprocessing uses random spatial crops consistent across frames, horizontal flipping, color jitter, and Gaussian blur, with grayscale conversion applied probabilistically.The listed probabilities are 0.5 for flipping, 0.8 for color randomization, and 0.2 for grayscale conversion.Source: §A.2 · p. 13
  • Flow uses random cropping independent of the narrow view and horizontal flipping, but omits scale jittering because it did not improve performance.Flow frames are resized to a shortest side of 128 and cropped to 112 × 112.Source: §A.2 · p. 13
  • Random convolutions use per-sample He-initialized kernels with odd sizes from 1 to 11, while audio features use 80-bin log-mel spectrograms from 48 kHz audio.The spectrograms use 320-sample Hanning windows and a stride of 160 samples.Source: §A.2 · p. 13

B. Downstream task evaluation

Downstream evaluation uses task-specific protocols for linear evaluation, fine-tuning, Kinetics-600, and AudioSet. The appendix also includes a study of syncing narrow and broad views.

B. Downstream task evaluation

  • For linear evaluation on HMDB51, UCF101, and ESC-50, the paper trains an SVM on extracted pretrained-backbone features and selects regularization using the first split.HMDB51 and UCF101 predictions average 30 clips; ESC-50 uses one 5-second test window.Source: B. Downstream task evaluation · p. 13
  • The 10-second narrow-view ablation is evaluated using 64-frame clips at 6.25 FPS and 112 × 112 crops, matching its training configuration.Source: B. Downstream task evaluation · p. 14
  • Fine-tuning on HMDB51 and UCF101 uses LARS, cross-validated learning rate, weight decay, and dropout, with hyperparameters selected on split 1 and results averaged across splits.Table 7 provides the fine-tuning hyperparameters; evaluation averages predictions over 30 clips.Source: B. Downstream task evaluation · p. 14, Table 7 · p. 14
  • Kinetics-600 linear evaluation trains a linear layer for 100 epochs with LARS and reports top-1 accuracy on the validation set.The test prediction averages 30 clips, each using three spatial crops.Source: B. Downstream task evaluation · p. 14
  • AudioSet evaluates frozen audio representations by training a one-hidden-layer MLP for 20 epochs and averaging 10 overlapping 5-second test crops.Source: B. Downstream task evaluation · pp. 14–15
  • The appendix includes a sync study examining the effect of syncing the narrow and broad views.Source: Table 8 · p. 15

C. Syncing audio and video

BraVe's broad-view experiments find that beginning-synchronized audio and video works best, while a single-frame narrow view can learn strong image and video features. The image-model experiment reports no improvement on detection and attributes this to the semantic nature of the loss.

C. Syncing audio and video

  • Beginning synchronization was the best option among start-synced, center-synced, and independently sampled audio-video views.The experiment also supports previously established evidence that synchronizing audio and video benefits self-supervised learning.Source: C. Syncing audio and video · p. 15

D. Using BraVe to learn image models

  • ImageNet linear scores were 56.4 for the visual-trained model and 59.2 for the audio-trained model; Kinetics-600 linear accuracy was 60.5 and 60.8, respectively.The narrow view was one frame; broad views were 10s visual (RGB+RC) or 5s audio.Source: D. Using BraVe to learn image models · p. 15
  • BraVe pretraining did not improve detection, which the authors attribute to a semantic loss that need not preserve spatial information.Source: D. Using BraVe to learn image models · p. 15

E. Evaluating the broad backbone

The broad-backbone comparison evaluates RGB broad views against narrow-backbone results and notes that the strongest results use a different or altered broad-view modality, though those results are not directly comparable.

E. Evaluating the broad backbone

  • The broad-view combinations table uses a 1.3s narrow extent and a 10s broad extent; RC means random convolutions.Source: Table 9 · p. 15
  • RGB broad-view performance was 63.4 on HMDB51 and 90.3 on UCF101, versus 63.2 and 90.5, respectively, for the narrow backbone.Source: E. Evaluating the broad backbone · p. 15
  • The best results came from broad views using a different or altered modality, such as flow, audio, or random convolutions, but those results were not comparable.Source: E. Evaluating the broad backbone · p. 15

F. Training BraVe with RGB and flow views

Using both RGB and flow broad views improves HMDB51 and UCF101 results but lowers K600 performance relative to two RGB views. This configuration uses two broad backbones, increasing computational resources, and is not directly comparable to a single-backbone broad view.

F. Training BraVe with RGB and flow views

Related papers

Loading 2103.16559v3…