PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Feature Prediction for Learning Visual Representations from Video

Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, Nicolas Ballas

arXiv:2404.08471v1cs.CVcs.AIcs.LG

TL;DR

The paper asks whether feature prediction can serve as a stand-alone objective for unsupervised video representation learning. It introduces V-JEPA, trained solely with self-supervised feature prediction, and finds versatile representations across image and video tasks, including frozen-backbone evaluation.

  • Problem

    The paper asks how effective feature prediction is as a stand-alone objective for unsupervised learning from video with modern tools.

  • Method

    V-JEPA combines masked modeling with a joint-embedding predictive architecture and trains on videos without pretrained image encoders, text, negative examples, human annotations, or pixel reconstruction.

  • Results

    V-JEPA learns versatile representations across downstream image and video tasks without adapting model weights, outperforming prior video representation methods in frozen evaluation.

  • Takeaways & Limitations

    Feature prediction can serve as an effective stand-alone objective for unsupervised video learning, with particular effectiveness on fine-grained motion tasks and good performance with few labels.

  • Takeaways & Limitations

    The video datasets used for V-JEPA and other video models may be too constrained and lack the visual diversity of internet-scale image pretraining data.

Abstract

from arXiv · show

This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision. The models are trained on 2 million videos collected from public datasets and are evaluated on downstream image and video tasks. Our results show that learning by predicting video features leads to versatile visual representations that perform well on both motion and appearance-based tasks, without adaption of the model's parameters; e.g., using a frozen backbone. Our largest model, a ViT-H/16 trained only on videos, obtains 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet1K.

1 Introduction

The paper revisits feature prediction as a stand-alone objective for unsupervised video representation learning and introduces V-JEPA, trained solely through feature prediction. V-JEPA produces versatile representations across motion- and appearance-based tasks, including frozen-backbone evaluation.

  • Method: V-JEPA integrates masked modeling, joint-embedding prediction, vision transformers, and large video datasets into a feature-prediction method.The models are trained without pretrained image encoders, text, negative examples, human annotations, or pixel reconstruction.
  • Research question: Feature prediction is studied as a stand-alone objective for unsupervised learning from video with modern tools.The paper explicitly frames this as its central research question.
  • Results: V-JEPA performs well across motion-based Something-Something-v2 and appearance-based Kinetics400 using the same frozen backbone.The evaluation covers downstream image and video tasks without adapting model parameters.
  • Results: +6% accuracy is V-JEPA’s improvement over the methods considered on Something-Something-v2, which requires fine-grained temporal understanding.The paper also reports competitiveness on Kinetics400, where appearance-based features are sufficient.
  • Comparison: Feature prediction outperforms pixel prediction under attentive probing and remains competitive under full fine-tuning while using significantly shorter training schedules.This comparison spans frozen and end-to-end evaluation protocols.
  • Label efficiency: Feature-prediction models are more label-efficient than pixel-prediction approaches, with the performance gap increasing as labeled examples decrease.The comparison is reported in Table 7.

2 Related Works

Related work spans slow temporal features, predictive feature learning, self-supervised vision transformers, masked image modeling, and the contrast between feature prediction and pixel reconstruction.

  • Slow Features: Earlier slow-feature methods encourage temporally adjacent representations to vary slowly while using spectral methods to prevent collapse.Examples include SFA, SSA, and Simulated Fixations.
  • Slow Features: This paper extends temporal invariance by exploring feature prediction through masked modeling.The approach moves beyond local invariance toward predicting masked representations.
  • Predictive Features: Prior predictive-feature methods either place a predictor on frozen encoders or jointly train encoders and predictors with supervised action forecasting to prevent collapse.These approaches include video feature prediction across different time steps.
  • Advances in Self-Supervised Learning: Vision transformers and joint-embedding architectures have become standard tools in self-supervised learning and enabled masked image modeling in pixel space.The cited advances improved representation quality for autoencoding methods.
  • Feature Prediction versus Pixel Reconstruction: Pixel prediction spends capacity modeling low-level visual detail, whereas latent prediction can discard irrelevant or unpredictable pixel-level information.Representation-space prediction is associated with versatile downstream representations.

3 Methodology: Video-JEPA

V-JEPA learns video representations by predicting masked regions in feature space with a JEPA architecture. Its training uses masked video tokens, collapse prevention, and large-scale video pretraining with frozen and fine-tuned evaluation.

  • Training objective: A JEPA predicts the representation of input y from input x, conditioning the predictor on z to specify the transformation between them.
  • Collapse prevention: The modified regression objective prevents collapse with stop-gradient targets, an exponential-moving-average target encoder, and an L1 loss found more stable.
  • Network parameterization: The predictor combines visible-token embeddings with learnable mask tokens and regresses its masked-token outputs toward y-encoder targets using L1 loss.
  • Prediction task: The video objective masks regions into target y and context x, using large spatio-temporal blocks to reduce information leakage and make prediction harder.
  • Network parameterization: V-JEPA uses a ViT video backbone that processes 16 × 16 pixel patches spanning 2 consecutive frames as spatio-temporal tokens.
  • Pretraining and evaluation: VideoMix2M combines HowTo100M, Kinetics-400/600/700, and Something-Something-v2 videos into approximately 2 million training videos for downstream image and video evaluation.

4 What Matters for Learning Representations from Video?

The study isolates feature prediction, data scale, pooling, and masking as key choices for learning video representations. Feature-space prediction, larger pretraining datasets, attentive pooling, and multi-block masking consistently improve downstream performance.

  • Predicting Representations versus Pixels: Feature-space prediction consistently improves over pixel-space prediction in both frozen evaluation and end-to-end fine-tuning.
  • Feature Pooling: +17.3 points on K400 and +16.1 points on SSv2 result from adaptive pooling with a learnable cross-attention layer.The frozen V-JEPA feature map is pooled with an attentive probe before linear classification.
  • Pretraining Data Distribution: Average performance across tasks increases monotonically with larger pretraining datasets, while task-specific optima depend on the data mixture.VideoMix2M achieves the best average performance across tasks.
  • Masking Strategy: Multi-block masking produces the best results, whereas causal multi-block masking decreases downstream performance.
  • Comparison with Pixel Prediction: V-JEPA outperforms pixel-reconstruction baselines on all frozen downstream tasks except ImageNet, where it reaches 74.8% versus 75.1% for OmniMAE.It also achieves the best fine-tuning performance among ViT-L models while processing significantly fewer pretraining examples.

5 Comparison with Prior Work

V-JEPA is compared with pixel-prediction video models and state-of-the-art image and video models under frozen and fine-tuned evaluation. It performs strongly across motion, appearance, image, and low-shot tasks while using less pretraining time or labeled data.

  • Comparison with Pixel Prediction: V-JEPA outperforms pixel-reconstruction baselines on all frozen downstream tasks except ImageNet and matches Hiera-L on SSv2 after fine-tuning.On ImageNet, it achieves 74.8% versus 75.1% for an OmniMAE model trained directly on ImageNet.
  • Comparison with State-of-the-Art: V-JEPA outperforms previous large-scale video models on every downstream video and image task in frozen evaluation.
  • Comparison with State-of-the-Art: Roughly 2× speedup is achieved relative to large pixel-prediction models at comparable reported performance.
  • Comparison with Image Models: Over +21 points on Something-Something-v2 separates V-JEPA from large-scale image baselines on fine-grained motion understanding.The paper also reports superior action localization relative to image-based pretraining.
  • Comparison with Image Models: 82.0% top-1 accuracy is achieved by the H/16 model on Kinetics-400, where appearance cues can often support classification without motion understanding.An attentive probe raises the g/14 frozen score to 83.4%, compared with DINOv2’s reported 78.4% linear-probe score.
  • Comparison with Image Models: 77.9% on ImageNet is reached with a two-layer attentive probe, narrowing the gap with image-pretrained models.
  • Label Efficiency: Reducing labeled examples by 10× lowers V-JEPA’s K400 accuracy by 12% to 68.2%, compared with a 30% drop for VideoMAEv2.

6 Evaluating the Predictor

The predictor’s feature-space outputs are decoded into pixels to inspect what information they contain. The visualizations indicate positional uncertainty, consistent motion, and object permanence in predicted masked regions.

  • Visualization Methodology: A conditional diffusion decoder maps predicted representations of masked video regions into interpretable pixels while the V-JEPA encoder and predictor remain frozen.The decoder receives predictions for missing regions without access to the unmasked video regions.
  • Qualitative Analysis: Predicted features exhibit spatio-temporal consistency with unmasked regions and capture positional uncertainty across visual samples.
  • Qualitative Analysis: Some decoded samples preserve object identity after partial occlusion, demonstrating evidence of object permanence in the predictions.
  • Qualitative Analysis: Figure 6 presents offline visualizations of V-JEPA feature-space predictions.

7 Conclusion

V-JEPA uses video-only self-supervised feature prediction to learn representations that transfer across image and video tasks, including fine-grained motion understanding and low-label settings.

  • Conclusion: V-JEPA models solve image and video tasks with frozen parameters and outperform prior video representation methods on several frozen evaluations.The reported tasks include action recognition, spatio-temporal action detection, and image classification.
  • Conclusion: V-JEPA is introduced as a collection of vision models trained solely with a self-supervised feature prediction objective.The method is presented as feature prediction used as a stand-alone objective for unsupervised video representation learning.
  • Conclusion: Video pretraining is particularly effective for downstream tasks requiring fine-grained motion understanding.The paper contrasts this with large-scale image models, which fall short on such tasks.
  • V-JEPA pipeline: The training pipeline represents a 16 × 224 × 224 × 3 video clip as 1568 × d tokens after 3D convolution, positional embeddings, and flattening.The convolution uses temporal stride 2 and spatial stride 16, producing an intermediate 8 × 14 × 14 × d tensor.
  • V-JEPA pipeline: V-JEPA predicts representations for masked video regions from unmasked contextual representations and compares them with target representations from the full clip.The x-encoder processes unmasked tokens, the predictor generates masked-region outputs, and the y-encoder supplies target representations.

C Pretraining details

V-JEPA pretraining uses ViT encoders, AdamW optimization, scheduled learning rates and weight decay, EMA target updates, and multi-block masking.

  • Architectures: V-JEPA pretraining uses ViT-L/16 and ViT-H/16 encoders at 224 or 384 spatial resolution with 16-frame clips sampled at temporal stride 4.The models flatten clips into non-overlapping spatio-temporal patches.
  • Optimization: The y-encoder is initialized from the x-encoder and updated as an exponential moving average whose momentum increases from 0.998 to 1.0.This target-encoder update follows the x-encoder during training.
  • Masking: V-JEPA uses short-range and long-range 3D multi-block masks with different numbers and spatial scales of sampled target blocks.Short-range masks unite 8 blocks at spatial scale 0.15, while long-range masks unite 2 blocks at scale 0.7.

D Evaluation details

Frozen evaluation pools frozen encoder tokens with an attentive probe, using multi-clip inputs for video tasks and adapted procedures for image and detection tasks.

  • Frozen classification: Attentive probing pools frozen V-JEPA tokens through learnable cross-attention, an MLP, LayerNorm, and a linear classifier.The probe uses a learnable query token and is jointly trained with the classifier while encoder parameters remain frozen.
  • Frozen classification: The attentive probe is optimized with AdamW, cosine learning-rate decay without warmup, fixed weight decay, and task-specific augmentations.Image tasks use AutoAugment, while other probe training uses random resized crops and horizontal flips.
  • Video evaluation: Frozen video evaluation uses 8 clips for Kinetics and 2 clips for Something-Something-v2 to increase temporal coverage.Clips are sampled from equal-length temporal segments and their feature maps are concatenated before probing.
  • Cross-domain evaluation: Video models are evaluated on images by duplicating images into still video clips, while image models are evaluated on videos by encoding frames independently.The resulting feature maps are passed to the same attentive probe framework.
  • Detection: AVA evaluation trains a linear classifier on frozen V-JEPA features extracted from regions of interest defined by precomputed Faster-RCNN masks.The AVA dataset contains 211k training and 57k validation video segments.

E.1 Frozen Evaluation.

Frozen evaluation favors attentive, multi-clip probing, while finetuning shows competitive V-JEPA performance, especially on Something-Something-v2.

  • Frozen evaluation: Attentive probing improves V-JEPA and VideoMAE performance on K400 and SSv2 compared with linear probing.The protocol also improves average performance for DINOv2 and OpenCLIP.
  • Frozen evaluation: Increasing temporal coverage from one clip to eight improves both V-JEPA and VideoMAE on K400 action classification.The paper therefore adopts multiclip attentive probing as its default evaluation pipeline.
  • Finetuning: With ViT-H/16 finetuning, V-JEPA outperforms VideoMAE by 1.2% and VideoMAEv2 by 0.3% on SSv2.V-JEPA has comparable K400 performance and similar SSv2 performance to MVD.
  • Finetuning: MVD achieves the best K400 performance among the compared models but uses ImageNet1K and substantially more pretraining samples than the video-only methods.The additional samples arise from teacher-encoder processing during a pre-pre-training step.

E.3 Sample Efficiency of pretraining

V-JEPA achieves strong sample efficiency during pretraining, using substantially fewer processed samples than previous methods and OpenCLIP.

  • An order of magnitude fewer samples than previous methods were processed during V-JEPA pretraining.
  • V-JEPA processed two orders of magnitude fewer samples than OpenCLIP during pretraining.

E.4 Masking Strategy

V-JEPA’s 3D masking strategy uses multiple blocks and masks with high spatial and temporal coverage. Ablations find that these choices improve the prediction task and downstream performance.

  • Ablation setup: The masking ablations use a ViT-B/16 pretrained on Kinetics-400 and evaluate linear-probe performance under varied 3D Multi-Block settings.
  • Masking coverage: Low spatial or temporal coverage creates a trivial prediction task and degrades downstream performance.
  • Masking coverage: The default masks remove roughly 90% of each frame and extend across the clip’s entire temporal dimension.A temporal masking ratio of 100% extends the spatial mask across all frames.
  • Block structure: Several smaller blocks perform better than a single large block at approximately fixed masking ratios.Masks are formed by sampling possibly overlapping blocks and taking their union.
  • Mask sampling: Sampling two masks per clip with different spatial block sizes is more effective than sampling one mask.The authors hypothesize that the two masks induce complementary tasks.
Loading 2404.08471v1…