PaperlayerView exact revision on arXiv

Source-linked AI summary

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas, Adrien Bardes

arXiv:2603.14482v3cs.CV

TL;DR

V-JEPA 2.1 addresses the challenge of learning visual representations that retain dense spatio-temporal structure alongside global understanding. It combines all-token dense prediction, deep self-supervision, modality-specific tokenizers, and scaling, achieving strong results across video, robotics, dense vision, and recognition benchmarks. The paper concludes that these designs advance dense visual understanding and world modeling, while noting remaining segmentation limitations and opportunities for further scaling and world-model development.

  • Problem

    Learning representations that preserve dense spatio-temporal structure while supporting global understanding remains an open challenge.

  • Method

    V-JEPA 2.1 combines dense prediction over visible and masked tokens, hierarchical deep self-supervision, modality-specific image-video tokenizers, and scaling in data and model capacity.

  • Results

    V-JEPA 2.1 achieves strong performance across predictive video, robotic, dense-vision, and global-recognition tasks, including 7.71 mAP on Ego4D and 40.8 Recall@5 on EPIC-KITCHENS-100.

  • Takeaways & Limitations

    The resulting representations are spatially structured, semantically coherent, and temporally consistent, supporting dense visual understanding and world-modeling tasks.

  • Takeaways & Limitations

    Performance on ADE20K and Cityscapes remains slightly behind the best image encoders, possibly because the training data contains fewer highly cluttered scenes.

Abstract

from arXiv · show

We present V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and videos while retaining strong global scene understanding. The approach combines four key components. First, a dense predictive loss uses a masking-based objective in which both visible and masked tokens contribute to the training signal, encouraging explicit spatial and temporal grounding. Second, deep self-supervision applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality. Third, multi-modal tokenizers enable unified training across images and videos. Finally, the model benefits from effective scaling in both model capacity and training data. Together, these design choices produce representations that are spatially structured, semantically coherent, and temporally consistent. Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging benchmarks, including 7.71 mAP on Ego4D for short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS for high-level action anticipation, as well as a 20-point improvement in real-robot grasping success rate over V-JEPA-2 AC. The model also demonstrates strong performance in robotic navigation (5.687 ATE on TartanDrive), depth estimation (0.307 RMSE on NYUv2 with a linear probe), and global recognition (77.7 on Something-Something-V2). These results show that V-JEPA 2.1 significantly advances the state of the art in dense visual understanding and world modeling.

1 Introduction

V-JEPA 2.1 addresses the challenge of learning representations that preserve dense spatio-temporal structure while supporting global understanding. It combines dense predictive loss, deep self-supervision, unified image-video training, and scaling to achieve strong results across predictive, robotic, dense-vision, and recognition tasks.

  • Motivation: V-JEPA 2.1 targets representations that preserve fine-grained spatial structure and motion while retaining global video understanding.Prior V-JEPA models showed strong global understanding but could be less suited to extracting local spatial structure, whereas other SSL methods primarily provided dense image features.
  • Method: Applying predictive loss to both masked and visible tokens substantially improves dense representations compared with supervising masked regions alone.The objective grounds tokens in their spatio-temporal locations and prevents visible tokens from acting as global aggregators.
  • Method: Deep self-supervision, modality-specific tokenizers, and scaling from 1M to 142M images and 300M to 2B parameters provide additional gains across downstream tasks.The model applies supervision at multiple intermediate encoder layers and jointly trains images and videos through a shared encoder.
  • Results: 7.71 mAP on Ego4D short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS-100 demonstrate state-of-the-art predictive video performance.The benchmarks cover localized interaction regions and time-to-interaction, and forecasting upcoming actions from partial temporal context.
  • Results: +20% success rate over VJEPA-2 AC on real Franka-arm grasping and 5.687 ATE on Tartan Drive show strong robotic manipulation and navigation performance.The navigation result is reported alongside 10x faster planning than previous work.
  • Results: V-JEPA 2.1 achieves 0.307 RMSE on NYUv2 depth estimation and 77.7% accuracy on Something-Something-v2, covering dense and global visual understanding.It also reports 85.0 mIoU on Pascal VOC, 72.7 J &F-Mean on YouTube-VOS, and 83.1 accuracy on PerceptionTest.

2 Methodology

V-JEPA 2.1 extends latent mask-denoising with supervision on visible and masked tokens, deep intermediate-layer supervision, modality-specific tokenization, and scaling to improve dense features while retaining global understanding.

  • V-JEPA foundation: V-JEPA predicts representations of masked video patches from visible context tokens, applying its original loss only to masked tokens.The encoder and predictor use Vision Transformers, with stop-gradient and an exponential-moving-average target encoder preventing collapse.
  • Motivation: V-JEPA 2 features show fragmented local structure and limited dense-task performance, including 22.2 mIoU on ADE20K and 0.682 RMSE on NYUv2.These observations motivate adding self-supervision to context patches.
  • Dense prediction loss: The context loss supervises both masked and context tokens, producing coherent local structure and improving ADE20K from 22.2 to 33.9 mIoU and NYUv2 from 0.682 to 0.473 RMSE.Distance-based weighting emphasizes context patches near masked regions to enforce local continuity.
  • Deep self-supervision: Context supervision alone improves dense vision but reduces global recognition, changing SSv2 from 72.8 to 62.5 and IN1K from 82.2 to 72.6.Deep self-supervision recovers global performance while further improving dense tasks, reaching 72.0 on SSv2, 80.8 on IN1K, 38.6 mIoU on ADE20K, and 0.463 RMSE on NYU.
  • Model scaling and resolution: Scaling from ViT-L to ViT-G improves all reported downstream metrics, while high-resolution cool-down yields 77.7 on SSv2, 85.5 on ImageNet, 47.9 mIoU on ADE20K, and 0.307 RMSE on NYUv2.The cool-down increases image resolution from 256 × 256 to 512 × 512 and video input from 16 frames at 256 × 256 to 64 frames at 384 × 384.

3 Results

V-JEPA 2.1 achieves strong results across predictive, robotic, dense-vision, video-segmentation, and global-recognition benchmarks. Its dense features support spatially accurate depth, segmentation, tracking, and manipulation while retaining video-dynamics understanding.

  • Predictive video understanding: 7.71 mAP on Ego4D short-term anticipation sets absolute state-of-the-art, approximately 35% above the previous best method.The gain is driven by next-action understanding, time-to-contact prediction, and precise next-active-object localization.
  • Predictive video understanding: 40.8 Action Recall@5 on EK100 establishes a new state of the art, improving 2.8% over the prior result at the 2B model scale.The 1B model is comparable to V-JEPA 2 under the same context and evaluation setup.
  • Robotic manipulation: A 10% grasp-success improvement over V-JEPA 2 reflects better depth understanding, while longer rollouts further improve V-JEPA 2.1 grasping performance.Observed failures mainly involve gripper-action planning rather than spatial understanding.
  • Dense visual understanding: 0.307 RMSE on NYUv2 establishes state-of-the-art linear-probe monocular depth estimation, with sharper and more coherent depth maps than V-JEPA 2.V-JEPA 2.1 also reaches 85.0 mIoU on Pascal VOC, 73.5 mIoU on Cityscapes, and 47.9 mIoU on ADE20K.
  • Limitations: Performance on ADE20K and Cityscapes remains slightly behind the best image encoders.The authors hypothesize that VisionMix contains comparatively fewer highly cluttered scenes, limiting exposure to the granularity required by these benchmarks.
  • Video and global understanding: V-JEPA 2.1 obtains 72.7 J &F on YouTube-VOS and 77.7 top-1 accuracy on Something-Something-v2, indicating temporally consistent tracking and strong global video understanding.It slightly improves over V-JEPA 2 on average across VidQA, but performs worse on TemporalBench and TOMATO.
  • Representations and scaling: Dense feature maps from V-JEPA 2.1 are spatially structured, and distillation substantially improves compact ViT-L and ViT-B models toward ViT-G performance.On SSv2, distilled ViT-L improves from 74.2% to 76.5%; on ADE20K, it improves from 42.0 to 46.7 mIoU.

4 Related work

Prior work spans handcrafted and contrastive self-supervision, video prediction and masked modeling, scaling generalist encoders, dense-feature objectives, and ViT adaptations for dense prediction.

  • Self-Supervised Learning: Early visual SSL used patch-position prediction, shuffled-patch reordering, inpainting, recolorization, and transformation prediction.Later view-invariant joint-embedding methods advanced SSL through contrastive and non-contrastive approaches.
  • Video Models: Video SSL has explored camera transformations, future-frame prediction, temporal patch consistency, object retrieval, temporal-order verification, contrastive learning, and masked modeling.VideoMAE extended image masking to video, while other work studied CLIP-space masking and joint image-video training.
  • Scaling: Scaling data and model size has enabled generalist video encoders, while JEPAs offer promising efficiency gains at large scale and support world-modeling applications.These developments motivate learning broad representations from extensive video observations.
  • Learning dense features: Dense-feature research uses local objectives based on spatio-temporal consistency, crop alignment, patch consistency, region proposals, and encoder distillation.Some methods instead improve dense features post hoc through clustering or related objectives.
  • Dense prediction architectures: Dense prediction has also been addressed architecturally by converting ViT token sequences into multi-scale feature maps with reassembly and convolutional decoding.The Dense Prediction Transformer addresses the token-to-pixel problem through this pipeline.

5 Conclusion and Future work

The paper concludes that dense spatio-temporal self-supervision, modality-specific tokenization, balanced data, and scaling yield strong dense and global representations. Future work targets larger models and datasets, world modeling, and robotics.

  • Conclusion: V-JEPA 2.1 combines a weighted context loss with a multi-level predictor to supervise intermediate encoder layers and produce structured, coherent, temporally consistent dense features.PCA visualizations and downstream results support this representation quality.
  • Conclusion: Modality-specific tokenizers, balanced image-video curation, and ViT-G scaling support strong performance and effective distillation into compact models.The conclusion links these design choices to predictive, depth, segmentation, and video-tracking results.
  • Future work: Future work will scale model size and data size, building on positive trends from 1B to 2B parameters and the benefits of large-scale datasets.The paper specifically identifies scaling to larger vision models as a promising direction.
  • Future work: Future research will explore world modeling on top of dense representations for navigation and fine-grained manipulation in robotics and autonomous agents.The stated applications require estimating physical-world state down to the pixel level.
  • Training protocol: Pretraining uses a primary phase followed by cooldown with progressively higher-resolution inputs, while video clips increase from 16 to 64 frames.The protocol applies shared hyperparameters across ViT-L, ViT-g, and ViT-G architectures.
  • Training protocol: The predictor has 24 blocks, deep self-supervision uses four intermediate encoder blocks, and λ is set to 0.5 for video and 0.7 for images.Distillation replaces the EMA encoder with a frozen teacher and omits deep self-supervision.
  • Training protocol: Distillation is staged across low- and high-resolution phases, using corresponding pre-cooldown and post-cooldown ViT-G teachers.The student is trained with the pretraining masking and JEPA-loss framework, subject to the stated distillation modifications.

C.1 Depth Estimation

Depth quality is evaluated on NYUv2 and KITTI using RMSE, with frozen V-JEPA 2.1 patch features and a learned linear probe.

  • Datasets and Metrics: Depth estimation is evaluated on NYUv2 and KITTI using root mean square error (RMSE).The datasets measure geometric quality of V-JEPA 2.1 dense features.
  • Evaluation Protocol: The evaluation trains a linear classifier on each dataset while keeping the V-JEPA 2.1 visual encoder patch features frozen.A learned BatchNorm layer further normalizes the frozen features, and single-image classifiers are trained and evaluated.

C.2 Semantic Segmentation

This section evaluates semantic segmentation with linear probes on ADE20K, Pascal VOC12, and Cityscapes, reporting mean intersection-over-union (mIoU).

  • Semantic segmentation is evaluated with linear probing on ADE20K, Pascal VOC12, and Cityscapes using mIoU.
  • Linear classifiers are trained on normalized last-block patch features from frozen encoders.
  • Evaluation uses 512px resolution for ADE20K and VOC12 and 1024px height for Cityscapes.

C.3 Video Object Segmentation Tracking

This section evaluates video object segmentation on DAVIS 2017 and YouTube-VOS using dense video annotations and feature-based label propagation.

  • Video object segmentation is evaluated on DAVIS 2017 and YouTube-VOS.DAVIS provides dense train/validation annotations, while the publicly available YouTube-VOS training set is split into training and validation videos.
  • Labels are propagated across frames using cosine similarity between patch features from a frozen backbone.The method combines first-frame and selected past-frame references with weighted k-NN within a spatial neighborhood.
  • Hyperparameters include context length, neighborhood size and shape, top-k neighbors, and similarity temperature.They are selected on the DAVIS training set and applied to all test splits at 480px input resolution.

C.4 Short Term Object Interaction Anticipation

This section evaluates short-term object-interaction anticipation on Ego4D, using spatially aligned attentive probing and established detection-based evaluation protocols.

  • Ego4D STA v2 contains 243 hours of annotated clips spanning 128 noun and 81 verb categories, with 98,276 training and 47,395 validation samples.Performance is reported with Top-5 Average Precision and Top-5 mean Average Precision, matching predictions to ground-truth boxes at IoU > 0.5.
  • The attentive probe uses four attention blocks followed by frame-guided temporal pooling to keep predictions aligned with the last video frame.
  • The prediction head combines RPN proposals, RoIAlign local features, and pooled scene context through residual fusion.
  • Optimization explores frame rates of 16, 8, 4, and 2 fps, clip lengths of 32, 16, 8, and 4 frames, and resolutions of 256px and 384px.
  • The same protocol is applied to DINOv2, DINOv3, and V-JEPA 2 models, reporting each model’s best performance.
  • Global recognition evaluations include ImageNet, Kinetics-400, Something-Something-v2, and Diving-48.
  • The attentive probe contains four transformer blocks with 16-head attention, ending in cross-attention with a learnable query token and residual addition.
  • Video evaluation samples multiple clip segments and three spatial views per segment, including 64 × 2 × 3 inputs for Something-Something-v2.

C.6 Action Anticipation

This section evaluates action anticipation on EPIC-KITCHENS-100, where models predict future noun, verb, and joint action labels from preceding egocentric context.

  • EPIC-KITCHENS-100 contains 100 hours of cooking videos across 45 kitchen environments and 3,568 unique action labels.
  • Action anticipation predicts noun, verb, and joint action from context preceding an action segment, with a default anticipation time of 1 second.
  • Performance is measured using mean-class Recall@5 for noun, verb, and action because multiple future actions may follow a context.
  • Probe training uses focal loss with α = 0.25 and γ = 2.0 on 32 frames sampled at 8 frames per second and 384 × 384 resolution.
  • Table 13 compares last-layer evaluation with concatenated four-intermediate-layer evaluation for a V-JEPA 2.1 ViT-L trained from scratch.
  • Table 14 compares cooldown-phase pretraining at 256 × 256 and 384 × 384 spatial resolutions.

D.1 Multi-scale evaluation

The study evaluates whether deep self-supervision reduces the need for multi-scale evaluation by comparing probe inputs from the last encoder layer with four intermediate layers.

  • D.1 Multi-scale evaluation: Table 13 compares models trained with and without Deep Self-Supervision across ADE20k segmentation, NYU depth estimation, and Diving-48 action recognition.Probes use either only the last encoder layer or four equally spaced intermediate layers.

D.2 Pretraining Spatial Resolution

The cooldown phase uses video pretraining at either 256 × 256 or 384 × 384 spatial resolution, with higher resolution benefiting downstream performance.

  • D.2 Pretraining Spatial Resolution: Higher-resolution pretraining at 384 × 384 outperforms 256 × 256 pretraining across all downstream tasks.The comparison concerns the video spatial resolution used during the cooldown phase of pretraining.
Loading 2603.14482v3…