PaperlayerView exact revision on arXiv

Source-linked AI summary

V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba, Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, Nicolas Ballas

arXiv:2506.09985v1cs.AIcs.CVcs.LGcs.RO

TL;DR

The paper addresses how AI can understand and act in the physical world largely from observation despite limited real-world interaction data. It pretrains V-JEPA 2 with self-supervised learning on internet-scale video, then post-trains an action-conditioned world model with limited robot data. The resulting system performs strongly across video understanding and anticipation and enables zero-shot robot manipulation planning in new environments.

  • Problem

    Limited real-world interaction data constrains scalable predictive world models, while existing video-based approaches show limited results in robot execution and planning.

  • Method

    V-JEPA 2 is pretrained action-free on internet-scale video in a learned representation space, then V-JEPA 2-AC is post-trained on 62 hours of unlabeled robot interaction data for action-conditioned prediction and planning.

  • Results

    V-JEPA 2 achieves strong motion understanding and state-of-the-art action anticipation, video question answering, and zero-shot robot manipulation planning across new environments.

  • Takeaways & Limitations

    Self-supervised video representations combined with a small amount of interaction data can support physical-world understanding, future-state prediction, and planning.

  • Takeaways & Limitations

    Long-horizon planning is limited by declining autoregressive prediction quality and an exponentially growing action-search space.

Abstract

from arXiv · show

A major challenge for modern AI is to learn to understand the world and learn to act largely by observation. This paper explores a self-supervised approach that combines internet-scale video data with a small amount of interaction data (robot trajectories), to develop models capable of understanding, predicting, and planning in the physical world. We first pre-train an action-free joint-embedding-predictive architecture, V-JEPA 2, on a video and image dataset comprising over 1 million hours of internet video. V-JEPA 2 achieves strong performance on motion understanding (77.3 top-1 accuracy on Something-Something v2) and state-of-the-art performance on human action anticipation (39.7 recall-at-5 on Epic-Kitchens-100) surpassing previous task-specific models. Additionally, after aligning V-JEPA 2 with a large language model, we demonstrate state-of-the-art performance on multiple video question-answering tasks at the 8 billion parameter scale (e.g., 84.0 on PerceptionTest, 76.9 on TempCompass). Finally, we show how self-supervised learning can be applied to robotic planning tasks by post-training a latent action-conditioned world model, V-JEPA 2-AC, using less than 62 hours of unlabeled robot videos from the Droid dataset. We deploy V-JEPA 2-AC zero-shot on Franka arms in two different labs and enable picking and placing of objects using planning with image goals. Notably, this is achieved without collecting any data from the robots in these environments, and without any task-specific training or reward. This work demonstrates how self-supervised learning from web-scale data and a small amount of robot interaction data can yield a world model capable of planning in the physical world.

1 Introduction

V-JEPA 2 uses self-supervised learning from internet-scale video to build representations for understanding and prediction, then adds limited robot interaction data for action-conditioned planning. The resulting system spans video understanding, anticipation, question answering, and zero-shot robotic manipulation.

  • Motivation: Limited real-world interaction data constrains the scalability of traditional predictive world-model approaches, while video-based methods have shown limited robot-execution results.These methods often emphasize prediction faithfulness and visual quality rather than planning capabilities.
  • Approach: V-JEPA 2 learns representations and predictive world knowledge by making predictions in a learned representation space from internet-scale video, including videos without direct action observations.JEPA focuses on predictable scene aspects while ignoring unpredictable pixel-level details emphasized by generative objectives.
  • Approach: V-JEPA 2 uses stage-wise training: action-free mask-denoising pretraining on more than 1 million hours of video, followed by interaction-data post-training.The encoder scales to up to 1 billion parameters and supports visual understanding through probes and language-model alignment.
  • Results: V-JEPA 2-AC uses 62 hours of unlabeled Droid interaction data to enable zero-shot Franka-arm manipulation planning in new environments.The model performs prehensile tasks such as grasping and pick-and-place without additional lab-robot data, task-specific training, or reward.
  • Results: 77.3 top-1 accuracy on Something-Something v2 demonstrates strong fine-grained motion understanding using an attentive probe.The encoder also supports downstream video question answering after alignment with a language model.
  • Results: 39.7 recall-at-5 on Epic-Kitchens-100 marks state-of-the-art human-action anticipation and a 44% relative improvement over the previous best model.The result uses an attentive probe and demonstrates prediction capability from large-scale self-supervised video pretraining.

2 V-JEPA 2: Scaling Self-Supervised Video Pretraining

V-JEPA 2 scales self-supervised video pretraining through larger datasets and models, longer training, and progressive resolution, improving visual understanding across motion and appearance tasks. The resulting recipe reaches an 88.2% average performance, a cumulative 4.0-point gain over the ViT-L/16 baseline.

  • Pretraining objective: Over 1 million hours of video support V-JEPA 2 pretraining with mask-denoising predictions in learned representation space.The approach extends V-JEPA with larger-scale models, more data, and progressive spatial and temporal resolution training.
  • Architecture: The encoder and predictor use vision transformers with 3D-RoPE, which stabilizes training for the largest models.The video is patchified into tubelets, while rotary embeddings encode temporal, height, and width positions.
  • Evaluation protocol: The evaluation freezes the encoder and trains a four-layer attentive probe across six motion and appearance classification tasks.The tasks are Something-Something v2, Diving-48, Jester, Kinetics, COIN, and ImageNet.
  • Scaling ingredients: A 1.0-point gain comes from increasing the dataset from 2 million to 22 million videos, while scaling the model from 300 million to 1 billion parameters adds 1.5 points.Extending training from 90K to 252K iterations contributes another 0.8-point improvement.
  • Scaling ingredients: 88.2% average performance follows from scaling spatial resolution from 256 to 384 and temporal duration from 16 to 64 frames, yielding a cumulative 4.0-point improvement.Each individual scaling change provides a positive impact over the ViT-L/16 baseline.
  • Efficient resolution scaling: Progressive-resolution training reduces GPU time by 8.4× for 64-frame, 384×384 inputs compared with full-resolution training from scratch.Longer clips also improve average performance by +0.7 percentage points when evaluation remains fixed at 16 frames, while longer than 64 frames adds no further improvement on these tasks.

3 V-JEPA 2-AC: Learning an Action-Conditioned World Model

V-JEPA 2-AC post-trains a latent action-conditioned world model on a small amount of unlabeled robot interaction data, using frozen V-JEPA 2 representations to predict future states. It supports goal-conditioned planning through autoregressive rollouts and receding-horizon control.

  • World-model formulation: V-JEPA 2-AC adds action conditioning to the pretrained encoder so the model predicts future video representations for robot control.The frame-causal predictor is trained on teleoperated Franka Panda data from Droid and is used in a model-predictive control loop.
  • Training data: The model uses approximately 62 hours of unlabeled Droid video, retaining raw video and end-effector state signals without reward or task-success metadata.The training clips are four seconds long at 256 × 256 resolution and 4 fps, producing 16-frame sequences.
  • Architecture: A frozen V-JEPA 2 encoder produces frame-level feature maps that a roughly 300M-parameter transformer predicts autoregressively.The predictor uses block-causal attention, allowing each timestep to attend to current and previous actions, states, and patch features.
  • Training losses: Teacher-forcing and rollout losses train the predictor both on next-step representations and on multi-step autoregressive forecasts.The rollout loss feeds predicted representations back as inputs, while the practical rollout loss uses T = 2 so gradients pass through one recurrent step.
  • Planning: Goal-conditioned planning minimizes the L1 distance between an imagined future representation and a goal representation over a fixed action horizon.The cross-entropy method updates Gaussian action distributions using top-performing trajectories, then returns the mean action sequence.
  • Planning: The controller executes only the first planned action before observing the new state and replanning, implementing receding-horizon control.This closed-loop procedure uses image goals encoded by the video encoder alongside the current robot state and observation.

4 Planning: Zero-shot Robot Control

V-JEPA 2-AC uses latent action-conditioned prediction and model-predictive control to perform visual-goal robot manipulation zero-shot in new environments. It reaches goals, supports object interaction, and exposes practical limits from camera positioning, long horizons, and image-goal specifications.

  • Experimental Setup: V-JEPA 2-AC generalizes zero-shot to new environments for reaching, grasping, and pick-and-place using visual goal specification.The robots operate in two labs absent from the Droid dataset, using uncalibrated monocular RGB input.
  • Results: <4 cm: the end-effector reaches within four centimeters of each single-goal target while monotonically decreasing error.The task uses a single goal image and tests action understanding and 3D spatial understanding from monocular RGB.
  • Results: The learned energy landscape is smooth and locally convex, with its minimum near the ground-truth action, facilitating planning without precision sensing.The minimum occurs near the correct Cartesian action in the reaching analysis.
  • Results: V-JEPA 2-AC achieves the highest success rate across grasp, reach-with-object, and pick-and-place tasks, although object type affects performance.Object interaction exposes failures from sub-optimal actions and object-specific grasping requirements.
  • Limitations: Camera positioning constrains performance because the model must infer action coordinates without explicit calibration, while long-horizon planning faces prediction error accumulation and exponential search growth.The current formulation also assumes visual goals rather than language-based task specifications.

5 Understanding: Probe-based Classification

V-JEPA 2 is evaluated with frozen-encoder attentive probes across motion and appearance classification tasks. It leads on motion understanding, remains competitive on appearance understanding, and achieves the best average performance across six tasks.

  • Evaluation: The evaluation balances motion tasks requiring multiple frames with appearance tasks that can generally be solved from a single frame.Appearance benchmarks include Kinetics400, COIN, and ImageNet.
  • Results: V-JEPA 2 outperforms state-of-the-art visual encoders on motion understanding while remaining competitive on appearance understanding.The comparison uses a common probing protocol, although the baseline encoders use different training data.
  • Evaluation: The attentive probe is a four-layer transformer trained on frozen encoder outputs, averaging classification logits across sampled clips.The final probe block uses cross-attention with a learnable query token.
  • Results: 75.3 top-1 accuracy: V-JEPA 2 ViT-g reaches this score on Something-Something v2, compared with 69.7 for InternVideo and 55.4 for PECoreG.It also reaches 84.6 on ImageNet, a +4.6-point improvement over V-JEPA.
  • Results: 88.2 average performance: the higher-resolution, longer-duration V-JEPA 2 ViT-g384 achieves further improvement across all six tasks.V-JEPA 2 obtains the best average performance among the evaluated image and video encoders.

6 Prediction: Probe-based Action Anticipation

V-JEPA 2 uses frozen representations and an attentive anticipation probe to predict future actions from video context. Performance scales with model size and surpasses task-specific prior approaches, while remaining limited by anticipation horizon and benchmark scope.

  • Task and Method: V-JEPA 2 predicts future actions from contextual video clips ending one second before the action begins.The anticipation probe combines encoded context with predicted future-frame representations.
  • Results: V-JEPA 2 significantly outperforms prior state-of-the-art action-anticipation approaches despite using only an attentive probe on pretrained representations.The baselines include supervised hand-object modeling and language-model-based methods.
  • Results: V-JEPA 2 action recall-at-5 scales linearly with model size across 300-million-to-1-billion-parameter encoders.The evaluation uses 32 frames sampled at 8 frames per second with 256 × 256 resolution, plus a 384 × 384 variant.
  • Results: +12.1 points: V-JEPA 2 ViT-g384 improves over PlausiVL on action recall-at-5, corresponding to a 44% relative improvement.The comparison is against PlausiVL’s 8 billion parameters, while the smaller V-JEPA 2 model also surpasses it.
  • Results: The model’s top-ranked predictions can include coherent alternative or ongoing actions, but failures may confuse the exact object identity.In the illustrated failure, it predicts plausible actions while missing that the object is a tea package.
  • Limitations: Accuracy degrades at longer anticipation horizons, and EK100’s kitchen-only, closed-vocabulary setting limits evidence about generalization to other environments or unseen actions.The benchmark uses fixed action categories, preventing prediction of categories absent from training.

7 Understanding : Video Question Answering

V-JEPA 2 is aligned with a large language model to support video question answering, using a video encoder pretrained without language supervision. Scaling the encoder, resolution, and alignment data yields competitive or state-of-the-art results across multiple benchmarks.

  • Approach: V-JEPA 2 is used as the visual encoder in a multimodal large language model for open-language video question answering.Its patch embeddings are projected into the language model’s input space.
  • Main results: V-JEPA 2’s video encoder was pretrained without language supervision yet achieves state-of-the-art performance on several video question-answering benchmarks after alignment.Reported 8B-class results include 84.0 test set accuracy on PerceptionTest and 76.9 multi-choice accuracy on TempCompass.
  • Controlled comparison: V-JEPA 2 obtains competitive frozen-encoder performance against DINOv2, SigLIP, and Perception Encoder under the same LLM, data, and training setup.It outperforms the compared encoders on all tested benchmarks except PerceptionTest, where it slightly underperforms SigLIP and PE.
  • Scaling: 0.9 points on PerceptionTest, 3.3 points on TVBench, and 1.2 points on MVBench result from scaling the encoder from 300M to 1B parameters at 256-pixel resolution.Increasing resolution to 512 pixels adds 2.2 points on PerceptionTest, 4.0 points on TemporalBench, and 3.3 points on TVBench.
  • Scaling alignment data: 88.5 million alignment samples produce state-of-the-art results on PerceptionTest, MVP, TempCompass, TemporalBench, and TOMATO.Relative to PerceptionLM 8B, the reported gains are 1.3, 4.8, 4.2, 8.4, and 7.1 points on those benchmarks, respectively.

8 Related Work

Related work spans world models, action-conditioned video generation, policy learning, vision-language-action control, and self-supervised vision encoders. V-JEPA 2 distinguishes itself by targeting task-agnostic generalization and model-predictive control from limited interaction data.

  • World models: Prior real-world robotics systems commonly train task-specific world models using interaction data from the deployment environment.Evaluation has focused more on explored task spaces than on generalization to new environments or unseen objects.
  • World models: V-JEPA 2 trains a task-agnostic world model and evaluates generalization to new environments and unseen objects.This contrasts with prior task-specific approaches described in the related work discussion.
  • Action-conditioned generation: Recent general-purpose action-conditioned video models can generate visually valid-looking robot plans but had not demonstrated using those models to control robots.V-JEPA 2 addresses this execution gap through a world model used for planning.
  • Control paradigms: Unlike policy-learning approaches that require expert trajectories for imitation learning, V-JEPA 2 uses model-predictive control; the approaches remain complementary.The paper identifies combining them as possible future work.
  • Vision foundation models: Self-supervised image and video foundation models use large-scale observation datasets to learn generalist vision encoders for diverse downstream tasks.This work extends that direction toward physical-world understanding, prediction, and planning.

9 Conclusion

The paper concludes that self-supervised learning from web-scale video plus limited robot interaction can support physical-world understanding, prediction, and planning. It reports successful zero-shot robotic pick-and-place, while identifying limits in horizon, goal specification, and model scale.

  • Conclusion: V-JEPA 2 combines web-scale self-supervised learning with a small amount of robot interaction data to build a world model for understanding, prediction, and planning.The conclusion presents this as the paper’s central result.
  • Conclusion: V-JEPA 2 achieves state-of-the-art action-classification and human-action-anticipation performance and outperforms previous vision encoders on video question answering after LLM alignment.Its action-conditioned extension enables zero-shot prehensile manipulation on real-world robots.
  • Limitations and future work: Planning currently targets predictions up to roughly 16 seconds into the future, supporting simpler manipulation tasks from a single goal image.Longer-horizon tasks such as pick-and-place without sub-goals require further modeling innovations.
  • Limitations and future work: V-JEPA 2-AC currently relies on image goals, although language-based goal specification may be preferable in some situations.The paper suggests embedding language goals into the V-JEPA 2-AC representation space.
  • Limitations and future work: The models are scaled only to a modest 1B parameters, leaving open whether performance improvements can be sustained at larger scales.The paper calls for scalable pretraining recipes beyond this scale.

A.2 Pretraining data

The pretraining data pipeline curates and reweights web video scenes to better match target datasets, then combines the curated collection with established image and video datasets. The resulting data and architecture support large-scale V-JEPA 2 pretraining.

  • Data curation: YT1B curation begins with scene extraction, discarding scenes shorter than 4 seconds and retaining 316 million scenes.PySceneDetect is used to split videos at scene transitions.
  • Data curation: DINOv2 scene embeddings are clustered into 1.5 million clusters, and only approximately 210,000 clusters containing target-distribution assignments are retained.The retained clusters contain 115 million scenes.
  • Data curation: Weighted sampling rebalances retained clusters toward the target distribution using dataset- and cluster-level weighting coefficients.The final curated dataset more closely matches handcrafted datasets from the literature.
  • Final dataset: The curated YT1B collection is combined with SSv2, Kinetics, HowTo100M, and ImageNet to form the final VM22M dataset.The dataset statistics are summarized in Table 11.
  • Architecture: V-JEPA 2 uses vision-transformer encoders with 16 × 16 patches, scaling from a 300M-parameter ViT-L to a 1B-parameter ViT-g while keeping the predictor fixed.Architecture details are summarized in Table 12.

A.4.1 Effect of Data Curation

Data curation, training schedules, and longer evaluation clips materially affect V-JEPA 2 performance, with benefits depending on model scale and training duration.

  • At ViT-L scale, curated YT1B improves performance across downstream classification tasks compared with uncurated YT1B.
  • At ViT-g scale, Mixed+Curated YT1B outperforms Mixed+Uncurated YT1B on all evaluated tasks.
  • After epoch 600, the uncurated-data model stops improving while curated and uncurated models initially improve similarly.
  • The cooldown phase using 64 frames and a ramped-down learning rate yields a benefit of over one point across evaluations.
  • +9.7 percentage points average improvement results from increasing evaluation video duration from 16 to 64 frames.This ablation uses single-clip evaluation because of memory constraints.

B.3 Visualizing World Model Predictions

The visualizations show that V-JEPA 2 representations preserve scene information useful for control, while V-JEPA 2-AC models action effects and exhibits camera-dependent coordinate errors.

  • Visualizing representations: Decoded V-JEPA 2 representations capture salient scene elements needed for vision-based control, although backgrounds appear blurry.A feedforward frame decoder maps representations to human-interpretable pixels using pixel reconstruction loss.
  • Action-conditioned predictions: With identical actions, open-gripper rollouts keep the cup’s predicted location unchanged, indicating object and shape constancy alongside gravity-consistent behavior.
  • Camera sensitivity: Camera position causes rotation error in the inferred action coordinate axis, which is measured by fitting a transformation from inferred to ground-truth actions.The analysis sweeps camera positions and uses 200 paired action observations per position.
  • Camera sensitivity: Because the error is primarily rotational, unsupervised calibration can rotate inferred actions using a least-squares transformation estimated from random robot actions.

C.2 Additional Results

Additional ablations examine probe size, encoder depth, and evaluation configuration for classification tasks, showing that deeper encoder information can matter for some datasets.

  • Probe size: The evaluation compares attentive probes with different numbers of layers while using 16-frame, 256 × 256 inputs.
  • Encoder multilayer: Diving-48 and Jester strongly benefit from feeding deeper encoder layers to the attentive probe.
  • Action anticipation setup: The action-anticipation probe uses four transformer blocks, including a final cross-attention layer with learnable query tokens and per-token classifiers.
  • Action anticipation setup: The default action-anticipation setup uses 32 frames at 8 frames per second, with task-specific spatial resolutions and anticipation times sampled between 0.25 and 1.75 seconds.

D.2 Additional results

Additional results characterize how probe inputs, context settings, prediction horizon, and multimodal training affect action anticipation and video question answering.

  • Action anticipation: Adding predictor outputs to encoder outputs provides a small but consistent improvement across Epic-Kitchens-100 action, verb, and object categories.Predictor outputs alone perform substantially below encoder outputs, indicating the task mainly requires semantic understanding.
  • Action anticipation: V-JEPA 2 benefits from longer context, higher frame rate, and higher resolution until performance saturates or slightly decreases.Optimal performance uses 32 frames, 8 frames per second, and 384 × 384 resolution.
  • Action anticipation: Action-anticipation performance sharply decreases as anticipation time increases from 1 to 10 seconds.The paper attributes this to the non-deterministic nature of forecasting the future in Epic-Kitchens-100.
  • Action anticipation: The most represented failure configurations include incorrect action classification, while the most represented success configuration correctly predicts verb, noun, and action.
  • Multimodal training: Image-text pairs are crucial for high downstream benchmark performance, and repeating images across frames is ineffective for extracting fine-grained information.The training pipeline instead uses Dynamic S2 processing to preserve image resolution granularity.
  • Multimodal training: The multimodal model uses staged image and video alignment, with the largest video-question-answering improvement occurring after video captioning and question-answering training.
  • Video question answering: As V-JEPA 2 receives more video frames during training and testing, downstream multimodal performance improves linearly, unlike DINOv2 performance.

E.3 Data scaling setup

The data-scaling setup adapts an 8B language-model pipeline to V-JEPA 2 and evaluates the resulting model against established video-language baselines.

  • The setup replaces the original visual encoder with V-JEPA 2 and uses Llama 3.1 8B Instruct as the language-model backbone.The model uses an MLP projector without pooling, producing 288 tokens per frame.
  • The training procedure follows three progressive stages, beginning by aligning the MLP pooler with image-captioning data.
  • The scaling runs are compared with Qwen2VL, Qwen2.5VL, InternVL-2.5, and PerceptionLM 8B.Baseline figures come directly from the cited papers, except for MVP, which the authors run themselves.
  • Evaluation follows a similar controlled pipeline using lmms-eval and reports results on 32 frames.
Loading 2506.09985v1…