PaperlayerView exact revision on arXiv

Source-linked AI summary

JEPA-VLA: Video Predictive Embedding is Needed for VLA Models

Shangchen Miao, Ningya Feng, Jialong Wu, Ye Lin, Xu He, Dong Li, Mingsheng Long

arXiv:2602.11832v1cs.CVcs.RO

TL;DR

Current VLAs have limited sample efficiency and generalization because common visual representations inadequately encode task-relevant environment understanding and policy priors. The paper analyzes video-predictive V-JEPA 2 and introduces JEPA-VLA to integrate it into existing VLAs, reporting consistent performance gains across multiple benchmarks and real-robot tasks.

  • Problem

    Current VLAs struggle with low sample efficiency and limited generalization, while their pretrained visual representations inadequately provide environment understanding and policy priors.

  • Method

    The paper analyzes V-JEPA 2 video-predictive representations and proposes JEPA-VLA, a general framework for integrating them into existing VLAs.

  • Results

    JEPA-VLA consistently boosts VLA performance across LIBERO, LIBERO-plus, RoboTwin2.0, and real-world tasks.

  • Takeaways & Limitations

    Video-predictive embeddings can provide task-relevant state representations and temporal regularities that act as policy priors for VLAs.

  • Takeaways & Limitations

    The work focuses on a straightforward fusion strategy, while more principled mechanisms for integrating predictive embeddings remain largely unexplored.

Abstract

from arXiv · show

Recent vision-language-action (VLA) models built upon pretrained vision-language models (VLMs) have achieved significant improvements in robotic manipulation. However, current VLAs still suffer from low sample efficiency and limited generalization. This paper argues that these limitations are closely tied to an overlooked component, pretrained visual representation, which offers insufficient knowledge on both aspects of environment understanding and policy prior. Through an in-depth analysis, we find that commonly used visual representations in VLAs, whether pretrained via language-image contrastive learning or image-based self-supervised learning, remain inadequate at capturing crucial, task-relevant environment information and at inducing effective policy priors, i.e., anticipatory knowledge of how the environment evolves under successful task execution. In contrast, we discover that predictive embeddings pretrained on videos, in particular V-JEPA 2, are adept at flexibly discarding unpredictable environment factors and encoding task-relevant temporal dynamics, thereby effectively compensating for key shortcomings of existing visual representations in VLAs. Building on these observations, we introduce JEPA-VLA, a simple yet effective approach that adaptively integrates predictive embeddings into existing VLAs. Our experiments demonstrate that JEPA-VLA yields substantial performance gains across a range of benchmarks, including LIBERO, LIBERO-plus, RoboTwin2.0, and real-robot tasks.

1. Introduction

Current VLAs struggle with sample efficiency and generalization because their visual representations inadequately support environment understanding and policy priors. The paper argues that video-predictive V-JEPA 2 addresses these gaps, motivating JEPA-VLA.

  • Motivation: Current VLAs can suffer up to ∼40% performance degradation on unseen tasks or distribution shifts despite training on millions of trajectories.The cited context notes that these datasets may cover only tens of tasks.
  • Motivation: Robotic control requires representations that capture task-relevant object attributes while discarding nuisance factors and anticipate state evolution under successful actions.These capabilities are termed environment understanding and policy priors.
  • Motivation: Common image-based and language-image representations inadequately provide the visual knowledge required for robotic control.The paper attributes these deficiencies to image-level augmentation biases and limited coverage beyond text-described information.
  • Approach: V-JEPA 2 uses video-based latent prediction to emphasize predictable task-relevant factors, suppress nuisances, and encode dynamic cues important for robotics.The representation is pretrained on internet-scale videos by predicting masked video patches in latent space.
  • Contribution: JEPA-VLA integrates V-JEPA 2 into existing VLAs and yields consistent improvements in sample efficiency and generalization across benchmarks and real-world tasks.The reported evaluation includes LIBERO, LIBERO-plus, RoboTwin2.0, and a real-world experiment.

2. An Analysis of Vision Representations for VLAs

The analysis evaluates whether visual representations encode task-relevant states, discard nuisance factors, and anticipate state transitions. V-JEPA 2 outperforms DINOv2 and SigLIP on these task-relevant probes while encoding less irrelevant visual information.

  • Analysis framework: The POMDP formulation emphasizes inferring latent environment states and their evolution from partial visual observations.The agent observes o_t = ϕ(s_t), while actions induce transitions to future states.
  • Analysis framework: The study probes frozen vision encoders with lightweight heads for task-relevant state regression, nuisance-factor regression, and future state-residual prediction.LIBERO-10 supplies current-state and transition probes, while LIBERO-plus supplies lighting and background perturbations.
  • Task-relevant state estimation: V-JEPA 2 achieves lower MSE than DINOv2 and SigLIP when regressing current task-relevant states.Its representations capture object-centric and motion-related cues more aligned with manipulation states.
  • Task-irrelevant state estimation: V-JEPA 2 encodes less information about lighting and background perturbations than DINOv2 and SigLIP.Higher nuisance-regression error indicates greater insensitivity to irrelevant variations.
  • Policy priors: V-JEPA 2 achieves lower MSE when predicting future task-relevant state residuals than DINOv2 and SigLIP.The result suggests transition-aware representations that capture temporal regularities aligned with successful task execution.

3. JEPA-VLA

JEPA-VLA incorporates frozen V-JEPA 2 video-predictive representations into existing VLA action models. It uses lightweight or gated fusion to improve environment understanding and action generation without replacing the VLA backbone.

  • Framework: JEPA-VLA combines an action model with a frozen, pretrained V-JEPA 2 encoder.The encoder maps a video clip to visual representations that condition the enhanced action model.
  • Framework: The V-JEPA 2 encoder maps video clips to spatiotemporal visual representations used by the action model.The representation tensor scales with the input video's spatial and temporal resolution.
  • Fusion strategies: JEPA-VLA introduces two fusion strategies for conditioning action models on additional visual representations.The strategies are early fusion and gated fusion.
  • Fusion strategies: Early fusion concatenates projected V-JEPA 2 embeddings with the original token sequence.This lightweight design is reported to work well when the policy is largely learned from scratch.
  • Fusion strategies: Gated fusion preserves pretrained VLA priors by adapting V-JEPA 2 information through gated cross-attention.Sparse insertion and lower learning rates for new fusion layers balance efficiency with stable integration.

4. Experiments

The experiments evaluate JEPA-VLA across simulation, domain-shifted, mainstream-VLA, and real-world settings, including limited-data regimes. Across these evaluations, integrating V-JEPA 2 improves task performance and generalization.

  • Benchmarks: JEPA-VLA is evaluated on LIBERO, LIBERO-plus, RoboTwin2.0, and a real-world pick-and-place task.The evaluation includes both controlled simulation and deployment-oriented settings.
  • Real-world results: In real-world experiments, JEPA-VLA outperforms the baseline while training on only one-fifth of the trajectories.The experiment uses a Piper robot arm and tests generalization under varied settings.
  • Implementation: The basic VLA uses a pretrained VLM backbone and linear action head, with V-JEPA 2 representations added as conditioning signals.The method extracts representations from the two most recent frames alongside the current observation and language instruction.
  • Basic VLA results: 7.4% on LIBERO and 6.7% on LIBERO-plus: V-JEPA 2 improves basic-VLA success rates.These experiments use only one-tenth of the LIBERO data.
  • Basic VLA results: V-JEPA 2 outperforms the official WorldVLA model on LIBERO-plus despite using one-tenth of the action-model data and no world-model training data.The result is reported as evidence of enhanced generalization.
  • Mainstream VLA results: 6.1% over the baseline on mainstream LIBERO, while RoboTwin2.0 gains average 18.7% clean and 8.4% domain-randomized success.The method also surpasses official OpenVLA-OFT results and transfers across mainstream VLA architectures.
  • Representation comparison: Controlled replacement experiments show marginal or negative gains from DINOv2 and modest gains from SigLIP, whereas V-JEPA 2 produces substantially larger improvements.The comparison directly swaps visual representations in the basic VLA baseline.

5. Related Work

Related work places JEPA-VLA at the intersection of vision-language-action models, pretrained robotic visual representations, and predictive JEPA architectures. These lines of work motivate stronger visual backbones for language-conditioned embodied control.

  • Vision-Language-Action Models: Vision-language-action models combine visual observations and task instructions to generate robotic manipulation actions, but still exhibit insufficient visual understanding.Recent VLAs improve language-conditioned control and can generalize to unseen tasks.
  • Pretrained Vision Representations for Robotics: Robotics-specific pretrained visual representations improve transfer in selected settings but are often not general enough to serve as universal VLA backbones.Recent VLAs commonly rely on vision backbones from pretrained vision-language models.
  • The Evolution of JEPA: JEPA research progresses from predictive world modeling to I-JEPA, V-JEPA, and V-JEPA 2 through self-supervised learning.The framework targets efficient understanding, prediction, and reasoning without large labeled datasets or traditional generative objectives.

6. Conclusion and Future Work

The paper concludes that video-predictive embeddings address missing environment understanding and policy priors in VLAs, while identifying more principled fusion mechanisms as future work.

  • JEPA-VLA addresses two visual-knowledge gaps in VLAs: task-relevant environment understanding and temporal policy priors.
  • The framework consistently improves VLA performance across multiple benchmarks.
  • Future Work: More principled mechanisms for integrating predictive embeddings remain largely unexplored beyond the paper’s straightforward fusion strategy.

A. Details of Vision Representation Analysis

The analysis evaluates whether frozen visual representations encode relevant states, ignore irrelevant factors, and predict transition-aware state changes. Experiments use standardized dataset splits, early stopping, and representation-specific regression or reconstruction heads.

  • The analysis uses 8:1:1 training, validation, and test splits, with early stopping after 10 validation epochs without improvement.The checkpoint with the lowest validation loss is selected for test evaluation.
  • The hyperparameters for the vision-representation analysis are summarized in Table 8.
  • Task-Relevant State Regression: Task-relevant state regression prepends a [CLS] token and predicts normalized relevant-state labels from frozen visual features.Labels are normalized to zero mean and unit variance for training stability.
  • Task-Irrelevant State Regression: Task-irrelevant sensitivity is assessed by regressing lighting parameters and reconstructing background texture from frozen representations.Lighting uses a [CLS]-based prediction head, while background reconstruction uses linear upsampling and MAE loss.
  • Task-Relevant State Prediction: Transition-aware representation quality is tested by predicting the residual between the current state and its state 10 time steps later.Residual prediction is used because adjacent LIBERO-10 states often change only subtly.

B. JEPA-VLA Fusion Method Details

JEPA-VLA adapts its fusion mechanism to the underlying VLA: concatenation injects V-JEPA 2 embeddings for models without extensive robotic pretraining, while gated cross-attention protects pretrained priors in larger pretrained models.

  • The fusion design distinguishes VLAs without extensive robotic pretraining from VLAs pretrained on large-scale robotic datasets.
  • VLAs Without Extensive Robotic Pretraining: For VLAs without extensive robotic pretraining, V-JEPA 2 representations are concatenated with the original token sequence as additional input embeddings.This injects temporal and state-centric information into the policy.
  • Pretrained VLAs: Naive concatenation can disrupt pretrained representations and degrade performance when a VLA already has strong task-specific or action-aligned priors.
  • Pretrained VLAs: For heavily pretrained VLAs, gated cross-attention uses original tokens as queries and V-JEPA 2 features as keys and values.Gating adaptively controls predictive-feature contributions while preserving pretrained priors.

C. LIBERO-plus Benchmark Implementation Details

LIBERO-plus evaluation trains each VLA separately on four task suites and averages scores from four selected checkpoints to reduce checkpoint-selection variance. Results are also reported separately for Spatial, Object, Goal, and Long tasks.

  • Each VLA is trained separately on the four LIBERO task suites under the official LIBERO-plus protocol.Four selected checkpoints are evaluated per suite and their scores are averaged.
  • Detailed results are reported separately for the Spatial, Object, Goal, and Long task suites.These results appear in Tables 9, 10, 11, and 12, respectively.

D. CortexBench Benchmark Implementation Details

The CortexBench evaluation is restricted to tasks where VC-1 baseline results can be reliably reproduced, and reports reproduced rewards for MetaWorld because success-rate mapping is unspecified.

  • VC-1 performance could not be reliably reproduced for all CortexBench tasks using the publicly released code and configurations.
  • CortexBench spans 17 robotic manipulation tasks, but evaluation is restricted to 10 tasks with consistently reproducible VC-1 results.The comparison uses V-JEPA 2 and VC-1 under identical experimental settings on this subset.
  • MetaWorld results report reproduced task rewards rather than binary success rates because the implementation provides no explicit reward-to-success mapping.This preserves a fair and transparent comparison between VC-1 and V-JEPA 2.

E. Real-World Experiment Details

The real-world experiment uses 100 demonstration trajectories to train baseline and JEPA-VLA models, with an additional one-fifth-data condition and evaluation across in-domain, lighting, and layout variations.

  • Data Collection: 100 demonstration trajectories are collected for a real-world pick-and-place task using a single robotic arm.Both the baseline and JEPA-VLA are trained on these trajectories, while an additional JEPA-VLA condition uses one-fifth of the data.
  • Training Details: JEPA-VLA adds V-JEPA 2 embeddings from the two most recent visual frames to the original VLA token sequence.Models are trained for 30 epochs with action chunk size 10, learning rate 5 × 10^-6, and batch size 32.
  • Evaluation Details: Evaluation uses future-action masking with parallel decoding and interpolated consecutive actions to produce smooth trajectories.
  • Evaluation Details: Each model is evaluated over 10 independent trials under in-domain conditions and variations in lighting and object layouts.Representative scenarios and successful trajectories are shown in Figure 5.
Loading 2602.11832v1…