Source-linked AI summary
VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
Jingwen Sun, Wenyao Zhang, Zekun Qi, Shaojie Ren, Zezhi Liu, Hanxin Zhu, Guangzhong Sun, Xin Jin, Zhibo Chen
TL;DR
Internet-video VLA pretraining can produce temporally predictive representations that are weakly tied to controllable structure, while existing multi-stage pipelines are complex and fragile. VLA-JEPA addresses this with leakage-free latent state prediction and achieves consistent robustness and generalization gains across benchmarks and real-world settings.
Problem
Internet-video latent-action pretraining can yield representations weakly tied to controllable structure, causing brittle behavior, poor transfer, and inefficient fine-tuning; existing multi-stage pipelines are also complex and fragile.
Method
VLA-JEPA uses leakage-free state prediction: a target encoder forms future latent targets, while the VLM pathway receives only the current observation and learns a JEPA latent world model.
Results
VLA-JEPA achieves consistent gains in robustness and generalization across LIBERO, LIBERO-Plus, SimplerEnv, and real-world settings, with competitive simulation and robotic performance.
Takeaways & Limitations
VLA-JEPA mitigates information leakage and representation degeneration, enabling the model to capture genuine inter-frame dynamics through a scalable human-video pretraining paradigm.
Takeaways & Limitations
In real-world experiments, VLA-JEPA is prone to grasping objects that do not align with textual commands because it lacks fine-grained textual reasoning.
Abstract
from arXiv · showhide
Pretraining Vision-Language-Action (VLA) policies on internet-scale video is appealing, yet current latent-action objectives often learn the wrong thing: they remain anchored to pixel variation rather than action-relevant state transitions, making them vulnerable to appearance bias, nuisance motion, and information leakage. We introduce VLA-JEPA, a JEPA-style pretraining framework that sidesteps these pitfalls by design. The key idea is leakage-free state prediction: a target encoder produces latent representations from future frames, while the student pathway sees only the current observation -- future information is used solely as supervision targets, never as input. By predicting in latent space rather than pixel space, VLA-JEPA learns dynamics abstractions that are robust to camera motion and irrelevant background changes. This yields a simple two-stage recipe -- JEPA pretraining followed by action-head fine-tuning -- without the multi-stage complexity of prior latent-action pipelines. Experiments on LIBERO, LIBERO-Plus, SimplerEnv and real-world manipulation tasks show that VLA-JEPA achieves consistent gains in generalization and robustness over existing methods.
1 Introduction
The introduction argues that video-based latent-action pretraining often learns pixel variation and nuisance dynamics instead of action-relevant state transitions. VLA-JEPA addresses these issues with leakage-free latent-state prediction and a simpler two-stage training workflow.
- Internet-scale video offers abundant, diverse demonstrations that can reduce reliance on costly, narrow robot interaction data.
- Why latent-action pretraining often drifts away: Pixel-level and frame-difference objectives bias representations toward texture, illumination, background clutter, viewpoint, and other visually dominant but weakly controllable factors.
- Why latent-action pretraining often drifts away: Future-context inputs create an information-leakage shortcut, allowing latent actions to encode future observations rather than meaningful state-transition structure.
- Why latent-action pretraining often drifts away: Multi-stage pipelines combining representation pretraining, latent-action alignment, and policy learning increase engineering complexity and stage-wise inconsistency.
- VLA-JEPA: VLA-JEPA predicts future latent states from current observations and history representations, using future frames only as targets and avoiding pixel reconstruction.
- Improved and robustness with a simpler workflow: Across LIBERO, LIBERO-Plus, SimplerEnv, and real-world settings, VLA-JEPA reports consistent gains in robustness and generalization while simplifying training.
2 Related Works
Related work reduces dependence on action-labeled robot data by extracting latent actions from video, but existing approaches often require visual priors, alignment mechanisms, or multi-stage training.
- VLA research increasingly uses multimodal models, while large-scale action-labeled robot data remain costly and difficult to scale.
- Latent-action methods extract discrete or continuous motion tokens from human and robot video before mapping them to real robot controls.
- Hand-crafted visual priors can align latent spaces with visual deltas rather than controllable signals, causing systematic failures in novel environments.
3 Methodology
VLA-JEPA jointly pretrains on action-free human videos and action-labeled robot data by learning latent state-transition dynamics, then conditions embodied action generation on the learned representations.
- VLA-JEPA jointly pretrains on action-free human videos and action-labeled robot data.Human videos use a world-model-based state-transition objective, while robot data additionally support action prediction.
- Learning from Human Videos: A V-JEPA2 encoder produces unified world-state representations by encoding observations and concatenating representations across multiple viewpoints.The resulting representation corresponds to the world state at each timestamp.
- Learning from Human Videos: The autoregressive world model predicts future state chunks from encoded state histories and corresponding latent-action conditioning variables.Attention is bidirectionally full within each timestep but strictly causal across timesteps, preventing access to future steps.
- Learning from Human Videos: The VLM maps learnable ⟨latenti⟩ tokens from initial multi-view observations and language into latent representations summarizing world dynamics.The representations are used as conditioning variables for subsequent world-state prediction.
- Learning from Human Videos: The JEPA-style objective trains an online predictor against frozen V-JEPA2 target states using latent-space reconstruction.Because the target encoder produces deterministic embeddings, the KL term vanishes and the ELBO reduces to latent-space reconstruction loss.
- Action Prediction with Joint Optimization Objectives: For robot data, latent representations condition a flow-matching action head that generates continuous action trajectories.The action-conditioning representation is derived from visual tokens, language, latent-action tokens, and embodied action tokens; inference integrates the learned vector field from noise to data space.
4 Experiments
VLA-JEPA is evaluated across simulation benchmarks and real-world manipulation settings, with analyses covering benchmark performance, robustness, human-video effects, and latent-action behavior. Results show strong generalization and robustness, while real-world deployment reveals a trade-off between instruction following and execution safety.
- Experimental setup: VLA-JEPA is evaluated on LIBERO, LIBERO-Plus, SimplerEnv, and a real-world Franka robot environment.The evaluation spans three simulation benchmarks and one real-world environment.
- LIBERO: VLA-JEPA achieves state-of-the-art performance on 2 of 4 LIBERO task suites and the highest average success rate overall.It outperforms prior latent-action VLAs and human-video-trained VLAs while using less training data than some top-performing baselines.
- SimplerEnv: VLA-JEPA achieves the best performance on 2 of 4 tasks for both Google Robot and WidowX Robot in SimplerEnv.It attains the highest average success rate on Google Robot and the second highest on WidowX Robot; robot-data quality substantially affects performance.
- LIBERO-Plus: VLA-JEPA achieves the best performance on 5 of 7 LIBERO-Plus perturbations, with significant advantages under Language, Light, Background, and Layout perturbations.These results indicate robustness to task-agnostic disturbances and generalization across perturbation dimensions.
- Real-world experiments: In real-world experiments, VLA-JEPA achieves state-of-the-art performance in ID and object-layout OOD settings and the second-best result in task OOD.Its trajectories are more stable and reliable than π0.5, although its generalization is less robust and its instruction following is weaker.
- Further analysis and ablation study: Increasing human-video data consistently improves LIBERO-Plus robustness, primarily strengthening existing skills rather than adding new action-execution capabilities.Human videos provide substantial gains on LIBERO-Plus, while removing them does not significantly reduce performance across LIBERO and SimplerEnv.
- Further analysis and ablation study: Compared with prior methods, VLA-JEPA's latent actions focus more precisely on the robot arm, hand, and manipulated objects rather than operation-irrelevant visual details.The analysis contrasts this behavior with LAPA's dense attention to irrelevant objects and UniVLA's attention to background elements.
- Further analysis and ablation study: VLA-JEPA's performance is best when the video horizon is close to the predefined action horizon; horizons that are too small or large degrade performance.Short horizons provide insufficient information, whereas excessive horizons add redundancy and particularly hurt fine-grained spatial tasks.
5 Conclusion
VLA-JEPA is presented as a unified framework that learns latent actions from human and robot videos through latent world modeling. It mitigates information leakage and representation degeneration, achieving competitive simulation and real-world performance while offering a scalable pretraining paradigm.
- VLA-JEPA learns latent actions from both human and robot videos through latent world modeling.
- Existing latent-action methods suffer from information leakage and representation degeneration that hinder meaningful temporal-dynamics learning.
- VLA-JEPA mitigates these issues and captures genuine inter-frame dynamics.
- The human-video pretraining paradigm is described as scalable and extensible to robot data and text-based reasoning data.
A.1 VLA-JEPA Architecture
The architecture uses Qwen3-VL-2B to produce latent action tokens and a V-JEPA2-based latent world model to predict physical dynamics with time-causal attention.
- Qwen3-VL-2B serves as the dense Transformer VLM backbone, combining a Vision Transformer with 3D convolutional modules.
- A V-JEPA2 encoder checkpoint and randomly initialized predictor form the latent world model, using VLM-produced latent action tokens as inputs.
- The vocabulary adds latent action and embodied action tokens so the VLM can output time-aware latent actions and embodied actions.
- The auto-regressive world model uses bidirectional attention within each time step and causal attention across time steps.
A.2 Training Details
Training uses standardized image and video preprocessing, selected camera-view handling, distributed optimization, and staged pretraining schedules; Figure 7 evaluates object-layout OOD behavior.
- Preprocessing: Observation images are resized to 224×224, while world-state video clips are resized to 256×256.
- Evaluation: Figure 7 compares π0, π0.5, and VLA-JEPA under the object-layout OOD setting.
- Camera views: When fewer than two camera views are available, the world-state representation is duplicated and concatenated; with more views, two are selected.
- Optimization: Training uses a global batch size of 256 across eight GPUs, with cosine learning-rate scheduling and linear warmup.
B Real-world Experiments Details
Real-world experiments evaluate manipulation across in-distribution, task-level OOD, and object-layout OOD settings using a Franka arm, gripper, and three cameras. VLA-JEPA shows approximately 50% banana-task success and more adaptive regrasping behavior after failed grasps.
- Setup: The setup uses a Franka Research 3 arm, Robotiq 2F-85 gripper, and three RealSense D435 cameras.Expert demonstrations cover picking and placing grapes, apples, mangoes, and oranges; each task uses 10 independent trials.
- Task-level OOD: Task-level OOD tests include banana-to-bowl, peach-to-plate, and grape-to-shelf manipulation tasks.
- Task-level OOD: Approximately 50% success is achieved by both π0.5 and VLA-JEPA on the banana-picking task.
- Task-level OOD: On the shelf task, none of the models successfully place the end-effector at the top level because shelves are absent from training data.VLA-JEPA approaches from behind the shelf rather than directly colliding with it, unlike π0 and π0.5.
- Object-layout OOD: Object-layout OOD evaluation randomly shuffles object layouts for three tasks selected from the training set.
- Object-layout OOD: After failed grasps, VLA-JEPA immediately opens the gripper and retries, whereas π0 and π0.5 do not reattempt grasping.The paper connects this behavior to human-video exposure containing repeated grasping actions and regrasp decisions.