PaperlayerView exact revision on arXiv

Source-linked AI summary

PLaW-VLA: Predictive Latent World Modeling for Vision-Language-Action Policies

Yu Liu, Hetian Guo, Tianlv Huang, Ziyi Cai, Wudi Chen, Hantang Wang, Qiutong Liu, Yingzhi Peng, Wei Han, Peijun Tang, Jianan Wang, Zipei Fan, Zhiyuan Zha, Xuan Song

arXiv:2610.12285v1cs.RO

TL;DR

Long-horizon VLA policies need predictive context, but reconstruction-oriented futures also model visual details irrelevant to control. PLaW-VLA predicts task-relevant futures in a prediction-oriented latent space and uses them to condition action generation. It reports gains on RoboTwin Hard Horizon III and zero-shot LIBERO-Plus, with lower inference latency than generative world-action modeling at comparable policy performance.

  • Problem

    For VLA control, it remains open which future representation should be predicted and how it should condition actions, since reconstruction-oriented targets also model control-irrelevant appearance variation.

    Source: §1 · p. 2

  • Method

    PLaW-VLA predicts task-relevant future states in a pretrained prediction-oriented latent space and conditions continuous action generation on them alongside observation history and task semantics.

    Source: §1 · p. 2

  • Results

    +11.8 percentage points over π0.5 on RoboTwin Hard Horizon III; PLaW-VLA achieves 72.7% task-weighted success on zero-shot LIBERO-Plus.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PLaW-VLA achieves about 1/19 the inference latency of generative world-action modeling at comparable policy performance.

    Source: §5 · p. 9

    The current video-to-robot training data remain limited in scale and diversity, and scalability of the training paradigm remains underexplored.

    Source: §6 · p. 9

Abstract

from arXiv · show

Learning to predict how the world evolves can provide vision-language-action (VLA) policies with predictive context for long-horizon control, but its effectiveness depends on what future representation is modeled and how it conditions action generation. We introduce PLaW-VLA, which models task-relevant future states in a pretrained prediction-oriented representation space, reducing the need to predict control-irrelevant visual details. Built on a Mixture-of-Transformers architecture, PLaW-VLA conditions action generation on observation history, current task semantics, and predicted future states through structured causal attention. Experiments show a +11.8 percentage-point (pp) gain over reactive policies on RoboTwin Hard Horizon III and a +1.77 pp gain over reconstruction-oriented latent prediction on zero-shot LIBERO-Plus, supporting improved long-horizon control and generalization under distribution shift, respectively. By avoiding low-level visual reconstruction, PLaW-VLA lowers the burden of future prediction, enabling a lightweight latent world model with parallel future prediction and about 1/19 the inference latency of generative world-action modeling at comparable policy performance.

1 Introduction

Long-horizon VLA control must track task evolution under partial observability and anticipate future states, but reconstruction-oriented prediction also models visual details irrelevant to control. PLaW-VLA predicts task-relevant futures in a pretrained prediction-oriented latent space, conditions actions on those futures, and uses staged video-to-robot training.

1 Introduction

  • Long-horizon manipulation needs temporal context from observation history and anticipation of future state evolution because partial observability and execution perturbations can make current observations insufficient.This motivates predictive context for action generation across successive interactions.Source: §1 · p. 2
  • Existing reconstruction- or generation-based targets model both task dynamics and appearance variations, increasing prediction burden and potentially propagating control-irrelevant errors into action generation.The central design questions are which future representation to predict and how it should condition action generation.Source: §1 · p. 2
  • PLaW-VLA predicts task-relevant future states in a pretrained prediction-oriented representation space and directly conditions action generation on them through structured causal attention.It combines observation history, current task semantics, and predicted futures to generate continuous robot actions.Source: §1 · p. 2
  • Training proceeds from action-free human and robot videos to joint prediction and action learning on robot trajectories, then target-task and embodiment adaptation.The three stages reuse predictive knowledge from large-scale videos for robot policy learning.Source: §1 · p. 2
  • PLaW-VLA outperforms π0.5 by 11.8 percentage points on RoboTwin Hard Horizon III and achieves 72.7% task-weighted success on zero-shot LIBERO-Plus, supporting long-horizon control and distribution-shift generalization.Source: §1 · p. 2

2 Related Work

Prior VLA work augments policies with memory, spatial representations, and hierarchical reasoning, while world-model approaches predict observations, latents, or video-action trajectories. PLaW-VLA combines a prediction-trained target space with direct future-state conditioning of action generation.

Vision-Language-Action Models

  • VLA policies transfer pretrained visual and language representations to robot control, with complementary work adding memory, spatial representations, or hierarchical task reasoning.PLaW-VLA complements these directions by adding predicted future visual representations as action-generation context.Source: §2 · p. 2

World Models for Robot Control

  • World-model methods use generated images or videos, autoregressive future observation-action models, or video-action generative architectures; PLaW-VLA instead predicts features in frozen V-JEPA 2’s prediction-trained representation space.The cited approaches include visual imagination [26] [27] [28] [29], autoregressive formulations [18] [30], and video-action generative architectures [13] [14] [10] [11] [15].Source: §2 · p. 3

Predictive Representations for Policy Learning

  • Predictive representation methods use future features as auxiliary supervision, latent action representations, or fused visual features, whereas PLaW-VLA uses predicted future states directly to condition action generation.The comparisons include FLARE [7], VLA-JEPA, and JEPA-VLA [9].Source: §2 · p. 3

3 Method

PLaW-VLA factorizes action generation into semantic grounding, latent future prediction, and action decoding. Its three-expert architecture uses structured causal attention, and its training objectives jointly supervise future-state prediction and continuous action generation.

3.1 Problem Formulation

  • The model factorizes action generation into semantic grounding from language and visual history, prediction of future latent states, and decoding of future actions.The latent world model predicts from grounded semantics and observation history; the action model uses grounded semantics, historical latents, and predicted futures.Source: §3.1 · p. 3

3.2 Model Architecture

  • Three causally ordered Mixture-of-Transformers experts handle semantic grounding, latent prediction, and continuous action generation, with structured attention regulating information flow.The world model predicts future representations in frozen V-JEPA 2 latent space using learnable queries, and the action expert attends to those tokens and observed context.Source: §3.2 · p. 3, Figure 2 · p. 4
  • Structured cross-expert attention preserves semantic-to-prediction-to-action information flow while preventing future information leakage and reducing cross-expert interference.The tokens denote contextual, historical-latent, future-query, and action-query inputs, respectively.Source: §3.2 · p. 4

3.3 Training Objective

  • Training uses a frozen V-JEPA 2 encoder to provide latent targets for future-state prediction and a flow-matching objective for continuous action chunks.During robot policy learning, the two objectives are jointly optimized; the predictive branch is stochastically dropped to avoid over-reliance on predictive context.Source: §3.3 · p. 5
  • The three training stages pretrain prediction on action-free videos, jointly learn prediction and action generation on robot trajectories, and adapt to the target task and embodiment.When the predictive branch is dropped, historical and predicted latents are omitted from action conditioning and the latent-world-model objective is disabled while action generation remains active.Source: §3.3 · p. 5

4 Experiments

PLaW-VLA performs strongly across simulation and real-world evaluation, with particularly large gains on long-horizon RoboTwin tasks. Ablations examine the contributions of predictive modeling, future conditioning, representation choice, pretraining, and inference-time use; the final comparison shows low added inference cost.

4.1 Simulation Benchmarks

  • PLaW-VLA reaches 97.4% average LIBERO success and 72.7% task-weighted zero-shot LIBERO-Plus success.On LIBERO, the improvement is most pronounced on LIBERO-Long (94.6% versus 92.4%); on LIBERO-Plus, it performs best under Robot, Layout, Light, and Background perturbations and outperforms WorldVLA across all seven perturbation types.Source: §4.1 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
  • +11.8 points on RoboTwin Hard Horizon III: PLaW-VLA reaches 79.2% success versus π0.5’s 67.4%.On Hard tasks overall, average success rises from 76.8% to 83.2%.Source: §4.1 · p. 6, Table 3 · p. 7

4.2 Real-World Evaluation

  • PLaW-VLA outperforms both baselines on all four seen real-world tasks and maintains higher progress on two zero-shot variants with changed appearances or receptacle geometry.The largest seen-task gains are on Fold Towel and Pack Toy, which require tracking intermediate interaction states.Source: Figure 4 · p. 7, §4.2 · p. 7

4.3 Ablation Studies

  • With matched Stage II data and training budgets, removing predictive world modeling lowers success by 3.56 points on LIBERO-Plus and 0.23 points on LIBERO.The results indicate a predictive-modeling gain even without additional video pretraining.Source: §4.3 · p. 7
  • On LIBERO-Plus, removing predictive supervision lowers success by 5.14 points, while removing future conditioning lowers it by 2.89 points.Replacing V-JEPA 2 with Cosmos VAE also lowers success, from 72.70% to 70.93% on LIBERO-Plus and from 97.40% to 96.65% on LIBERO.Source: §4.3 · p. 8
  • Removing Stage I, Stage II, or both lowers LIBERO-Plus success by 3.65, 6.67, and 11.90 percentage points, respectively.Stage I learns predictive dynamics from action-free videos; Stage II grounds them in robot trajectories through joint prediction and action learning.Source: §4.3 · p. 8
  • Masking all predicted future states reduces LIBERO success from 97.40% to 93.05%; masking only far- or near-future states reduces it to 95.45% or 95.80%.Both horizons contribute to average success, with suite-level exceptions; the predicted features also change with future horizon and robot-arm motion.Source: §4.3 · pp. 8–9

4.4 Inference Efficiency

  • PLaW-VLA achieves comparable LIBERO success to Motus at about 1/19 of its inference latency; versus π0.5, it adds 0.9B parameters and approximately 70 ms while increasing success from 96.9% to 97.4%.Its latent model predicts multiple future states in parallel without video generation or decoding.Source: §4.4 · p. 9

5 Conclusion

PLaW-VLA predicts task-relevant state evolution in a prediction-oriented latent space and uses those future states to condition action generation. The reported results show gains on long-horizon and zero-shot benchmarks, alongside substantially lower inference latency than generative world-action modeling at comparable policy performance.

5 Conclusion

  • PLaW-VLA achieves a +11.8 pp gain over π0.5 on RoboTwin Hard Horizon III, 72.7% average success on zero-shot LIBERO-Plus, and about 1/19 the inference latency of generative world-action modeling at comparable policy performance.Ablations identify contributions from predictive supervision, future conditioning, representation space, and pretraining.Source: §5 · p. 9
  • PLaW-VLA models task-relevant state evolution in a prediction-oriented latent space and directly conditions action generation on predicted futures.The design is intended to increase control relevance of future representations while reducing the influence of control-irrelevant prediction errors on policy outputs.Source: §5 · p. 9

6 Limitations

The paper identifies coupled prediction and closed-loop control errors, and underexplored scalability of its video-to-robot training paradigm, as limitations.

6 Limitations

  • Execution deviations can degrade future-state prediction, while inaccurate predictions may affect later actions; training-data scale and diversity also remain limited, leaving transfer scaling underexplored.The authors identify predictive uncertainty, adaptive horizons, recovery mechanisms, and study of video scale and distribution as directions for future work.Source: §6 · p. 9

A.1 Extended LIBERO Results

Extended comparisons show PLaW-VLA has the highest average success on LIBERO and the highest task-weighted average on LIBERO-Plus. Its advantage is not uniform across LIBERO suites.

A.1 Extended LIBERO Results

  • 97.4% average success gives PLaW-VLA the highest listed LIBERO result, although VLA-JEPA scores higher on LIBERO-Goal and LIBERO-Long.Table 5 extends the comparison to additional baselines; its caption notes that some averages are recomputed from displayed suite scores.Source: §A.1 · p. 14, Table 5 · p. 14

A.2 Extended LIBERO-Plus Results

  • PLaW-VLA has the highest task-weighted average among the methods in the full LIBERO-Plus comparison.Table 6 includes OpenVLA and NORA results omitted from the main-text comparison.Source: §A.2 · p. 14, Table 6 · p. 14

A.3 Task-Level RoboTwin Results

Across 50 RoboTwin tasks, PLaW-VLA’s advantage over π0.5 is broad but uneven: all Hard Horizon III tasks improve, while some individual tasks regress.

A.3 Task-Level RoboTwin Results

  • On Hard tasks, PLaW-VLA beats π0.5 on 36 tasks, ties 4, and declines on 10; all 5 Hard Horizon III tasks improve.The task-level results include regressions on Handover Mic and Place Dual Shoes that horizon averages do not reveal.Source: §A.3 · p. 15, Table 7 · p. 15

B.1 Pretraining and Training Dynamics

Pretraining benefits vary across checkpoints and evaluation shifts, while Stage III LIBERO performance plateaus after about 30k updates. Inference-time masking also shows that future-state conditioning affects suites differently.

B.1 Pretraining and Training Dynamics

  • At 50k steps, Stage I–II pretraining retains a 2.83-point LIBERO-Plus advantage overall, but the advantage narrows on LIBERO and reverses on LIBERO-Long.The full model’s LIBERO advantage narrows from 3.35 points at 30k to 1.00 point at 50k; the no-pretraining variant does better under sensor noise.Source: §B.1 · p. 16, Table 8 · p. 16, Table 9 · p. 16

B.2 Training Dynamics and Checkpoint Robustness

  • 97.40% LIBERO success at 30k updates is followed by 96.47%–97.40% through 50k, despite continuing training-loss decreases.The trajectory indicates a downstream performance plateau after approximately 30k updates, with less than one percentage point of variation across later evaluated checkpoints.Source: §B.2 · p. 16, Figure 6 · p. 17

B.3 World-Model Branch Dropout

  • Reducing α from 0.3 to 0.1 preserves 97.4% 30k-step LIBERO success but increases training time by approximately 10%; the selected value is α = 0.3.With m = 0, stochastic branch dropout omits historical and future tokens and Lwm, unlike removing future conditioning.Source: §B.3 · p. 16

B.4 Future Conditioning and Latent Visualization

  • The model predicts K = 2 future latent states from a six-frame clip divided into three two-frame temporal units.Near- and far-future masking target the first and second future units, respectively; all-future masking targets both.Source: §B.4 · p. 16
  • Masking all future states lowers LIBERO-Spatial success from 98.90% to 89.60%, while LIBERO-Goal remains at 97.40% versus 97.10% for the full model.The aggregate reduction therefore does not imply a benefit on every suite; Table 10 reports inference-time masking success rates.Source: §B.4 · p. 17, Table 10 · p. 17
  • The predicted-state figure is qualitative and does not measure prediction accuracy.It provides a visual view of predicted states, not an accuracy evaluation.Source: §B.4 · p. 17

C.1 Task Specifications and Scoring

The real-robot evaluation covers four manipulation tasks with task-specific scoring. Its protocol uses shared spatial configurations and includes two variants evaluated without additional adaptation.

C.1 Task Specifications and Scoring

  • The four real-robot tasks cover towel folding, toy packing, basketball placement, and stationery organization, with binary scoring for towel folding and partial-credit scores for the other tasks.Maximum scores are 3 for Pack Toy, 2 for Play Basketball, and 6 for Organize Stationery.Source: §C.1 · p. 18

C.2 Evaluation Protocol

  • Each setting uses 10 spatial configurations and three trials per configuration, for 180 trials per method across four seen-task settings and two zero-shot variants.All methods use the same configurations; the variants change towel material and color or replace the pen holder with a paper cup, without additional adaptation.Source: §C.2 · p. 18
  • Average task progress normalizes each trial’s score by its task maximum; it equals success rate for Fold Towel and credits partial completion on the other tasks.The task maxima are 1, 3, 2, and 6, respectively.Source: §C.2 · p. 19

D.1 Training Details

Training proceeds from action-free visual-dynamics pretraining to joint prediction and action learning, then target-task adaptation. The pipeline also standardizes robot actions across embodiments and converts LIBERO action targets to deltas from the current state.

D.1 Training Details

  • Stage I learns visual dynamics without action labels; Stage II jointly optimizes future-state prediction and action generation; Stage III adapts the policy to the target task and embodiment.The Stage III ablations hold Stages I–II fixed and change only the final-stage setup.Source: §D.1 · p. 19
  • The loss and conditioning ablations use the same Stage III budget and three-seed evaluation, varying prediction loss or access to predicted states.Removing prediction loss retains the world-model expert and future conditioning; removing future conditioning blocks action attention to predicted states while preserving prediction loss.Source: §D.1 · p. 19

D.2 Data Corpus and Action Alignment

  • Stage I uses visual sequences, including action-free EgoDex videos; Stage II uses action-annotated robot trajectories; Stage III uses task-specific demonstrations.All datasets are resampled to 10 Hz, and robot action annotations are ignored in Stage I.Source: §D.2 · p. 20, Table 12 · p. 20
  • Action-annotated trajectories map to a shared EEF representation: each arm has an 8-dimensional action, and bimanual actions concatenate both arms into 16 dimensions.The representation includes relative translation, normalized orientation quaternion, and gripper command; Stage III may add platform-specific dimensions.Source: §D.2 · p. 20
  • For LIBERO Stage III, each action target in a predicted chunk is converted to a delta from the same current robot state.Translation and gripper targets are offset from the current state, while orientation uses a normalized relative quaternion.Source: §D.2 · p. 20
Loading 2610.12285v1…