PaperlayerView exact revision on arXiv

Source-linked AI summary

World Guidance: World Modeling in Condition Space for Action Generation

Yue Su, Sijin Chen, Haixin Shi, Mingyu Liu, Zhengshen Zhang, Ningyuan Huang, Weiheng Zhong, Zhengbang Zhu, Yuxiao Liu, Xihui Liu

arXiv:2602.22010v1cs.ROcs.CV

TL;DR

Existing future representations struggle to balance prediction efficiency with the fine-grained information needed for precise action generation. WoG compresses future observations into action conditions, then trains a VLA to predict those conditions alongside actions; experiments report improved performance, generalization, and learning from human manipulation data. The paper also identifies persistent challenges with fine-grained spatial constraints and foundation-model capability gaps.

  • Problem

    Existing approaches trade off redundant rich future representations against compact latent actions that lack the precision required for fine-grained control.

  • Method

    WoG injects future observations into action inference to learn compact conditions, then trains the VLA to predict those conditions jointly with actions.

  • Results

    WoG substantially improves performance over existing future-prediction methods across simulation and real-world experiments, with further gains from large-scale human manipulation data.

  • Takeaways & Limitations

    Condition-space world modeling balances efficient future forecasting with rich manipulation knowledge and generalizes to large-scale human manipulation data.

  • Takeaways & Limitations

    Fine-grained spatial constraints remain a persistent challenge, and WoG cannot supply semantic information missing from foundation models' intrinsic capabilities.

Abstract

from arXiv · show

Leveraging future observation modeling to facilitate action generation presents a promising avenue for enhancing the capabilities of Vision-Language-Action (VLA) models. However, existing approaches struggle to strike a balance between maintaining efficient, predictable future representations and preserving sufficient fine-grained information to guide precise action generation. To address this limitation, we propose WoG (World Guidance), a framework that maps future observations into compact conditions by injecting them into the action inference pipeline. The VLA is then trained to simultaneously predict these compressed conditions alongside future actions, thereby achieving effective world modeling within the condition space for action inference. We demonstrate that modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities. Moreover, it learns effectively from substantial human manipulation videos. Extensive experiments across both simulation and real-world environments validate that our method significantly outperforms existing methods based on future prediction. Project page is available at: https://selen-suyue.github.io/WoGNet/

1 Introduction

Existing future-modeling approaches either retain redundant task-agnostic representations or compress information too aggressively for precise control. WoG addresses this trade-off by learning compact future conditions within the action-generation pipeline and predicts them alongside actions, with validation in simulation and real-world settings.

  • Motivation: World Action Models predict rich future modalities or semantic features, but their task-agnostic spaces contain substantial redundancy for manipulation.These representations provide perceptual cues about dynamics, motion, and spatial geometry, while limiting pretraining efficiency and cross-scenario scalability.
  • Motivation: Latent Action Models offer compact, embodiment-agnostic representations for high-level planning but provide only coarse guidance for fine-grained action generation.Their representations are learnable from large-scale video data, yet lack the precision needed for fine-grained control.
  • Motivation: The central challenge is finding a predictive space that is tractable for VLA forecasting while sufficiently expressive for accurate action generation.WoG frames this as a trade-off between redundant rich representations and compact spaces that capture only coarse motion trends.
  • WoG: WoG maps future observations into an action-relevant condition space by incorporating them as conditions in the action inference pipeline.The resulting representation is intended to be non-redundant and sufficient for action generation because it is learned through the action-prediction process.
  • WoG: WoG uses a two-stage curriculum: future observations guide action prediction first, then the VLA predicts compressed future conditions alongside actions.The first stage learns the condition space and uses it for precise action prediction; the second transfers future-condition knowledge into the VLA.
  • Results: WoG shows substantial improvements over existing methods in simulation and real-world experiments, and gains further performance from large-scale human manipulation videos.The human-video setting includes both action-annotated and unannotated data, as well as UMI data.

2 Related Work

Related work models future information through explicit world representations or compact latent actions to improve VLA action inference. These approaches respectively face redundancy or insufficient precision for fine-grained manipulation.

  • World Action Models: Future-modeling methods integrate video-model features, future images, depth, or optical flow into action modules to capture manipulation dynamics.Some approaches use intermediate video-generation features, while others model future images directly through VLM backbones.
  • Latent Action Models: Latent Action Models assume actions have compact, skill-related, embodiment-agnostic representations that capture coarse motion trends for high-level planning.They were developed partly to train VLA models on large-scale heterogeneous datasets.
  • Latent Action Models: Mainstream LAMs compress heterogeneous actions using visual reconstruction objectives, then decode predicted discretized latent actions into downstream fine-grained actions.The reconstruction representations may be generative or discriminative, but both target compact latent action spaces.
  • Latent Action Models: Prior studies report that LAM compression can resemble PCA-like maximum-variance extraction, producing coarse planning representations and sometimes action-weakly-correlated noise.Recent methods add action reconstruction to generative objectives to strengthen the mapping between latent actions and control.

3 Method

WoG first learns a compact future condition space by conditioning action prediction on compressed future observations, then trains the VLA to infer those conditions from current observations alone. Its two-stage design also supports learning from robot data, human videos, and UMI data.

  • Problem Formulation: The task is to predict future actions from the current observation and language instruction, with the VLM encoding them into a latent representation for the action head.The action head generates actions by maximizing the likelihood of the future action sequence conditioned on the encoded current input.
  • Stage I: World Guidance: In Stage I, future observations are compressed into condition representations and supplied with the current latent representation to guide action generation.The VLA thereby learns both to use complementary future-derived conditions and to encode a compact representation of future observations.
  • Stage II: World Inference: Stage II is required because future-derived conditions are unavailable at test time and must be inferred from the current observation under deterministic environmental dynamics.The training objectives supervise future-condition prediction and marginal action prediction while transferring condition information into the VLM backbone.
  • Stage I: World Guidance: Stage I forms conditions with future observations encoded by frozen vision models and compressed by a trainable Q-former-based Future Encoder.The conditions are combined with VLM-encoded current observations and instructions for action prediction.
  • Stage II: World Inference: In Stage II, the future encoder is frozen, and the VLA jointly predicts future conditions and actions so future-condition knowledge is transferred into its current-observation representation.Only the VLM output is subsequently passed to the DiT action head, making test-time inference rely on the current input latent.
  • Learning from Human Data: WoG extends to human manipulation videos through annotated and unannotated-data strategies, with condition supervision enabling additional learning beyond robot action labels.The paper validates both strategies and also evaluates UMI data for condition modeling under egocentric observations and unseen embodiments.

4 Simulation Experiments

Simulation experiments evaluate WoG across two robot configurations and compare it with conventional VLAs, latent-action, world-action, and hybrid video-prediction methods. WoG generally improves trajectory planning and pose prediction, while spatially constrained tasks remain challenging.

  • Setup: Evaluation uses closed-loop SIMPLER with single-RGB observations on Google Robot and WidowX configurations.The benchmark includes predominantly pick-and-place tasks requiring trajectory planning, obstacle avoidance, and precise end-effector pose prediction.
  • Baselines: WoG is compared with conventional VLAs, latent action models, world action models, and methods combining latent actions with future video generation.The baselines include π0, π0-FAST, OpenVLA, GR00T-N1, Moto, UniVLA, DeFI, VITA, and ViPRA.
  • Main Results: WoG outperforms existing methods across most Google Robot and WidowX tasks, with notable gains in trajectory planning and collision avoidance.The strongest improvements occur in scenarios involving interfering objects and broad pick-and-place behavior.
  • Main Results: WoG substantially improves future grasp and placement pose prediction on tasks such as Pick Coke and Put Spoon.Its compact condition space selectively extracts critical future semantics instead of reconstructing all visual information, reducing visual prediction error propagation into action prediction.
  • Pretrained Encoder Configuration: Among encoder configurations, dino-vae reaches the highest Google Robot overall success rate of 70.9%, while dino-siglip achieves 33.0% versus dino-vae’s 29.2% on Stack Green on Yellow.The results associate VAE features with trajectory planning and SigLIP features with fine-grained spatial reasoning.
  • Future Encoder Ablation: Future Encoder ablations show that WoG consistently outperforms variants without this component across most tasks, especially grasping and picking phases.However, WoG provides no distinct advantage on spatially sensitive placement tasks such as Stack, where foundation-model deficiencies remain uncompensated.

5 Real-World Experiments

WoG is evaluated on rigid, articulated, and deformable manipulation tasks under both in-distribution and shifted real-world conditions. It outperforms comparison methods, preserves generalization across OOD scenarios, and benefits from co-training and human manipulation data.

  • Setup: The evaluation covers rigid-body Pick and Place, articulated-object Close the Microwave, and deformable Fold the Towel tasks.Pick and Place requires collision-aware placement, Close the Microwave tests rotational dynamics, and Fold the Towel succeeds when corner distance is below 5 cm.
  • Setup: Real-world experiments compare WoG with UniVLA, VPP, training-stage ablations, and models using different data sources.UniVLA represents latent-action modeling, while VPP uses future video prediction; each method receives 20 trials per task under matched randomized initial scenes.
  • Fine-grained action prediction: WoG consistently outperforms VPP and provides finer action guidance than high-level latent planning, especially for deformable manipulation and articulated dynamics.The condition space distills manipulation-relevant dynamics such as cloth deformation, and WoG achieves near-perfect performance on Close the Microwave.
  • Generalization ability in OOD scenes: WoG maintains superior performance with minimal degradation across background and novel-object OOD scenarios, unlike baselines that degrade under visual distribution shifts.Its conditions are constructed from informative features of frozen pretrained visual encoders, retaining manipulation-relevant distinctions while being invariant to visual nuisances.
  • Efficient training strategy: WoG improves over vanilla VLA across all ID tasks, while the no-co-training variant remains comparable to vanilla VLA and falls behind full WoG.These results identify co-training as necessary for aligning future conditions with the VLM backbone without significantly degrading baseline action generation.
  • Learning from human data: 220h of action-annotated human data enables WoG to outperform its robot-only counterpart across ID and OOD settings, while unannotated human videos can also improve selected outcomes.Human-video benefits depend on task similarity: gains appear on Pick and Place, whereas performance degrades on deformable-object manipulation when human and robotic behaviors are less aligned.

6 Conclusion

WoG compresses future observations into low-dimensional conditions and jointly predicts them within a VLA to support action generation. The approach balances efficient future forecasting with rich manipulation knowledge, with a 42% improvement on P&P and 33% on Fold after UMI training.

  • WoG compresses future observations into a low-dimensional condition space for action generation.
  • Jointly predicting compact future conditions within a VLA balances efficient forecasting with acquiring rich manipulation knowledge.
  • 42% improvement in P&P and 33% in Fold followed UMI-data training compared with training solely on robot data.

A Detailed Architecture

WoG uses query mechanisms to extract action-relevant conditions from future visual features and to predict conditions from VLM hidden states. The same Q-Former-based encoder operates across both training stages, changing from trainable to frozen in the second stage.

  • The Q-Former-based encoder remains consistent across stages, transitioning from trainable in stage one to frozen in stage two.
  • In the second stage, the VLM processes the current observation to produce last hidden states whose trailing tokens distill visual and linguistic context.
  • The future encoder uses learnable queries to extract low-dimensional, action-relevant conditions from pretrained vision-model features.
  • During second-stage condition prediction, learnable query embeddings cross-attend to the VLM’s last hidden states.

B.1 Training Settings

Simulation training uses OXE pretraining followed by second-stage training on Bridge and Fractal data, with no additional fine-tuning on either individual dataset.

  • The first stage pretrains on Open X-Embodiment for 100k steps with a global batch size of 1024.
  • The second stage trains on Bridge and Fractal for 50k steps using the same batch size, with sampling ratios of about 51% and 49%.
  • No additional fine-tuning is needed on individual Bridge or Fractal datasets.

B.2 Evaluation Settings

SIMPLER evaluates simulation performance as a real-to-sim proxy for Fractal and Bridge, including protocols with different degrees of domain shift for Google Robot tasks.

  • SIMPLER functions as a real-to-sim evaluation proxy for the Fractal and Bridge datasets.
  • Its Google Robot evaluation protocols differ in domain shift: Visual Matching minimizes visual discrepancies, while Variant introduces a distinct shift.

C Real-World Experiments

The real-world experiments pretrain and fine-tune WoG and comparison models under matched evaluation settings. WoG uses a two-stage setup, while the ablation removes future-condition co-training from its second stage.

  • Training setup: WoG is pretrained on OXE for 100k steps in stage one and 50k steps in stage two, using a global batch size of 1024.Real-world fine-tuning uses 30 epochs with a global batch size of 512.
  • Evaluation setup: The real-world policy predicts 16 future actions from the current RGB observation and executes the first 8 steps.Evaluation uses the Maniunicon control strategy and D435 camera observations.
  • Baselines: Vanilla VLA receives supervision only for action prediction and matches WoG’s total pretraining duration with 150k OXE steps.It is then fine-tuned on expert demonstrations for 30 epochs.
  • Ablation: The WoG w/o cotrain variant retains the first-stage future-observation-guided setup but excludes auxiliary future-condition supervision during its second stage.Its second stage still lasts 50k steps, followed by 30-epoch fine-tuning on expert demonstrations.

C.4 Human Data Learning Settings

The human-data experiments use a large manipulation-video corpus, with action annotations available for only a subset. Variants differ in whether human videos provide action supervision, condition supervision, or both, while UMI data is reserved for final fine-tuning.

  • Data presentation: Figure 6 presents samples from the human manipulation videos, while Table 11 reports the corresponding human-data sampling ratios.The training-strategy description points to section 3.4 for implementation details.
  • Dataset: The human manipulation dataset contains 650k trajectories totaling approximately 1,920 hours, including a 220-hour action-annotated subset.The dataset is collected with PICO 4 Ultra Enterprise.
  • Action representation: Human arm actions are represented using wrist pose and gripper-state changes, formulated as state deltas between consecutive frames.The wrist pose is expressed relative to the PICO coordinate system, and deltas use the preceding frame’s reference frame.
  • Human-video supervision: In the w. human v. variant, all 1,920 hours of unannotated human videos provide condition supervision, while robot data provides both condition and action supervision.This variant uses 50k second-stage training steps with OXE and human videos.
  • Human-video supervision: In the w. human v./a. variant, action supervision covers OXE and the 220-hour annotated subset, while future-condition supervision covers all data samples.The full 1,920-hour human-video corpus is introduced in the second stage.
  • UMI data: UMI data contains only egocentric observations and defines actions as motions relative to the headset coordinate system.It is used solely during final fine-tuning alongside expert demonstrations, with a sampling weight matching the other expert-demonstration tasks.
Loading 2602.22010v1…