PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoSem-WAM: Geometry- and Semantic-Aware World Action Models

Fulong Ma, Daojie Peng, Wenjun Yue, Jiahang Cao, Bintao Wang, Qiang Zhang, Jun Ma

arXiv:2606.03188v1cs.RO

TL;DR

WAMs’ effectiveness is increasingly attributed to predictive training that builds latent representations, but RGB-based future prediction offers limited structural and semantic understanding. GeoSem-WAM adds geometry and semantic prediction to future RGB prediction during training. The authors report consistent improvements in action prediction, robustness, and scene understanding while avoiding explicit future generation at inference.

  • Problem

    WAMs’ primary benefit is understood to lie in predictive-training representation learning rather than test-time future imagination, while RGB-based future prediction provides limited structural and semantic understanding.

    Source: §1 · pp. 1–2

  • Method

    GeoSem-WAM supplements future RGB prediction with geometry and semantic prediction branches during training, while avoiding explicit future sequence prediction at inference.

    Source: §1 · p. 2

  • Results

    Structured supervision consistently improves action prediction accuracy, robustness, and scene understanding across diverse embodied tasks, particularly in challenging scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Training-only structured supervision is reported to enhance spatial motion awareness and task-relevant reasoning without adding test-time computational overhead.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent World Action Models (WAMs) have demonstrated impressive capabilities in embodied decision-making. However, whether their effectiveness stems from explicit future imagination during inference or representation learning induced by predictive training remains an open question. Emerging evidence suggests the primary advantage lies in learning robust latent representations rather than generating future observations at test time. Nevertheless, existing WAMs mainly rely on RGB-based future prediction, which provides limited structural and spatial understanding of complex environments. To address this, we propose a structured world modeling framework that enhances latent representations through geometric and semantic supervision. Alongside future RGB prediction, our model introduces two auxiliary prediction branches for future geometry and semantic representations, enabling it to jointly capture scene dynamics, spatial geometry, and semantic context within a unified latent space. Crucially, our approach preserves efficient inference by avoiding explicit future rollout or video generation at test time. Extensive experiments show that incorporating structured world supervision consistently improves action prediction accuracy, scene understanding, and robustness under challenging embodied scenarios, highlighting its potential for advancing scalable and efficient WAMs.

1 Introduction

The paper argues that WAMs benefit primarily from predictive training that learns dynamics-aware latent representations, while RGB-only prediction provides limited structural and semantic understanding. GeoSem-WAM adds geometry and semantic supervision to future RGB prediction, reporting improved embodied-task performance while avoiding explicit future generation at inference.

1 Introduction

  • The paper frames WAMs’ primary benefit as training-phase dynamics-aware representation learning rather than explicit future imagination or rollout at test time.Predictive supervision is described as a structured self-supervised objective.Source: §1 · pp. 1–2
  • Existing WAMs mainly use RGB future prediction, which offers limited structural understanding and lacks explicit geometric reasoning and high-level semantic awareness.The passage notes that appearance-based supervision may capture short-term visual dynamics without richer environmental structure and object-level semantics.Source: §1 · p. 2
  • GeoSem-WAM augments future RGB prediction with geometry and semantic branches to learn spatial structure, 3D consistency, object-level semantics, and scene context.The framework jointly models appearance, geometry, and semantics in a structured latent world representation.Source: §1 · p. 2
  • Structured supervision consistently enhances embodied decision-making across diverse tasks while maintaining efficient inference by avoiding explicit future generation.The authors report improvements in performance and robustness in simulation and real-world experiments.Source: §1 · p. 2

2 Related Works

The related work situates GeoSem-WAM among VLA policies and WAMs, imitation-learning pipelines, and foundation-model scaling research. It positions structured world modeling as a way to enhance representations without relying solely on larger models or more training data.

2.1 Vision-Language-Action (VLA) Policies and World Action Models (WAMs)

  • WAMs use predictive supervision to learn environmental dynamics, while GeoSem-WAM adds geometric and semantic cues beyond RGB-only designs.The passage contrasts this work with VLA deployment optimization and RGB-only WAM approaches.Source: §2.1 · p. 2
  • Figure 1 distinguishes the overall training-phase architecture from the model-inference stage shown inside the dashed box.The caption identifies the dashed-box region as the inference stage.Source: Figure 1 · p. 3

2.2 Imitation Learning and Robotic Data Learning

  • Structured world modeling is presented as a representation-enhancement module that can be embedded in imitation-learning pipelines using demonstration data.The passage also identifies BridgeData V2 and CALVIN and LIBERO as robotic datasets and evaluation benchmarks, respectively.Source: §2.2 · p. 3

2.3 Scaling Laws and Foundation Model Representation Learning

  • The work follows foundation-model representation research by using customized multimodal world-prediction supervision rather than blindly increasing model scale or training-data volume.The passage connects this direction to scaling-law research and pretrained visual-language models.Source: §2.3 · p. 3

3 Methodology

GeoSem-WAM adds geometry and semantic prediction to future RGB and action training, while using a single forward pass over the first observation’s latent tokens to generate actions at inference. Its architecture combines video and action transformers with dense auxiliary prediction heads; future video generation and the auxiliary heads are not used at deployment.

3.1 Overview

  • At inference, GeoSem-WAM uses the first observation’s latent tokens to generate actions without future video rollout, discarding the DPT branches and requiring no geometry or semantic annotations as inputs.The action chunk is initialized from Gaussian noise and iteratively denoised, conditioned on the latent world representation.Source: §3.1 · pp. 3–4, §3.2 · p. 4

3.2 Architecture

  • The model uses a Wan2.2-5B video DiT backbone, an action-expert DiT, and a Mixture-of-Transformer architecture with shared attention between video and action branches.Task instructions use the T5 encoder, and visual observations are encoded into latent video tokens by the pretrained VAE.Source: §3.2 · p. 4
  • Future video latents are corrupted with Gaussian noise during training, and the video DiT predicts a flow target conditioned on the current observation and language instruction.The video modeling objective is defined over that prediction.Source: §3.2 · p. 4
  • The action branch predicts future action chunks through denoising, using the latent world representation as conditioning and an action objective during training.At inference, the action chunk is iteratively denoised from Gaussian noise without generating future video frames.Source: §3.2 · p. 4
  • A DPT-style head aggregates intermediate video tokens from multiple Transformer blocks, then fuses and decodes multi-resolution representations into dense geometry and semantic predictions.The reassemble and fusion modules are extended to 3D for video inputs.Source: §3.2 · pp. 4–5
  • Training jointly optimizes RGB, geometry, semantic, and action prediction objectives.The geometry branch uses an L1 reconstruction objective, while the semantic branch uses pixel-wise cross-entropy.Source: §3.2 · p. 5

4 Experiments

GeoSem-WAM improves success rates in simulation and real-robot manipulation, with especially strong gains on real-world generalization and challenging tasks. Its geometry and semantic supervision also improves latent semantic and geometric representations, while combined supervision performs best in ablations.

4.1 Experimental Setup

  • LIBERO and RoboTwin 2.0 evaluate success rate across task suites and clean or randomized settings, respectively.LIBERO has four suites with 500 expert demonstrations per suite; RoboTwin randomizes scene clutter, background textures, illumination, and tabletop height.Source: §4.1 · p. 5

4.2 Comparisons with State-of-the-Art Methods

  • GeoSem-WAM tokens show clearer semantic clustering and yield depth predictions closer to ground truth than Fast-WAM tokens.The depth probe uses a frozen backbone and intermediate tokens on LIBERO.Source: Figure 3 · p. 6, §4.2 · p. 6
  • 98.55% average success rate on LIBERO, up from Fast-WAM’s 97.60%.Each task was evaluated for 50 trials across four task suites.Source: §4.2 · p. 5, Table 1 · p. 6
  • 92.52% average success rate on RoboTwin 2.0, improving over Fast-WAM by 0.8% and outperforming LingBot-VA without embodied pre-training.The evaluation covers 50 tasks under clean and random settings.Source: §4.2 · pp. 5–6, Table 2 · p. 7

4.3 Ablation Study

  • 98.6% average success rate with both auxiliary branches, compared with 98.2% for geometry-only and 98.1% for semantics-only supervision.The combined branches yielded a +1.02% improvement over the 97.6% Fast-WAM baseline.Source: §4.3 · p. 6, Table 3 · p. 7

4.4 Real-World Experiments on Franka Emika Panda Robot

  • Real-robot evaluation covers four manipulation tasks, two mat-background tests, and a height test with a 4 cm elevated platform.Each core task used 50 teleoperation trajectories and 50 repeated evaluation trials.Source: §4.4 · p. 7, Figure 4 · p. 8
  • 88.9% to 95.4% overall average success rate (+6.6%) across real-world scenarios, with gains on background, height, multi-object, and long-horizon tasks.Easy-Pick reached 100% with both methods; gains were +10%, +8%, and +12% on Easy-Pick-B1, Easy-Pick-B2, and Easy-Pick-D, respectively.Source: §4.4 · p. 7, Table 4 · p. 8

5 Conclusion

GeoSem-WAM adds geometric and semantic supervision to world-action modeling while keeping these auxiliary predictions training-only. The conclusion also identifies annotation scarcity and potential gradient conflicts as limitations.

5 Conclusion

  • GeoSem-WAM adds auxiliary DPT-style heads to intermediate video-expert tokens during training, leaving the deployment action-inference pipeline unchanged.The added supervision covers geometry and semantics, alongside visual dynamics.Source: §5 · p. 8
  • The authors conclude that structured geometric and semantic prediction is a valuable auxiliary signal for learning richer representations in world-action modeling.They associate it with task-relevant spatial layouts, object-level semantics, spatial motion awareness, and task logical reasoning.Source: §5 · p. 8
  • Pixel-level annotations for the auxiliary DPT head are scarce in real RGB-only datasets and may require potentially noisy pseudo-labels.The authors propose exploring foundation-model self-supervised features to extract spatial and categorical priors from raw RGB videos.Source: §5 · p. 8
  • Jointly optimizing heterogeneous losses increases training complexity and risks gradient conflicts.The authors propose gradient deconfliction methods, such as gradient surgery, to mitigate task interference.Source: §5 · p. 8

A Simulation Environments and Multi-Modal Observations

The simulation evaluation covers LIBERO and RoboTwin tasks with paired visual, depth, and semantic observations. The section also reports GeoSem-WAM's overall success-rate comparison on RoboTwin.

Libero Benchmark

  • LIBERO comprises four task suites of increasing complexity, while RoboTwin evaluates bimanual manipulation in clean and randomized environments.The LIBERO suites are Goal, Object, Spatial, and Libero-10; randomized RoboTwin scenes include clutter and distractors.Source: A Simulation Environments and… · pp. 12–13

A Simulation Environments and Multi-Modal Observations

RoboTwin Benchmark

  • GeoSem-WAM achieves the best overall success rate against previous state-of-the-art methods on RoboTwin 2.0.Table 5 covers 50 bimanual manipulation tasks across two difficulty levels.Source: Table 5 · p. 12, A Simulation Environments and… · p. 13
  • RoboTwin is a bimanual manipulation platform with clean and randomized evaluation settings.Its benchmark description includes 50 tasks, five robot embodiments, and structured randomization across clutter, lighting, background texture, tabletop height, and language instructions.Source: A Simulation Environments and… · p. 13

B Example Episodes of Real World Experiments on Franka

The Franka Pick-Pour example presents synchronized multi-modal observations across the manipulation sequence from both third-person and ego-centric views.

B Example Episodes of Real World Experiments on Franka

Loading 2606.03188v1…