PaperlayerView exact revision on arXiv

Source-linked AI summary

GAWM: A Compact Latent World-Action Model for Multi-Task Robot Manipulation

Yang Liu, Xiaoli Liu, Liuzhao Li, Yuduo Song, Muyuan Liu, Xiang Gao

arXiv:2610.37091v1cs.RO

TL;DR

Many world-action models depend on large backbones that are costly to train and slow to run. GAWM uses a compact task-conditioned latent predictor to forecast future scenes for reactive action decoding. It outperforms billion-parameter WAMs on RoboTwin 2.0 and performs comparably to billion-parameter VLAs on LIBERO, with lower action-prediction error than a fine-tuned billion-parameter VLA on a dual-arm robot.

  • Problem

    Many WAMs use billion-parameter vision-language or video-generation backbones, incurring high training cost and inference latency.

    Source: §I · p. 1

  • Method

    GAWM predicts two task-conditioned horizons of frozen-encoder visual tokens and decodes action chunks from current and forecast tokens.

    Source: §I · pp. 1–2

  • Results

    GAWM-S outperforms billion-parameter WAMs on RoboTwin 2.0 and performs comparably to billion-parameter VLAs on LIBERO; on a dual-arm robot, it has lower action-prediction error than a fine-tuned billion-parameter VLA.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    A compact task-conditioned latent predictor can provide a reactive policy with foresight, and the world model improves performance when training data or time is limited.

    Source: §V · pp. 7–8

    GAWM targets a closed set of tasks identified by demonstration-derived codes and does not interpret language instructions.

    Source: §V · p. 8

Abstract

from arXiv · show

World-action models (WAMs) incorporate the prediction of future observations into robot policy learning. However, most WAMs build on billion-parameter vision-language or video-generation backbones, which incur high training cost and inference latency. In this letter, we present GAWM (Geometry-Aware World Model), a compact multi-task WAM that predicts future observations in the latent space of a frozen DINOv3 encoder. GAWM compresses the features of each camera view into a small set of tokens. A lightweight predictor, conditioned on the task rather than on actions, forecasts these tokens at two future horizons. The forecast is supervised with the dense patch features of future frames, which requires it to encode the spatial layout of the future scene. An action decoder directly regresses a chunk of future actions from the current and predicted tokens. With fewer than 100M parameters and no robot or video pretraining, GAWM outperforms billion-parameter WAMs on RoboTwin 2.0 and performs comparably to billion-parameter VLAs on LIBERO. On a dual-arm robot, it achieves lower action-prediction error on held-out demonstrations than a fine-tuned billion-parameter VLA, with substantially lower inference latency. Ablation studies show that the latent world model substantially improves performance when training data or training time is limited. Code is available at https://github.com/gaoxiang12/ga-wm.

I. INTRODUCTION

Existing world-action models often rely on large backbones and costly future-frame generation, while compact reactive policies could better suit fixed-task deployment. GAWM predicts task-conditioned future latents for action decoding, and achieves strong benchmark results with a compact model.

I. INTRODUCTION

  • Existing WAMs often use billion-parameter video generators or VLA backbones, making training costly and inference difficult for onboard computers.Video-frame generation can spend capacity on appearance details irrelevant to control; LingBot-VA takes 4.5 s per query.Source: §I · p. 1
  • For fixed task sets, compact policies support high-rate onboard execution, motivating foresight without a large backbone or open-ended language understanding.The proposed hierarchy assigns instruction interpretation to a language model and task execution to a compact policy.Source: §I · p. 1
  • Action-conditioned JEPA predictors support planning over candidate sequences toward a goal image, whereas reactive control needs task-conditioned forecasts without search.Evaluating many candidates limits control rate and requires a goal image for each task.Source: §I · p. 1
  • GAWM encodes camera views into frozen-DINOv3 tokens, predicts two task-conditioned future horizons, and decodes action chunks from current and forecast tokens.Dense future patch-feature supervision constrains the forecast to encode the positions of the gripper and objects.Source: §I · p. 1
  • With 77M parameters and no robot or video pretraining, GAWM-S reaches 93.9% success on RoboTwin 2.0 clean and 93.1% randomized, outperforming billion-parameter WAMs.Fig. 1 compares success rate and model size; GAWM-S is the smallest and best-performing plotted RoboTwin method.Source: §I · p. 2, Figure 1 · p. 2
  • Ablations show that the world model yields large gains when training data or training time is limited, and its forecast encodes future scene layout.The action decoder attends to the forecast in temporal order.Source: §I · p. 2

II. RELATED WORK

Related work spans instruction-conditioned VLAs, pixel- and latent-prediction WAMs, and JEPA world models used for planning. GAWM differs by bringing task-conditioned future prediction into a compact reactive policy.

II. RELATED WORK

  • VLAs map observations and instructions to actions, but on closed task sets the instruction contributes little to success and can be replaced by a task identifier with only a one-point TurboVLA decrease [29].GAWM adopts LiLa-WAM’s demonstration-derived task code instead of an instruction.Source: §II · p. 2
  • Pixel-space WAMs predict future frames, while latent WAMs predict representations; LaWAM uses predicted DINOv3 features in a 2.1B-parameter policy.LiLa-WAM decodes foresight tokens from its action transformer and supervises them with frozen DINOv3 patch features.Source: §II · pp. 2–3
  • WAMs vary in whether predicted futures affect test-time actions: Fast-WAM and LiLa-WAM use prediction only in training, while Faster-WAM reports its importance under distribution shift [13].GAWM conditions its action decoder on the forecast at every step, with a 4–6M-parameter task-conditioned predictor.Source: §II · p. 3
  • Unlike action-conditioned JEPA planners, GAWM predicts from the task and acts without candidate search, using dense patch features and one shared predictor across tasks.Its frozen-encoder targets cannot collapse, so a regularizer such as SIGReg is unnecessary.Source: §II · p. 3

III. METHOD

GAWM is a reactive policy for a closed set of manipulation tasks: it forecasts future latent tokens from the task and uses them with current visual tokens to decode an action chunk.

III. METHOD

  • At each query, GAWM forecasts scene tokens given the task and decodes an action chunk from current and forecast tokens.This summarizes the policy’s factorization for reactive control.Source: §III · p. 3

A. Problem Formulation

  • The policy observes multi-camera RGB images, robot state, and task identity, then outputs a chunk of H actions.The formulation concerns a closed set of manipulation tasks.Source: A. Problem Formulation · p. 3
  • The predictor uses current visual tokens and task code to forecast two future representations; the decoder regresses actions from those forecasts, current tokens, robot state, and task code.Unlike action-conditioned predictors, this formulation requires no search over candidate actions.Source: A. Problem Formulation · p. 3

B. Visual Encoding

GAWM encodes each camera view into tokens and uses a task-conditioned predictor to forecast scene tokens at two future horizons. The forecasts are residual updates of the current tokens, with the far and near horizons aligned to the action chunk and its midpoint.

B. Visual Encoding

  • Each view’s fused features from three DINOv3 layers are compressed into 64 tokens using learned queries, then stacked across views with camera embeddings.For two cameras, the stacked representation contains 128 tokens.Source: B. Visual Encoding · p. 3

C. Task-Conditioned Latent Predictor

  • A task-conditioned predictor forecasts both horizons in parallel, using current tokens, horizon queries, temporal and positional embeddings, and projected task codes.Bidirectional transformer blocks apply self-attention across views and temporal positions.Source: C. Task-Conditioned Latent Predictor · p. 4
  • The forecasts are residual updates of the current tokens; the far horizon equals the action-chunk length, while the near horizon is its midpoint.The output projection is initialized to zero, so the predictor initially forecasts a static scene and learns changes as residuals.Source: C. Task-Conditioned Latent Predictor · p. 4

D. Forecast Supervision

GAWM trains its forecasts against dense patch features from recorded future frames. This target constrains spatial layout without depth or 3D supervision.

D. Forecast Supervision

  • A training-only patch decoder maps each view’s 64 forecast tokens to patch features, which are compared with features from the recorded future frame.The forecast target uses the frozen encoder’s patch-feature space.Source: D. Forecast Supervision · p. 4
  • Forecast and current-feature supervision are combined with VICReg-style penalties that discourage token similarity and low variance within each frame.Future frames beyond an episode’s end are masked; the expectation spans horizons, views, and patches.Source: D. Forecast Supervision · p. 4
  • Location-specific patch features constrain where objects and the gripper will be, so the forecast captures each view’s spatial layout without depth or 3D supervision.The target comes from a fixed frozen encoder and cannot collapse during training.Source: D. Forecast Supervision · p. 4

E. Action Decoder and Training Objective

GAWM’s action decoder regresses a full action chunk from current and forecast tokens, robot state, and task information. Joint training shapes both forecasts and control, while inference drops the patch decoder and avoids iterative sampling or search.

E. Action Decoder and Training Objective

  • H learned action queries attend to memory containing current tokens, both forecasts, two state tokens, and one task token; an MLP regresses the action chunk.The decoder follows an ACT-style design.Source: E. Action Decoder and Training Objective · p. 4
  • The decoder uses deterministic ℓ1 regression on normalized actions, avoiding iterative inference sampling and performing on par with flow matching for a compact LIBERO policy.Source: E. Action Decoder and Training Objective · p. 4
  • All modules except the encoder are trained jointly; action loss also backpropagates through the forecast into the predictor and adapter, and inference discards the patch decoder without search.The temporal loss applies only on RoboTwin 2.0.Source: E. Action Decoder and Training Objective · p. 4

IV. EXPERIMENTS

The experiments compare GAWM with published VLAs, WAMs, and compact policies across LIBERO and RoboTwin 2.0, and include ablations and a real-robot evaluation. The setup specifies the benchmark protocols, training conditions, and reporting conventions.

IV. EXPERIMENTS

  • The experimental plan compares success rate, model size, and computational cost, then examines forecast encoding and decoder attention before evaluating GAWM on a real robot.Source: §IV · p. 4

A. Experimental Setup

  • LIBERO uses one policy across 40 tasks, while RoboTwin 2.0 uses one policy across 50 bimanual tasks evaluated in clean and randomized scenes.LIBERO Standard uses 1,693 demonstrations; Extended data totals 6,585 trajectories, and RoboTwin training uses 50 clean plus 500 randomized demonstrations per task.Source: A. Experimental Setup · p. 4
  • Table I reports success rates on LIBERO and RoboTwin 2.0; its parameter counts include action-dependent modules and frozen encoders, but exclude training-only modules.The table notes robot-data or embodied-video pretraining, task-input type, evaluation protocols, and exceptions to reported parameter counts.Source: Table I · p. 5, A. Experimental Setup · p. 5
  • Training uses AdamW at a 2×10^-4 learning rate, with 80k steps for LIBERO and 765k steps for RoboTwin; RoboTwin also has a 9.6k-step fine-tuning stage.Success rates use Wilson 95% confidence intervals, while model differences use paired statistics.Source: A. Experimental Setup · p. 5
  • The world-model ablation evaluates GAWM-S against variants with the world model removed under reduced-data LIBERO and shortened-training RoboTwin conditions.The LIBERO conditions use a quarter of Standard demonstrations or Extended data; RoboTwin training uses 191k rather than 765k steps.Source: A. Experimental Setup · p. 5, Table II · p. 5

B. Main Results

GAWM-S delivers strong benchmark results with a compact model: it outperforms billion-parameter WAMs on RoboTwin 2.0 and scores above π0 on LIBERO, while maintaining low inference latency.

RoboTwin 2.0

  • 93.94% clean and 93.10% randomized success with 77M parameters put GAWM-S above the four strongest billion-parameter WAMs on RoboTwin 2.0.The baselines score 91.9–93.0% clean and 88.8–91.8% randomized; their results fall below GAWM-S's 95% intervals.Source: B. Main Results · pp. 5–6

Inference latency

  • 8.8 ms on RoboTwin 2.0 and 7.1 ms on LIBERO make GAWM-S the lowest-latency compared method for one action chunk.Other compact policies take 17.6–143.5 ms, VLAs 41.2–68.2 ms, and WAMs 114–3,763 ms.Source: Figure 3 · p. 6, B. Main Results · p. 6

LIBERO

  • 95.6% on LIBERO with 38M parameters exceeds π0 (3.2B, 94.2%) and Evo-1 (0.78B, 94.8%), but trails X-VLA by 2.5 points.Source: B. Main Results · p. 6

Model size and cost

  • GAWM-S is the smallest model in the comparison, using 77M parameters on RoboTwin 2.0 and 38M on LIBERO without a VLM or video generator.Training requires 45 GPU-hours on LIBERO and 1,749 on RoboTwin 2.0, versus 2,903 for ViT-L.Source: B. Main Results · p. 6
  • A larger ViT-L encoder does not improve success over ViT-S on RoboTwin 2.0 or Extended LIBERO.On RoboTwin, ViT-L scores 93.78% clean and 93.42% randomized, versus 94.36% and 93.48% for the 765k-step GAWM-S model; the Extended LIBERO difference is −0.5 points [−3.5, 2.5].Source: B. Main Results · p. 6

C. Ablation and Analysis

Ablations show that the latent world model matters most under constrained training, while forecast and attention analyses indicate that it captures future scene structure used across action steps.

World-model ablation

  • Removing the world model drops RoboTwin 2.0 success from 90.7% to 18.8% under the 191k-step schedule, and the full model wins on all 50 tasks.With a quarter of LIBERO demonstrations, removal lowers success by 9.5 points [5.2, 14.2]; the gain is 2.5 points [−0.8, 6.1] with Extended LIBERO data.Source: C. Ablation and Analysis · p. 6
  • On quarter-data LIBERO, removing the world model lowers success by 21.0 points on LIBERO-Long but yields a 3.6-point gain on LIBERO-Object.The authors note the Long tasks comprise several subtasks, while Object tasks differ mainly in the grasped object.Source: C. Ablation and Analysis · p. 6

Forecast quality

  • 0.970 and 0.967 mean per-patch cosine similarity at +16 and +32 frames exceed the current frame's 0.929 and 0.892, respectively.The forecast is closer to true future features on 75% and 91% of anchors at those horizons.Source: C. Ablation and Analysis · p. 6

Attention

  • Action-decoder attention shifts from current tokens to near forecasts around h1 and then far forecasts, whose share rises from 20% to 44%.This progression holds for every anchor; adapter attention peaks on gripper fingers and manipulated objects.Source: C. Ablation and Analysis · p. 6

D. Real-Robot Evaluation

On held-out dual-arm demonstrations, GAWM-S predicts actions more accurately than fine-tuned π0.5 models and succeeds more often in the reported closed-loop trials, with lower latency.

Platform and data

  • The four tasks share an otherwise identical bottle scene, so the task code is the only cue to which bottle the robot must transfer.The dual-arm setup uses 340 teleoperated demonstrations for training and checkpoint selection, with 81 separate-day demonstrations as test data.Source: D. Real-Robot Evaluation · p. 6, Figure 6 · p. 7

Forecast visualization

  • Fig. 4 compares frames at t, t+16, and t+32 with PCA projections of true future and decoded-forecast patch features, labeled by mean per-patch cosine similarity.Source: Figure 4 · p. 7

Attention visualization

  • Fig. 5's top panel maps adapter attention over image patches, while its bottom panel tracks decoder attention shares across current, near-forecast, far-forecast, and state tokens by action step.Source: Figure 5 · p. 7

Results

  • 46 of 50 yellow-bottle closed-loop trials succeed for GAWM-S (92%, [81.2, 96.8]), versus 39 (78%, [64.8, 87.2]) for π0.5 (60k).GAWM-S uses 77M rather than 3.4B parameters and takes 9.3 ms per query versus 63.8 ms for π0.5.Source: D. Real-Robot Evaluation · p. 7
  • 0.072 rad (4.1°) first-action joint error is 26% and 22% lower than the two π0.5 models, and GAWM-S has the lowest gripper error.Across the full chunk, its 0.088 rad error over 16 steps is below π0.5's 0.101–0.106 rad over 20 steps; the reported table summarizes the evaluation.Source: Table III · p. 7, D. Real-Robot Evaluation · p. 7

V. CONCLUSION

GAWM uses task-conditioned latent forecasts to provide reactive manipulation policies with future-scene information. Its reported scope is closed task sets, and it does not interpret language instructions.

V. CONCLUSION

  • GAWM forecasts frozen-encoder latent tokens at two future horizons and regresses action chunks from current and forecast tokens.The paper reports that it needs neither a vision-language model nor a video generator, and performs comparably to billion-parameter VLAs on LIBERO.Source: §V · p. 7
  • The forecast encodes future scene layout and substantially improves success when training data or training time is limited.Source: §V · p. 8
  • GAWM targets a closed task set identified by demonstration-derived codes and does not interpret language instructions.The authors propose adding a large language model to map open-ended instructions to known tasks.Source: §V · p. 8

Related papers

Loading 2610.37091v1…