PaperlayerView exact revision on arXiv

Source-linked AI summary

World Dreamer v5-lite: A Multi-Robot World Action Model with One Dreamer

Guoning Wu, Zijian Cai, Yuhang Zhang, Ying Liu, Yongbin Zheng

arXiv:2610.37518v1cs.RO

TL;DR

Multi-robot control must integrate distributed camera views while predicting separate action sequences, but most VLA and world-action models target one robot and concatenated views do not establish workspace correspondence. World Dreamer v5-lite builds a calibrated shared world state that robot-specific queries read and exchange pose-aware information through, with masked-view cooperative training. It exceeds evaluated baselines on RoboFactory and four real-robot tasks and achieves 58.08% overall on RoboCasa365.

  • Problem

    Multi-robot control requires integrating distributed, partial camera views while coordinating separate action sequences, but most VLA and world-action models target one robot and concatenated views alone do not align workspace locations.

    Source: §1 · p. 1

  • Method

    World Dreamer v5-lite builds a shared workspace state from calibrated video features, uses pose-aware queries for each robot, and withholds selected views during cooperative training while retaining all action targets.

    Source: §1 · p. 1

  • Results

    World Dreamer v5-lite beats the strongest evaluated baselines in mean success by 23.7 points on RoboFactory and 13.8 points across four real-robot tasks, and reaches 58.08% overall on RoboCasa365.

    Source: §1 · p. 2, §4.4 · p. 7

  • Takeaways & Limitations

    The framework combines shared visual context for multi-robot coordination with separate action streams for each robot.

    Source: §5 · p. 8

Abstract

from arXiv · show

Multi-robot control requires integrating distributed camera views while predicting separate action sequences. Most vision-language-action and world-action models target one robot. Combining multiple views alone does not establish spatial correspondence between them. World Dreamer v5-lite is a multi-robot world-action model built around a shared world state for robot-specific action prediction. Calibrated features from a pretrained video encoder populate this state in workspace coordinates; its spatial size remains fixed as views are added. A vision-language model provides scene context and one query per robot. The queries read the shared state and exchange pose-aware information, while their action streams remain separate. Masked-view cooperative training withholds selected camera views from both visual branches while retaining all action targets. This encourages the use of observations shared by other robots. Pretraining uses more than 10,000 hours of trajectories from 20 simulated robot types, RoboFactory, and real robots. On RoboFactory, World Dreamer v5-lite achieves 74.0% mean success, 23.7 percentage points above the strongest evaluated baseline. Across four real-robot tasks, it reaches 78.8% mean success, a margin of 13.8 points. It also reaches 58.08% overall on RoboCasa365.

1. Introduction

Multi-robot manipulation distributes partial camera observations across robots and couples their decisions despite separate controls. World Dreamer v5-lite addresses the resulting coordination and view-alignment challenges with a shared workspace state and robot-specific queries.

1. Introduction

  • Most VLA and world-action models predict actions for one robot, while concatenating camera tokens alone does not align views in workspace coordinates.Source: §1 · p. 1
  • Multi-robot tasks distribute partial camera views, couple decisions across robots with different embodiments, and can require acting on observations available only to teammates.Cameras move with robots and may be occluded by another robot or the manipulated object.Source: §1 · p. 1
  • World Dreamer v5-lite builds a shared workspace state from calibrated video features; robot queries read it and exchange pose-aware information while action streams remain separate.Selected views are withheld from both visual branches during training, while every robot retains its action target.Source: §1 · p. 1
  • Evaluation spans RoboFactory, RoboCasa365, and four real-robot tasks with homogeneous LeKiwi and heterogeneous LeKiwi–OpenArm teams.Source: §1 · pp. 1–2
  • World Dreamer v5-lite exceeds the strongest evaluated baselines in mean success by 23.7 points on RoboFactory and 13.8 points across four real-robot tasks, and is competitive on RoboCasa365.Source: §1 · p. 2

2. Related Work

Prior work includes pretrained vision-language-action policies, world-action models that couple actions with visual dynamics, multi-robot policies, and explicit 3D visuomotor representations. World Dreamer v5-lite combines pretrained visual features with calibrated multi-view state and pose-aware robot queries.

Vision–language–action models

  • VLA policies process one robot’s observations as image tokens; World Dreamer v5-lite also aggregates all robots’ views in a calibrated shared state while retaining VLM semantic context [11] [16].Source: §2 · p. 2

World-action models

  • World-action models couple action prediction with visual dynamics, and several use future-video prediction only during training; World Dreamer v5-lite instead places observed video features in workspace coordinates [7] [12] [30].UVA, Fast-WAM, and GigaWorld-Policy skip future-frame generation at inference [12] [25] [24].Source: §2 · pp. 2–3

Multi-robot policies

  • GauDP redistributes a 3D Gaussian field across agents, while CLS-DP distills privileged dynamics into a latent inferred from each agent’s own camera [23] [17].World Dreamer v5-lite instead uses pretrained vision and video models, pose-aware queries, and masked-view training.Source: §2 · p. 3

3D visuomotor policies

  • DP3 and PDFactor use point-cloud representations, whereas World Dreamer v5-lite builds a plane-layout state from pretrained video features across moving cameras, including patches without depth [26].For patches without depth, it uses candidate points along viewing rays and accumulates observations with visibility-gated updates.Source: §2 · p. 3

3. Method

World Dreamer v5-lite predicts action chunks for multiple robots sharing a workspace by combining VLM scene context with calibrated video features in a shared world state. Its decoder keeps robot action streams separate while sharing spatial information, and masked-view training retains action supervision when cameras are withheld.

3.1. Vision–language conditioning

  • One learnable query per robot follows a shared instruction-and-image prefix, allowing a single VLM pass to condition the robot queries on the same scene.The decoder receives each contextualized query and a shared cache of prefix keys and values.Source: §3.1 · p. 3

3.2. Video encoder

  • The video encoder processes recent camera histories and uses future video steps only for an auxiliary training objective, so action prediction uses observed features without video generation at inference.Observed outputs are prevented from attending to future steps.Source: §3.2 · pp. 3–4

3.3. Shared world state ϕ

  • The shared world state ϕ places calibrated camera features in workspace coordinates on three planes, giving all robots a fixed-size representation as views are added.Without depth, patches contribute through candidate points along their viewing rays; visibility-gated updates preserve regions not seen in the current frame.Source: §3.3 · pp. 4–5, Figure 2 · p. 4
  • The three-plane state uses 3G^2 readout locations rather than G^3 in a dense 3D grid at the same resolution.Aggregated planes are mixed by lightweight MLPs, and normalizing by accumulated weight keeps feature scale independent of how many views observe a location.Source: §3.3 · pp. 4–5

3.4. Action Decoder

  • Each robot has a separate action stream, while pose-aware query exchange lets robots share context without combining their heterogeneous action tokens.Queries first read the shared state; action streams then use the VLM scene-prefix cache and remain isolated from other robots’ streams.Source: §3.4 · pp. 5–6, Figure 3 · p. 5

3.5. Masked-view cooperative training

  • Selected cameras are withheld from both visual branches during training, while every robot’s demonstrated action chunk remains a target.The withheld views may be all cameras of one randomly selected robot or a random subset of views.Source: §3.5 · p. 6

4. Experiments

Evaluation spans RoboFactory multi-robot collaboration, RoboCasa365 household manipulation, and homogeneous and heterogeneous real-robot teams.

4.1. Datasets and protocols

  • Pretraining combines more than 10,000 hours of trajectories from 20 simulated robot types, RoboFactory, and real robots.Evaluation covers three RoboFactory tasks with increasing agent counts, 50 RoboCasa365 tasks, and four real-robot tasks; Composite-Unseen is evaluated zero-shot because those tasks are absent from Human300.Source: §4.1 · pp. 6–7

4.2. Baselines and metrics

  • Comparisons use six baselines on RoboFactory and real-robot tasks and recent leaderboard methods on RoboCasa365; the metric is task success rate.Means are unweighted across tasks for RoboFactory and real-robot evaluations; RoboCasa365 overall averages all 50 tasks, weighting splits by their task counts.Source: §4.2 · p. 7

4.3. Implementation details

  • The vision–language backbone is initialized from Qwen3-VL; its vision encoder and language model are trained while input embeddings and the language-modeling head are frozen.Each vision–language forward pass is reused for four flow-matching samples of the action decoder.Source: §4.3 · p. 7

4.4. Main results

  • 74.0% mean success on RoboFactory exceeds the strongest baseline, π0.5, at 50.3%.World Dreamer v5-lite also exceeds π0.5 by 36, 11, and 24 percentage points on Place Food, Camera Alignment, and Take Photo, respectively.Source: Table 1 · p. 7, §4.4 · p. 7
  • 78.8% mean success across four real-robot tasks exceeds the strongest baseline, π0.5, at 65.0%.The tasks include teams of two LeKiwi robots and teams of two LeKiwi robots with one OpenArm.Source: Table 2 · p. 7, §4.4 · p. 7
  • 58.08% overall success on RoboCasa365 exceeds PatchWAM’s highest compared overall score of 47.6% by 10.48 percentage points.Scores are 80.8%, 60.0%, and 30.6% on Atomic-Seen, Composite-Seen, and Composite-Unseen, respectively.Source: §4.4 · p. 7, Figure 4 · p. 8

5. Conclusion

World Dreamer v5-lite uses a shared world state for multi-robot action prediction while preserving separate robot action streams. It reports results across simulated, real-robot, and household benchmarks.

5. Conclusion

  • Calibrated camera observations populate a shared world state that gives robot queries a common spatial reference while retaining separate action streams.Vision–language context and relative-pose-aware query exchange support coordination.Source: §5 · p. 8
  • Masked-view cooperative training withholds selected cameras from both visual branches while retaining action supervision for every robot.The approach encourages robots to use observations shared by teammates.Source: §5 · p. 8
  • World Dreamer v5-lite reaches 74.0% mean success on RoboFactory, 78.8% across four real-robot tasks, and 58.08% overall on RoboCasa365.Source: §5 · p. 8

Related papers

Loading 2610.37518v1…