PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding

Feng Jiang, Shu Chen, Yuanfei Wang, Yuan Li, Zhenhao Shen, Jiaqi Liang, Yang Chen, Lei Wei, Chen Xie, Ruihai Wu

arXiv:2610.13164v1cs.RO

TL;DR

Robot data is difficult to scale because teleoperation is labor-intensive, simulation has a visual gap, and human videos lack robot-action annotations. Grounded-Dreamer combines video world models with simulator-grounded trajectories and verification; across real-world tasks, its synthetic-data-trained policies are competitive with human-demonstration-trained policies and generalize better out of distribution.

  • Problem

    Robot data is hard to scale because teleoperation is labor-intensive, simulation has a visual sim-to-real gap, and human videos lack robot-action annotations.

    Source: §1 · pp. 1–2

  • Method

    Grounded-Dreamer reconstructs real scenes in simulation, uses grounded trajectories to post-train a video world model and IDM, and replays generated actions to filter unsuccessful or inconsistent trajectories.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across seven real-world tasks, policies trained exclusively on grounded synthetic data are competitive with human-demonstration-trained policies and outperform real-data policies in out-of-distribution scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Grounded-Dreamer supports policy learning from autonomously generated grounded synthetic data without human teleoperation demonstrations.

    Source: §1 · p. 3

    The pipeline requires a one-time set of reference images, and synthesized initial frames succeed approximately 65% of the time.

    Source: A LIMITATION · p. 14

Abstract

from arXiv · show

Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale. Existing methods either depend on human teleoperation, rely on simulation with limited visual realism, or use human videos that lack embodiment-specific annotations. In this paper, we introduce Grounded-Dreamer, a framework for synthesizing physically grounded robot manipulation data with video world models while avoiding human teleoperation demonstrations. Grounded-Dreamer combines the visual realism of generative video models with simulator physics engines. Starting from a small set of real images depicting robot task initial states, our method automatically reconstructs task-relevant objects and camera geometry, aligns them in simulation, and collects physically valid trajectories. These trajectories are used to post-train both a video world model and an inverse dynamics model, enabling the generation of visually realistic robot manipulation videos and corresponding robot actions from real images. To filter physically implausible rollouts, Grounded-Dreamer replays generated trajectories in the reconstructed simulation and retains only those that successfully complete the tasks. The real-world robot policies are trained solely on the grounded synthetic trajectories without any human demonstrations. Empirical results in real-world tasks demonstrate that Grounded-Dreamer can synthesize physically grounded data and substantially facilitate policy learning without requiring human teleoperation effort.

1 INTRODUCTION

Grounded-Dreamer addresses the challenge of scaling robot data without relying on human teleoperation, combining video-world-model realism with simulator-based physical grounding. Its real-world evaluations show competitive performance against policies trained on human demonstrations and stronger generalization in out-of-distribution scenarios.

Motivation

  • Robot data is difficult to scale, and existing sources trade off quality and quantity across costly teleoperation, visually limited simulation, and human videos without robot-action annotations.UMI collection improves efficiency but still requires manual operations.Source: §1 · p. 1
  • Simulation-based automation leaves the visual sim-to-real discrepancy unresolved, while applying human-centric video world models to robot tasks can produce physically inconsistent motions and contacts.The models can be paired with an Inverse Dynamics Model to convert videos into action-labeled trajectories.Source: §1 · p. 2

Approach

  • Grounded-Dreamer reconstructs and aligns real scenes and task objects in simulation, collects trajectories from simulator states, and uses them to post-train the video world model and IDM.The framework synthesizes manipulation data without human teleoperation demonstrations.Source: §1 · pp. 2–3
  • Generated videos are converted into robot actions, then replayed in the reconstructed simulation so physically inconsistent trajectories are excluded from policy training.Only successful, grounded trajectories are retained.Source: §1 · p. 2

Results

  • Across seven real-world tasks, policies trained only on grounded synthetic data are competitive with human-demonstration-trained policies and outperform real-data policies in out-of-distribution scenarios.The evaluations span short-to-long-horizon tasks and tool use.Source: §1 · pp. 2–3

2 RELATED WORK

Prior robot data synthesis uses simulation, adapts collected demonstrations, or generates rollouts with world models. Grounded-Dreamer combines real scene inputs and reconstructed geometry with simulator-grounded world-model rollouts.

2.1 ROBOT DATA SYNTHESIS

  • Simulation platforms and visual or dynamics randomization have supported diverse robot data generation, from standardized manipulation environments to scalable physics and household manipulation.Examples include RLBench, robosuite, SAPIEN, Isaac Gym, and RoboCasa.Source: §2.1 · p. 3
  • Task-generation frameworks use language or foundation models, but their data quality depends on generated programs, planners, solvers, or simulator assumptions; local object pose and camera-frame errors can mislead physically plausible trajectories.Reconstruction methods recover task-relevant structure and view-consistent geometry or appearance.Source: §2.1 · p. 3
  • Demonstration-reuse methods adapt collected trajectories, while large real-robot datasets require substantial collection effort; Grounded-Dreamer instead grounds synthetic data in real scene-object inputs and expands it with world-model rollouts.The cited approaches include DemoGen, DROID, and Open X-Embodiment.Source: §2.1 · p. 3

2.2 WORLD MODELS FOR ROBOT LEARNING

  • Video world models can generate realistic rollouts, but videos often lack actions, embodiment constraints, and reliable interactions; Grounded-Dreamer couples rollouts with reconstructed scenes and simulation-based grounding before policy learning.Inverse-dynamics models can recover actions from observations, enabling generated trajectories to train robot policies.Source: §2.2 · p. 3

3 METHODOLOGY

Grounded-Dreamer turns a small set of real-world reference images into synthetic robot trajectories through scene reconstruction, simulator data collection, image synthesis, and closed-loop verification. The resulting data is intended to combine visual realism with physical consistency.

3 METHODOLOGY

  • The pipeline has four stages: task-oriented simulation reconstruction, simulator data collection and interaction-aware masking, initial-frame synthesis, and trajectory synthesis with closed-loop verification.The stages support synthetic-data generation without human-collected real-world trajectories.Source: §3 · p. 4

3.1 TASK-ORIENTED SIMULATION RECONSTRUCTION

  • The reconstruction stage calibrates simulated and real viewpoints and creates task-relevant object geometry, scaling rigid meshes to real dimensions and assembling articulated objects from components and joint specifications.The scene uses calibrated camera parameters and a lightweight workspace simulation.Source: §3.1 · pp. 4–5

3.2 SIM DATA COLLECTION AND INTERACTION-AWARE MASKING

  • An LLM-based agent generates and executes task data-collection scripts from templates, task descriptions, and object metadata; interaction-aware masking retains the robot arm and interacting objects for model fine-tuning.The collected 100–150 trajectories per task vary with task complexity and object count.Source: §3.2 · p. 5

3.3 INITIAL FRAME SYNTHESIS

  • Initial frames are made by compositing randomized simulated objects onto real backgrounds, then refining them with real-object photographs and augmenting lighting and tablecloth textures.The image variations are intended to improve robustness to distribution shifts.Source: §3.3 · p. 5

3.4 DATA SYNTHESIS AND CLOSED-LOOP VERIFICATION

  • The fine-tuned world model generates rollout videos from augmented frames, and an IDM extracts actions; simulator replay classifies trajectories by task success and filters inconsistent or failing samples.The validation example contrasts a simulated failure from incorrect gripper placement with a hallucinated successful video outcome.Source: §3.4 · p. 6, Figure 4 · p. 6

4 EXPERIMENTS

Across seven real-world manipulation tasks, Grounded-Dreamer data supports policies competitive with real-demonstration training and stronger OOD generalization than Real-only training. Co-training, trajectory scaling, interaction-aware masking, and replay verification further shape performance.

4.1 EXPERIMENTAL SETUP

  • Grounded-Dreamer is evaluated on seven diverse real-world manipulation tasks spanning short-horizon interactions to tool use, with DP and π0.5 backbones compared across Sim-only, Real-only, low-data Real, and RIGVid regimes.Success rate is the primary metric; scaling-law analyses use 30 trials per task, while other comparative evaluations use 20.Source: §4.1 · p. 7

4.2 MAIN RESULTS AND COMPARATIVE ANALYSIS

  • GD-only policies are competitive with Real-only, including π0.5 parity on Pressing Button, while RIGVid performs substantially worse.The reported RIGVid degradation is attributed to hand-pose extraction inaccuracies and viewpoint jitter.Source: §4.2 · p. 8
  • 8.6% for DP and 14.3% for π0.5 are the Low-data Real results; adding 50 GD trajectories yields a leap that matches or exceeds 50-sample Real-only.Co-training also outperforms the GD-data-only model.Source: §4.2 · p. 8

4.3 GENERALIZATION AND ROBUSTNESS TO ENVIRONMENT VARIATION

  • 7.1% for DP and 16.4% for π0.5 are Real-only OOD success rates; GD-trained models generalize better in novel environments.The reported diversity comes from augmenting initial frames with varied backgrounds, tablecloth textures, and lighting before generating 50 GD trajectories.Source: §4.3 · p. 8

4.4 SCALING LAWS OF SYNTHETIC WORLD MODEL DATA

  • Success rates improve on Close Drawer and Put in Drawer as GD trajectories increase, then plateau at approximately 150 trajectories.Further increases beyond approximately 150 trajectories yield only marginal improvements.Source: Figure 6 · p. 8, §4.4 · p. 9

4.5 ABLATION STUDIES: KEY COMPONENTS OF GROUNDED-DREAMER

  • Task success rises with the proportion of verified trajectories, with a precipitous drop at 50% sub-optimal data.The ablation supports simulation replay as a filter for synthetic training data.Source: Table 4 · p. 9, §4.5 · p. 9
  • Without interaction-aware masking, generated videos show distorted pixels and unrealistic scene deformations, and downstream success rates drop significantly.The ablation supports masking task-irrelevant regions during world-model fine-tuning.Source: Figure 7 · p. 9, Table 3 · p. 9, §4.5 · p. 9

A LIMITATION

Grounded-Dreamer still requires reference images to initialize an environment, and the scale demonstrated so far was constrained by computational resources. The paper also flags world-model bias and deployment safety as broader-impact concerns.

A LIMITATION

  • Initialization requires a one-time reference set, typically one scene overview and 1–4 object perspectives; synthesized initial frames succeed approximately 65% of the time.The authors identify image-conditioned editing models as rapidly evolving and expect this rate to improve as foundational generative models mature.Source: A LIMITATION · p. 14
  • Experiments establish effectiveness with hundreds of trajectories per task, with scale constrained primarily by computational resources rather than algorithmic bottlenecks.The authors anticipate larger-scale generation as hardware constraints are alleviated.Source: A LIMITATION · p. 14

B BROADER IMPACTS

  • The paper urges attention to potential world-model biases and safety of policies trained on synthetic data, especially for real-world human-centric deployment.It recommends maintaining rigorous safety protocols in those environments.Source: B BROADER IMPACTS · p. 14

C ADDITIONAL QUALITATIVE VISUALIZATIONS

C.1 AUTOMATED SIMULATION-BASED DATA ACQUISITION

Figure 8 contrasts original simulation frames with frames after interaction-aware masking.

C.1 AUTOMATED SIMULATION-BASED DATA ACQUISITION

  • Figure 8’s top row shows original simulation frames and its bottom row shows frames after interaction-aware masking.Source: §C.1 · p. 14

C.2 VIDEO ROLLOUTS FROM GROUNDED-DREAMER

The appendix presents additional world-model video rollouts as qualitative evidence of predicted physically plausible future states.

C.2 VIDEO ROLLOUTS FROM GROUNDED-DREAMER

  • Figure 9 presents additional world-model video rollouts that demonstrate prediction of physically plausible future states.Source: §C.2 · p. 14

D DETAILS OF INITIAL FRAME SYNTHESIS

Grounded-Dreamer synthesizes realistic initial frames by harmonizing simulated objects with real-world references while preserving the simulated objects’ positions and orientations. The section provides qualitative visualizations of synthesized frames and generated data.

D DETAILS OF INITIAL FRAME SYNTHESIS

  • Nano Banana 2 synthesizes initial frames from a composite of simulated objects overlaid on a real scene and a reference photograph of the actual objects.The reference provides structural and illumination constraints.Source: D DETAILS OF INITIAL FRAME SYNTHESIS · p. 14
  • The structured prompt replaces each simulated object with a realistic version while preserving its original position and orientation.The prompt specifies that the replacement should not be aligned with the real object in the reference image.Source: D DETAILS OF INITIAL FRAME SYNTHESIS · p. 14
  • Figure 10 shows synthesized frames integrating virtual assets into diverse physical contexts.Additional visualizations cover automated simulation data collection and Grounded-Dreamer video generation.Source: Figure 8 · p. 15, Figure 9 · p. 16, D DETAILS OF INITIAL FRAME SYNTHESIS · p. 17

E EXPERIMENTAL DETAILS

The experimental details specify the Cosmos fine-tuning configuration and computational requirements for reproducibility.

E EXPERIMENTAL DETAILS

E.1 FINE-TUNING DETAILS OF COSMOS

  • Cosmos Predict2.5 is fine-tuned for 1,400–2,000 iterations with a global batch size of 32 and task-dependent temporal lengths of 125 or 93 frames.The learning rate is 4.315 × 10−5; training uses eight NVIDIA A800 GPUs with context parallelism and requires 74 GB VRAM per GPU.Source: §E.1 · p. 17

E.2 TRAINING DETAILS OF THE INVERSE DYNAMICS MODEL (IDM)

The IDM combines a SigLIP2-Large encoder with a flow-matching action head and is trained in mixed precision across eight GPUs.

E.2 TRAINING DETAILS OF THE INVERSE DYNAMICS MODEL (IDM)

  • The IDM uses a SigLIP2-Large encoder and a flow-matching action head.Source: §E.2 · p. 17
  • Training runs for 5,000–10,000 steps with a global batch size of 512 and a 10−4 learning rate.Source: §E.2 · p. 17
  • The IDM is trained across eight A800 GPUs using BF16 mixed precision for numerical stability.Source: §E.2 · p. 17

E.3 SYNTHETIC DATA GENERATION AND ACTION EXTRACTION

Inference generates video rollouts from replicated initial frames and extracts robot actions with the IDM; generation-phase task success averages approximately 45%.

E.3 SYNTHETIC DATA GENERATION AND ACTION EXTRACTION

  • Approximately 45% is the average task success rate during the data-generation phase.Source: §E.3 · p. 17
  • Video generation conditions rollouts on five replicated initial frames, using 125 frames for Put in Drawer and 93 frames for other tasks.Inference requires approximately 16 GB VRAM per GPU.Source: §E.3 · p. 17
  • The IDM extracts robot actions from generated sequences in 0.5 seconds per trajectory, using 4 GB VRAM per GPU.Source: §E.3 · p. 17
Loading 2610.13164v1…