PaperlayerView exact revision on arXiv

Source-linked AI summary

DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptation

Makoto Sato, Tatsuya Matsushima, Yutaka Matsuo, Yusuke Iwasawa

arXiv:2608.29078v1cs.RO

TL;DR

DREAM addresses the costly need for workspace-specific action-labeled data when pretrained VLAs are deployed in new environments. It reconstructs the workspace, translates language into validated goals and success criteria, and uses TAMP to generate and render augmented demonstrations for fine-tuning. Experiments report strong sim-to-real ranking correlation and improved real-world performance as automatically generated data increases, without additional task-specific human demonstrations.

  • Problem

    Pretrained VLA performance can be insufficient in new workspaces, while collecting workspace-specific action-labeled supervision through human teleoperation is costly.

  • Method

    DREAM reconstructs a workspace from video, derives symbolic goals and success criteria from language, plans feasible trajectories with TAMP, and renders augmented demonstrations for VLA fine-tuning.

  • Results

    Simulated and real policy rankings correlate strongly, with r = 0.98 on BlockIntoBowl and r = 0.99 on FruitPacking, while real-world performance improves as more DREAM data is generated.

  • Takeaways & Limitations

    DREAM provides workspace-specific VLA supervision from captured deployment scenes without requiring a task-specific human demonstration.

Abstract

from arXiv · show

Vision-language-action (VLA) models have made strong progress in language-conditioned robot manipulation, but improving their performance in a new workspace still often requires action-labeled data from that environment. Collecting such data by human teleoperation is costly, especially when each workspace, object arrangement, or task may require new demonstrations. We present DREAM, a framework that generates fine-tuning data for a pretrained VLA from a captured workspace and a language instruction, without requiring a task-specific human demonstration. DREAM reconstructs the workspace, automatically translates the instruction into symbolic task goals and success criteria using a large language model, and uses task-and-motion planning to generate feasible robot trajectories. The planned trajectories are augmented across randomized object configurations, verified by the generated success criteria, and rendered into image-action examples for VLA fine-tuning. Through real-robot experiments on language-conditioned manipulation tasks, we study whether DREAM can serve as a scalable data-collection system for the deployment workspace by examining whether fine-tuning on its automatically generated data improves success over direct deployment and how its data-collection cost compares with human teleoperation when adapting a VLA to a new workspace.

I. INTRODUCTION

DREAM addresses the costly need for workspace-specific action-labeled supervision by generating VLA fine-tuning data from a captured workspace and language instruction without task-specific demonstrations. It evaluates whether reconstructed environments support scalable data generation and whether automatically generated data improves real-world performance.

  • Out-of-the-box VLA performance can remain insufficient in new workspaces, where object instances, poses, clutter, surfaces, and reachability affect executable actions.
  • Human teleoperation is costly because each workspace, arrangement, or task requires operator time, robot access, scene resets, and safety monitoring.
  • The framework reconstructs a deployment workspace, derives symbolic goals and success criteria with a large language model, and uses TAMP to generate feasible trajectories.
  • DREAM converts a short workspace video and language instruction into fine-tuning data without any task-specific human demonstration.
  • DREAM validates reconstruction through sim-and-real performance correlation and measures how real-world performance scales with automatically generated data relative to human teleoperation.

II. RELATED WORK

Related work reduces manual robot-data collection through planning, demonstration augmentation, and reconstructed scenes, but existing approaches commonly retain seed demonstrations, real-robot interaction, or task-specific scripting. DREAM instead plans multi-step demonstrations directly from language goals in a reconstructed deployment workspace.

  • Planning-based data generation: Planning-based methods use language decomposition, grasp planning, contact planning, or TAMP to generate demonstrations, including for long-horizon manipulation.
  • Data augmentation from demonstrations: Demonstration-augmentation systems expand one or a few human demonstrations across configurations, object instances, embodiments, or reusable skills.
  • Data generation in reconstructed scenes: Reconstructed-scene methods use digital twins, scans, tracked motion, or Gaussian Splatting to reduce visual gaps or transfer policies to real environments.
  • DREAM capabilities: The paper characterizes DREAM as teleop-free, demo-free, real-to-sim, language-task-driven, and capable of long-horizon demonstration generation.
  • DREAM’s distinction: DREAM combines real-world reconstruction with TAMP to generate long-horizon image–action demonstrations without a task-specific seed demonstration.

III. PROPOSED SYSTEM

DREAM generates deployment-workspace training data from a short workspace video, language instruction, robot model, and camera calibration under assumptions of static scenes and rigid objects. Its design separates visual rendering from geometric planning to obtain realistic and feasible demonstrations.

  • DREAM assumes a static workspace, rigid objects, a recorded capture-time joint configuration, and calibrated fixed and wrist-mounted deployment cameras.
  • The pipeline uses workspace capture and reconstruction, demonstration synthesis with TAMP, data rendering, and policy fine-tuning to produce deployment-specific training data.
  • The reconstructed environment supplies visual deployment context, while a compact geometric world supplies the robot, movable objects, support surface, and planning structure.
  • This separation produces image–action data grounded in the deployment workspace and geometrically feasible for the target robot without requiring watertight scene meshes.

A. Digital-Twin Reconstruction

DREAM reconstructs and metrically aligns a captured workspace, decomposes it into independently movable assets, and instantiates randomized scenes for rendering and planning. The resulting digital twin supports photorealistic image generation while preserving robot-relative geometry.

  • A handheld RGB video is processed with COLMAP and Gaussian Splatting to reconstruct the workspace using calibrated, metric scene geometry.
  • DREAM automatically recovers a SIM(3) transform from the robot’s appearance to register the monocular reconstruction with the robot base frame.
  • The aligned splat is factorized into a robot-free static background and independently posable per-link robot assets for moving the robot and objects.
  • Scene metadata records movable objects, initial poses, randomization regions, language instructions, and camera calibration, instantiating an IsaacLab environment without per-task code.
  • The pipeline converts the reconstructed digital twin into photorealistic image–action demonstrations for fine-tuning a pretrained VLA.

B. Demonstration Synthesis Using TAMP

DREAM converts language instructions and reconstructed scenes into symbolic goals, success criteria, and feasible robot plans. It then executes, verifies, and records successful simulated trajectories with planner-provided action alignment.

  • Task specification: The language model maps instructions to typed symbolic goals and staged success rubrics from a fixed predicate and checker vocabulary.Outputs are grammar-checked for predicate arity and scene-object references, while malformed responses fail closed.
  • Task specification: The initial symbolic state is inferred from scene geometry, including object supports, hand occupancy, and movable-object or surface status.Support relations are determined by comparing object poses and extents rather than generated by the language model.
  • Planning: Breadth-first search enumerates PDDL plan skeletons, whose free grasps, placements, and robot configurations are resolved by GPU-parallel optimization over 1000 particles.The domain includes MoveFree, MoveHolding, Pick, Place, and Push operators.
  • Execution and recording: Each plan is executed open-loop in simulation, aborted after failed grasp detection, and recorded only when it passes the staged success rubric.Recorded trajectories contain robot states, gripper commands, object states, and symbolic operator labels at every frame.

C. Demonstration Augmentation

DREAM expands a small pool of physically valid TAMP trajectories into a larger dataset by reusing symbolic plan structure across new object configurations. Planner annotations provide direct subtask boundaries for augmentation and trimming.

  • C. Demonstration Augmentation: MimicGen-style augmentation multiplies a small source pool of TAMP trajectories across new object configurations without requiring a planning rollout for every example.The source trajectories remain physically valid but expensive because each requires a planning call and simulator rollout.
  • C. Demonstration Augmentation: Symbolic plans expose subtask boundaries directly, allowing augmentation to shift boundaries past post-grasp lifting and trim settling and homing motions.This avoids inferring boundaries heuristically from low-level signals.

D. Rendering and Dataset Construction

DREAM renders augmented trajectories into camera observations and packages them as policy-training data. The resulting policies are trained by imitation learning and deployed on real observations without planning at inference time.

  • D. Rendering and Dataset Construction: Gaussian-splat replay combines the background, posed robot links, and posed objects before rasterizing RGB observations from calibrated policy cameras.Augmentation itself produces no images; observations are attached afterward from recorded states.
  • D. Rendering and Dataset Construction: The final dataset stores language instructions, 640×480 RGB observations, robot states, and action labels in LeRobot format.Policy-specific input-output representations are derived from this stored format.
  • D. Rendering and Dataset Construction: Policies trained on DREAM demonstrations receive real camera observations and robot states at deployment, while language-conditioned policies also receive the task instruction.TAMP is used during data generation, not for direct action prediction during deployment.
  • D. Rendering and Dataset Construction: The evaluation asks whether DREAM supports scalable data collection in unseen workspaces through reconstructed-environment reliability and substitution for manual collection.These questions compare simulated and real policy performance and assess automatically generated demonstrations as an alternative to manual data collection.

A. Experimental Setup

The experiments use two tabletop tasks, two VLA policies, and demonstrations from human teleoperation or DREAM simulation. Policies are evaluated on the real robot using held-out configurations, with success, generator, cost, and compute metrics.

  • A. Experimental Setup: BlockIntoBowl tests single-step placement of a blue block into a red bowl, whereas FruitPacking tests sequential placement of a banana and peach onto a red plate.The tasks run on an xArm7 and include both single-step and longer-horizon manipulation.
  • A. Experimental Setup: Two Intel RealSense D435i cameras capture fixed external and wrist views, with calibrated extrinsics reproduced in the reconstructed simulation.Calibration uses ArUco markers for the external-camera-to-base and wrist-camera-to-end-effector transforms.
  • A. Experimental Setup: SmolVLA and π0.5 are fine-tuned from pretrained initializations with supervised imitation learning, while Diffusion Policy is trained from scratch for broader sim-to-real correlation analysis.All policies use RGB and architecture-required proprioception; the VLA policies additionally receive language.
  • A. Experimental Setup: Human teleoperation records real rollouts, while DREAM synthesizes TAMP trajectories in a reconstructed digital twin, augments them, and renders image-action demonstrations.Both training sources are evaluated on the physical robot under the same test conditions.
  • A. Experimental Setup: Task success is measured over 15 held-out object-configuration trials per policy and task, alongside trajectory-validity, planning, augmentation, timing, and human-effort metrics.Human-effort time includes workspace capture, calibration and alignment, and task specification.
  • A. Experimental Setup: All non-training computational stages run on three 48 GB NVIDIA RTX A6000 GPUs, while policy training uses NVIDIA GB200 GPUs.The reported stages include scene optimization, asset generation, demonstration generation, simulation, and rendering.

B. Experiment 1: Sim-to-Real Correlation

Experiment 1 tests whether reconstructed simulation performance tracks real-robot performance when comparing policy checkpoints. Across both tasks, the reconstruction preserves policy rankings while underestimating absolute success.

  • Evaluation protocol: Experiment 1 evaluates matched policy checkpoints in reconstructed simulation and on the robot using identical object configurations and success criteria.The evaluation includes three policy architectures and checkpoints trained with the teleoperated demonstrations for each task.
  • Evaluation protocol: Figure 4 compares real-world success on the horizontal axis with reconstructed-simulation success on the vertical axis, separating tasks by color and architectures by marker shape.The dashed diagonal denotes equal simulated and real-world performance.
  • Results: MMRV = 0.000 on both tasks, with Pearson’s r = 0.98 for BlockIntoBowl and r = 0.99 for FruitPacking.These values indicate that the reconstruction reproduces the real ranking of the three policies in both tasks.
  • Results: Simulated success is 7 to 14 percentage points lower than real success, except for SmolVLA on FruitPacking at the low end, where the rates coincide.The reconstruction is therefore pessimistic about absolute performance while preserving policy comparisons.

C. Experiment 2: Scaling with Data Collection Time

Experiment 2 compares DREAM-generated and teleoperated data as acquisition time increases. Generated data can surpass teleoperation at sufficient scale, while costing far less per demonstration after DREAM’s fixed setup cost.

  • Cost and protocol: DREAM uses fixed capture, reconstruction, and planning costs followed by per-demonstration augmentation and rendering, whereas teleoperation charges operator time for each demonstration.The experiment evaluates nested dataset sizes on physical-robot performance for SmolVLA and π0.5.
  • Policy performance: On BlockIntoBowl, π0.5 reaches 93.3% with 1000 generated demonstrations versus 86.7% with 100 teleoperated demonstrations, while SmolVLA reaches 80.0% versus 73.3%.
  • Policy performance: On FruitPacking, π0.5 reaches 86.7% with generated data versus 80.0% with teleoperation, but SmolVLA reaches only 6.7% versus 26.7%.Adding generated demonstrations beyond 500 does not help SmolVLA on FruitPacking.
  • Cost and scalability: DREAM becomes cheaper in wall-clock time beyond 73 demonstrations on BlockIntoBowl and 168 on FruitPacking.Its fixed cost is 36 and 95 minutes respectively, followed by a few seconds per generated demonstration; teleoperation remains linear in operator time.
  • Cost and scalability: Generated demonstrations cost an order of magnitude less to produce than teleoperated ones, but their lower per-example quality means policy improvement requires scaling beyond teleoperated dataset sizes.Randomization introduces frequent wrist rotations and slightly redundant motions, while increasing dataset size consumes machine time more slowly than operator time.

V. CONCLUSION

DREAM converts a captured workspace and language task into workspace-specific VLA fine-tuning data without a task-specific human demonstration. The current framework is scoped to approximately static tabletop scenes with rigid objects, leaving broader manipulation settings for future work.

  • Conclusion: DREAM reconstructs an interactable digital twin and uses task-and-motion planning to synthesize and validate feasible trajectories for adapting pretrained VLAs.The framework turns a captured workspace and language task specification into workspace-specific fine-tuning data without human demonstration of the task.
  • Scope and future work: The current framework targets approximately static tabletop scenes with rigid objects.
  • Scope and future work: Extending DREAM to deformable and articulated objects and more contact-rich tasks is identified as an important direction for future work.
Loading 2608.29078v1…