Source-linked AI summary
DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning
Zhengrong Xue, Shuying Deng, Zhenyang Chen, Yixuan Wang, Zhecheng Yuan, Huazhe Xu
TL;DR
DemoGen addresses the high human-data demands of visuomotor policy learning caused by limited spatial generalization. It generates spatially augmented demonstrations synthetically by adapting actions and editing 3D point clouds, improving policy performance across real-world tasks while supporting additional capabilities such as disturbance resistance and obstacle avoidance.
Problem
Visuomotor policies require substantial human-collected data because limited spatial generalization necessitates demonstrations across different object configurations.
Method
DemoGen uses TAMP-based action adaptation and 3D point-cloud manipulation to generate spatially augmented demonstrations from one source demonstration.
Results
DemoGen significantly enhances policy performance across 8 real-world tasks, averaging 74.6%, at a generation cost of 0.01 seconds per demonstration trajectory.
Takeaways & Limitations
DemoGen supports spatial generalization across varied real-world platforms and objects and can extend imitation learning to disturbance resistance and obstacle avoidance.
Takeaways & Limitations
Synthetic demonstrations from one source demonstration are less effective than the same number of human-collected demonstrations because of visual mismatch between synthetic and real point clouds.
Abstract
from arXiv · showhide
Visuomotor policies have shown great promise in robotic manipulation but often require substantial amounts of human-collected data for effective performance. A key reason underlying the data demands is their limited spatial generalization capability, which necessitates extensive data collection across different object configurations. In this work, we present DemoGen, a low-cost, fully synthetic approach for automatic demonstration generation. Using only one human-collected demonstration per task, DemoGen generates spatially augmented demonstrations by adapting the demonstrated action trajectory to novel object configurations. Visual observations are synthesized by leveraging 3D point clouds as the modality and rearranging the subjects in the scene via 3D editing. Empirically, DemoGen significantly enhances policy performance across a diverse range of real-world manipulation tasks, showing its applicability even in challenging scenarios involving deformable objects, dexterous hand end-effectors, and bimanual platforms. Furthermore, DemoGen can be extended to enable additional out-of-distribution capabilities, including disturbance resistance and obstacle avoidance.
I. INTRODUCTION
Visuomotor policies require substantial human-collected data partly because their spatial generalization is limited. DemoGen addresses this cost with fully synthetic, TAMP-based action adaptation and 3D point-cloud observation synthesis, improving performance across varied real-world settings.
- State-of-the-art visuomotor policies often require tens to hundreds of demonstrations, and intricate long-horizon tasks may require thousands.
- Policies typically generalize only near demonstrated object configurations, so covering the full tabletop workspace requires repeated data collection.
- DemoGen replaces costly on-robot rollout-based generation with a fully synthetic pipeline that produces spatially augmented demonstrations for policy training.
- DemoGen adapts source trajectories by transforming contact-rich skill segments as units and replanning free-space motion segments with TAMP-based motion planning.
- DemoGen-generated datasets use one human demonstration per task and support single-arm, bimanual, parallel-gripper, dexterous-hand, third-person, egocentric, rigid, deformable, and fluid settings.
- DemoGen combines TAMP principles for synthetic data generation with closed-loop visuomotor policies for task resolution.
C. Data Generation for Robotic Manipulation
Prior automated approaches generate execution plans but may depend on costly on-robot rollouts, whereas DemoGen produces real-world training demonstrations synthetically. Experiments examine how spatial coverage, precision, representations, and demonstration count affect generalization.
- MimicGen-style methods adapt human demonstrations to novel configurations, but their execution plans require on-robot rollouts to verify actions and capture observations.
- DemoGen replaces those expensive rollouts with fully synthetic generation, producing real-world demonstrations ready for policy training.
- A. Visualization of Spatial Effective Range: Spatial generalization is the ability to perform tasks with object configurations not seen during training.
- A. Visualization of Spatial Effective Range: The benchmark evaluates Button-Large and precision-demanding Button-Small tasks across a 30 cm × 40 cm tabletop randomization area, with 441 placements sampled for visualization.
- A. Visualization of Spatial Effective Range: The spatial effective range approximates the union of areas surrounding demonstrated placements, while higher precision requirements shrink the range and increase coverage needs.
- B. Benchmarking Spatial Generalization Capability: Object randomization strongly affects demonstration requirements; 3D representations and pretrained 2D encoders improve spatial generalization but do not fundamentally resolve it.
IV. DemoGen METHODS
DemoGen augments a source observation-action trajectory for a different initial object configuration. Its formulation represents demonstrations as paired observations and actions, with actions and point-cloud observations transformed through the generation pipeline.
- A. Problem Formulation: A visuomotor policy maps visual observations o ∈ O directly to predicted actions a ∈ A, while a source demonstration is a trajectory of paired observations and actions conditioned on an initial configuration.
- A. Problem Formulation: DemoGen generates a new demonstration by conditioning the source trajectory on a different initial object configuration.
- A. Problem Formulation: The initial configuration is defined by the set of initial poses of the sequentially manipulated objects.
- A. Problem Formulation: Actions include target end-effector poses and hand commands, which may be gripper open/close signals or higher-dimensional dexterous-hand joint controls.
- A. Problem Formulation: Raw point clouds are cropped, clustered, and downsampled, while semantic masks guide parsing of the source action trajectory into motion and skill segments.
- A. Problem Formulation: Observations include point-cloud data and proprioceptive feedback from the robot.
B. Pre-processing the Source Demonstration
DemoGen preprocesses a source demonstration by segmenting it into motion and skill stages, then adapting arm actions to new object configurations while preserving hand commands. Motion segments are replanned to connect transformed skill segments, with IK-based absolute-pose control used for failure-free execution.
- Parsing the source demonstration: The source trajectory is parsed into alternating motion segments in free space and skill segments involving contact-based object manipulation.Skill segments are identified by the end-effector’s distance to an object’s point-cloud center; intermediate trajectories are motion segments.
- Adapting actions to the new configuration: Robot-hand commands remain unchanged because they are invariant to spatial transformations, whereas robot-arm commands are spatially equivariant to object movements.The action representation separates robot-arm and robot-hand commands.
- Adapting actions to the new configuration: DemoGen transforms entire skill segments according to the corresponding object movements while adapting robot-arm commands spatially to the altered configuration.The method uses a TAMP-based procedure for motion and skill segments involving each object.
- Adapting actions to the new configuration: Motion-stage arm commands are planned between neighboring skill segments, using their endpoint poses and either linear interpolation or obstacle-aware motion planning.The starting pose comes from the previous skill segment’s final frame, and the ending pose comes from the current skill segment’s first frame.
- Failure-free action execution: IK controllers with target absolute end-effector poses are used to support failure-free execution without on-robot rollouts for filtering.The authors report that these adjustments help minimize compounding control errors.
D. Fully Synthetic Observation Generation
DemoGen synthesizes augmented observations from point clouds and proprioceptive states by applying transformations consistent with the adapted actions. Objects are handled differently across to-do, doing, and done stages, with doing-stage object points merged with the end-effector.
- Adapting proprioceptive states: Proprioceptive states undergo the same transformations as actions because they share the same semantics.The observations combine point-cloud data with proprioceptive states.
- Adapting proprioceptive states: Directly replacing the current state with the next target-pose action may impair performance because IK controllers may not achieve the exact target pose.This observation constrains how proprioceptive states should be adapted during synthesis.
- Synthesizing point-cloud observations: Synthetic point-cloud observations are generated by segmenting scene elements and transforming them according to the target configurations and adapted proprioceptive states.The strategy requires first-frame segmentation masks for the manipulated objects in addition to the target transformations.
- Synthesizing point-cloud observations: In the to-do stage, each object is transformed from its source to its initial target configuration; in the doing stage, its point cloud is merged with the end-effector.In the done stage, the object remains in its final state.
- Synthesizing point-cloud observations: The robot and doing-stage objects can be separated by subtracting to-do- and done-stage object point clouds from the scene point cloud.This procedure relies on the assumption of a cropped workspace.
A. Effectiveness: One-Shot Imitation
DemoGen is evaluated in simulation using one source demonstration per task and spatially augmented datasets, showing improved generalization with substantially less human data. Performance eventually saturates as synthetic coverage or density increases, partly because visual mismatch grows with spatial distance.
- Evaluation setup: The simulated evaluation adapts eight MetaWorld tasks with enlarged object-randomization ranges to emphasize spatial generalization.Policies consume point-cloud and proprioception observations, with equal training cost across dataset sizes.
- Generation setup: DemoGen generates 100 demonstrations for one-object tasks and 200 for tasks involving two objects from a single source demonstration.Two-object tasks receive more generated demonstrations because their object configurations are more diverse.
- Evaluation and results: DemoGen-generated datasets significantly outperform the one-source-demonstration baseline and approach performance from 25 source demonstrations.The study reports potential to maintain policy performance with over 20× reduced human data-collection effort.
- Limitations: Synthetic demonstrations are less effective than the same number of human-collected demonstrations, attributed to visual mismatch under single-view observations.The mismatch arises because synthetic point clouds preserve a fixed observed side while real object appearance changes with perspective.
- Performance saturation: Performance plateaus after synthetic demonstrations sufficiently cover the workspace or reach adequate spatial density.Increasing distance from the source configuration intensifies visual mismatch, reducing the benefit of additional demonstrations.
A. Single-Arm Platforms
On single-arm platforms, DemoGen is evaluated across diverse tasks and configurations using point-cloud observations and small perturbations around target placements. Generated demonstrations improve success across the evaluated tasks while preserving adaptive closed-loop behavior.
- Evaluation protocol: The evaluation covers seven single-arm tasks using parallel-gripper and dexterous-hand end-effectors across feasible tabletop configurations.Configurations are sampled across the robot’s maximum-reach workspace and evaluated repeatedly.
- Generation strategy: Each real-world task starts from one demonstration replayed twice, yielding three source point-cloud trajectories to reduce overfitting to sensor artifacts.Replay diversity addresses flickering holes and projective smearing in real-world point clouds.
- Generation strategy: Nine ±1.5 cm perturbations are added around each target configuration to model imprecision in human object placement.The generated dataset therefore includes both evaluated configurations and nearby placement variations.
- Results: DemoGen consistently increases success rates across the evaluated single-arm tasks compared with policies trained only on source demonstrations.Source-only policies severely overfit by blindly replicating demonstrated trajectories, whereas generated data supports more adaptive responses.
- Results: In Dex-Rollup, the generated-data policy dynamically adjusts wrapping motions from 2 to 5 according to each piece’s plasticity.This observation indicates that DemoGen does not eliminate closed-loop replanning behavior.
B. Bimanual Humanoid Platform
DemoGen extends to a bimanual humanoid task with egocentric observations and out-of-distribution object orientations. A near bird’s-eye view supports point-cloud editing, and the generated-data policy achieves high success across orientation changes.
- Task setup: The Fruit-Basket task requires simultaneous bimanual manipulation, with one arm placing a basket and the other placing a banana inside it.The task uses an egocentric camera mounted on the robot’s head.
- Generation strategy: DemoGen independently adapts both arms’ actions to the transformations of their respective objects after replaying the human demonstration twice.Small perturbations are omitted because this task has relatively lower precision requirements.
- Generation strategy: A stooping posture provides a near bird’s-eye view that enables more effective point-cloud editing for full-directional yaw rotations.The viewpoint addresses the limited front-facing appearance captured by the single camera.
- Results: 90.8% success demonstrates DemoGen’s effectiveness on the bimanual humanoid platform and for out-of-distribution orientations.The Fruit-Basket task varies banana orientation by 45°, 90°, and 135° while randomizing basket position.
- Disturbance resistance: ADR augments DemoGen by simulating crust displacements and end-effector re-approach motions for disturbance resistance.The ADR-enhanced policy performs comparably to human experts and remains robust under up to five successive disturbances.
B. Obstacle Avoidance
DemoGen is extended to synthesize obstacle-involved demonstrations for a dexterous teddy-bear transfer task. The resulting policy avoids diverse obstacles in most trials while retaining a lower trajectory when no obstacle is present.
- Motivation: A source-only policy fails to account for potential collisions, including knocking over a coffee cup placed in the workspace.The task transfers a teddy bear into a box using a dexterous hand.
- Generation strategy: Obstacle-involved point clouds are created by fusing sampled points from simple geometries such as boxes and cones into the original scene.Collision-free trajectories are generated with a motion-planning tool.
- Evaluation and results: The augmented-data policy bypasses obstacles in 22 of 25 trials across five everyday objects with diverse shapes.Without obstacles, it follows the lower trajectory from the source demonstrations, indicating environmental responsiveness.
- Conclusion: DemoGen’s broader conclusion includes generating demonstrations for both disturbance resistance and obstacle avoidance.The paper presents these extensions as additional capabilities imparted through targeted demonstrations.
- Limitations: DemoGen relies on segmented point clouds and is affected by visual mismatch, limiting applicability in highly cluttered or unstructured environments.The method is also unsuitable when spatial generalization is not required.
APPENDIX
The appendix analyzes spatial generalization across policy architectures, object randomization, and demonstration counts. It finds interpolation but limited extrapolation, with performance shaped by precision, workspace coverage, and diminishing returns from additional demonstrations.
- Visuomotor policies show spatial interpolation, but higher precision requirements reduce interpolation capability.The effective range with 9 demonstrations exceeds nine times the single-demonstration range in the sparse setting.
- More demonstrations near the workspace center do not significantly extend the effective range to distant areas, making extrapolation challenging.
- The spatial generalization range is approximated by adjacent areas around demonstrated configurations, with extent influenced by required fault tolerance.
- DP3 exhibits the highest spatial generalization capacity among the evaluated 2D-based counterparts.
- Precision requirements and spatial randomization jointly contribute to task difficulty; precision alone is insufficient when positions are not fully randomized.
- Performance generally improves with more demonstrations, but gains diminish beyond a threshold: increasing from 100 to 150 improves performance by 37%, versus 6% from 150 to 200.These figures are reported for the full-workspace setting with DP3.
C. Increased Human-Collected Demonstrations
This section details real-world task settings, evaluation procedures, and analyses of human-collected and synthetic demonstrations. It also reports visual mismatches under orientation changes and physical constraints in mobile manipulation.
- C. Increased Human-Collected Demonstrations: The comparison evaluates DemoGen-generated data against 3 human-collected source demonstrations on the Dex-Cube task.
- Orientational augmentation can cause action mismatches when generated orientations differ substantially from the source demonstration.
- In mobile manipulation, terrain constraints prevent the base from approaching objects at arbitrary distances, so spatial generalization remains limited.
- The sauce-coverage metric segments background, sauce, and uncovered crust in HSV space, then applies smoothing, dilation, and erosion to reduce fragmentation artifacts.
- The disturbance evaluation uses 25 trials for each strategy, from 5 repetitions across 5 disturbance directions.
- The simulated evaluation enlarges object randomization ranges and uses a slightly larger generation range to avoid degradation near workspace boundaries.