PaperlayerView exact revision on arXiv

Source-linked AI summary

Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulation

Sizhe Yang, Wenye Yu, Jia Zeng, Jun Lv, Kerui Ren, Cewu Lu, Dahua Lin, Jiangmiao Pang

arXiv:2504.13175v1cs.RO

TL;DR

Visuomotor imitation policies need broader visual coverage, but real-world demonstration collection is costly and existing 2D or simulation-based augmentation has important constraints. RoboSplat reconstructs scenes with 3DGS and edits them to generate diverse demonstrations across six generalization types. In real-world experiments, it reaches 87.8% average success versus 57.2% for policies trained on hundreds of demonstrations with additional 2D augmentation.

  • Problem

    Imitation policies often lack robustness under visual conditions differing from expert demonstrations, while collecting diverse human demonstrations is time-consuming and labor-intensive.

    Source: §I · pp. 1–2

  • Method

    RoboSplat reconstructs manipulation scenes with 3DGS and directly edits Gaussian representations to generate demonstrations across six generalization types.

    Source: §I · p. 2, A. Reconstruction and Preprocessing · p. 4

  • Results

    87.8% average success versus 57.2% for policies trained on hundreds of real-world demonstrations with additional 2D augmentation.

    Source: §I · p. 2

  • Takeaways & Limitations

    RoboSplat generates diverse, high-quality data from a single collected demonstration and enhances visuomotor-policy robustness under varied real-world disturbances.

    Source: §I · p. 2, §VII · p. 10

    Naive 3DGS cannot handle deformable objects and lacks physical constraints for contact-rich and dynamic tasks.

    Source: §VI · p. 10

Abstract

from arXiv · show

Visuomotor policies learned from teleoperated demonstrations face challenges such as lengthy data collection, high costs, and limited data diversity. Existing approaches address these issues by augmenting image observations in RGB space or employing Real-to-Sim-to-Real pipelines based on physical simulators. However, the former is constrained to 2D data augmentation, while the latter suffers from imprecise physical simulation caused by inaccurate geometric reconstruction. This paper introduces RoboSplat, a novel method that generates diverse, visually realistic demonstrations by directly manipulating 3D Gaussians. Specifically, we reconstruct the scene through 3D Gaussian Splatting (3DGS), directly edit the reconstructed scene, and augment data across six types of generalization with five techniques: 3D Gaussian replacement for varying object types, scene appearance, and robot embodiments; equivariant transformations for different object poses; visual attribute editing for various lighting conditions; novel view synthesis for new camera perspectives; and 3D content generation for diverse object types. Comprehensive real-world experiments demonstrate that RoboSplat significantly enhances the generalization of visuomotor policies under diverse disturbances. Notably, while policies trained on hundreds of real-world demonstrations with additional 2D data augmentation achieve an average success rate of 57.2%, RoboSplat attains 87.8% in one-shot settings across six types of generalization in the real world.

I. INTRODUCTION

RoboSplat addresses limited robustness and costly data collection in visuomotor imitation by editing reconstructed 3D scenes to generate diverse demonstrations from a single trajectory. Real-world experiments report improved robustness across varied disturbances and generalization settings.

  • Imitation policies often lack robustness because demonstrations insufficiently cover visual domains, while collecting more human demonstrations is time-consuming and labor-intensive.Source: §I · p. 1
  • Real-to-Sim-to-Real methods face inaccurate geometric reconstruction and monotonous scenes, whereas image augmentation remains primarily limited to 2D observations.Source: §I · pp. 1–2, B. Data Augmentation for Policy Learning · p. 2, C. Gaussian Splatting in Robotics · p. 3
  • 3DGS preserves spatial information, supports consistent multi-view rendering, and enables interpretable editing of reconstructed scenes for novel manipulation configurations.Source: §I · p. 2, C. Gaussian Splatting in Robotics · pp. 2–3
  • RoboSplat uses 3D Gaussian replacement, equivariant transformations, visual attribute editing, novel view synthesis, and 3D content generation across six generalization types.Source: §I · p. 2
  • 87.8% average success versus 57.2% for policies trained on hundreds of real-world demonstrations with additional 2D augmentation demonstrates stronger real-world robustness.Source: §I · p. 2

III. PRELIMINARIES

RoboSplat reconstructs manipulation scenes as 3D Gaussians, aligns them to robot coordinates through differentiable rendering, and prepares editable components for demonstration generation.

Reconstruction

Frame alignment

  • Differentiable rendering refines ICP alignment by optimizing transformations against corresponding URDF and Gaussian-rendered robot masks.The optimized transformation is applied to the reconstructed scene to place it in the real-world coordinate frame.Source: A. Reconstruction and Preprocessing · p. 4
  • The alignment objective compares masks rendered from Gaussian Splatting and URDF across canonical camera views.Gradient descent optimizes the relative transformation, including translation, rotation, and scale, before applying it to the Gaussians.Source: Figure 4 · p. 4, A. Reconstruction and Preprocessing · p. 4

Scene decomposition

  • RoboSplat decomposes the scene into object and robot-link Gaussians using task-related segmentation, URDF geometry, and point-cloud distances.This component-level representation supports subsequent editing of objects and robot configurations.Source: §IV · p. 3, A. Reconstruction and Preprocessing · p. 4

Camera alignment

  • RoboSplat uses differentiable rendering to estimate deployed camera poses by optimizing image similarity between expert and reconstructed Gaussian renderings.The objective uses SSIM between the expert image and the rendered image.Source: A. Reconstruction and Preprocessing · p. 5

Pose generation

  • Forward kinematics transforms link-wise Gaussians from the default robot configuration to arbitrary joint configurations, while manipulated objects receive analogous transformations.The transformed link Gaussians are composed to represent the robot under each new configuration.Source: A. Reconstruction and Preprocessing · p. 5

B. Novel Demonstration Generation

RoboSplat systematically augments one expert demonstration across six generalization axes using editable 3D Gaussians.

B. Novel Demonstration Generation

  • RoboSplat augments demonstrations across object poses, object types, camera views, embodiment types, scene appearance, and lighting conditions.The resulting augmented demonstrations are used to train visuomotor policies for deployment on real robots.Source: B. Novel Demonstration Generation · p. 5

1) Object Pose

RoboSplat generates object-pose, object-type, camera-view, and embodiment variations by transforming scene geometry and reusing or adapting action representations.

1) Object Pose

  • Object-pose augmentation extracts keyframes at gripper toggles or near-zero joint velocities, then applies rigid transformations to target objects.End-effector poses are transformed equivariantly with the objects.Source: 1) Object Pose · p. 5

2) Object Type

  • 3D content generation creates graspable object variants from generated object names and synthesizes corresponding grasp poses.Object-relative grasp poses are composed with object poses to produce keyframe poses for new demonstrations.Source: 2) Object Type · p. 5

3) Camera View

  • Novel-view demonstrations randomize camera positions in spherical coordinates around a target point constrained to remain within the manipulation scene.This preserves meaningful observations while introducing camera diversity.Source: 3) Camera View · p. 5

4) Embodiment Type

  • Robot-embodiment augmentation replaces the robot Gaussians, reuses embodiment-agnostic keyframe end-effector poses, and derives new joint trajectories through motion planning.Policies trained on these demonstrations are directly deployed on novel embodiments.Source: 4) Embodiment Type · p. 5

5) Scene Appearance

RoboSplat varies scene appearance and lighting directly in reconstructed 3D scenes, while the experiments evaluate robustness across multiple deployment settings.

5) Scene Appearance

  • Scene-appearance augmentation attaches images to table-top and background Gaussian planes, combining reconstructed 3D scenes with COCO images.Reconstructed 3D scenes provide consistent geometry across camera views, while 2D images increase appearance diversity.Source: 5) Scene Appearance · p. 5

6) Lighting Conditions

  • Lighting augmentation modifies each Gaussian’s diffuse color through random scaling, offsets, and noise.The method changes visual attributes while retaining the reconstructed scene representation.Source: 6) Lighting Condition · p. 6

Experiments

  • The experiments examine generated-data efficiency, policy performance, scaling with generated data, and robustness to varied deployment settings.Evaluated disturbances include object types, camera views, scene appearance, lighting conditions, and embodiment types.Source: §V · p. 6

A. Experimental Setup

The real-world evaluation uses five manipulation tasks and measures success across varied deployment settings. The pipeline is also assessed for demonstration-generation efficiency.

Hardware and collection

  • Expert demonstrations are collected on a Franka Research 3 robot using two eye-on-base RGB cameras and 10 Hz SpaceMouse teleoperation.Policy inference runs on an NVIDIA RTX4090 GPU with 0.1 s latency imposed.Source: A. Experimental Setup · p. 6

Tasks

  • The experiments use Pick Object, Close Drawer, Pick-Place-Close, Dual Pick-Place, and Sweep tasks.Together, these tasks cover pick-and-place, articulated-object manipulation, long-horizon behavior, tool use, and functional motion.Source: A. Experimental Setup · pp. 6–7, Figure 6 · p. 7

Evaluation protocol

  • Each policy is evaluated with success rate over 30 trials for each setting.Object-pose generalization is included in all experiments, alongside object types, camera views, embodiments, lighting, and scene appearance.Source: A. Experimental Setup · p. 7

Data generation

  • The pipeline generates demonstrations automatically using eight parallel processes on an NVIDIA RTX4090 GPU.The comparison covers all five tasks and contrasts generated demonstrations with manually collected real-world demonstrations.Source: B. Efficiency of Augmenting… · p. 7
  • 0.64s is the average generation time per demonstration, making the pipeline more than 29 times faster than real-world collection.The reported average is measured across all five tasks.Source: B. Efficiency of Augmenting… · p. 7

C. Performance of the Policy Trained on Augmented Data

RoboSplat improves policy robustness across object poses and multiple deployment changes, while performance scales as generated demonstrations increase. Its advantages are reported for lighting, appearance, camera views, object types, and robot embodiments.

Object-pose generalization

  • 800 generated demonstrations achieve comparable performance to 200 manually collected demonstrations, while 1800 generated demonstrations reach 94.7% average success rate.The Dual Pick-Place task reaches 96.7% with generated data, nearly 20% above the manually collected baseline.Source: C. Performance of the Policy Trained… · p. 7

Lighting conditions

  • 3200 demonstrations with lighting and object-pose augmentation achieve over 80% average success across three tasks and improve over unaugmented real-world data by 70%.The policies also outperform pose-only generated data and color-jitter augmentation.Source: D. Robustness when Facing Various… · p. 7

Scene appearance

  • Appearance and object-pose augmentation produces over a 70% success-rate increase across three tasks compared with data without appearance augmentation.Pick Object reaches 100% success under varied background appearance.Source: D. Robustness when Facing Various… · p. 8

Camera view

Object type

Embodiment type

VI. LIMITATIONS

RoboSplat enhances visuomotor-policy robustness across diverse disturbances, but naive 3D Gaussian Splatting cannot handle deformable objects and lacks physical constraints for contact-rich and dynamic tasks.

  • Naive 3D Gaussian Splatting cannot handle deformable objects.Source: §VI · p. 10
  • The pipeline lacks physical constraints, limiting its suitability for contact-rich and dynamic tasks.Source: §VI · p. 10
  • RoboSplat significantly enhances visuomotor-policy robustness against various disturbances using a single collected demonstration.Source: §VII · p. 10

APPENDIX

The appendix describes Gaussian transformations, demonstration augmentation for object pose and camera view, and lighting-condition parameterization used in the augmentation process.

  • Gaussian transformations update positions, orientations, scales, and Spherical Harmonics coefficients.Positions are scaled, rotated, and translated; orientations combine rotations; scales multiply by the scaling factor; SH coefficients use Wigner D matrices.Source: A. Applying Transformation and Scaling… · p. 14
  • The appendix presents augmented demonstrations spanning object pose, lighting condition, scene appearance, object type, camera view, and embodiment type.Source: B. Details of Demonstration… · p. 14

Object pose

Camera view

Lighting condition

  • Lighting augmentation normalizes pixel RGB values to the range 0 to 1 before sampling lighting hyperparameters.Color-scale parameters are sampled from Uniform(0.3, 1.8), while color-offset parameters are sampled from Uniform(-0.3, 0.3).Source: B. Details of Demonstration… · p. 14

C. Policy Architecture

The policy combines image and robot-state inputs through modality-specific encoders, a GPT-2-style multimodal transformer, and action decoders that predict ten-step action chunks.

  • Images use a ResNet-18 encoder and linear layer, while robot states use a multilayer perceptron to produce modality-specific tokens.Source: C. Policy Architecture · p. 14
  • A GPT-2-style multimodal transformer uses readout tokens to attend across image and state embeddings as action latents.Source: C. Policy Architecture · p. 14
  • The action decoder predicts chunks of 10 future actions to provide temporal action consistency and robustness to idle actions.Source: C. Policy Architecture · p. 14

D. Training Details

Training uses paired camera images and robot states, with task-specific real-world experiments and documented implementation hyperparameters.

  • Each timestep uses two eye-on-base camera images and robot state containing arm and binary gripper states.The arm state is 7-dimensional for Franka FR3 and 6-dimensional for UR5e.Source: D. Training Details · p. 14
  • The policy uses history length 1 and an action chunk size of 10.Source: D. Training Details · p. 14
  • Augmented demonstrations cover object pose, lighting condition, scene appearance, object type, camera view, and embodiment type.Source: Figure 12 · p. 15
  • Training uses a batch size of 256, learning rate 1e-4, and one NVIDIA RTX 4090 GPU.The number of training epochs varies with the number of demonstrations.Source: D. Training Details · p. 15

Lighting generalization

Appearance generalization

Related papers

Loading 2504.13175v1…