PaperlayerView exact revision on arXiv

Source-linked AI summary

Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulation

Sizhe Yang, Wenye Yu, Jia Zeng, Jun Lv, Kerui Ren, Cewu Lu, Dahua Lin, Jiangmiao Pang

arXiv:2504.13175v1cs.RO

TL;DR

Visuomotor imitation policies need broader visual coverage, but real-world demonstration collection is costly and existing 2D or simulation-based augmentation has important constraints. RoboSplat reconstructs scenes with 3DGS and edits them to generate diverse demonstrations across six generalization types. In real-world experiments, it reaches 87.8% average success versus 57.2% for policies trained on hundreds of demonstrations with additional 2D augmentation.

  • Problem

    Imitation policies often lack robustness under visual conditions differing from expert demonstrations, while collecting diverse human demonstrations is time-consuming and labor-intensive.

  • Method

    RoboSplat reconstructs manipulation scenes with 3DGS and directly edits Gaussian representations to generate demonstrations across six generalization types.

  • Results

    87.8% average success versus 57.2% for policies trained on hundreds of real-world demonstrations with additional 2D augmentation.

  • Takeaways & Limitations

    RoboSplat generates diverse, high-quality data from a single collected demonstration and enhances visuomotor-policy robustness under varied real-world disturbances.

  • Takeaways & Limitations

    Naive 3DGS cannot handle deformable objects and lacks physical constraints for contact-rich and dynamic tasks.

Abstract

from arXiv · show

Visuomotor policies learned from teleoperated demonstrations face challenges such as lengthy data collection, high costs, and limited data diversity. Existing approaches address these issues by augmenting image observations in RGB space or employing Real-to-Sim-to-Real pipelines based on physical simulators. However, the former is constrained to 2D data augmentation, while the latter suffers from imprecise physical simulation caused by inaccurate geometric reconstruction. This paper introduces RoboSplat, a novel method that generates diverse, visually realistic demonstrations by directly manipulating 3D Gaussians. Specifically, we reconstruct the scene through 3D Gaussian Splatting (3DGS), directly edit the reconstructed scene, and augment data across six types of generalization with five techniques: 3D Gaussian replacement for varying object types, scene appearance, and robot embodiments; equivariant transformations for different object poses; visual attribute editing for various lighting conditions; novel view synthesis for new camera perspectives; and 3D content generation for diverse object types. Comprehensive real-world experiments demonstrate that RoboSplat significantly enhances the generalization of visuomotor policies under diverse disturbances. Notably, while policies trained on hundreds of real-world demonstrations with additional 2D data augmentation achieve an average success rate of 57.2%, RoboSplat attains 87.8% in one-shot settings across six types of generalization in the real world.

I. INTRODUCTION

RoboSplat addresses limited robustness and costly data collection in visuomotor imitation by editing reconstructed 3D scenes to generate diverse demonstrations from a single trajectory. Real-world experiments report improved robustness across varied disturbances and generalization settings.

  • Imitation policies often lack robustness because demonstrations insufficiently cover visual domains, while collecting more human demonstrations is time-consuming and labor-intensive.
  • Real-to-Sim-to-Real methods face inaccurate geometric reconstruction and monotonous scenes, whereas image augmentation remains primarily limited to 2D observations.
  • 3DGS preserves spatial information, supports consistent multi-view rendering, and enables interpretable editing of reconstructed scenes for novel manipulation configurations.
  • RoboSplat uses 3D Gaussian replacement, equivariant transformations, visual attribute editing, novel view synthesis, and 3D content generation across six generalization types.
  • 87.8% average success versus 57.2% for policies trained on hundreds of real-world demonstrations with additional 2D augmentation demonstrates stronger real-world robustness.

III. PRELIMINARIES

RoboSplat reconstructs manipulation scenes as 3D Gaussians, aligns them to robot coordinates through differentiable rendering, and prepares editable components for demonstration generation.

  • Reconstruction: 3D Gaussian Splatting reconstructs the entire scene from multi-view images, camera poses, and depth estimates.The reconstruction includes robot Gaussians that initially reside in an arbitrary scene frame.
  • Frame alignment: Differentiable rendering refines ICP alignment by optimizing transformations against corresponding URDF and Gaussian-rendered robot masks.The optimized transformation is applied to the reconstructed scene to place it in the real-world coordinate frame.
  • Frame alignment: The alignment objective compares masks rendered from Gaussian Splatting and URDF across canonical camera views.Gradient descent optimizes the relative transformation, including translation, rotation, and scale, before applying it to the Gaussians.
  • Scene decomposition: RoboSplat decomposes the scene into object and robot-link Gaussians using task-related segmentation, URDF geometry, and point-cloud distances.This component-level representation supports subsequent editing of objects and robot configurations.
  • Camera alignment: RoboSplat uses differentiable rendering to estimate deployed camera poses by optimizing image similarity between expert and reconstructed Gaussian renderings.The objective uses SSIM between the expert image and the rendered image.
  • Pose generation: Forward kinematics transforms link-wise Gaussians from the default robot configuration to arbitrary joint configurations, while manipulated objects receive analogous transformations.The transformed link Gaussians are composed to represent the robot under each new configuration.

B. Novel Demonstration Generation

RoboSplat systematically augments one expert demonstration across six generalization axes using editable 3D Gaussians.

  • B. Novel Demonstration Generation: RoboSplat augments demonstrations across object poses, object types, camera views, embodiment types, scene appearance, and lighting conditions.The resulting augmented demonstrations are used to train visuomotor policies for deployment on real robots.

1) Object Pose

RoboSplat generates object-pose, object-type, camera-view, and embodiment variations by transforming scene geometry and reusing or adapting action representations.

  • 1) Object Pose: Object-pose augmentation extracts keyframes at gripper toggles or near-zero joint velocities, then applies rigid transformations to target objects.End-effector poses are transformed equivariantly with the objects.
  • 2) Object Type: 3D content generation creates graspable object variants from generated object names and synthesizes corresponding grasp poses.Object-relative grasp poses are composed with object poses to produce keyframe poses for new demonstrations.
  • 3) Camera View: Novel-view demonstrations randomize camera positions in spherical coordinates around a target point constrained to remain within the manipulation scene.This preserves meaningful observations while introducing camera diversity.
  • 4) Embodiment Type: Robot-embodiment augmentation replaces the robot Gaussians, reuses embodiment-agnostic keyframe end-effector poses, and derives new joint trajectories through motion planning.Policies trained on these demonstrations are directly deployed on novel embodiments.

5) Scene Appearance

RoboSplat varies scene appearance and lighting directly in reconstructed 3D scenes, while the experiments evaluate robustness across multiple deployment settings.

  • 5) Scene Appearance: Scene-appearance augmentation attaches images to table-top and background Gaussian planes, combining reconstructed 3D scenes with COCO images.Reconstructed 3D scenes provide consistent geometry across camera views, while 2D images increase appearance diversity.
  • 6) Lighting Conditions: Lighting augmentation modifies each Gaussian’s diffuse color through random scaling, offsets, and noise.The method changes visual attributes while retaining the reconstructed scene representation.
  • Experiments: The experiments examine generated-data efficiency, policy performance, scaling with generated data, and robustness to varied deployment settings.Evaluated disturbances include object types, camera views, scene appearance, lighting conditions, and embodiment types.

A. Experimental Setup

The real-world evaluation uses five manipulation tasks and measures success across varied deployment settings. The pipeline is also assessed for demonstration-generation efficiency.

  • Hardware and collection: Expert demonstrations are collected on a Franka Research 3 robot using two eye-on-base RGB cameras and 10 Hz SpaceMouse teleoperation.Policy inference runs on an NVIDIA RTX4090 GPU with 0.1 s latency imposed.
  • Tasks: The experiments use Pick Object, Close Drawer, Pick-Place-Close, Dual Pick-Place, and Sweep tasks.Together, these tasks cover pick-and-place, articulated-object manipulation, long-horizon behavior, tool use, and functional motion.
  • Evaluation protocol: Each policy is evaluated with success rate over 30 trials for each setting.Object-pose generalization is included in all experiments, alongside object types, camera views, embodiments, lighting, and scene appearance.
  • Data generation: The pipeline generates demonstrations automatically using eight parallel processes on an NVIDIA RTX4090 GPU.The comparison covers all five tasks and contrasts generated demonstrations with manually collected real-world demonstrations.
  • Data generation: 0.64s is the average generation time per demonstration, making the pipeline more than 29 times faster than real-world collection.The reported average is measured across all five tasks.

C. Performance of the Policy Trained on Augmented Data

RoboSplat improves policy robustness across object poses and multiple deployment changes, while performance scales as generated demonstrations increase. Its advantages are reported for lighting, appearance, camera views, object types, and robot embodiments.

  • Object-pose generalization: 800 generated demonstrations achieve comparable performance to 200 manually collected demonstrations, while 1800 generated demonstrations reach 94.7% average success rate.The Dual Pick-Place task reaches 96.7% with generated data, nearly 20% above the manually collected baseline.
  • Lighting conditions: 3200 demonstrations with lighting and object-pose augmentation achieve over 80% average success across three tasks and improve over unaugmented real-world data by 70%.The policies also outperform pose-only generated data and color-jitter augmentation.
  • Scene appearance: Appearance and object-pose augmentation produces over a 70% success-rate increase across three tasks compared with data without appearance augmentation.Pick Object reaches 100% success under varied background appearance.
  • Camera view: Under novel and moving camera views, the method exceeds 80% success on Pick Object, exceeds 50% on Pick-Place-Close, and approaches 100% on Close Drawer.The method outperforms policies without augmentation and VISTA.
  • Object type: Training on 50 object types improves novel-object success over baseline models by over 40%.The deployment objects differ from all objects used during training.
  • Embodiment type: Policies generated from one Franka Research 3 demonstration achieve a success rate close to 100% when deployed on a UR5e robot.The result surpasses the baseline in cross-embodiment transfer.

VI. LIMITATIONS

RoboSplat enhances visuomotor-policy robustness across diverse disturbances, but naive 3D Gaussian Splatting cannot handle deformable objects and lacks physical constraints for contact-rich and dynamic tasks.

  • Naive 3D Gaussian Splatting cannot handle deformable objects.
  • The pipeline lacks physical constraints, limiting its suitability for contact-rich and dynamic tasks.
  • RoboSplat significantly enhances visuomotor-policy robustness against various disturbances using a single collected demonstration.

APPENDIX

The appendix describes Gaussian transformations, demonstration augmentation for object pose and camera view, and lighting-condition parameterization used in the augmentation process.

  • Gaussian transformations update positions, orientations, scales, and Spherical Harmonics coefficients.Positions are scaled, rotated, and translated; orientations combine rotations; scales multiply by the scaling factor; SH coefficients use Wigner D matrices.
  • The appendix presents augmented demonstrations spanning object pose, lighting condition, scene appearance, object type, camera view, and embodiment type.
  • Object pose: Object-pose augmentation transforms key-frame end-effector poses equivariantly with the target object and removes redundant gripper-axis rotation.
  • Camera view: Camera-view augmentation samples target-point coordinates and spherical coordinates from uniform distributions.
  • Lighting condition: Lighting augmentation normalizes pixel RGB values to the range 0 to 1 before sampling lighting hyperparameters.Color-scale parameters are sampled from Uniform(0.3, 1.8), while color-offset parameters are sampled from Uniform(-0.3, 0.3).

C. Policy Architecture

The policy combines image and robot-state inputs through modality-specific encoders, a GPT-2-style multimodal transformer, and action decoders that predict ten-step action chunks.

  • Images use a ResNet-18 encoder and linear layer, while robot states use a multilayer perceptron to produce modality-specific tokens.
  • A GPT-2-style multimodal transformer uses readout tokens to attend across image and state embeddings as action latents.
  • The action decoder predicts chunks of 10 future actions to provide temporal action consistency and robustness to idle actions.

D. Training Details

Training uses paired camera images and robot states, with task-specific real-world experiments and documented implementation hyperparameters.

  • Each timestep uses two eye-on-base camera images and robot state containing arm and binary gripper states.The arm state is 7-dimensional for Franka FR3 and 6-dimensional for UR5e.
  • The policy uses history length 1 and an action chunk size of 10.
  • Augmented demonstrations cover object pose, lighting condition, scene appearance, object type, camera view, and embodiment type.
  • Training uses a batch size of 256, learning rate 1e-4, and one NVIDIA RTX 4090 GPU.The number of training epochs varies with the number of demonstrations.
  • Lighting generalization: Real-world lighting experiments alternate red and blue flashing light at 4Hz, with 6 trials per lighting condition.
  • Appearance generalization: Real-world appearance experiments use 5 trials per scenario.
Loading 2504.13175v1…