PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning

Jiawei Zhang, Chang Liu, Litao Deng, Tao Zhang, Qin Jin

arXiv:2610.20336v1cs.RO

TL;DR

Real-to-sim pipelines can still require substantial human effort to turn reconstructed workspaces into reusable assets and executable robot trajectories. AutoRSR automates that conversion and generates simulation episodes from reference motions; across three tasks, its policies approach teleoperated-data baselines with substantially less human data-generation time.

  • Problem

    Turning reconstructed real workspaces into reusable simulation assets and executable robot data still involves human intervention in object preparation, coordinate alignment, and trajectory collection or specification.

    Source: §I · p. 1

  • Method

    AutoRSR reconstructs and registers a scanned workspace, extracts reusable objects, and uses reference motions to generate and augment executable simulation trajectories.

    Source: §I · p. 2

  • Results

    73.3–86.7% real-world success with 1K automatically generated episodes per task, within 10 percentage points of 150-demo baselines, with an order-of-magnitude reduction in human data-generation time.

    Source: §I · p. 2

  • Takeaways & Limitations

    Automatically reconstructed real workspaces can provide a practical intermediate representation for scalable robot learning.

    Source: §I · p. 2

Abstract

from arXiv · show

Scaling embodied robot learning requires large and diverse manipulation datasets, but collecting real-world demonstrations remains expensive in human effort and robot time. Simulation provides a scalable alternative, yet converting a real workspace into a robot-learning-ready environment still requires substantial manual effort for object extraction, asset processing, coordinate calibration, and trajectory specification. We present AutoRSR, a highly automated real-to-sim-to real framework that converts scanned real-world workspaces into reusable embodied simulation assets and scalable manipulation data. Given multi-view observations, robot states, and open-vocabulary semantic queries, AutoRSR reconstructs the workspace, extracts individual object assets, and automatically registers the reconstructed scene to the robot base frame at metric scale. Starting from a small number of reference motions, AutoRSR then generates executable manipulation trajectories and scales them through trajectory interpolation and domain randomization. After workspace scanning and semantic queries, the remaining pipeline requires no manual object annotation, scene-to-robot calibration, or robot teleoperation. Across three real-world manipulation tasks, policies trained on 1K automatically generated episodes per task achieve 73.3--86.7% real-world success rates, comparable to pi 0.5 trained from 150 human teleoperated demonstrations, while reducing the human time required for data generation by an order of magnitude. These results demonstrate that real-world observations can be converted into metrically aligned, executable, and scalable simulation data with substantially reduced human effort.

I. INTRODUCTION

AutoRSR addresses the human effort between real-scene reconstruction and executable, scalable robot-learning data. It automates asset extraction, metric scene-to-robot registration, and trajectory generation, and achieves real-robot performance near teleoperated-data baselines on three tasks.

I. INTRODUCTION

  • Reconstruction-based simulation still requires human work to prepare reusable objects and make scenes and trajectories executable for a robot.Existing workflows may require manual segmentation and object processing, coordinate calibration, teleoperation, or manually specified trajectories.Source: §I · p. 1
  • AutoRSR uses synchronized wrist-camera poses and robot states with open-vocabulary queries to reconstruct workspaces and extract objects without manually annotated masks.The camera’s rigid attachment and synchronized poses support automatic metric registration to the robot base frame.Source: §I · pp. 1–2
  • Reference demonstrations seed executable trajectories, which trajectory interpolation and domain randomization expand into diverse simulation episodes without further real-robot interaction.Human effort is concentrated on workspace scanning and semantic specification; object extraction, registration, trajectory generation, and augmentation are automated.Source: §I · p. 2
  • 73.3–86.7% real-world success with 1K generated episodes per task, within 10 percentage points of 150-demo baselines, while reducing data-generation time by an order of magnitude.Evaluated on three real-world manipulation tasks.Source: §I · p. 2
  • AutoRSR converts scanned workspaces into reusable, metrically aligned, robot-executable environments for manipulation data generation.The paper presents automated object extraction, scene-to-robot registration, and trajectory generation as contributions of the pipeline.Source: §I · p. 2

II. RELATED WORKS

Prior robot-data approaches draw on human demonstrations or generate data in simulation, while Real2Sim2Real methods reconstruct real environments for simulated data collection and real-world transfer. AutoRSR is situated against the data-collection and sim-to-real challenges described in these approaches.

A. Robot Data Collection Paradigms.

  • Human teleoperation is a prevalent source of manipulation data but requires extensive effort and time; alternative sources include human behavior recordings and handheld-gripper demonstrations.Simulation augmentation can transfer, perturb, replay, and compose a small number of demonstrations, while LLM-based methods plan behaviors and generate simulated data.Source: A. Robot Data Collection Paradigms · pp. 2–3

B. Real2Sim2Real Data Generation.

  • Real2Sim2Real methods reconstruct real environments, collect and augment data in simulation, and transfer trained policies back to the real world, but simulation still introduces a sim-to-real gap.3D Gaussian Splatting and collision meshes are used in digital twins for visual fidelity and physical interactions.Source: B. Real2Sim2Real Data Generation · p. 3

III. METHOD

AutoRSR turns scans, robot states, a demonstration video, and semantic queries into aligned, augmented simulation data. Its initialization propagates semantic labels across views and densifies sparse anchors while respecting semantic boundaries.

III. METHOD

  • Users provide scene-scanning data, a demonstration video, and open-vocabulary queries; the remaining reconstruction, alignment, trajectory retargeting, and augmentation pipeline runs automatically.Scanning combines a robot-mounted wrist camera with third-party cameras while recording robot states.Source: §III · p. 3

A. Initialization

  • Semantic masks from a reference frame are propagated across views, then transferred to sparse points by majority voting over valid projections.The candidate labels for each point are determined from visible views with depth-occlusion relationships considered.Source: A. Initialization · p. 3
  • Semantic-constrained anchor growing densifies sparse regions by adding midpoints only between semantically consistent neighbors, then re-voxelizing to remove duplicates.New anchors inherit their semantic label, reducing propagation across semantic boundaries.Source: A. Initialization · p. 3

B. Scene Reconstruction and Object Segmentation

The section uses object-aware anchors to represent the reconstruction as neural Gaussians, trained by rendering views and supervising appearance and semantics. Densification adds labeled anchors where more representation capacity is needed.

B. Scene Reconstruction and Object Segmentation

  • Object-aware anchors generate multiple neural Gaussians whose color, opacity, covariance, and rotation are predicted by separate MLPs.Each anchor includes a spatial position, learnable feature, instance label, local scale, and learnable offsets.Source: 1) Object-Aware Gaussian… · pp. 3–4
  • Rendered views provide RGB images, alpha maps, and semantic scores for a reconstruction objective that includes semantic supervision and configured auxiliary penalties.The semantic loss uses 2D object-ID masks while ignoring background label 0; auxiliary penalties can include scale, depth, normal, distortion, opacity, and background-semantic regularization.Source: 1) Object-Aware Gaussian… · p. 4
  • During densification, child-Gaussian screen-space gradients identify regions for new anchors, which inherit the label of the parent with the strongest gradient contribution.Child Gaussians from each anchor share its one-hot semantic vector, coupling object identity with local geometry generation.Source: 1) Object-Aware Gaussian… · p. 4

2) Object Extraction and Asset Generation:

The section extracts queried objects from reconstructed instance IDs and converts the resulting representations into explicit Gaussian assets and simulation geometry.

2) Object Extraction and Asset Generation:

  • A user query selects a target mask on a rendered view, which is matched to reconstructed instance IDs by overlap scoring against the semantic-ID map.The selected anchors yield object-only and scene-without-object representations; spatial neighborhoods are expanded during removal to suppress residual related Gaussians.Source: 2) Object Extraction and Asset… · p. 4
  • View-dependent anchor attributes are decoded across training cameras and aggregated into explicit 3DGS assets, with multi-view distillation used when needed to reduce conversion artifacts.Object export uses opacity-based cross-view consensus and median aggregation; scene export averages colors and opacities over accepted views.Source: 2) Object Extraction and Asset… · p. 4
  • SuGaR reconstructs meshes from explicit Gaussians, and isolated objects receive watertight collision meshes through support-face filtering, voxelization, and surface reconstruction.Articulate-Anymesh can further process meshes when articulated assets are required.Source: 2) Object Extraction and Asset… · p. 4

C. Demonstration Alignment and Motion Retargeting

The section registers reconstructed coordinates to the robot using calibrated wrist-camera poses, then retargets a demonstrated object motion into a feasible robot trajectory. Grasp candidates are guided by image-space hand cues, while achieved robot motion governs rigid-object transport and release.

1) Metric Coordinate Alignment:

  • Reconstructed and robot-derived camera poses determine a similarity transform through RANSAC and robust refinement using camera positions and orientations.Metric camera poses are computed from synchronized robot configurations, robot-base extrinsics, and hand–eye calibration; the same scale is applied to Gaussian assets and collision meshes.Source: 1) Metric Coordinate Alignment · p. 4
  • The demonstration is interpreted using the calibrated head-camera pose, rather than assigning it to an arbitrary scanning image.Object poses are mapped from reconstruction coordinates into metric robot coordinates.Source: 1) Metric Coordinate Alignment · p. 4

2) Object Tracking and Grasp Retargeting:

  • Object motion is recovered from the demonstration video by optimizing pose with silhouette, Dice, and robust foreground photometric losses, using temporal regularization and bounded image-centroid priors.Initialization stays near the reconstructed object pose, and each later frame starts from the preceding estimate.Source: 2) Object Tracking and Grasp Retargeting · p. 4
  • Grasp candidates vary in approach direction, gripper-closing direction, and position; projected grasp locations are ranked against the demonstrated thumb–index midpoint without treating monocular hand depth as metric ground truth.Candidate generation accounts for gripper opening range and object geometry.Source: 2) Object Tracking and Grasp Retargeting · pp. 4–5
  • The tracked object motion and selected local grasp pose determine end-effector motion, with joint-limit-constrained inverse kinematics and full-sequence position and orientation checks.During rigid-object transport, the object follows achieved end-effector motion while preserving the local grasp transform; at release, the achieved placement is retained. For articulated or pushing motions, prescribed object kinematics remain, with contact checked against achieved robot motion.Source: 2) Object Tracking and Grasp Retargeting · p. 5

D. Motion Trajectory and Visual Augmentation

The section expands feasible motion templates through object placement and geometric variation while adapting trajectories to preserve task-relative relationships. It screens candidates for geometric feasibility before replay.

D. Motion Trajectory and Visual Augmentation

  • Objects selected by semantic queries receive independently randomized tabletop placements and geometric modifications while preserving initial heights, validated grasp relationships, and object scales.Planar translations use a bounded distribution, and yaw rotations are applied about each object’s initial position rather than the world origin.Source: 1) Motion Trajectory Augmentation · p. 5
  • Task-relative adaptation retains the manipulated object’s sampled initial pose and adjusts its terminal pose to preserve the demonstrated relationship to the independently placed receiver.A smooth correction blends positions in world coordinates and interpolates relative orientation; a specified interaction interval can enforce the relationship throughout that interval.Source: 1) Motion Trajectory Augmentation · p. 5
  • Candidates expand into synchronized approach, gripper-closing, and task-motion sequences, with placement settling, opening, and retreat appended only when geometric support is confirmed.Required-placement candidates are rejected if terminal support cannot be established; settling is bounded contact adjustment, not randomized vertical placement.Source: 1) Motion Trajectory Augmentation · p. 5
  • Batch solving and replay screen grasp support, joint limits, pose residuals, finger contact, and excessive robot–object mesh penetration, but do not guarantee force closure or dynamic task success.Terminal support is rechecked after rigid objects are bound to achieved forward kinematics.Source: 1) Motion Trajectory Augmentation · p. 5

2) Visual Augmentation:

Visual augmentation expands the visual configurations of augmented motion templates while holding geometry, motion, actions, and timing fixed. It randomizes object and scene textures, Gaussian exposure and white balance, and robot-renderer lighting.

2) Visual Augmentation:

  • Visual augmentation varies appearance while keeping geometry, motion, actions, and timing unchanged.It combines many augmented motion templates with multiple visual configurations.Source: 2) Visual Augmentation · p. 5
  • Object and scene textures, Gaussian exposure and white balance, and robot-mesh-renderer lighting are varied.Source: 2) Visual Augmentation · p. 5
  • Rendering reuses runtime resources and cached Gaussian assets, batching Gaussian data and camera requests for compatible episodes.Robot meshes and Gaussian layers use consistent camera poses and are composited using depth.Source: 2) Visual Augmentation · p. 5
  • Head-camera observations, wrist-camera observations, proprioceptive states, actions, and gripper openings share aligned frame indices and timestamps.Wrist-camera poses are recomputed from robot joint configurations and hand–eye transformations.Source: 2) Visual Augmentation · p. 5

A. Setups

Across tasks, policies trained on 1,000 AutoRSR-generated episodes achieved success rates within 10 percentage points of policies trained on 150 teleoperated demonstrations. The evaluation also compared data mixtures and augmentation ablations.

A. Setups

  • The setup generated 1,000 augmented trajectories and collected 150 teleoperated demonstrations for each of three tasks, then fine-tuned π0.5 using LoRA.Fine-tuning ran for 30,000 steps with batch size 32 on each dataset.Source: A. Setups · p. 5

B. Manipulation

C. Alignment accuracy

The alignment evaluation compares the proposed aligned coordinate frame with the original COLMAP reconstruction against tabletop calibration-board ground truth. The paper also provides tables comparing pipeline human involvement and processing-time scopes.

C. Alignment accuracy

  • Calibration-board corners at ten random tabletop positions were measured in the aligned frame and the COLMAP frame after a Sim(3) transformation.Errors were computed relative to ground-truth coordinates in the real workspace.Source: C. Alignment accuracy · pp. 6–7
  • A stage-by-stage table compares human involvement across the pipeline; H marks required human involvement and A marks automated stages.GASE has blank demonstration and augmentation entries because it does not perform those stages.Source: Table VI · p. 7
  • The aligned coordinate frame achieved substantially lower mean error and RMSE than the original reconstruction, with estimated positions closely matching ground truth.Source: C. Alignment accuracy · p. 7, Table V · p. 7
  • The paper includes separate timing comparisons for upfront processing until generation and for collecting 150 teleoperated demonstrations versus generating 1,000 synthetic demonstrations.The latter comparison specifies one human operator for teleoperation and a single GPU for AutoRSR.Source: Table VII · p. 7, Table VIII · p. 7

D. Time cost and human involvment comparison

AutoRSR reduces the time and human involvement required for data generation while automating most pipeline stages. The paper also reports effective real-robot performance with a small real-to-sim gap.

D. Time cost and human involvment comparison

  • AutoRSR generally generates 1,000 augmented episodes in no more than half the time needed to collect 150 teleoperated demonstrations.Teleoperation requires continuous human involvement, while AutoRSR augmentation requires almost none.Source: D. Time cost and human involvment… · p. 7
  • Users provide only scene scans, demonstration data, and open-vocabulary semantic queries; the remaining stages are fully automatic.Other systems require substantial human intervention throughout the pipeline.Source: D. Time cost and human involvment… · pp. 7–8
  • AutoRSR’s end-to-end processing time is broken down for the task Pour into the yellow bowl.The passage identifies Tab.VII as presenting this breakdown but gives no processing-time values.Source: D. Time cost and human involvment… · p. 8

V. CONCLUSION

  • Real-robot experiments validate AutoRSR’s effectiveness and demonstrate a small real-to-sim gap.Source: §V · p. 8

Related papers

Loading 2610.20336v1…