R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation

I. INTRODUCTION

• Spatial generalization is a fundamental data-efficiency bottleneck because skills are harder to learn than free-space motions, while demonstrations often devote effort to motion and must cover varying object layouts and robot-base positions.For a task such as putting an apple on a plate, the paper notes that hundreds of demonstrations may be needed despite identical objects and pick-and-place skills.Source: §I · pp. 1–2

II. RELATED WORK

• Prior generators reduce recollection effort, but simulator-based approaches require on-robot rollouts, while DemoGen is limited by fixed-base and camera settings, input assumptions, and visual mismatch.The stated DemoGen assumptions include cropped environment point clouds, limited object counts, and one target object per skill.Source: §I · p. 2

I. INTRODUCTION

• R2RGen converts observations and actions into a shared 3D space, applies group-wise augmentation with backtracking, and uses camera-aware processing to match sensor pointcloud distributions.The framework is described as supporting mobile robots, different camera setups, raw pointclouds, and varied numbers of objects and interaction modes.Source: §I · p. 2, A. Problem Statement · p. 3

A. Problem Statement

• The formal objective is to generate spatially diverse data from one human-collected demonstration and train a generalized policy directly from those generated samples.The proposed policy takes pointclouds as visual input.Source: A. Problem Statement · p. 3

B. Source Demonstration Pre-processing

• Scene parsing separates object, environment, and robot-arm pointclouds, and completes object pointclouds so novel viewpoints are not limited to the camera-visible surfaces.For wrist-camera data, the object pointclouds are transformed from the wrist frame into the world coordinate system.Source: Figure 2 · p. 3, B. Source Demonstration Pre-processing · p. 3
• The pre-processing setup also acquires a complete environment pointcloud and measures the arm-base pose used to generate the robot-arm pointcloud.The environment is scanned in the world coordinate system with relevant objects removed, and the robot-arm pointcloud is ignored at this stage.Source: B. Source Demonstration Pre-processing · pp. 3–4
• Trajectory parsing divides actions into interleaved motion and skill segments and records the objects associated with each skill.The annotation specifies target and in-hand object IDs, which may be null, singular, or multiple.Source: B. Source Demonstration Pre-processing · pp. 3–4
• The annotation system uses RGB video to label skill boundaries and object IDs, taking less than 60 seconds per demonstration.Frames between skill segments are classified as motion segments.Source: B. Source Demonstration Pre-processing · p. 4

C. Group-wise Data Augmentation

• Group-wise augmentation preserves structural constraints among multiple objects, avoiding the broken spatial relationships that can result when objects are transformed independently.The paper’s bridge example requires the two piers to remain at a specific relative distance for the deck-placement skill.Source: C. Group-wise Data Augmentation · p. 4
• Backtracking assigns each skill a group containing its target and in-hand objects, applies the same geometric transformation to that group, and traverses skills in reverse order to avoid causal conflicts.The fixed-object set determines whether a group can be augmented; after updating the set, the in-hand object is released because its earlier state is independent of the current skill’s constraints.Source: C. Group-wise Data Augmentation · pp. 4–5
• For an unfixed skill group, the method transforms the end-effector pose and the associated object pointclouds together.The gripper state remains unchanged during skill augmentation.Source: C. Group-wise Data Augmentation · p. 5
• Motion planning connects adjacent skills, and the augmented object, environment, and arm pointclouds form the generated scene representation.For bimanual manipulation, additional constraints are introduced to keep generated demonstrations executable.Source: C. Group-wise Data Augmentation · p. 5
• Base augmentation changes the robot base position and orientation, then uses inverse kinematics to generate the corresponding complete arm pointcloud.The robot geometry is converted from its URDF model using the solved joint angles and transformed into the world coordinate system.Source: C. Group-wise Data Augmentation · p. 5

D. Camera-aware 3D Post-processing

• The pipeline transforms the augmented cloud to the deployment camera frame, projects it to the image plane, crops out-of-frame pixels, depth-filters overlaps, fills empty pixels, and back-projects it.The Z-buffer keeps the nearest point; patch-wise filtering lets nearby front-surface points hide deeper points.Source: D. Camera-aware 3D Post-processing · p. 5
• Augmented clouds can be over-complete for a given camera viewpoint and spatially shifted by robot-base augmentation, unlike raw RGB-D observations.The raw point cloud is complete only from its capture viewpoint.Source: D. Camera-aware 3D Post-processing · p. 5
• The adjusted point cloud matches the RGB-D camera’s distribution and can be fed directly to the 3D policy during training.Source: D. Camera-aware 3D Post-processing · p. 5

IV. EXPERIMENT

• The experiments evaluate R2RGen for one-shot imitation learning and scaling with additional source demonstrations.Source: §IV · p. 5
• The authors conduct ablation studies to explore design choices.Source: §IV · p. 5
• The experiments also show extensions to appearance generalization and mobile manipulation.Source: §IV · p. 5

A. Experimental Setup

• Experiments used iDP3, a 7-DoF UR5 with mobile base, and a bimanual MobileAloha platform across exterior-exterior, exterior-wrist, and wrist-wrist setups.Source: A. Experimental Setup · pp. 5–6

B. Results: One-shot Imitation Learning

• With one human demonstration, R2RGen outperforms DemoGen across tasks, matches policies trained with 25 human demonstrations, and surpasses 40 on several difficult tasks.Source: Table I · p. 6, B. Results: One-shot Imitation Learning · p. 6
• For exterior-wrist and wrist-wrist settings, R2RGen works well, while naive exterior-to-wrist deployment catastrophically fails.Source: Table II · p. 6, B. Results: One-shot Imitation Learning · p. 6

A. Experimental Setup

• The eight evaluation tasks span simple, complex, and bimanual manipulation; testing varied object poses and robot viewpoints, including out-of-distribution samples.Source: A. Experimental Setup · p. 6

C. Results: Performance-annotation Tradeoff

• Success rate gradually saturates as generated demonstrations increase, while more source demonstrations yield a higher saturation performance.Source: C. Results: Performance-annotation… · pp. 6–7, Figure 4 · p. 7

D. Ablation Study

• Removing pointcloud completion produced unrealistic data under large spatial augmentation, while removing environment operations reduced viewpoint robustness.The ablation analysis is conducted on Place-Bottle.Source: D. Ablation Study · p. 7
• The camera-aware processing operations played a critical role in final performance; the paper also compares shrinking and expanding for Fill.The two Fill choices are described in Appendix A5.Source: D. Ablation Study · p. 7

E. Extension and Application

• R2RGen can extend to articulated and deformable objects by replacing rigid-object FoundationPose with suitable vision foundation models, including SAM 3D.The authors designed two tasks to validate the framework on articulated and deformable objects.Source: E. Extension and Application · p. 7, Table V · p. 7
• 43.8% success rate with four R2RGen demonstrations contrasts with 25% using 40 human demonstrations on the combined appearance-and-spatial generalization task.The R2RGen setting uses one demonstration per bottle-base pair.Source: Figure 5 · p. 8, E. Extension and Application · p. 8
• iDP3 trained with R2RGen generalized across docking points more than 5 cm apart, using raw RGB-D observations without calibrated environment cropping.Figure 6 visualizes generalization to different camera views.Source: Figure 6 · p. 8, E. Extension and Application · p. 8

V. CONCLUDING REMARK

• The conclusion reports experiments across multiple real-world tasks and extensions to foundation models, appearance generalization, and mobile manipulation.It characterizes these extensions as evidence of generalizability and scalability for broad real-world deployment.Source: §V · p. 8

1) Annotation System:

• The source setup captures an empty-scene RGB-D image for the environment and separately scans each object to obtain its 3D template.The robot base is fixed during task execution.Source: 1) Annotation System · p. 11
• Users label objects in the initial frame, mark skill-segment key frames, and annotate segment boundaries and target and in-hand object IDs.The annotation UI provides Play, Stop, and Rollback controls for selecting key frames.Source: 1) Annotation System · p. 11, Figure 7 · p. 12
• FoundationPose tracks each object across the video and uses its estimated 6-DoF pose to transform the object template into the world coordinate system.The tracking input includes object masks, the RGB-D video, and object templates.Source: 1) Annotation System · p. 11

2) Pipeline of Augmentation:

• The augmentation pipeline takes parsed motion and skill trajectories, scene pointclouds, base pose, target and in-hand objects, and transformations, and outputs augmented observations and actions.The scene pointcloud is divided into environment, robot-arm, and object components.Source: 2) Pipeline of Augmentation · p. 11
• For each object, the method backtracks skills involving it and cumulatively applies each skill’s spatial transformation to that skill and earlier timestamps.The environment pointcloud is static across frames, while the arm pointcloud depends on end-effector and base pose.Source: 2) Pipeline of Augmentation · p. 12

3) Bimanual Tasks:

• Bimanual annotation extends in-hand information to separate left- and right-hand fields, allowing single-arm and dual-arm operations to share the annotation scheme.The method does not segment skill and motion separately for each arm.Source: 3) Bimanual Tasks · p. 12
• The arm pointcloud is clustered into two parts for the two arms.The passage identifies one resulting pointcloud as the left-arm pointcloud.Source: 3) Bimanual Tasks · p. 12
• When both hands hold the same non-null object during a skill, both arm trajectories maintain a fixed spatial relationship; otherwise, each arm’s motion is interpolated individually.Source: 3) Bimanual Tasks · p. 12

4) Data Generation and Training:

• Each source demonstration is replayed three times, and each generated demonstration receives three perturbations of up to 1.5 cm translation and ±20° rotation.The total generated demonstrations are calculated as 3 × N × 3, where N is the number of augmented location-and-rotation combinations.Source: 4) Data Generation and Training · p. 12
• iDP3 uses observation, prediction, and execution horizons of 2, 16, and 8, respectively, runs at 5 Hz, and is trained for 6,000 epochs.Training uses one RTX 4090 GPU, batch size 64, and Adam; the selected checkpoint has the lowest validation loss.Source: 4) Data Generation and Training · p. 13

5) Design Choices of Fill Operation:

• Shrinking and expanding achieve comparable Fill performance, and the authors choose shrinking for operational simplicity and no additional processing requirements.When shrinking is used, raw RGB-D observations are also shrunk at deployment.Source: 5) Design Choices of Fill Operation · p. 13

B. Hardware Setup

• The setup includes a movable-base UR5e platform with a 7-DoF arm and a Mobile Aloha platform with four arms and a 14-dimensional overall action space.The UR5e uses a base- or wrist-mounted RGB-D camera; the mobile platform uses a head-mounted RGB-D camera.Source: B. Hardware Setup · p. 13, Figure 9 · p. 13

• Eight real-world tasks comprise six single-arm tasks and two bimanual tasks.The section introduces them as the task suite for evaluation.Source: 1) Task Definition · p. 13
• Single-arm tasks include opening a jar, placing a bottle, putting food in a pot and covering it, hanging cups, stacking bricks, and building a bridge.The task descriptions specify these object interactions and sequences.Source: 1) Task Definition · pp. 13–14
• The bridge task places two piers at a proper distance before setting a deck on them.Source: 1) Task Definition · p. 14
• Bimanual tasks include lifting a box with both grippers and storing a banana in a box while the other gripper lifts it.Source: 1) Task Definition · p. 14

2) Evaluation Protocol:

• Object positions are sampled from 32 workspace points, rotations from -20 to 20 degrees, and the robot base from three distinct locations.For multi-object tasks, location ranges are constrained to reduce combinations.Source: 2) Evaluation Protocol · p. 14
• Single-arm tasks use task- and object-specific location subsets, with separate ranges specified for Open-Jar through Build-Bridge.Source: 2) Evaluation Protocol · p. 14
• The dual-arm protocols likewise specify location subsets for the box in Grasp-Box and for the box and banana in Store-Item.Source: 2) Evaluation Protocol · p. 15
• Black crosses mark locations that may have appeared in training demonstrations, while red crosses mark locations excluded from training.Source: 2) Evaluation Protocol · p. 15

3) Auxiliary Tasks:

• Two auxiliary tasks test non-rigid-object handling: Store-Box uses an articulated cabinet, and Cover-Object uses deformable cloth.Store-Box involves placing a box inside the cabinet and closing its door; Cover-Object places cloth over a target object.Source: 3) Auxiliary Tasks · p. 15, Figure 12 · p. 15