PaperlayerView exact revision on arXiv

Source-linked AI summary

Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration

Matthew Strong, Ankush K. Dhawan, Julien Poffet, Giuse Pham, Jinho So, Monroe Kennedy III

arXiv:2610.28880v1cs.ROcs.CV

TL;DR

Robot learning from human video is constrained by methods that require depth or robot overlays, or generate data only for demonstrated objects. R2Sem2R turns one human RGB demonstration into robot demonstrations across digital cousins through semantic alignment and simulation. Its policies match or exceed teleoperation across tasks and robots, while the method still requires metric object measurements.

  • Problem

    Human-video robot learning methods may require depth sensing or robot overlays, and some generate data only for demonstrated objects.

    Source: §I · p. 1

  • Method

    R2Sem2R converts one human RGB video into robot demonstrations across digital cousins by aligning whole objects and functional regions and generating episodes in simulation.

    Source: §I · p. 2, C. Semantic Augmentation · p. 5

  • Results

    On every task and robot, R2Sem2R leads R2R2R+ by 26–54 points and matches or exceeds teleoperation.

    Source: C. Real-Robot Results · p. 7

  • Takeaways & Limitations

    R2Sem2R data supports policies that perform robustly in the real world across many cousins, matching or exceeding teleoperation.

    Source: §V · p. 8

    R2Sem2R requires metric measurements of demonstrated objects, and it handles rigid objects only.

    Source: §V · p. 8

Abstract

from arXiv · show

Human video scales easily and does not need robot hardware, but existing methods require depth sensing, overlay the robot onto 2D video, or generate data only for the demonstrated objects. We present Real2Sem2Real (R2Sem2R), a real-to-semantic-to-real pipeline that turns one monocular RGB video of a human demonstration into thousands of robot demonstrations across the semantic object instances of a category. R2Sem2R tracks the SE(3) pose of each object with DINOv3-distilled Gaussian splats, outperforming existing methods and FoundationPose with estimated metric depth. R2Sem2R then aligns digital cousins, each generated from one or a few photographs, to the demonstrated objects with a semantic alignment. This alignment matches whole objects and the functional regions of target objects, such as a coffee machine's tray. R2Sem2R maps the human hand to a feasible robot grasp that extends to the cousins. Robot demonstrations are generated either in IsaacSim, for photorealistic rendering, or MuJoCo, for contact physics, and R2Sem2R can generate over 2,000 episodes in about two hours. On two real robots and four tasks, we train a vision-language-action policy on data from one human video per task, and policies trained on our data match or exceed policies trained on teleoperation across diverse object instances. The project page is available at https://real2sem2real.github.io/.

I. INTRODUCTION

R2Sem2R addresses the data burden of robot learning by turning one human RGB demonstration into robot data across object instances. It combines object tracking, semantic alignment, hand-to-gripper retargeting, and parallel simulation, with policies matching or exceeding teleoperation.

I. INTRODUCTION

  • Robot learning is data hungry: trajectories require human time, and new object instances require collection on those instances.The introduction notes that hundreds of demonstrations for one object do not generalize to other objects of the same type.Source: §I · p. 1
  • Prior alternatives leave gaps: some need privileged state or accurate 3D observations, while others require hardware, robot overlays, or data limited to demonstrated objects.The introduction describes these constraints across augmentation, simulation, grippers, and human-video approaches.Source: §I · p. 1
  • Parallel generation in IsaacSim or MuJoCo produces over 2,000 episodes in about two hours, after about 25 minutes of setup.An interactive viewer supports scene creation that ports directly to simulation.Source: §I · p. 2
  • R2Sem2R tracks objects with DINO-distilled Gaussian splats and semantically aligns digital cousins, including functional regions for placement and motion.The alignment supports demonstrations across cousin pairings, beyond single-object pick-and-place.Source: §I · p. 2
  • Policies trained from one human video per task match or outperform two hours of teleoperation on nearly 2,000 real rollouts across two robots.The contributions summary reports the rollout and policy comparison.Source: §I · p. 2

II. RELATED WORK

Prior work addresses human-video learning, object tracking, and demonstration generation, but the cited approaches retain limitations in hardware requirements, object coverage, or transfer across object instances. R2Sem2R is presented as combining monocular RGB input with transfer across both grasped and target objects.

Learning from human video

  • Human-video methods either require motion-capture or depth hardware, overlay a robot onto 2D video, or generate data only for demonstrated objects.Phantom needs 250–350 videos per task and WARPED needs 30, while R2Sem2R starts from one.Source: §II · p. 2

Object pose tracking

  • RGB-D trackers use depth sensing, while monocular substitutes depend on estimated depth; RGB-only alternatives can fail on textureless objects or start from generated models.R2Sem2R instead tracks a splat distilled from a scan of the real object, initialized upright on the table plane.Source: §II · p. 2

Demonstration generation

  • Demonstration-generation methods often need teleoperation, privileged state, or depth, and prior RGB replay methods vary object placement rather than object identity.R2R2R and its followers widen spatial augmentation but do not vary which objects are demonstrated.Source: §II · p. 2

Digital cousins and category-level generation

  • Digital-cousin systems use trajectories from teleoperation, scripted skills, or planners, and overall-geometry registration does not locate where a placed object should rest.Other transfers cited need RGB-D inputs, annotated contacts, keypoints, or learned warping, and some vary only the manipulated object.Source: §II · pp. 2–3
  • R2Sem2R uses one monocular RGB human video without teleoperation, annotation, or trained warping, and varies both grasped and target objects.Distilled DINOv3 features align whole objects and the demonstrated contact region to cousins.Source: §II · p. 3

III. METHOD

R2Sem2R takes a human RGB task video, a phone scan of demonstrated objects, and photographs of cousin objects, then computes object-frame trajectories and valid grasps for simulation and robot use.

III. METHOD

  • The pipeline takes one RGB video, a phone scan of demonstrated objects, and photographs of cousin objects.The video supplies the demonstration from which trajectories and grasps are computed.Source: §III · p. 3
  • From one demonstration, R2Sem2R computes object trajectories and valid grasps in the object’s frame.Source: §III · p. 3
  • Inverse kinematics can use these outputs for any manipulator with a parallel-jaw gripper.Source: §III · p. 3

A. Human Demonstration Parsing

The method builds metric object splats from phone scans and tracks their poses through video with render-and-compare losses. It then retargets the human hand to a collision-free parallel-jaw grasp selected from the carry phase.

A. Human Demonstration Parsing

  • Phone scans produce segmented Gaussian splats with distilled DINOv3 features, and measured object height scales each splat to metric dimensions.The table plane is fitted from the same scan; mask membership across views filters the object Gaussians.Source: A. Human Demonstration Parsing · p. 3
  • Object pose is optimized by rendering the splat and combining feature, RGB, silhouette, and relative-depth-ranking losses.The loss weights are optimized over poses in SE(3).Source: A. Human Demonstration Parsing · p. 3
  • Tracking initializes upright orientations from the table normal, refines and ranks them, and excludes hand pixels when occlusion distorts the silhouette.Later frames start from the prior pose with light smoothing.Source: A. Human Demonstration Parsing · p. 3

B. Hand-to-Gripper Retargeting

  • The gripper pose is stored in the object frame: fingertip midpoint sets the tool center point, fingertip separation sets the closing axis, and wrist-to-midpoint sets approach.Source: B. Hand-to-Gripper Retargeting · p. 3
  • The best grasp is selected from carry frames, where the hand pinches and the object is lifted over 2 cm, by searching nearby perturbations for a collision-free antipodal grasp.Candidates are checked against gripper, object, and table collisions; the final choice across frames favors opposed gripper and object-mesh normals.Source: B. Hand-to-Gripper Retargeting · p. 3

C. Semantic Augmentation

Semantic augmentation extends a demonstration from its reference object to digital cousins through staged whole-object and task-weighted alignment. The resulting alignments transfer demonstrated trajectories to cousin pairings while preserving task-relevant regions.

Semantic alignment and grasp transfer

  • The figures illustrate cousins carried to task poses and show grasp transfer from a scanned mug to cousin handles.Figure 4 marks task-specific anchor regions, including the machine tray, poured mug rim, mug handle and branch, and drill handle.Source: Figure 4 · p. 4, Figure 5 · p. 5

Cousin assets

  • Cousin assets can be generated from one or several photographs, with ten cousins taking about seven minutes on one RTX 5090.The assets are scaled to real height and receive the same DINOv3 feature distillation.Source: C. Semantic Augmentation · p. 4

Semantic ICP

  • Semantic ICP restricts correspondences to matching DINOv3 feature clusters, pairing semantically similar object parts during whole-object alignment.The procedure starts with bidirectional ICP from candidate orientations, clusters features with K-means, and matches clusters by cosine similarity.Source: C. Semantic Augmentation · p. 4

Task-weighted ICP

  • Task-weighted alignment anchors cousins at semantically relevant interaction regions, such as a tray, rim, or handle, rather than relying on whole-object alignment alone.The method selects task-specific regions and weights their points 20×; it may then project rotation onto table-normal yaw and retranslate to keep the region fixed.Source: C. Semantic Augmentation · pp. 4–5

Trajectory transfer

  • Each aligned cousin inherits the tracked reference-object pose through its rigid alignment, preserving the demonstrated goal relation for every cousin pairing.A separate ground-alignment step moves the grasped object only in z to start level on the table without changing semantic alignment at the target state.Source: C. Semantic Augmentation · p. 5

D. Grasp Transfer and Inverse Kinematics

The demonstrated grasp is transferred to each cousin through its alignment, then checked for feasibility before inverse kinematics produces robot end-effector targets.

Grasp transfer

  • The reference-frame grasp follows each cousin through its semantic alignment before final ground alignment.Transferred grasps are checked for force closure and collision on the cousin mesh, with local resampling if a grasp collides.Source: D. Grasp Transfer and Inverse Kinematics · p. 5

Inverse kinematics

E. Scene Construction and Camera Optimization

A shared interactive scene layout is ported to IsaacSim and MuJoCo, while camera calibration aligns the simulated view to the real setup. Data generation varies object pairings and poses, then uses the simulators for complementary rendering and contact-physics needs.

E. Scene Construction and Camera Optimization

  • The interactive Viser layout places the robot, table, and posterboard, then ports the shared scene directly to IsaacSim and MuJoCo.Objects use textured meshes with alignment and metric scale baked in; MuJoCo adds convex collision hulls within 3 mm of the surface.Source: E. Scene Construction and Camera… · p. 5
  • Camera extrinsics are optimized from a user initialization using robot-mask IoU plus masked RGB and DINOv3 feature losses.Because rendering is non-differentiable, simultaneous perturbation estimates gradients with two renders per step in coarse-to-fine passes.Source: E. Scene Construction and Camera… · p. 5

F. Data Generation

  • Episodes select cousin pairings and new object poses, warping the demonstrated trajectory to the transferred goal while preserving the demonstrated motion at the goal.The start offset decays to zero along the arc length.Source: F. Data Generation · pp. 5–6
  • The generated training data and real rollouts are shown for both robots, with the manipulated object outlined as it moves.Source: Figure 7 · p. 6
  • IsaacSim provides photorealistic rendering with kinematic object carry, whereas MuJoCo simulates contact and retains episodes only when objects settle within goal tolerance.Episodes randomize table texture, lighting, skybox, and camera pose around the calibrated pose; generation keeps about 2,000 episodes in about two hours on one RTX 5090.Source: F. Data Generation · p. 6

IV. EXPERIMENTS

The experiments cover four tasks on two robot platforms and compare teleoperation with generated-data conditions across object instances. Policies use RGB and proprioception, and evaluation uses repeated real-robot trials with task-specific success criteria.

A. Experimental Setup

  • The four tasks test mug placement, continuous pouring, mug-tree hanging, and drill reorientation, with object variation differing across tasks.The drill task tests grasp recovery beyond mugs; its drill tree is not varied, while the mug tree is fixed in the hanging task.Source: A. Experimental Setup · p. 6
  • The xArm7 and Franka Panda use native grippers and an external RealSense camera, with RGB as the camera input.The Panda fingers receive a 3D-printed part to improve surface coverage; the camera is behind the xArm7 and in front of the Panda.Source: A. Experimental Setup · p. 6
  • The π0.5 policy receives a 224×224 third-person RGB frame and an 8-D proprioceptive state, then predicts ten-action chunks at 10 Hz.Generated datasets train for five epochs, 58k to 93k steps; teleoperated datasets train for 30k steps.Source: A. Experimental Setup · p. 6
  • The comparison includes 150 teleoperated demonstrations, 2,000 R2R2R+ episodes for the demonstrated pair, and 2,000 R2Sem2R episodes across cousin pairings.The coffee-machine and pour tasks hold out three of seven instances per object category from teleoperation training; the drill comparison uses identical objects in both data sources.Source: A. Experimental Setup · p. 6
  • Each task and robot is evaluated in 50 real-robot trials, except drill reorientation with 40, using randomized placements within the training-data spatial distribution.Success is task-specific: the object must be level on the tray, pour over the bowl, hang on a branch, or stand upright within target orientation.Source: A. Experimental Setup · p. 6

B. Object Tracking Accuracy

The tracker is reported as the most accurate on the paper’s objects across tested conditions and leads monocular baselines on YCBInEOAT. Table II covers mask IoU on the paper’s objects and AUC on YCBInEOAT.

B. Object Tracking Accuracy

  • The comparison on the paper’s seven objects uses mean per-frame mask IoU against the same SAM masks, with each method given the same TSDF mesh model.The mesh is fused from the splat’s renders at the measured object size.Source: B. Object Tracking Accuracy · p. 6
  • Table II reports mask IoU on the paper’s objects and AUC (%) on YCBInEOAT, with the latter starting from ground-truth pose.The table caption also specifies per-video depth-scale calibration on the first frame.Source: Table II · p. 7
  • FoundationPose with depth gets 95.8 ADD-S and 93.5 ADD.Source: B. Object Tracking Accuracy · p. 7
  • The tracker is the most accurate on every tested object under every condition, and leads every monocular baseline on average on YCBInEOAT.The YCBInEOAT evaluation starts each method from the ground-truth first pose and uses MoGe-3 depth for FoundationPose.Source: B. Object Tracking Accuracy · p. 7

C. Real-Robot Results

On every task and robot, R2Sem2R leads R2R2R+ and matches or exceeds teleoperation. The accompanying analyses associate performance with object diversity, data scale, and simulator choice.

C. Real-Robot Results

  • R2Sem2R leads R2R2R+ by 26–54 points on every task and robot, and matches or exceeds teleoperation.Both approaches use 2,000 episodes from the same human video; they differ in whether episodes span cousins.Source: C. Real-Robot Results · p. 7, Figure 8 · p. 7

D. What Drives Performance

  • On placement, object diversity matters more than data source; single-pair teleoperation and R2R2R+ are the two weakest policies on both robots.Figure 9 contrasts one-pair data with several-object data across teleoperation and generated data.Source: D. What Drives Performance · p. 7, Figure 9 · p. 7
  • On xArm7 coffee-machine placement, R2Sem2R matches the best teleoperated policy at 1,320 episodes and passes it at 2,000.Single-pair policies plateau: teleoperation at 10–12% and R2R2R+ at 36–38% from 660 episodes onward.Source: D. What Drives Performance · p. 7
  • On xArm7 placement, MuJoCo lowers success from IsaacSim’s 45/50 (90%) to 31/50 (62%), the same as teleoperation.Other generated-data results use IsaacSim; the authors suggest visual realism may matter more than contact dynamics for this RGB policy.Source: D. What Drives Performance · p. 7

E. Collection Cost

R2Sem2R generates 2,000 episodes in about two hours after roughly 25 minutes of setup, using about a minute of human time and 2.5 hours end to end. The conclusion reports real-world policies matching or exceeding teleoperation, while noting metric-measurement and task-scope limitations.

E. Collection Cost

  • Generating 2,000 R2Sem2R episodes costs about a minute of human time and 2.5 hours end to end, compared with about two hours of operator time for 150 teleoperated demonstrations.The 2.5 hours includes about 25 minutes of GPU time for reconstruction, tracking, cousin generation and alignment, and motion planning, followed by about two hours on one GPU. The shared 30-minute robot setup is excluded.Source: E. Collection Cost · p. 7

V. CONCLUSION

  • R2Sem2R policies perform robustly in the real world, matching or exceeding teleoperation.The conclusion describes semantically augmented demonstrations generated from one human RGB video.Source: §V · p. 8
  • R2Sem2R requires metric measurements of demonstrated objects, handles rigid objects only, and does not yet transfer articulated cousins.Recovering a screw axis and transferring it through alignment is future work; IsaacSim’s kinematic replay also omits finger-object contact, limiting contact-rich tasks.Source: §V · p. 8
Loading 2610.28880v1…