PaperlayerView exact revision on arXiv

Source-linked AI summary

H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos

Hai Ci, Xiaokang Liu, Pei Yang, Yiren Song, Mike Zheng Shou

arXiv:2512.09406v1cs.ROcs.AIcs.CV

TL;DR

Robot learning from human videos could reduce reliance on costly robot demonstrations, but human and robot embodiments differ and paired training videos are expensive to collect. H2R-Grounder uses unpaired robot footage and a shared pose-and-background representation to generate robot videos from human demonstrations. In a DexYCB user study, it led preferences across all four evaluated aspects, with its strongest first-rank rates in visual quality and physical plausibility.

  • Problem

    Robot manipulation data collection is costly and lab-constrained, while human and robot appearances and motions differ and paired human–robot videos are costly to collect at scale.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    H2R-Grounder trains on unpaired robot videos using a shared representation of pose trajectories and background videos, then applies equivalent human-hand pose cues to generate robot videos.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    In the DexYCB user study, H2R-Grounder had the highest first-rank preference across all four aspects, including 61.4% for visual quality and 63.6% for physical plausibility.

    Source: §4.2.4 · p. 7

  • Takeaways & Limitations

    The framework generates motion-consistent, realistic robot videos from human interaction videos without calibration or paired supervision.

    Source: §5 · p. 8

    The current framework supports only single-hand-to-single-arm translation and produces Franka-style outputs; other robot embodiments require fine-tuning or robot-specific LoRA adapters.

    Source: §5 · p. 8

Abstract

from arXiv · show

Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos with realistic, physically grounded interactions. Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper's position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions. We fine-tune a SOTA video diffusion model (Wan 2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos. Project page: https://showlab.github.io/H2R-Grounder/

1. Introduction

H2R-Grounder translates human manipulation videos into robot videos without paired human–robot training data. It uses a shared pose-and-background representation and generative video modeling to produce motion-consistent, physically grounded interactions.

1. Introduction

  • The approach addresses costly, lab-constrained robot data collection and the visual embodiment gap between human hands and robot grippers.Prior rendering-based methods can produce floating or misaligned arms and depend on accurate calibration and pose estimation.Source: §1 · p. 2, Figure 2 · p. 2
  • H2R-Grounder uses unpaired robot videos to train a generative model that reconstructs robot arms from gripper-pose cues over inpainted backgrounds.The model learns from real robot footage, which includes physical contacts and occlusions, without seeing humans during training.Source: §1 · p. 2
  • At test time, human hands are replaced by equivalent pose indicators, which condition generation of robot videos that follow the human’s motion and interact with scene objects.The human is inpainted from the frames before the pose indicator is overlaid.Source: §1 · p. 2
  • H2R-Grounder’s contributions include H2Rep, a transferable intermediate representation, and in-context fine-tuning of diffusion video models for realistic, temporally consistent generation.The method uses minimal 2D pose indicators rather than strict 3D alignment.Source: §1 · p. 2

2. Related Work

Prior work bridges human and robot embodiments with intermediate representations, edits human videos by compositing robot renderings, and studies transfer between similar robot morphologies. H2R-Grounder instead uses pose sequences and background videos to directly generate robot videos from human inputs.

Intermediate Representations for Bridging Humans and Robots

  • H2Rep combines pose sequences with background videos to preserve motion and scene context, then uses this representation to synthesize robot videos directly from human inputs.Earlier intermediate representations were generally used for feature alignment and could lose information or produce visual misalignment.Source: §2 · p. 3

Translating Human Videos into Robot Videos

  • Rendering-based methods such as Phantom, Masquerade, and H2R can produce implausible occlusions or contacts and require camera–robot calibration unavailable for in-the-wild videos.H2R-Grounder conditions generation on abstract 2D pose indicators and learns occlusion and contact from real robot data.Source: §2 · p. 3

Cross-Robot Embodiment Transfer

  • Cross-robot transfer studies focus on robots with similar morphology, whereas human-to-robot transfer spans full-body humans and structurally different robotic manipulators.Source: §2 · p. 3

Generative Robot Video Prediction

  • Robot video prediction typically conditions on robot actions such as 3D end-effector poses, while H2R-Grounder uses 2D pose sequences and background videos for pose-consistent motion and scene coherence.RoboMaster requires manual object-mask and trajectory annotations; the adapted baseline is reported to trail H2R-Grounder in consistency and realism.Source: §2 · p. 3

3. Methodology

H2R-Grounder represents manipulation as pose cues over a background video, creating a shared input for human and robot demonstrations. It trains a conditional video generator on unpaired robot videos, then applies it to human videos to synthesize robot interactions.

3.1. A shared abstraction for human and robot videos

  • H2Rep combines a manipulator pose trajectory with background video, using aligned hand and gripper poses as a shared carrier of action semantics and scene state.For robot videos, the background preserves scene layout and manipulated-object state.Source: §3.1 · p. 3

3.2. Training data construction from robot videos

  • Robot-video H2Rep is constructed by projecting the gripper’s 6-DoF trajectory into image space and rendering position and orientation as a dot-and-arrow overlay.The arm is segmented and removed with video inpainting; Minimax-Remover preserves the background and removes the arm more reliably than E2FGVI.Source: §3.2 · pp. 3–4
  • The representation blends the rendered pose with the inpainted background, pairing gripper motion and scene evolution with the physically grounded robot video target.The selected inpainting model is Minimax-Remover [66].Source: §3.2 · p. 4

3.3. In-context learning for physically grounded robot video generation

  • The model fine-tunes Wan 2.2 as a conditional video generator, using robot H2Rep to reconstruct real robot videos with genuine physical interactions.The flow-matching objective trains a conditional velocity field; at inference, the trained generator produces robot videos from H2Rep.Source: §3.3 · pp. 4–5

3.4. Human video →robot video

  • For human videos, the pipeline estimates hand position and orientation, removes the person, and conditions the trained generator on the resulting H2Rep to produce a robot video.The paper attributes strong out-of-distribution generalization to fine-tuning only lightweight LoRA adapters while keeping the base generator frozen.Source: §3.4 · p. 5

4.1. Experimental Setup

The experiments train on Droid robot-manipulation videos and evaluate on held-out robot data and out-of-distribution human videos. The setup also tests transfer to internet clips and uses both quantitative measures and human or VLM evaluation where robot-video ground truth is unavailable.

4.1. Experimental Setup

  • Droid provides approximately 76K third-person Franka-arm manipulation videos; 50 videos are reserved for validation, and SSIM and LPIPS evaluate consistency and high-level visual feature distance.Training videos are standardized to 1280×720 and 10 fps, with clips sampled up to 49 frames during fine-tuning.Source: §4.1 · pp. 5–6
  • The OOD test set uses 100 DexYCB videos from subject 01 in a top-down view, with automatic annotations rather than supplied human masks or object poses.Because no ground-truth robot videos are available, evaluation uses VLM scoring and human studies on motion, background, visual quality, and physical plausibility.Source: §4.1 · p. 5
  • The setup also includes internet videos with varied backgrounds, occlusions, viewpoints, and camera motion for qualitative baseline comparisons.Figure 5 shows internet examples in the top row and DexYCB examples in the bottom row.Source: §4.1 · p. 5, Figure 5 · p. 6
  • The primary generator is Wan 2.2 TI2V-5B, fine-tuned for 200 steps; the study also trains VACE for two epochs as an alternative generator.The generator is reported to generalize to human videos with different frame rates at inference.Source: §4.1 · pp. 6–7

4.2. Comparison with Baselines

The comparison evaluates rendering-based, animation-based, commercial editing, and generative translation approaches. On DexYCB, H2R-Grounder leads human preferences and performs best or comparably on most VLM criteria, with qualitative tests also showing generalization to in-the-wild videos.

4.2.1. Rendering-Based Methods

  • Rendering-based methods are excluded because their calibration and camera-parameter requirements are unavailable for in-the-wild human videos.Source: §4.2.1 · p. 7

4.2.2. Animation-Based Methods

  • RoboMaster requires semi-manual preparation, including a reference frame, pose surrogates, object segmentation and tracking, phase labels, and a motion caption.This enables its robot–object interaction animations but requires heavy manual preparation.Source: §4.2.2 · p. 7

4.2.3. Commercial Video-Editing Methods

  • Kling and Runway Aleph are tested as prompted Franka-arm subject-replacement systems, representing appearance-level editing rather than true generative translation.Source: §4.2.3 · p. 7

4.2.4. Quantitative Results on DexYCB

  • H2R-Grounder leads all four human-study aspects, including visual quality (61.4%) and physical plausibility (63.6%), and leads or ties most VLM criteria, though Kling scores higher on visual quality.Human preferences are 54.5% for motion consistency and 56.8% for background consistency; VLM scores include 3.7 for motion consistency, 4.9 for background consistency, and 4.4 for physical plausibility, versus Kling's 4.1 to H2R-Grounder's 4.0 on visual quality.Source: §4.2.4 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.2.5. Qualitative Results

  • On internet videos and DexYCB sequences, H2R-Grounder maintains backgrounds, aligns motion, and positions the gripper to grasp a banana tip, while baselines distort backgrounds or robot-arm structure.Although fine-tuned only on the DROID indoor dataset, it generalizes to in-the-wild videos across viewpoints.Source: §4.2.5 · p. 8

4.3. Ablation Study

The ablations identify the pose cue and in-context fine-tuning as important components, while larger models do not yield clear quality gains and reduce efficiency.

  • Removing the H2Rep pose indicator causes motion drift, while removing LoRA fine-tuning prevents the model from generating a robot arm.Source: §4.3 · p. 8
  • VACE yields lower SSIM and higher LPIPS than the in-context video generator; scaling to 14B offers no clear quality gain and reduces sequence length from 49 to 17 frames.The 5B in-context configuration is adopted for its accuracy-efficiency balance.Source: §4.3 · p. 8

5. Conclusion and Limitation

H2R-Grounder uses H2Rep to translate human interaction videos into physically grounded robot videos without paired supervision or calibration. Its current scope is single-hand-to-single-arm translation with Franka-style outputs.

  • H2R-Grounder translates human interaction videos into motion-consistent, realistic robot manipulation videos without calibration or paired supervision.Source: §5 · p. 8
  • The framework currently supports only single-hand-to-single-arm translation and produces Franka-style outputs; other embodiments require fine-tuning or lightweight LoRA adapters.Source: §5 · p. 8

6. Motivation of H2Rep

H2Rep overlays pose cues on the background video using controlled transparency. This α-blended design reduces the computation and memory costs of dual-stream inputs while retaining pixel alignment.

  • H2Rep uses α-blending to overlay the pose sequence on the background, rather than representing pose and background as separate video streams.Source: §6 · p. 13
  • Dual video streams would double input tokens and make computation and memory scale 4×; α-blending reduces those costs while preserving pixel alignment with the human reference and generated robot video.Source: §6 · p. 13

7. Inference Efficiency

The 5B in-context model generates a 49-frame 704×1280 video in about 648 seconds on one H200 GPU, with peak memory use of 63 GB.

7. Inference Efficiency

  • About 648 seconds are required to generate a 49-frame 704×1280 video on a single H200 GPU.Peak memory consumption is 63 GB.Source: §7 · p. 13
  • The model runs at about 13 seconds per frame.Source: §7 · p. 13
  • Peak memory consumption reaches 63 GB during generation.Source: §7 · p. 13

Related papers

Loading 2512.09406v1…