PaperlayerView exact revision on arXiv

Source-linked AI summary

BeyondRetarget: Learning Executable Humanoid Motions Directly from Monocular Video

Tianyu Xiong, Yi Lu, Jinrui Wang, Ziqi Liang, Dandan Lei, Xiaoyang Zhou, Xiao-xiao Long, Qiu Shen, Xun Cao

arXiv:2609.29850v1cs.ROcs.CV

TL;DR

Humanoid-robot motion data are costly to capture, while human-representation-based video pipelines face morphology mismatch and compounded errors. BeyondRetarget maps monocular video directly to robot motion and applies contact-aware refinement; experiments report improved performance over two-stage pipelines in simulation and on real robots.

  • Problem

    Motion-capture data require specialized hardware, while human-representation-based pipelines face morphology differences and compounded reconstruction-retargeting errors.

    Source: §1 · p. 2

  • Method

    BeyondRetarget directly maps monocular video to robot motion using an implicit, shared robot-oriented representation and contact-aware refinement.

    Source: §1 · p. 2

  • Results

    BeyondRetarget improves accuracy, robustness, executability, and real-time performance over two-stage pipelines, with stable execution in simulation and real-robot settings.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The framework offers a convenient approach to large-scale acquisition of motion data that can adapt to diverse humanoid embodiments.

    Source: §1 · p. 2, §5 · p. 9

    Global trajectory estimation can err on large-range motions or with noticeably moving cameras, and the method does not cover complex terrains or contact-rich human-environment interactions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Learning executable motions from human videos offers a scalable solution for humanoid robots to acquire demonstration motions. However, existing pipelines typically first construct an explicit human motion representation and then convert it into robot motions via motion retargeting. Although such methods can effectively leverage large volumes of existing human data for training, the substantial differences between humans and humanoid robots in locomotion mechanisms and joint degree-of-freedom configurations make motions generated by this human-representation-centric approach difficult to execute on robots. Furthermore, errors introduced during human motion estimation inevitably propagate to the retargeting stage and cannot be eliminated via joint optimization. We propose BeyondRetarget, an end-to-end framework that directly maps monocular RGB videos to robot motions. Discarding the explicit human representation, this framework learns robot-oriented implicit representations directly from visual observations, enabling the model to capture cross-morphology motion structures. To generate motions more suitable for robot execution, we further design a contact-aware motion optimization mechanism to improve temporal consistency and physical plausibility. Experiments show that BeyondRetarget significantly improves the accuracy and robustness of generated robot motions, while achieving higher execution success rates and lower latency in both simulation environments and real humanoid robots.

1 INTRODUCTION

BeyondRetarget maps monocular video directly to executable robot motion, addressing the mismatch and error accumulation of human-representation-based pipelines. Its experiments report improved motion quality and execution across simulation and real-robot settings.

1 INTRODUCTION

  • Monocular RGB video is abundant and inexpensive to collect, unlike motion-capture and wearable-sensor data that require specialized hardware and controlled environments.Existing video-to-robot methods commonly reconstruct human motion before retargeting it to a robot (Roux et al., 2026).Source: §1 · p. 2
  • Human-centric intermediate representations do not inherently describe robot motion well, and reconstruction errors can compound with retargeting errors in two-stage pipelines.Humans and humanoid robots differ in body proportions, joint degrees of freedom, and motion dynamics.Source: §1 · p. 2
  • The end-to-end framework learns an implicit, shared robot-oriented representation from video, decodes it for different humanoid embodiments, and refines motion using contact and support information.A unified human-robot supervision mechanism and semantic correspondences support robot-motion learning from SMPL-based data.Source: §1 · p. 2
  • BeyondRetarget improves accuracy, robustness, executability, and real-time performance over two-stage pipelines, with stable execution in simulation and real-robot settings.It also reduces severe motion collapse while more faithfully reproducing human motions.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work acquires human motion from capture systems or monocular video, then commonly transfers it to robots through retargeting. BeyondRetarget's architecture instead encodes video into a shared motion representation for direct robot-motion decoding, with contact-aware refinement.

2 RELATED WORK

  • Human-motion acquisition ranges from specialized optical and wearable systems to monocular-video methods that use temporal context and motion priors to produce structured human motion.SMPL supplies a shared pose-and-shape interface, while AMASS unifies 15 optical motion-capture datasets in an SMPL-based representation.Source: §2 · p. 3
  • Retargeting methods progress from hand-designed joint correspondences and inverse kinematics to optimization and learned mappings, each with trade-offs in robot-specific design, stability, or embodiment adaptation.Latent-command approaches reduce reliance on explicit retargeting but provide less interpretable robot motion and limited access to complete trajectories.Source: §2 · p. 3
  • Figure 2 depicts RGB video passing through visual and temporal encoders into a shared motion representation, which is decoded directly into target-robot motion and then contact-refined.The caption states that contact-aware refinement improves support and temporal consistency.Source: Figure 2 · p. 4
  • Existing robot-motion acquisition methods decouple human motion capture from robot retargeting, creating information bottlenecks and error accumulation that limit generated-motion quality.The broader pipeline first recovers explicit human motion and then retargets it to robots (Roux et al., 2026).Source: §2 · pp. 3–4

3 METHOD

The method maps video-derived temporal features into a shared motion representation decoded for a target robot, then trains and refines robot motion using unified human-robot supervision and contact-aware constraints.

3.1 PROBLEM FORMULATION AND FRAMEWORK OVERVIEW

  • A visual encoder and temporal context module produce a shared motion representation that a target-robot decoder maps to robot motion as residuals around a robot-specific nominal state.For additional robots, the decoder adapts robot-specific degrees-of-freedom and root-orientation branches while reusing shared components.Source: §3.1 · p. 4

3.2 UNIFIED MOTION SUPERVISION AND TRAINING

  • Human annotations supervise robot motion in a unified geometric space by adapting SMPL to robot morphology and scaling links independently across 18 shared keypoints.The pelvis-rooted, robot-proportioned targets let human annotations supervise robot motions without native robot trajectories; keypoint positions and directed segments supervise robot pose.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
  • The training objective combines geometry, direction, root translation and orientation, temporal consistency, and robot joint-limit losses.Geometry and direction losses supervise relative pose; the remaining terms constrain root motion, consecutive-frame consistency, and joint limits.Source: §3.2 · p. 5

3.3 CONTACT-AWARE REFINEMENT

  • After freezing the motion estimator, a pressure-supervised contact predictor classifies foot contact and support, guiding root and lower-body corrections that are temporally smoothed.A contacted foot is support only below a horizontal-velocity threshold; contact triggers root-height correction, while support also enables horizontal correction and constrained inverse kinematics.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments assess BeyondRetarget against two-stage methods, test adaptation across humanoid morphologies, ablate model components, and evaluate real-robot execution and teleoperation.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses MotionPRO’s approximately 12,000,000 frames and over 400 action categories, trains on Unitree G1, and adapts additional robots with robot-specific decoder branches.Metrics cover pose, trajectory, temporal continuity, physical plausibility, and executability.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 COMPARISON WITH TWO-STAGE METHODS

  • BeyondRetarget outperforms two-stage methods across MAMPJPE, RTE, Jitter, and Foot Slide; in simulation it achieves 347/362 (95.86%) success and 35.04 mm Exec. MAMPJPE.No evaluated BeyondRetarget window exceeds either the 65 mm or 100 mm MAMPJPE threshold, unlike all two-stage methods.Source: §4.2 · pp. 7–8, Table 1 · p. 7

4.3 MULTI-ROBOT EXPERIMENTS

  • Across eight humanoids, MAMPJPE ranges from 26.36 mm to 72.05 mm, and BeyondRetarget outperforms GVHMR→GMR on all four robots both methods support.Figure 5(b) shows all eight robots reproducing the same visual action, including actions from in-the-wild Internet videos.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 ABLATION STUDIES

  • Removing non-uniform morphology alignment raises MAMPJPE from 26.36 mm to 36.08 mm; removing Ldir, Ltemp, or contact-aware refinement degrades their respective reported metrics.The reported effects include poorer local body-part orientations without Ldir, worse Jitter, Accel, and FS without Ltemp, and worse RTE, FS, and Viol. without contact-aware refinement.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 REAL-ROBOT AND TELEOPERATION EXPERIMENTS

  • On the real robot, BeyondRetarget better matches the input motions and maintains stable body motion and foot support than the four two-stage baselines; it also has lower latency and memory use and higher throughput than both two-stage baselines.The efficiency supports responsive real-time visual teleoperation.Source: §4.5 · p. 9, Table 4 · p. 9, Figure 6 · p. 9

5 CONCLUSION AND LIMITATIONS

BeyondRetarget generates executable humanoid motions from monocular videos and reports higher accuracy, robustness, executability, and real-time performance than two-stage pipelines. It scales to diverse humanoid embodiments, but errors remain for some large-range or camera-moving motions, and complex terrains and contact-rich interactions are not covered.

5 CONCLUSION AND LIMITATIONS

  • BeyondRetarget achieves higher accuracy, robustness, executability, and real-time performance than two-stage pipelines.The conclusion also describes adaptation to diverse humanoid embodiments and a scheme for large-scale multi-robot motion-data acquisition.Source: §5 · p. 9
  • The framework scales to diverse humanoid embodiments and offers a scheme for convenient, large-scale acquisition of multi-robot motion data.Source: §5 · p. 9
  • Global trajectory estimation can still err on large-range human motions or when the camera is noticeably moving.The limitation is attributed to model capacity and training-data distribution.Source: §5 · p. 9
  • The current method does not cover complex terrains or contact-rich human–environment interactions.Source: §5 · p. 9
Loading 2609.29850v1…