PaperlayerView exact revision on arXiv

Source-linked AI summary

ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback

Zimo He, Tianyi Bian, Zizhong Yang, Hongjie Li, Nan Jiang, Jieming Cui, Baoxiong Jia, Yizhou Wang, Siyuan Huang, Yixin Zhu

arXiv:2610.32966v1cs.ROcs.CV

TL;DR

Existing video-to-humanoid pipelines separately reconstruct and retarget human motion, limiting system-level scalability and making downstream tracking outcomes difficult to use for direct optimization. ViT-H generates humanoid reference motions directly from video and post-trains the generator using tracking feedback. It reports greater executability and local-pose accuracy than reconstruction-retargeting-tracking pipelines, with video-only tracking success increasing after post-training.

  • Problem

    Reconstruction-retargeting-tracking pipelines separately optimize stages, limiting system-level scalability and making it difficult to use tracking outcomes to directly improve the complete video-to-humanoid mapping.

    Source: §1 · p. 2

  • Method

    ViT-H directly generates humanoid reference motions from video with a diffusion model and post-trains the generator using tracker-scored preference pairs.

    Source: §1 · p. 2

  • Results

    ViT-H produces more executable and locally accurate references than state-of-the-art reconstruction-retargeting-tracking pipelines; video-only success rises from 74.8% to 80.0% offline and 49.6% to 63.8% online after post-training.

    Source: §4.1 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    Post-training gains carry to trackers not used for feedback, and the autoregressive variant supports real-time teleoperation of a physical humanoid.

    Source: §1 · p. 3, §5 · p. 10

    The scaling study is limited by available training data and computational resources, and the current framework does not model dexterous hand poses for fine-grained manipulation.

    Source: §5 · p. 10, E LIMITATIONS AND FUTURE WORK · p. 26

Abstract

from arXiv · show

Learning humanoid control from monocular human videos offers a scalable way to acquire diverse whole-body behaviors. Yet, existing reconstruction-retargeting-tracking pipelines are limited in system-level scalability by separately optimized intermediate stages and cannot directly optimize reference motion executability using downstream tracking feedback. We present ViT-H, a video-to-humanoid control framework following a generation-and-tracking paradigm. ViT-H directly generates humanoid reference motions from videos with a diffusion model, bypassing human motion reconstruction and retargeting, and uses motion-tracking outcomes to post-train the generator with direct preference optimization. We further introduce an autoregressive variant that enables real-time humanoid video teleoperation with sub-400 ms latency. Extensive experiments on the Unitree G1 demonstrate that ViT-H outperforms state-of-the-art reconstruction-retargeting-tracking pipelines in both motion quality and executability. We further show that ViT-H scales favorably with model size and with both pre-training and post-training data scale, while motion-tracking feedback consistently improves executability across different motion trackers.

1 INTRODUCTION

ViT-H replaces the separately optimized reconstruction-retargeting-tracking pipeline with direct humanoid motion generation and tracking-feedback post-training. Its experiments report better motion quality and executability than state-of-the-art pipelines, alongside favorable scaling and transfer of post-training gains.

1 INTRODUCTION

  • Existing pipelines separately reconstruct, retarget, and track motion, leaving system-level scalability constrained by the remaining stages and execution outcomes disconnected from end-to-end optimization.Reconstruction is not optimized for the target humanoid; retargeting cannot revisit discarded visual evidence after upstream errors propagate.Source: §1 · p. 2
  • ViT-H uses a video-conditioned humanoid motion diffusion model, pretrained on video-motion pairs and motion-only data, and tracker feedback for DPO post-training.Deployable and privileged trackers score candidates, and preference construction does not require ground-truth motion.Source: §1 · p. 2
  • An autoregressive generator produces temporal motion segments for online control and real-time teleoperation of a physical Unitree G1 from a single RGB camera.The abstract reports sub-400 ms latency, but that figure is not stated in this section's supplied passages.Source: §1 · p. 3
  • ViT-H directly generates executable humanoid motions from monocular videos, bypassing human-motion reconstruction and retargeting.Its tracker outcomes then provide feedback for post-training the generator without requiring motion annotations.Source: §1 · p. 3
  • Experiments report that ViT-H outperforms state-of-the-art reconstruction-retargeting-tracking baselines in motion quality and executability, with favorable scaling and post-training gains that generalize to held-out trackers.The reported scaling covers model capacity and pre- and post-training data size.Source: §1 · p. 3

2 RELATED WORK

Prior work studies human-motion reconstruction, human-to-humanoid retargeting, and humanoid motion generation and tracking as related but distinct problems. ViT-H instead generates robot-space motions directly from video and uses tracking feedback to post-train for executability.

Human motion reconstruction

  • Human-motion reconstruction methods estimate pose and shape from images or videos, with later work targeting globally consistent trajectories and physically plausible motion [2015; 2019; 2024; 2025; 2026].ViT-H does not seek physically plausible human motion as an intermediate representation; it directly generates robot-space references for downstream tracking.Source: §2 · p. 3

Human-to-humanoid motion retargeting

  • Retargeting converts human motion into humanoid joint space; GMR reduces kinematic artifacts [2026], PHUMA adds physical constraints [2026], and NMR learns a direct human-to-humanoid mapping [2026].ViT-H bypasses retargeting during inference and generates robot-specific references from monocular video.Source: §2 · p. 3

Humanoid motion generation and whole-body tracking

  • Whole-body trackers execute diverse kinematic references, while prior generators produce text-conditioned, multimodally conditioned, or loco-manipulation motions [2025; 2024; 2026].ViT-H uses tracking feedback to post-train its generator for executability rather than optimizing generation and tracking independently.Source: §2 · p. 3
  • Dedicated policies conditioned on motion latents enable retargeting-free language-, audio-, and video-driven control [2026], but require a specialized policy and are less able to leverage general motion trackers.This contrasts with ViT-H's use of a general tracker.Source: §2 · p. 3

Post-training of motion generation models

  • Prior tracker-feedback post-training improves physical plausibility for text-conditioned human-motion generators [2025; 2026], but still requires retargeting and optimizes in human-motion space.ViT-H instead post-trains a video-conditioned generator directly in the target humanoid motion space.Source: §2 · p. 4

3 VIT-H

ViT-H decomposes video-driven humanoid control into reference-motion generation and whole-body tracking, with offline and autoregressive online generation modes. Tracking outcomes score generated candidates and supply preference pairs for DPO post-training of the generator.

3 VIT-H

  • The framework separates video-conditioned reference-motion generation from whole-body tracking, which executes the reference and converts policy actions into joint torques.Offline generation predicts the full motion at once; online generation predicts temporal segments conditioned on preceding motion.Source: §3 · p. 4

3.1 HUMANOID MOTION GENERATION FROM VIDEOS

  • The diffusion generator maps video-derived camera orientations, bounding boxes, 2D keypoints, and image features directly to humanoid motion, without inference-time human-motion reconstruction or retargeting.Its motion representation includes global trajectory parameters, local joint rotations, camera-relative scale and translation, and end-effector stationary probabilities.Source: §3 · p. 4, §3.1 · pp. 4–5
  • The generator learns a reverse diffusion process that denoises samples and predicts clean motion; losses also use recovered global trajectories and forward-kinematics joint and vertex positions.The supplied passages describe an X0-prediction parameterization and losses on joints, vertices, root translation, end-effector stationarity, and 2D reprojection.Source: §3.1 · p. 5

3.2 POST-TRAINING WITH MOTION TRACKING FEEDBACK

  • Deployable and privileged trackers score sampled candidates using local-pose, global-motion, and tracking-success rewards, with the privileged tracker observing the root-pose offset.The deployable actor does not observe that offset, so global rollout deviations combine predicted-trajectory errors with tracker drift.Source: §3.2 · p. 5
  • Preference pairs require a reward advantage exceeding margin δ along each reward dimension, and the preferred candidate must succeed with both trackers; the pair with the largest total reward gap is selected.DPO fine-tuning lowers denoising loss on the preferred motion relative to the dispreferred motion, measured against a frozen reference model, and adds supervised fine-tuning on the preferred motion.Source: §3.2 · p. 6

Appendix B.4 details preference pair construction and post-training.

The autoregressive variant supports real-time video-to-humanoid teleoperation by generating motion in segments, using overlap with the preceding segment while retaining the ability to generate an initial segment without prior context.

3.3 AUTOREGRESSIVE GENERATION FOR VIDEO TELEOPERATION

  • The autoregressive generator enables segment-by-segment motion generation for real-time video-to-humanoid teleoperation.Source: §3.3 · p. 6
  • Each fixed-length segment is conditioned on its predecessor through L overlapping frames, while randomly dropping predecessor context during fine-tuning preserves initial-segment generation.The first L noisy frames are replaced with the preceding segment’s last L frames.Source: §3.3 · p. 6
  • During post-training, predecessor conditioning is used for in-distribution samples with ground-truth motion, but not for video-only samples lacking that motion.Source: §3.3 · p. 6

4 EXPERIMENTS

Across the evaluated datasets, ViT-H compares favorably with reconstruction-retargeting-tracking baselines in motion quality and executability. Analysis shows gains from tracker feedback, model and data scaling, and post-training transfer across trackers; the conclusion notes current limits in scaling resources and dexterous hand poses.

4.1 COMPARISONS

  • ViT-H's offline generator outperforms reconstruction-and-retargeting baselines across motion accuracy and quality metrics, while both post-trained generators achieve higher tracking success than all pipelines on every test set.The tracked offline motions are also closer to ground truth than those of any pipeline.Source: §4.1 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.2 ANALYSIS OF DPO POST-TRAINING

  • Deployable-tracker rewards raise success from 74.8% to 75.8% offline and 49.6% to 61.5% online; adding global-motion feedback raises it to 80.0% and 63.8%.Because evaluations use the deployable tracker, the added gain indicates global tracking feedback supplies supervision local feedback misses.Source: §4.2 · pp. 8–9, Figure 3 · p. 9
  • The best post-training data mix uses in-distribution data equal to half the video-only data, while fine-tuning the last 8 of 16 Transformer layers performs best.More in-distribution pre-training data degrades performance; updating fewer or all 16 layers also lowers performance.Source: §4.2 · pp. 9–10, Figure 3 · p. 9

4.3 SCALABILITY AND CROSS-TRACKER GENERALIZATION

  • Increasing model size from 49.9M to 114.6M and 224.7M parameters, pre-training data, or post-training data improves reported tracking success and Trk. MPJPE on the video-only test set.Model-size and pre-training-data results use generators without post-training; post-training data increases improve both metrics for offline and online generators.Source: Figure 4 · p. 9, §4.3 · p. 10
  • Post-training raises success and lowers Trk. MPJPE for both generators with Humanoid-GPT, ScaleBFM, and HoloMotion, none used during post-training.The authors interpret this as improved executability rather than overfitting to the feedback tracker.Source: §4.3 · p. 10, Figure 5 · p. 10

5 CONCLUSION

  • ViT-H generates motions directly in Unitree G1 motion space, and its autoregressive variant supports real-time teleoperation of a physical humanoid.The conclusion suggests a mature general-purpose tracker can serve as both executor and supervisor of the generator.Source: §5 · p. 10
  • The scaling study is limited by available training data and compute, and the framework does not model dexterous hand poses, restricting fine-grained object manipulation.Source: §5 · p. 10

D Additional Experimental Results A8

The supplement collects additional experimental results, implementation details, and experimental setup. It also concludes with limitations and future directions.

D Additional Experimental Results A8

A OVERVIEW

  • The supplement covers implementation details, the experimental setup, and further experimental results before discussing limitations and future directions.It recommends the project page for demonstrations across diverse scenarios.Source: A OVERVIEW · p. 18

B DETAILS ON VIT-H

  • Appendix B details the motion-generator representation, architecture, pre- and post-training, autoregressive adaptation, tracker training, and offline and online deployment.Source: B DETAILS ON VIT-H · p. 18

B.1 DATA REPRESENTATION OF MOTION GENERATOR

The generator conditions on video-derived camera, person, keypoint, and image information. For datasets without paired videos, these conditions are simulated from motion data.

B.1 DATA REPRESENTATION OF MOTION GENERATOR

  • Each video is represented by relative camera orientations, 2D bounding boxes, 2D keypoints, and image features.The passage names DPVO, YOLOv8, ViTPose, and Fast SAM 3D Body as the respective extraction tools.Source: §B.1 · p. 18
  • For datasets without paired videos, the pipeline simulates camera trajectories, generates boxes, projects 3D motions to 2D keypoints, and sets image features to zero.Source: §B.1 · p. 19
  • After prediction, the framework discards variables used only for loss computation and recovers global root motion in the gravity-view coordinate system.It then obtains world-coordinate joint and vertex positions through forward kinematics with the Unitree G1 kinematic model.Source: §B.1 · p. 19

B.2 NETWORK ARCHITECTURE OF MOTION GENERATOR

The reference motion generator uses a Transformer to denoise motion conditioned on video inputs. The appendix specifies its architecture and three model sizes used in scaling experiments.

B.2 NETWORK ARCHITECTURE OF MOTION GENERATOR

  • A Transformer backbone receives noisy motion concatenated with independently MLP-projected conditions and outputs a denoised motion sequence.The backbone uses rotary positional embeddings, and an output MLP projects its outputs.Source: §B.2 · p. 19
  • The main generator has 16 encoder layers, 16 attention heads, a 1,024-dimensional hidden state, and 224.7M parameters.It uses GELU activations.Source: §B.2 · p. 19
  • Scaling experiments also use 49.9M-parameter small and 114.6M-parameter medium variants.The small variant has 12 layers, 8 heads, and hidden dimension 512; the medium has 14 layers, 12 heads, and hidden dimension 768.Source: §B.2 · p. 19

B.3 PRE-TRAINING OF MOTION GENERATOR

The appendix describes generator pre-training losses and setup, tracking-informed preference construction and post-training, and adaptations for autoregressive generation and online teleoperation.

B.3 PRE-TRAINING OF MOTION GENERATOR

  • Pre-training adds local joint and vertex Huber losses, global translation L1 loss, and end-effector stationary-state binary cross-entropy to the diffusion loss.The local vertex loss uses 384 sampled Unitree G1 surface vertices, with denser sampling around hands and feet; stationary labels come from joint velocities.Source: §B.3 · p. 19
  • Training also applies Huber losses to reprojected 2D joints, vertices, and camera-coordinate root translation, rescaling camera translation by 0.8 for the human-to-G1 body-scale difference.Source: §B.3 · p. 19
  • Pre-training uses 1,000-step DDPM, 120-frame sequences at 30 fps, global batch size 256, and 730k optimization steps on four H100 GPUs.The learning rate is 2 × 10^-4.Source: §B.3 · p. 19

B.4 POST-TRAINING OF MOTION GENERATOR

  • Tracking-informed candidate scoring combines local-pose, global-motion, and tracking-success rewards, with errors normalized using an exponential kernel and empirically calibrated scales.The passage defines root translation, root orientation, root-relative joint position, and joint rotation errors as reward inputs.Source: §B.4 · p. 20
  • Preference-pair filtering accounts for whether ground-truth humanoid motion is available and weights tracking-success reward 10 times more than other reward dimensions when selecting pairs.For video-only data, root and local reward components are treated as separate dimensions; continuous margins are set to zero to retain internet-video data.Source: §B.4 · p. 20

B.5 ADAPTATION OF AUTOREGRESSIVE MOTION GENERATOR

  • The autoregressive generator uses 120-frame segments with either no prior-motion context or overlap of all but five frames, distinguished by learnable conditioning tokens.Source: §B.5 · p. 21
  • Autoregressive post-training follows the same DPO objective, using four DDIM denoising steps and balancing samples with and without preceding-motion conditioning for pre-training data.Internet-video samples are used without preceding-motion conditioning; the combined pre-training sample duration is kept roughly comparable to that of internet-video samples.Source: §B.5 · p. 21
  • Online inference conditions each segment on the preceding 115 frames and generates five new frames, aligning them to a global motion queue and damping root transforms to reduce accumulated drift.Source: §B.5 · p. 21

B.6 TRAINING OF MOTION TRACKER

The motion trackers are fine-tuned from SONIC, and real-world deployment uses both offline full-video generation and online segmentwise teleoperation. The system supports real-time control with sub-400 ms latency on a 29-DoF Unitree G1.

B.6 TRAINING OF MOTION TRACKER

  • The deployable tracker is fine-tuned from SONIC with PPO, while the privileged tracker adds relative robot-to-reference root pose to actor observations.The deployable tracker adds anchor-body horizontal-position and yaw rewards; the privileged tracker freezes the SONIC encoders and auxiliary kinematic decoder.Source: §B.6 · p. 21

B.7 REAL-WORLD DEPLOYMENT

  • Online deployment uses four DDIM denoising steps for autoregressive generation, while offline generation uses 50 steps without motion post-processing.Generated references are tracked by the deployable policy; online references are produced in segments from a fixed RGB camera stream.Source: §B.7 · p. 22
  • Average end-to-end latency is below 300 ms, with a reserved 100 ms margin for stable teleoperation below 400 ms.The setup uses a RealSense D455 RGB stream at 640×480 and 30 Hz, with the camera fixed during teleoperation.Source: §B.7 · pp. 22–23

C DETAILS ON EXPERIMENTS

  • Experiments use the 29-DoF Unitree G1 for both simulation and real-world deployment.The appendix section covers dataset conversion, the human study, evaluation metrics, and baseline adaptations.Source: C DETAILS ON EXPERIMENTS · p. 23

C.1 DATASET STATISTICS AND DATA CONVERSION

The appendix details the datasets, evaluation procedures, and adapted baselines, then reports that the offline generator outperforms all pipelines across the ground-truth-camera comparison metrics. Post-training improves executability while keeping local accuracy comparable.

C.1 DATASET STATISTICS AND DATA CONVERSION

  • Pre-training uses 383.8 hours of motion data, including 79 hours paired with video, while post-training uses 8,000 videos totaling 16.7 hours and testing uses 282 videos totaling 39.1 minutes.For unpaired datasets, GMR retargets human motion to the G1; its adapted procedure runs multiple initialization seeds and retains the candidate with the most valid frames, splitting localized invalid regions when needed.Source: §C.1 · pp. 23–24
  • BONES-SEED motion in SOMA representation is converted to SMPL-X for baseline training, using transferred vertices to fit body shape before fitting the remaining parameters.The generator itself takes human keypoints and does not rely on human-body-parameter supervision.Source: §C.1 · p. 24

C.2 HUMAN STUDY

  • The human study presents the input video above two side-by-side humanoid results in randomized left-right order and asks participants which better transfers the motion to the G1.The study recruited 20 participants with computer-vision or robotics research backgrounds; 12 had prior humanoid-research experience.Source: §C.2 · p. 24

C.3 EVALUATION METRICS

  • Motion quality is evaluated with local MPJPE, PA-MPJPE, root translation error, jitter, and self-penetration, while executability includes tracking success and tracking errors.For EMDB and RICH, tracking success is averaged over 250-frame segments; video-only test sets have no fixed segment-length constraint.Source: §C.3 · p. 24

C.4 ADAPTATION OF BASELINES

  • Adapted GVHMR and GEM use the generator’s training data, comparable model capacity, and SAM 3D Body visual features, with a matched 730k-step training budget.They retain their original training hyperparameters and are trained on four NVIDIA H100 GPUs.Source: §C.4 · pp. 24–25

D ADDITIONAL EXPERIMENTAL RESULTS

  • Additional experiments include qualitative comparisons and quantitative comparisons with baselines and ablations using ground-truth cameras.The reported ground-truth-camera trends match those obtained with estimated cameras.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 25, §D.1 · p. 25

D.1 COMPARISONS WITH GROUND-TRUTH CAMERA

  • The offline generator outperforms all pipelines on every metric in both ground-truth-camera tables, and both post-trained generators achieve higher success rates and lower tracking errors than all pipelines.Post-training reduces self-penetration in every setting and jitter in most, lowers tracking errors, and raises or preserves success rates while keeping local accuracy comparable.Source: Table A2 · p. 25, Table A3 · p. 25, §D.1 · p. 26

E LIMITATIONS AND FUTURE WORK

The paper’s scaling study is limited to the available data and compute regime, and its motion representation and tracking setup do not model dexterous hand poses.

E LIMITATIONS AND FUTURE WORK

  • Available training data and compute limit the scaling study, so it does not explore substantially larger models or datasets beyond the studied regime.The authors describe favorable scaling only across the model and data scales considered.Source: E LIMITATIONS AND FUTURE WORK · p. 26
  • Without dexterous hand poses, the object interactions studied are primarily whole-body manipulation of relatively large, simple objects rather than fine-grained tasks such as grasping small objects.The authors identify extending the representation and tracking framework to model hand poses as future work.Source: E LIMITATIONS AND FUTURE WORK · p. 26

Related papers

Loading 2610.32966v1…