PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowHMR: Physically Plausible Motion Capture from Video

Zhanke Wang, Chengfeng Zhao, Qing Shuai, Jingzhong Lin, Heng Li, Zeyu Ling, Yuxin Wen, Jing Li, Di Kang, Chunchao Guo, Linchao Bao

arXiv:2610.03691v1cs.CV

TL;DR

Recovering global 3D human motion from monocular video is challenging because depth is ambiguous and image motion entangles human and camera movement. FlowHMR treats motion capture as video-conditioned generation, then post-trains with fidelity and physical-tracking rewards. Human evaluators preferred FlowHMR’s motion fidelity, and its physical tracking success rate was 82.47% versus 62.82% for GVHMR.

  • Problem

    Monocular video motion capture must recover global 3D motion despite ambiguous depth, occlusions, and entangled human and camera movement.

    Source: §1 · p. 2

  • Method

    FlowHMR learns video-conditioned motion generation with flow matching, then uses reinforcement-learning post-training with fidelity and physical-tracking rewards.

    Source: §1 · p. 2

  • Results

    82.47% physical tracking success rate versus 62.82% for GVHMR, and human evaluators preferred FlowHMR over state-of-the-art methods in motion fidelity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Wild-4K provides an evaluation set of approximately 4K internet videos spanning everyday activities, dynamic movements, and severe occlusions.

    Source: §1 · p. 2

    Physical execution is evaluated in a simplified ground-plane environment, while simulation-based post-training requires repeated rollouts that become costly as data scale, sequence length, or sample count grows.

    Source: §6 · p. 14

Abstract

from arXiv · show

We present FlowHMR, a framework for recovering physically plausible global 3D human motion from monocular video. Previous learning-based methods typically regress human motion directly from video and train the network with geometric supervision. However, recovering human motion from monocular video is inherently ambiguous in depth, and direct regression tends to collapse toward an averaged solution. Moreover, the recovered motions are not guaranteed to be physically plausible, so physics-based tracking of them often fails. To address these challenges, we formulate video motion capture as a video-conditioned motion generation problem and first pretrain a flow matching model for this task. Given an input video, the pretrained model generates diverse motion candidates, but not all of them are faithful to the video or physically trackable. We therefore post-train the model using Group Relative Policy Optimization (GRPO) with two rewards. A fidelity reward encourages consistency with the input video. A tracking reward favors motions that a physics-based controller can track successfully. Together, these rewards shift the model's output preference, so the post-trained model stays faithful to the input video while producing more physically plausible motion. We further introduce Wild-4K, a large and diverse dataset of about 4K internet videos, for evaluating human motion recovery in the wild. Qualitative and quantitative experiments on Wild-4K show that our method outperforms state-of-the-art methods in overall motion fidelity and achieves a physical tracking success rate of 82.47%, compared with 62.82% for the strongest baseline, GVHMR.

1 INTRODUCTION

FlowHMR treats monocular motion capture as video-conditioned motion generation, then uses fidelity and simulation-tracking rewards to favor motions that match video and remain physically plausible. On Wild-4K, human evaluators prefer it over state-of-the-art methods, and it exceeds the strongest baseline in physical tracking success.

1 INTRODUCTION

  • Monocular capture is ambiguous in depth, and occlusion and camera movement complicate observations; deterministic regressors tend toward over-smoothed, averaged predictions under that ambiguity.Geometric objectives also do not explicitly model motion dynamics and contact, while auxiliary skating or penetration losses provide only indirect dynamics supervision.Source: §1 · p. 2
  • 82.47% physical tracking success for FlowHMR, versus 62.82% for the strongest baseline, GVHMR; human evaluators also prefer FlowHMR over state-of-the-art methods for motion fidelity.Evaluation combines blind pairwise user studies, kinematic metrics, and physical tracking on Wild-4K.Source: §1 · p. 2
  • FlowHMR models global 3D motion as video-conditioned generation, jointly representing body pose, hand articulation, and the sequence’s root trajectory in a world coordinate frame.The model is pretrained on approximately 3,000 hours of synthetic videos paired with ground-truth 3D motion.Source: §1 · p. 2
  • Its RL post-training combines a fidelity reward with a physical tracking reward to keep recovered motion faithful to video while favoring physical plausibility.The fidelity reward compares against ground-truth motion; the tracking reward measures how accurately a pretrained, frozen controller tracks generated motion in simulation.Source: §1 · p. 2
  • Wild-4K contributes approximately 4K internet videos for evaluating motion fidelity and physical tracking success in the wild.The evaluation set covers everyday activities, highly dynamic movements, and severe occlusions.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work spans optimization-based and feed-forward video motion capture, physics-aware motion methods, and reward-based generator post-training. FlowHMR learns a video-conditioned distribution of world-space motion and uses physical feedback during training.

Video-based human motion capture

  • Video motion capture has progressed from optimization with learned motion priors to feed-forward world-space regression and generative models; FlowHMR learns a video-conditioned distribution of world-space motion from paired video and motion data.Prior generative approaches include GENMO’s unified estimation and generation and DuoMo’s camera-space reconstruction followed by world-space diffusion refinement.Source: §2 · p. 3

Physics-aware motion capture and generation

  • Physics enters prior motion-capture and generation methods through optimization, dynamics modeling, simulated control, projection, or execution feedback; FlowHMR uses simulation only during training.A frozen PHC+ controller tracks sampled motions, and tracking performance plus framewise ground-truth agreement reward generator updates; inference produces motion without simulation-based selection or correction.Source: §2 · p. 3

Reward-based post-training of motion generators

  • Prior generator post-training includes kinematic-artifact rewards, semantic and kinematic rewards, and simulation-based tracking or alignment rewards; FlowHMR targets video-conditioned capture with framewise fidelity and physical tracking rewards.The paper builds on MixGRPO, a flow-based GRPO variant that mixes ODE and SDE sampling.Source: §2 · p. 3

Training-data figure

  • Figure 2 organizes training-data scale and diversity through synthetic examples, motion and paired-video duration versus BEDLAM, and sampled camera-trajectory proportions.The caption describes trajectories as simulating camera motion in in-the-wild internet videos.Source: Figure 2 · p. 4

3 DATA

FlowHMR builds large paired synthetic training data with varied motions, cameras, and appearances, then evaluates in-the-wild performance on Wild-4K internet clips. The two datasets serve distinct purposes: supervised pretraining and evaluation without ground-truth motion.

3.1 SYNTHETIC TRAINING DATA

  • Approximately 3,000 hours of synthetic video are rendered from 650 hours of source motion, expanding on the 74.52 hours of video in BEDLAM2.0.Public motion datasets and animation assets are used to scale paired supervision for body, hand, and global-motion recovery.Source: §3.1 · p. 4
  • Motion sources span locomotion, dance, exercise, and hand motion; they are converted to SMPL-H, and sequences with abnormal joints, velocities, or foot sliding are removed.The cleaning procedure follows HY-Motion.Source: §3.1 · p. 4
  • Static and moving cameras vary viewpoint, distance, and focal length, while generated trajectories keep the person in view and include full-body views and partial truncation.This variation is designed to distinguish human movement from apparent motion caused by the camera.Source: §3.1 · p. 4
  • Appearance diversity comes from randomized textures, lighting, ground materials, and foreground occluders; videos are rendered at 1080p and 30 fps in portrait, landscape, and square formats.Rendered videos retain motion and camera annotations.Source: §3.1 · p. 4
  • Approximately 571k motion pairs pass the post-training tracking filter of score above 0.6 with no early termination; pretraining uses all paired data regardless of tracking outcome.Original motion annotations are retained as fidelity targets for post-training.Source: §3.1 · p. 4

3.2 WILD-4K: INTERNET VIDEO EVALUATION

  • Wild-4K comprises 4,182 internet clips curated to assess generalization across broader actions, scenes, and visibility conditions than annotated benchmarks cover.Clips are sourced from Koala-36M; the cited benchmarks have limited environments or sequence counts, including 60 sequences in 3DPW.Source: §3.2 · pp. 4–5
  • Wild-4K selects single-adult clips with full-body framing while allowing temporary occlusion, excludes underwater, aerial, and suspended settings, and supports judgments, artifact measurements, and simulation tracking without ground-truth motion.The excluded settings match the ground-plane simulation environment; clips are grouped as normal, difficult, or self-occluded, and camera poses are estimated with VGGT-Ω.Source: §3.2 · p. 5

4 METHOD

FlowHMR frames monocular video motion capture as conditional motion generation, then combines video fidelity and physics-based tracking feedback to shape generated motions. Its two-stage method learns a video-conditioned motion distribution before reward-guided post-training.

4.1 OVERVIEW

  • FlowHMR first learns video-conditioned flow matching, then post-trains by reinforcement learning with fidelity and physical-tracking rewards.The post-training uses a pretrained, frozen humanoid controller for tracking feedback.Source: §4.1 · p. 5

4.2 GENERATIVE VIDEO-TO-MOTION PRETRAINING

  • Depth ambiguity and occlusion allow multiple 3D motions to explain the same observations, motivating conditional generation of global motion from video and camera observations.Source: §4.2 · p. 5

4.2.1 CONDITIONAL MOTION MODEL

  • The conditional model encodes body and hand articulation with global trajectory, and conditions a 0.46B-parameter MM-DiT on per-frame visual features and camera orientations.Sequences use a gravity-aligned world frame with heading set by the first-frame camera orientation.Source: §4.2.1 · p. 6

4.2.2 PRETRAINING WITH FLOW MATCHING

  • Flow matching trains on paired videos and 3D motions by interpolating Gaussian noise with ground-truth motion and predicting clean motion.The objective also applies feature-group losses and unaligned global-joint rotation and position losses.Source: §4.2.2 · pp. 6–7

4.3 PHYSICS-BASED POST-TRAINING

  • Post-training adds physics-simulation feedback because geometric pretraining does not explicitly account for motion dynamics and contact.Tracking alone could favor motions that are executable but deviate from the video, so it is combined with fidelity feedback.Source: §4.3 · p. 7

4.3.1 REWARDS

  • The fidelity reward compares generated and ground-truth joint positions in world coordinates, while a frozen PHC+ controller supplies the physical-tracking reward.Tracking succeeds when mean body-position error stays at or below 0.5 m at every step.Source: §4.3.1 · pp. 7–8

4.3.2 REWARD-GUIDED POLICY OPTIMIZATION

  • MixGRPO updates only the generator at stochastic denoising transitions, using group-relative advantages from standardized fidelity and tracking rewards.Training retains groups containing both successful and failed tracking outcomes.Source: §4.3.2 · p. 8

5 EXPERIMENTS

The experiments evaluate motion fidelity and physical trackability on Wild-4K using blinded human comparisons, simulation tracking, and geometric measurements. FlowHMR is preferred over external methods and achieves a higher tracking success rate than the strongest baseline, while post-training also reduces several physical errors.

5.1 EXPERIMENTAL SETUP

  • The evaluation combines randomized blinded pairwise judgments with PHC+ tracking and pre-simulation Penetration, Floating, and Skating metrics.Human judgments cover naturalness, video fidelity, global trajectory, and ground-contact artifacts; tracking success requires rtrack > 0.6, no early termination, and completion of the scored horizon.Source: §5.1 · pp. 8–9
  • The experiments compare GVHMR, PromptHMR, GENMO, and DuoMo with FlowHMR before and after RL post-training.Wild-4K tracking uses up to 360 frames per clip at 30 Hz; inference generates one motion per video with seed 0 and 20 sampling steps, without simulation-based selection or motion correction.Source: §5 · p. 8, §5.1 · p. 9

5.2.1 BLINDED HUMAN EVALUATION

  • FlowHMR wins 61.5–79.2% of valid annotations against four external methods, with losses of 3.8–10.0%.Against the pretrained model, post-training receives 34 wins (26.4%), 76 ties (58.9%), and 19 losses (14.7%) among 129 valid annotations.Source: §5.2.1 · p. 9

5.2.2 TRACKING AND PHYSICAL PLAUSIBILITY

  • After post-training, median Penetration falls from 0.60 to 0.06 mm, Floating from 0.53 to 0.20 mm, and Skating from 2.48 to 2.17 mm.GVHMR retains lower Skating at 1.59 mm.Source: §5.2.2 · p. 9
  • In the illustrated sequence, FlowHMR tracks successfully in simulation while all compared baselines fail.Figure 4 pairs each method’s reconstructed output in pink with its simulation tracking result in blue.Source: Figure 4 · p. 10

5.3 RECONSTRUCTION AND PHYSICAL EXECUTION ON RICH

On RICH, FlowHMR’s post-training improves PHC+ tracking success and tracking error, alongside small increases in two reconstruction errors. Reconstruction accuracy and controller tracking provide complementary evidence about the recovered motions.

5.3 RECONSTRUCTION AND PHYSICAL EXECUTION ON RICH

  • RICH provides annotated reconstruction and physical-execution comparisons on 185 sequences; Table 2 includes tracking failures and reports tracking MPJPE against generated reference motion.WA-MPJPE, W-MPJPE, and PA-MPJPE measure different joint-position errors; RTE and Jitter are evaluated over complete sequences.Source: §5.3 · p. 10, Table 2 · p. 10
  • On RICH, post-training raises PHC+ Success Rate from 48.65% to 51.35% and reduces tracking MPJPE from 133.5 to 118.8 mm.The final model has the highest success rate and lowest tracking error among evaluated methods.Source: §5.3 · p. 10, Table 2 · p. 10
  • Post-training slightly increases WA-MPJPE from 55.70 to 56.68 mm and PA-MPJPE from 30.78 to 31.40 mm, while Jitter and foot sliding decrease.DuoMo has lower WA-MPJPE, W-MPJPE, and RTE, but a lower PHC+ Success Rate than FlowHMR.Source: §5.3 · pp. 10–11
  • Figure 5 depicts an input video, FlowHMR’s reconstructed motion, and SONIC tracking in Isaac Lab on a Unitree G1.The caption distinguishes reconstructed motion in pink from tracking in blue.Source: Figure 5 · p. 11

5.4 EXECUTION UNDER SONIC

FlowHMR improves physical execution under SONIC and evaluates pretraining and post-training choices that balance reconstruction fidelity with tracking success.

5.4 EXECUTION UNDER SONIC

  • 77.72% success rate for FlowHMR exceeds GVHMR’s 54.15% under SONIC tracking on Wild-4K.Evaluation retargets reconstructions to the Unitree G1; a clip succeeds when its reference sequence finishes without a non-timeout termination.Source: Table 3 · p. 11, §5.4 · p. 11

5.5.1 PRETRAINING DESIGN

  • 3,000 hours of pretraining data improves all six reported metrics over 200 hours, reducing WA-MPJPE from 149.44 to 67.82 mm and hand-rotation error from 25.17 to 16.08 degrees.Table 4 reports mean results across motion categories on Studio, which has 300 sequences with optical motion-capture ground truth.Source: §5.5.1 · p. 11, Table 4 · p. 12
  • Flow Matching reduces WA-MPJPE from 73.28 to 67.82 mm and improves RTE, Jitter, and Foot-sliding, while hand-rotation error rises from 14.55 to 16.08 degrees.MPJPE is similar: 43.49 versus 43.44 mm.Source: §5.5.1 · pp. 11–12
  • Increasing model size from 0.2B to 1B reduces WA-MPJPE from 67.82 to 62.84 mm and Jitter from 11.75 to 10.93, but the 0.46B model has the lowest MPJPE, RTE, and Foot-sliding.The gains are not monotonic across all metrics.Source: §5.5.1 · p. 12

5.5.2 POST-TRAINING DESIGN

  • Joint-reward GRPO reaches 82.47% Success Rate with smaller reconstruction costs than projected-target SFT, retaining the original annotation as its fidelity reference.Projected-target SFT reaches 80.25% Success Rate, but raises WA-MPJPE to 70.80 mm and PA-MPJPE to 42.13 mm; filtered-original SFT does not improve Wild-4K tracking over pretraining.Source: §5.5.2 · p. 12
  • Tracking-only GRPO reaches 99.62% Success Rate, but WA-MPJPE and PA-MPJPE rise to 255.67 and 139.20 mm, illustrating reward hacking when tracking is assessed alone.Fidelity-only GRPO does not improve reconstruction or tracking over the pretrained model.Source: §5.5.2 · pp. 12–13
  • In Figure 6, joint-reward GRPO better preserves the input action while maintaining upright tracking, unlike tracking-only GRPO’s stationary character or projected-target SFT’s degraded pose match.The pretrained model falls during tracking in the illustrated example.Source: Figure 6 · p. 13, §5.5.2 · p. 13
  • Replacing the physical tracking reward with a Phys-Err kinematic proxy yields 77.98% Success Rate, below pretraining’s 78.79%; adding it to joint rewards gives 82.57% versus 82.47% but higher reconstruction errors.The two-reward configuration is used for better reconstruction at comparable tracking performance.Source: §5.5.2 · p. 13

5.6 SAMPLING VARIABILITY

The sampling study compares fixed pretrained and final GRPO checkpoints across eight seeds on a 300-video Wild-4K subset, reporting improved mean tracking and reduced seed sensitivity after post-training.

5.6 SAMPLING VARIABILITY

  • The 300-video subset contains 153 normal, 99 self-occluded, and 48 difficult motions, and the study evaluates fixed pretrained and final GRPO checkpoints with eight seeds.Table 6 reports variation among generated motions and execution quality under PHC+.Source: §5.6 · p. 13, Table 6 · p. 13
  • 79.71% mean PHC+ Success Rate after GRPO exceeds the pretrained model’s 76.33%, while standard deviation across eight seeds falls from 1.02 to 0.42 percentage points.The comparison uses a 300-video subset of Wild-4K.Source: Table 6 · p. 13, §5.6 · p. 14
  • Pairwise seed MPJPE decreases from 57.25 to 22.86 mm after GRPO, and tracking MPJPE decreases from 8.35 to 6.96 cm.Reduced variation across generated samples is distinct from movement amplitude within a motion sequence.Source: Table 6 · p. 13, §5.6 · p. 14

5.7 QUALITATIVE COMPARISONS

The qualitative comparison contrasts physical tracking of several methods and shows FlowHMR remaining upright with poses closer to its generated reference.

5.7 QUALITATIVE COMPARISONS

  • FlowHMR remains upright during PHC+ tracking, with torso orientation and arm poses closer to the reference.Figure 4 compares generated references with controller tracking.Source: §5.7 · p. 14
  • GVHMR and DuoMo fall during PHC+ tracking, while GENMO deviates substantially from its generated reference.These examples contrast physical execution across methods.Source: §5.7 · p. 14
  • Visually plausible reconstruction alone does not guarantee stable physical execution.Additional qualitative examples and comparisons appear in Figures 6 to 8.Source: §5.7 · p. 14

6 LIMITATIONS AND FUTURE WORK

The paper identifies the simplified simulation environment and rollout costs as limitations, and points to object interactions and more efficient simulation as future directions.

6 LIMITATIONS AND FUTURE WORK

  • Physical execution is currently evaluated in a simplified environment containing only a ground plane.This bounds the reported simulation evaluation setting.Source: §6 · p. 14
  • Simulation-based post-training requires repeated rollouts, making training costly as data scale, sequence length, or sample count grows.Source: §6 · p. 14
  • Future work could reconstruct objects and support surfaces, model their contacts in simulation, and develop more efficient rollout strategies for longer sequences and more videos.Internet videos often contain interactions with objects and support surfaces.Source: §6 · p. 14

7 CONCLUSION

FlowHMR combines Flow Matching pretraining with physics-based reinforcement learning to recover global 3D human motion, jointly optimizing reconstruction fidelity and simulation tracking. The paper introduces Wild-4K and reports improved overall motion quality and tracking performance over evaluated baselines.

7 CONCLUSION

  • FlowHMR combines Flow Matching pretraining with physics-based reinforcement learning to jointly optimize reconstruction fidelity and simulation tracking.Source: §7 · p. 15
  • The paper introduces Wild-4K for evaluating global 3D human motion recovery from monocular video.Source: §7 · p. 15
  • FlowHMR improves overall motion quality and tracking performance over the evaluated baselines.Source: §7 · p. 15

Related papers

Loading 2610.03691v1…