Source-linked AI summary
PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generation
Yidong Huang, Zun Wang, Han Lin, Dong-Ki Kim, Shayegan Omidshafiei, Jaehong Yoon, Jaemin Cho, Yue Zhang, Mohit Bansal
TL;DR
Existing video rewards often miss the 3D physical constraints underlying realistic human motion, limiting reliable reward-based post-training. PhyMotion grounds recovered 3D human trajectories in simulation and evaluates kinematic, contact/balance, and dynamic feasibility, achieving stronger human alignment and improving post-trained video generators.
Problem
Existing video rewards rely mainly on 2D or coarse signals that inadequately evaluate articulated human motion’s 3D configuration, contacts, balance, and dynamics.
Method
PhyMotion converts generated videos into 3D human motion, grounds trajectories in MuJoCo, and scores kinematic, contact/balance, and dynamic feasibility.
Results
80% average pairwise agreement and ρ=0.376 show stronger human alignment than existing rewards, while PhyMotion also improves RL-post-trained generators across automatic and human evaluations.
Takeaways & Limitations
PhyMotion provides an interpretable physics-grounded reward that improves motion realism while preserving general video generation quality across different generator backbones.
Takeaways & Limitations
RL post-training uses a curated prompt distribution focused on human motion across diverse actions and scenarios.
Abstract
from arXiv · showhide
Generating realistic human motion is a central yet unsolved challenge in video generation. While reinforcement learning (RL)-based post-training has driven recent gains in general video quality, extending it to human motion remains bottlenecked by a reward signal that cannot reliably score motion realism. Existing video rewards primarily rely on 2D perceptual signals, without explicitly modeling the 3D body state, contact, and dynamics underlying articulated human motion, and often assign high scores to videos with floating bodies or physically implausible movements. To address this, we propose PhyMotion, a structured, fine-grained motion reward that grounds recovered 3D human trajectories in a physics simulator and evaluates motion quality along multiple dimensions of physical feasibility. Concretely, we recover SMPL body meshes from generated videos, retarget them onto a humanoid in the MuJoCo physics simulator, and evaluate the resulting motion along three axes: kinematic plausibility, contact and balance consistency, and dynamic feasibility. Each component provides a continuous and interpretable signal tied to a specific aspect of motion quality, allowing the reward to capture which aspects of motion are physically correct or violated. Experiments show that PhyMotion achieves stronger correlation with human judgments than existing reward formulations. These gains carry over to RL-based post-training, where optimizing PhyMotion leads to larger and more consistent improvements than optimizing existing rewards, improving motion realism across both autoregressive and bidirectional video generators under both automatic metrics and blind human evaluation (+68 Elo gain). Ablations show that the three axes provide complementary supervision signals, while the reward preserves overall video generation quality with only modest training overhead.
1 Introduction
PhyMotion addresses the inadequacy of existing 2D motion rewards by grounding recovered 3D human trajectories in physics simulation and decomposing motion quality into structured feasibility dimensions. It serves as both a reliable evaluator and an effective RL reward, achieving strong agreement and correlation with human judgments.
- Motivation: Existing video rewards inadequately assess human motion because 2D classifiers and perceptual signals are structurally blind to articulated motion and its physical constraints.These limitations can produce misleading scores for physically implausible motion.
- Method: PhyMotion recovers SMPL meshes from generated videos, retargets 3D trajectories in a physics simulator, and evaluates motion across multiple dimensions of physical feasibility.The simulator exposes observables such as joint-level kinematics, contacts, and motion-driving torques.
- Method: The reward decomposes motion quality into kinematic, contact/balance, and dynamic feasibility scores targeting joint consistency, interaction correctness, and force-motion consistency.Each score addresses a distinct failure mode identified in the paper.
- Results: 80% average pairwise agreement with human judgments and ρ=0.376 aggregate Spearman correlation were achieved on 1,200 video pairs.The evaluation covered body structure, balance, and motion naturalness.
2 Related Work
Prior work advances video generation through RL-based post-training and motion-aware generation or evaluation, but existing evaluators provide limited fine-grained evidence about articulated human motion and physical failures.
- RL-based post-training and video reward models: RL-based post-training has become central for generative models, with diffusion methods including DDPO, DPOK, Diffusion-DPO, DanceGRPO, Flow-GRPO, and DiffusionNFT.These methods optimize policy gradients or preference objectives during reverse sampling, according to the supplied passage.
- Human motion in video generation and evaluation: Motion-aware video generation methods modify generators or incorporate motion priors, including external 2D poses, 3D SMPL priors, and jointly predicted motion representations.The passage cites pose-conditioned methods alongside VideoJAM and EchoMotion as examples of these strategies.
- Human motion in video generation and evaluation: PhyMotion highlights kinematic, contact, and dynamic failure modes that existing 2D metrics overlook.The figure describes these as articulated-body inconsistency, unstable body–environment interaction, and physically infeasible motion, respectively.
- Human motion in video generation and evaluation: Existing evaluators largely rely on perceptual video cues, while VLM-based evaluators remain limited for fine-grained human articulation.The passage also identifies MBench and MotionCritic as related 3D motion evaluators.
3 Physics-Grounded 3D Motion Evaluation
Existing 2D motion evaluators can reward videos with kinematic, contact, balance, or dynamic failures that become visible only after reconstructing 3D human motion. PhyMotion addresses this by retargeting recovered body trajectories in MuJoCo and scoring continuous feasibility across complementary kinematic, contact, and dynamic axes.
- Failure modes: 2D evaluators may score perceptually convincing videos despite self-penetration, floating or inconsistent contact, and dynamically excessive forces.Examples include a hand penetrating the hip, airborne support during a flying kick, and excessive joint torques during a baseball pitch.
- Physics-grounded protocol: PhyMotion recovers SMPL-X trajectories from generated videos, retargets them to a MuJoCo human model, and uses inverse dynamics to estimate joint torques and ground reaction forces.The simulator includes explicit mass, inertia, joint limits, and contact geometry.
- Kinematic feasibility: Kinematic feasibility combines angular-velocity, self-penetration, and joint-limit violations to measure smoothness and anatomical validity.The component penalizes abnormal velocities, intersecting non-adjacent mesh triangles, and joint angles outside valid MuJoCo ranges.
- Contact feasibility: Contact feasibility measures foot sliding, ground penetration, floating, and balance violations using inferred left- and right-foot ground contacts.Contacts are inferred from foot height and velocity over time.
- Dynamic feasibility: Dynamic feasibility penalizes excessive joint torques, implausible ground reaction forces, and unusually high joint effort, while the three axes jointly provide continuous, interpretable signals for evaluation and reward-based post-training.Together, the axes cover articulation quality, environment interaction, and physical realizability.
4 RL Post-Training with PhyMotion
PhyMotion is used as an optimization reward by equally aggregating its kinematic, contact, and dynamic scores, then maximizing this reward through KL-regularized policy learning. Its decomposed axes support per-dimension diagnosis and early detection of reward hacking during training.
- Reward construction: The three PhyMotion scores are combined into a single motion reward for optimization.The aggregation is equally weighted across Fkin(v), Fcon(v), and Fdyn(v).
- Problem formulation: The video generator is fine-tuned as a policy to maximize expected reward while remaining close to a reference policy through KL regularization.The KL coefficient λ controls regularization strength, helping prevent mode collapse and preserve general video-generation capability.
- Policy optimization: Training uses a forward-process RL formulation with normalized rewards, noisy samples, and interpolated positive and negative implicit policies.The method defines v+ and v− by interpolating the current and old velocity predictors with strength β, then contrasts them against the target forward velocity.
- Policy optimization: The trajectory-free formulation uses clean generated samples without backpropagating through the reverse sampling chain, enabling memory-efficient and solver-agnostic training.This implementation avoids dependence on the reverse sampler during optimization.
- Reward integration: The three-axis reward enables per-dimension improvement analysis and makes disproportionate gains that could indicate reward hacking immediately visible.Monitoring the independent axes helps detect when one dimension improves at the expense of the others.
5 Experiments
Experiments show that PhyMotion aligns more strongly with human judgments than existing video metrics and improves human-motion generation when used for RL post-training. These gains extend across automatic metrics, human preference comparisons, broader prompts, and practical training overhead.
- 5.1 Metric Alignment: PhyMotion achieves the strongest and most consistent alignment with human judgments across body structure, balance, and motion naturalness.Kinematic feasibility reaches 82.9% agreement on body structure, while contact and dynamic feasibility remain between 78–81%; existing metrics mostly remain in the 50–66% range.
- 5.2 RL Post-Training: PhyMotion RL post-training improves external motion-quality metrics beyond the metrics used for optimization on both Causal Forcing and FastWan.For Causal Forcing, VideoAlign video quality improves by +25.2%; overall PhyMotion feasibility improves by +3.5%, while FastWan gains +7.0% in overall feasibility.
- 5.2 Human Preference: The PhyMotion-optimized model achieves the highest overall human preference Elo score and consistently wins direct comparisons against all baselines, including Wan2.2 14B.The evaluation uses 1,487 randomly sampled video pairs, and ties count as 0.5 in pairwise win rates.
- Ablations: PhyMotion preserves general-domain video quality while improving VBench-2.0 performance relative to the Causal Forcing 1.3B base model.Average VBench performance remains comparable at 0.654 versus 0.657 for the base model.
- Ablations: PhyMotion requires 2.80s per video for standalone reward computation but adds only 7% effective training overhead, compared with 35% for HPSv3.Most PhyMotion computation is hidden behind video sampling, making the structured 3D reward practical for RL post-training.
6 Conclusion · Appendix · A Additional Qualitative Results
PhyMotion is a structured, fine-grained reward that grounds generated human videos in 3D motion and physics simulation to evaluate motion feasibility. It aligns better with human judgments than existing video rewards and supports post-training, as reported in the conclusion.
- 6 Conclusion: PhyMotion introduces a structured, fine-grained motion reward for physics-grounded human video generation.It is designed specifically to evaluate human motion realism.
- 6 Conclusion: The method lifts generated videos into 3D human motion rather than relying only on perceptual or learned video-level signals.This provides an explicit motion representation for reward evaluation.
- 6 Conclusion: PhyMotion grounds recovered motion in a physics simulator to assess physical feasibility.The simulator-based formulation connects reward scoring to physical motion constraints.
- 6 Conclusion: Its evaluation covers kinematic consistency, contact and balance consistency, and dynamic consistency.These three axes represent distinct aspects of motion feasibility.
- 6 Conclusion: PhyMotion aligns better with human judgments than existing video rewards.The conclusion reports stronger agreement without specifying a numerical correlation value.
- 6 Conclusion: The method is also used in a post-training experiment for human video generation.The supplied conclusion identifies post-training as an application but provides no further quantitative result.
A.1 Qualitative Analysis of Individual Reward Components
Qualitative examples show that PhyMotion’s three submetrics diagnose distinct motion failures by isolating videos that score poorly on one component while scoring relatively well on the other two. Kinematic, contact/balance, and dynamic feasibility therefore provide complementary supervision for decomposed motion evaluation.
- Diagnostic setup: Filtering isolates videos that fail primarily on one target submetric while performing relatively well on the other two.This selection highlights each component’s diagnostic role.
- Kinematic feasibility: Kinematic feasibility detects articulated-body errors including self-penetration, extra limbs, and impossible joint angles.These failures concern the plausibility of the recovered body configuration.
- Contact/balance feasibility: Contact/balance feasibility captures unstable support, balance violations, and foot sliding.These artifacts reflect inconsistencies in support and balance.
- Dynamic feasibility: Dynamic feasibility identifies motions requiring implausible forces and torques, including excessive horizontal ground force.The examples support decomposing the reward because the three submetrics capture complementary artifacts.
B Per-category Comparison on General Video Benchmarks. · C Human Study Detail
The category-level evaluation finds the model competitive across broad perceptual, temporal, and compositional dimensions, including human- and motion-related categories. Human preference studies assess whether PhyMotion’s feasibility metrics align with judgments of motion quality and compare post-trained models against baselines.
- B Per-category Comparison on General Video Benchmarks.: The model remains competitive across broad perceptual dimensions.
- B Per-category Comparison on General Video Benchmarks.: It remains competitive across temporal dimensions.
- B Per-category Comparison on General Video Benchmarks.: It remains competitive across compositional dimensions.
- B Per-category Comparison on General Video Benchmarks.: The model retains strong performance on subject consistency, motion smoothness, and temporal flickering.
- B Per-category Comparison on General Video Benchmarks.: It also retains strong performance on human action and human anatomy.
- C Human Study Detail: The human preference evaluation uses two pairwise-comparison studies.One study validates automatic feasibility metrics against human preferences, while the other ranks post-trained and baseline video generation models.
- C Human Study Detail: Both studies evaluate generated human-motion quality against human judgments.
- C Human Study Detail: Participants were non-author students and researchers in related areas who compared short generated videos.
C.1 Task Design … C.4 Human Study Significance
The human study compares synchronized, anonymized video pairs across body structure, balance, and motion naturalness, using a diverse, filtered model mixture. High inter-annotator agreement supports consistent criteria, while supplementary analyses report metric significance and per-question Elo uncertainty.
- C.1 Task Design: Annotators compare synchronized videos from the same prompt with hidden model identities and independently randomized left/right order.Each example contains two short videos generated by different models.
- C.1 Task Design: The three preference questions assess body structure, balance, and motion naturalness as distinct perceptual dimensions.Body structure covers anatomical correctness; balance concerns physical support; motion naturalness concerns resemblance to real human action.
- C.1 Task Design: Each question allows Video A, Video B, or Tie, with Tie as the default to avoid introducing preference through unanswered items.All annotators receive the same instructions, and the interface is illustrated in Fig. 8.
- C.1 Task Design: PhyMotion submetrics expose complementary failure modes in kinematic feasibility, contact/balance feasibility, and dynamic feasibility.Examples are selected to score poorly on one target submetric while scoring relatively well on the other two, with normalized z-scores reported.
- C.2 Study Composition: The dataset spans Causal Forcing 1.3B, FastWan 1.3B, Wan 2.1 1.3B, Wan2.2 5B, Wan2.2 14B, and EchoMotion 5B, pairing videos from different models under the same prompt.Unsafe videos, nearly identical pairs, and pairs where both videos already have very high feasibility scores are filtered out.
- C.2 Study Composition: PhyMotion remains competitive across general perceptual, temporal, and compositional VBench and VBench-2.0 categories while performing strongly on human- and motion-related dimensions.The radar-chart comparison supports improving human-centered physical plausibility without substantially sacrificing general video generation quality.
- C.3 Inter-Annotator Agreement: Inter-annotator agreement is high across all three questions when Tie is treated as compatible with either video and only hard opposite choices count as disagreements.Agreement is computed on overlapping video pairs in the pilot study.
- C.4 Human Study Significance: Supplementary human-study analyses report statistical significance for metric–human correlations and full per-question Elo ratings with bootstrap uncertainty.These results are provided in Tables 8 and 9 alongside the main-paper summaries.
D Detailed Definition of PhyMotion Metrics … D.3 Dynamic Feasibility
PhyMotion converts generated human videos into interpretable physical rewards by recovering SMPL-X trajectories, retargeting them in MuJoCo, and measuring kinematic, contact, and dynamic feasibility. The metrics identify anatomical and temporal errors, implausible body-ground interaction, and excessive forces, torques, or mechanical effort.
- D Detailed Definition of PhyMotion Metrics: PhyMotion recovers SMPL-X body trajectories, retargets them to a MuJoCo human model, and computes physics-based signals for evaluation and RL post-training.The implementation uses GVHMR-recovered camera-frame trajectories at f = 16.
- D Detailed Definition of PhyMotion Metrics: 55 SMPL-X joints, 10,475 mesh vertices, and 20,908 triangular faces define the recovered body representation used for metric computation.The retargeted motion also provides joint torques and ground reaction forces, with ∆t = 1/f.
- D.1 Kinematic Feasibility: Kinematic feasibility penalizes excessive joint speed, self-penetration, and joint-limit violations, rewarding smooth, anatomically plausible, and structurally stable motion.Joint-limit checks compare reconstructed poses with valid MuJoCo joint ranges, including per-degree-of-freedom averaging for multidimensional joints.
- D.2 Contact Feasibility: Contact feasibility evaluates foot sliding, ground penetration, floating feet, and balance to determine whether body-ground interactions are physically plausible.Balance uses the projected center of mass relative to the support polygon and remains continuous rather than binary to provide a more useful training signal.
- D.2 Contact Feasibility: The contact score treats a foot as contacting the ground only when it is sufficiently near the ground and nearly stationary.The implementation also checks sustained non-contact and whole-body floating at the sequence level.
- D.3 Dynamic Feasibility: Dynamic feasibility penalizes motions requiring excessive ground reaction forces, joint torques, or total mechanical effort.It combines ground-reaction-force, torque, and mechanical-work proxies into a final dynamic score.
- D.3 Dynamic Feasibility: 3 times body weight is the vertical-force penalty threshold, while 0.5 times body weight is the horizontal-force threshold.The force estimate uses m = 70 kg and g = 9.81 m/s2 under Newton’s second law.
D.4 Overall Reward
The final PhyMotion reward averages three feasibility axes, with normalized scores where higher values indicate more physically plausible human motion. Its decomposition separately measures kinematic, contact, and dynamic violations for interpretability.
- Overall Reward: The final PhyMotion reward is the average of kinematic, contact, and dynamic feasibility scores.It combines Fkin(v), Fcon(v), and Fdyn(v) into one overall reward.
E Training Details · F Licenses for Existing Assets
The paper specifies a shared training and rollout configuration, reports the checkpoint at training step 330, and documents licenses for all external assets used. The licensing table covers assets, models, datasets, codebases, and evaluation tools.
- E Training Details: The full training and rollout configuration is provided in Table 10.This configuration is used for the method’s training and rollout setup.
- E Training Details: Unless otherwise specified, all experiments use the same configuration.The shared setup serves as the default across experiments.
- E Training Details: The reported checkpoint corresponds to training step 330.The paper explicitly identifies step 330 as the checkpoint used for reporting.
- F Licenses for Existing Assets: The paper shows the licenses of all assets used in Table 11.This licensing documentation is presented for the existing assets used in the work.
- F Licenses for Existing Assets: Table 11 documents licenses for external assets used in the work.The table is specifically framed as a licensing record for external resources.
- F Licenses for Existing Assets: The licensing coverage includes models, datasets, codebases, and evaluation tools.These categories are listed alongside external assets in Table 11.