PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoMDM: Towards 3D Human Motion Generation From 2D Supervision

Amir Mann, Gal Michael Harari, Merav Keidar, Or Litany

arXiv:2606.13364v1cs.LGcs.CV

TL;DR

Training 3D human motion models from diverse monocular videos is difficult because reliable 3D supervision is scarce, while accurate 2D poses are available. VideoMDM learns a 3D diffusion prior using 2D reprojection supervision and nearly matches fully 3D-supervised MDM on HumanML3D, while earning human preference on real-video datasets.

  • Problem

    The central challenge is training generative 3D motion models from accurate 2D poses in monocular videos, which lack reliable 3D supervision.

  • Method

    VideoMDM trains a 3D diffusion model from 2D supervision by denoising lifted 3D poses and matching their depth-aware 2D reprojections to observed keypoints.

  • Results

    FID 0.88 versus 0.54 for fully 3D-supervised MDM on HumanML3D, while VideoMDM is human-preferred on Fit3D and NBA.

  • Takeaways & Limitations

    VideoMDM provides a path toward learning coherent 3D motion priors from abundant monocular videos without MoCap or 3D supervision.

  • Takeaways & Limitations

    The method depends on a pretrained 2D-to-3D lifter and has been evaluated mainly with no or minimal occlusions, limiting fully in-the-wild deployment.

Abstract

from arXiv · show

We introduce VideoMDM, a diffusion-based framework that trains 3D human motion priors directly from accurate 2D poses extracted from monocular videos, without any 3D ground truth. A pretrained 2D-to-3D lifter provides approximate 3D pose sequences that serve as a noisy teacher: these are diffused, denoised by the model in 3D, and supervised in 2D by reprojecting the prediction and comparing against accurate keypoints. We show that, under mild assumptions, a depth-weighted 2D reprojection loss is equivalent in expectation to direct 3D supervision, and we adapt standard 3D motion regularizers - velocity consistency and over-parameterized representation alignment - to this 2D setting. Unlike methods that lift 2D to 3D only at inference, VideoMDM learns a coherent 3D motion manifold during training. On HumanML3D it nearly closes the gap to fully 3D-supervised MDM (FID 0.88 vs 0.54); On real video datasets Fit3D and NBA the method learns to generate motions consistently preferred by humans, with strong quantitative results.

1 Introduction

VideoMDM trains a coherent 3D human-motion diffusion prior directly from accurate 2D poses in monocular videos, avoiding the need for 3D ground truth. Its depth-aware reprojection, motion regularization, and evaluation results support scalable learning from diverse in-the-wild data.

  • Motivation and contribution: VideoMDM is a 2D-supervised diffusion framework that trains 3D human-motion priors directly from monocular videos without 3D supervision.It addresses the limited diversity of controlled MoCap datasets and the ambiguity of monocular 3D estimation by using accurate 2D keypoints.
  • Method: A pretrained 2D-to-3D lifter supplies noisy pose sequences that are diffused, denoised in 3D, and supervised by 2D reprojection against video keypoints.This noisy-teacher scheme grounds the learned model in 2D observations while enabling a coherent 3D motion manifold.
  • Method: Depth-aware reprojection weighting is provably equivalent in expectation to standard 3D MSE under mild data and camera-distribution assumptions.The framework also adapts velocity-based and representation-level regularizers to constrain motion channels without 3D ground truth.
  • Evaluation: 0.88 FID is achieved on 2D-only HumanML3D, nearly matching fully 3D-supervised MDM at 0.54 and improving on the strongest 2D-supervised baseline by roughly x2.On real-world Fit3D monocular fitness video without 3D supervision, VideoMDM halves joint error against WHAM on motions far outside the reported training distribution.

2 Related Work

Prior 3D human motion generation advances include VAEs, diffusion models, and VQ-VAE-based autoregressive approaches, but these methods typically rely on high-quality MoCap-derived 3D motion.

  • Human Motion Generation in 3D: 3D human motion generation has progressed from VAEs to diffusion models and VQ-VAE-based autoregressive methods that improve motion quality.MDM is cited as substantially improving fidelity among diffusion models, while VQ-VAEs with autoregressive and bidirectional autoregressive models establish state-of-the-art quality.
  • Human Motion Generation in 3D: These approaches typically train on high-quality 3D motion captured by MoCap systems, including HumanML3D built from AMASS.The passage identifies HumanML3D and AMASS as examples of MoCap-derived 3D motion data.

3 Method

VideoMDM trains a 3D diffusion motion model from accurate 2D joint trajectories and approximate 3D poses produced by a pretrained lifter. It denoises diffused 3D poses while applying depth-aware 2D reprojection, velocity, and representation-alignment supervision.

  • Inputs and objective: A pretrained 2D-to-3D lifter converts accurate monocular-video joint trajectories into approximate 3D motions that serve as noisy teachers.The input 2D trajectories have shape y ∈ R^{J×2×F}, while the lifter produces ˜x0 ∈ R^{J×3×F}.
  • Diffusion training: The model diffuses the lifter’s 3D predictions and denoises them in 3D, but supervises predictions in 2D through a camera projection operator.The diffused input is xt = √αt˜x0 + √(1−αt)ϵ, with ϵ∼N(0,I), and the projection uses camera intrinsics, rotation, and translation.
  • Depth-aware reprojection: Depth-aware weighting removes perspective-induced 1/d scaling, making 2D reprojection loss equivalent in expectation to direct 3D MSE under two mild assumptions.A depth threshold excludes joints with unreliable predicted depths, especially joints behind or very close to the camera.
  • Motion regularization: The method adds 2D velocity consistency to enforce temporal similarity between generated motions and their supervision.The 2D motion is obtained by projecting the denoised 3D prediction as ˆy0 = Πc(ˆx0).
  • Motion regularization: Representation alignment derives 2D-consistent pseudo-targets for redundant motion channels, supervising the over-parameterized representation without 3D ground truth.The redundant channels include rotations, joint velocities, and foot-contact labels, which can be derived from joint positions.
  • Training schedule: Training first warms up on lifter-predicted 3D motions, then applies the full loss for t > t∗ and multi-step denoising for t ≤ t∗.This staged schedule transitions from lifter-based supervision to fully 2D-derived constraints.

4 Experiments

VideoMDM is evaluated across synthetic 2D-supervised HumanML3D, real monocular Fit3D, and centered NBA basketball data. Across these settings, it reduces lifter artifacts, generalizes to out-of-distribution motions, and performs favorably against established baselines.

  • Experimental Settings: Experiments span 2D-only HumanML3D, real monocular Fit3D fitness videos, and centered NBA basketball sequences, isolating supervision, distribution shift, and comparison with MAS.HumanML3D isolates supervision from pose-estimation errors; Fit3D tests motions outside lifter distributions; NBA provides a head-to-head setting favorable to MAS.
  • HumanML3D: On HumanML3D, VideoMDM produces clean, prompt-consistent trajectories, whereas MDM trained on lifter outputs inherits sliding, drifting, and unrealistic-pose artifacts.The comparison includes MDM/MotionBERT and MDM/MVLift baselines; the latter is better but still lacks coherence.
  • Fit3D: On Fit3D, VideoMDM is evaluated on 611 training sequences across 37 exercises, including motions absent from HumanML3D, using RTMPose 2D poses and WHAM as the noisy teacher.Accurate 3D ground truth is used only for evaluation, while camera positions are provided or estimated with PnP.
  • HumanML3D: 0.332 FID is the remaining gap between Ours/MVLift and 3D-supervised MDM on the HumanML3D test split.Table 1 evaluates text-to-motion models trained with 2D supervision and reports Ours/MVLift relative to 3D-supervised MDM.
  • Fit3D: On Fit3D text-to-motion evaluation, Ours/WHAM consistently outperforms all baselines in human preference, while its PnP-camera variant loses to WHAM baselines.Qualitative examples also show coherent, text-aligned motions for exercises outside the baselines’ training distribution.
  • NBA: On NBA, VideoMDM is compared with MAS under centered, no-text conditions using ElePose lifted motions as the noisy teacher and the released evaluation protocol.The protocol uses embeddings from a 2D VAE trained on the same data; the paper additionally reports Recall† anchored to real-distribution spread because standard Recall can favor exaggerated diversity.

5 Conclusion, Limitations and Future Work · A Weights for 3D to 2D Loss Equivalence · B Projection of a 3D Point onto a 2D Camera Ray

VideoMDM demonstrates that 2D-supervised diffusion can approach fully 3D-supervised performance while remaining limited by camera, lifter, pose-extraction, and occlusion challenges. The appendices establish the weighted 2D-to-3D loss equivalence and define orthogonal projection of 3D points onto camera rays.

  • 5 Conclusion, Limitations and Future Work: VideoMDM generates high-quality 3D motions using only 2D supervision and nearly matches fully 3D-supervised methods in some settings.On synthetic HumanML3D it recovers most of the performance gap, while Fit3D shows a larger drop from compounded camera, lifter, and pose-extraction noise.
  • 5 Conclusion, Limitations and Future Work: VideoMDM depends on a pretrained 2D-to-3D lifter, limiting application to domains without a reasonable lifter, including non-human motion.The learned prior generalizes substantially beyond the lifter’s distribution, but this dependency remains a boundary of the method.
  • 5 Conclusion, Limitations and Future Work: Current evaluations contain no or minimal occlusions, so extending VideoMDM to heavily occluded in-the-wild settings remains necessary.The passage identifies occluded settings such as those in [11] as a target for future deployment.
  • A Weights for 3D to 2D Loss Equivalence: Under the constant-depth-denominator assumption, depth-proportional weights Wu and Wv make the 2D loss equal the 3D loss.The derivation considers a single 3D point, decomposes the loss over coordinates, and defines perspective projection using camera elevation ψ and azimuth θ.
  • A Weights for 3D to 2D Loss Equivalence: The equivalence derivation combines image u-axis and v-axis contributions, whose coordinate-error coefficients sum to one for the x and z terms.The calculation uses linearity of integration and trigonometric identities over the camera angles.
  • B Projection of a 3D Point onto a 2D Camera Ray: For a calibrated pinhole camera, normalized image coordinates define a camera-frame ray that can be transformed into a world-frame ray using the camera extrinsics.The camera center is expressed in world coordinates, and the ray is represented as a line through that center.
  • B Projection of a 3D Point onto a 2D Camera Ray: The orthogonal projection of a world point onto the camera ray is its closest point on that ray under squared Euclidean distance.The operation uses the camera center and ray direction, with orthogonality of the rotation matrix simplifying the expression.
  • B Projection of a 3D Point onto a 2D Camera Ray: VideoMDM applies this ray projection separately to each frame and joint, denoted PΠ(ˆx0, y) in the paper.The projection operates on the predicted 3D point and its normalized image coordinate.

C Experiments Technical Details · C.1 HumanML3D Data Processing

The HumanML3D training set is converted to a 2D-only benchmark by random camera projection and removal of 3D ground truth. Noisy 3D teachers are then reconstructed by lifting the projected 2D motions with MotionBERT or MVLift.

  • C.1 HumanML3D Data Processing: HumanML3D motions are paired with cameras sampled using azimuth ∼U[−π, π] and elevation ∼U[0, π/8].Camera position is constrained so the closest joint is at least 3 units from the camera.
  • C.1 HumanML3D Data Processing: The sampled cameras project each 3D motion into 2D keypoint sequences.
  • C.1 HumanML3D Data Processing: The 3D ground truth is discarded from the resulting training set.
  • C.1 HumanML3D Data Processing: Noisy 3D teachers are obtained by lifting the 2D motions back to 3D.The lifting step supplies the teacher required by VideoMDM.
  • C.1 HumanML3D Data Processing: The experiments use either MotionBERT [57] or MVLift [25] for 2D-to-3D lifting.
  • C.1 HumanML3D Data Processing: Because MotionBERT produces centered poses, the processing recovers an uncentered representation.The supplied passage ends after stating that uncentering is recovered, without specifying the recovery procedure.

C.2 HumanML3D Evaluation Metrics

The HumanML3D evaluation measures motion quality, diversity, text–motion alignment, fidelity to ground-truth motions, and within-prompt variation. Results are averaged over 20 generations with 95% confidence intervals.

  • FID measures statistical similarity between real and generated motion features, while Diversity measures average distance between random motion pairs.
  • R-Precision Top-3 evaluates text–motion alignment in a shared embedding, and Multimodal Distance measures distance to ground-truth motions.
  • Multimodality captures variance across samples generated from the same prompt; all metrics use 20 generations and 95% confidence intervals.

C.3 Fit3D Data Processing · C.4 Clip Extraction for Fit3D

Fit3D processing smooths RTMPose 2D sequences, converts them to SMPL joints, and creates HumanML3D-style text descriptions. A dynamic-programming procedure then extracts natural clips with controlled lengths and motion-aware split points, producing 1,161 clips.

  • C.3 Fit3D Data Processing: RTMPose 2D sequences are smoothed with weighted temporal mean weights (0.25, 0.5, 0.25) and converted to SMPL joint format.The conversion uses manually picked weights.
  • C.3 Fit3D Data Processing: Processed sequences are split into clips matching the HumanML3D length range using the dynamic-programming procedure.Text descriptions are generated from exercise names in HumanML3D’s natural-language style.
  • C.3 Fit3D Data Processing: WHAM outputs raw-image poses directly in SMPL format, whereas MVLift requires conversion to 17-keypoint skeletons.Fit3D work relying on 2D pose estimation uses RTMPose pseudo-GT 2D poses.
  • C.4 Clip Extraction for Fit3D: Dynamic programming segments variable-duration raw sequences into clips that meet a desired length range, avoid high-motion frames, and favor rest-like poses.The split cost combines local-pose deviation from the rest pose with nearby local velocity magnitude.
  • C.4 Clip Extraction for Fit3D: The procedure uses λpose = 30000 and λvel = 15000 to weight pose and velocity terms in the split cost.The pose term is an MSE against the rest local pose, while the velocity term is the average local velocity magnitude around the candidate frame.
  • C.4 Clip Extraction for Fit3D: Lmin = 60, Ltarget = 150, and Lmax = 300 define the clip-length constraints used during segmentation.Dynamic programming evaluates valid previous split points and backtracks from the best terminal split to recover segment boundaries.
  • C.4 Clip Extraction for Fit3D: 1,161 clips are produced across all sequences, including subject s09, with stable alignment, avoided erratic split points, and consistent durations.The resulting clip-length distribution is summarized in Figure 7.

C.5 NBA Data Processing

NBA motions are scaled and centered, making recovery of a real pinhole camera infeasible. The method therefore follows MAS’s camera convention and uses ElePose as a noisy teacher.

  • Scaled and centered NBA motions make recovering a real pinhole camera from the data infeasible.
  • The processing adopts MAS’s convention of placing the camera 7 units along the Z-axis at π/16 elevation.
  • ElePose serves as the noisy teacher for NBA motion processing.

C.6 Compute Cost Estimates … Participant Consent Form

The appendices document compute requirements, auxiliary depth and recall formulations, hyperparameter choices, diffusion procedures, and human-evaluation protocols. Together, these sections specify implementation costs, methodological assumptions, sampling details, and survey procedures.

  • C.6 Compute Cost Estimates: All experiments used one NVIDIA RTX 4090 GPU, while approximately 1,120 GPU-hours were spent on the hyperparameter sweep and excluded from per-method costs.Lifter preprocessing was a one-time shared cost, with pseudo-3D labels reused across training runs.
  • D Naive Root Depth Estimation: The naive root-depth estimator uses the maximum projected length among three root-adjacent bones, assuming uniformly random orientations and known bone lengths.The assumptions imply at least one root-adjacent bone is likely nearly perpendicular to the camera optical axis.
  • E Formal Definition of Recall†: A recall variant computes nearest neighbors in the real distribution to avoid bias toward exaggerated diversity from generated-neighbor hyperspheres.The standard procedure defines closeness using each generated sample’s K-th nearest neighbor within the generated manifold.
  • F Hyper Parameter Choices: Hyperparameter search selected λvel = 287 and t∗= 30 with #steps = 3 on HumanML3D, versus λvel = 187, t∗= 23, and #steps = 4 on NBA.MDM follows the original configuration with T = 50 diffusion steps as one stated exception.
  • G.1 DDIM Used in Training: Multistep training samples three DDIM time buckets for t∗= 12 and b = 3, producing t3 ∼U(12, 17), t2 ∼U(6, 11), and t1 ∼U(0, 5).Batches with t ≥t∗ use the standard diffusion loss, while lower-t batches are iteratively denoised and supervised.
  • G.2 DDPM Used in Lifting: DDPM sampling provides the full-sampling procedure for lifting, with 2D ray-projection guidance inserted during inference.The guidance is described as a specific instance of guided diffusion and a relaxation of a method for unknown cameras.
  • H Human Preference Survey: Human surveys used anonymous Prolific crowdworkers, randomized method identities and left/right positions, and required both clips to finish before choices were enabled.The NBA survey involved 20 participants and the Fit3D survey involved 52 participants, using 10 and 20 comparison pairs respectively.
  • Participant Consent Form: The consent form described a 2–4 minute study involving 20 video-pair judgments, anonymous responses, voluntary participation, no known risks, and compensation upon completion.Participants evaluated perceptual quality by indicating which generated motion appeared more realistic.

Motion Generation Survey · I Camera Parameters Estimation · J Explicit HumanML Channel Partitioning

The paper evaluates motion realism through pairwise human preference surveys, estimates missing camera parameters with EPnP and refinement, and explicitly partitions HumanML3D motion channels into position, rotation, and root-motion components. It also specifies the resulting channel dimensions for x and r.

  • Motion Generation Survey: Participants evaluate 20 motion-clip pairs for the same text prompt by choosing the animation that looks more realistic and natural.Both clips play automatically, and participants may replay them before choosing.
  • Motion Generation Survey: The survey takes approximately 2–4 minutes, with participants compensated at an average effective rate of £15.53/hour.The reported rate covers both surveys.
  • I Camera Parameters Estimation: When camera parameters are unavailable, the method estimates them by solving EPnP over the first 24 pose frames and refining the pose with Levenberg-Marquardt.It uses MVLift on HumanML3D, WHAM on Fit3D, and OpenCV solvers with default parameters.
  • J Explicit HumanML Channel Partitioning: HumanML3D’s representation includes root angular velocity, planar root velocity, root height, and three channels per non-root joint for position.Joint positions use X and Z in the root coordinate frame and Y globally.
  • J Explicit HumanML Channel Partitioning: Each non-root joint contributes six channels for continuous 6D rotation representations relative to the rest pose and three channels for joint features.Each rotation is calculated as normalized displacement from the joint’s ancestor.
  • J Explicit HumanML Channel Partitioning: The explicit partition defines x with AJ = 4+(J −1)×3 channels and r with BJ = (J −1)×6+J ×3+4 channels.These formulas summarize the position-like and rotation-related channel groups.
Loading 2606.13364v1…