Source-linked AI summary
CQF-HMR: Continuous Quaternion Flows for Probabilistic 3D Human Mesh Recovery from a Single Image
Cuong Le, Bao-Long Tran, Pavlo Melnyk, Tahereh Dehdarirad, Bastian Wandt, Mårten Wadenbäck
TL;DR
Single-view 3D human mesh recovery is ambiguous because depth is missing, while SMPL joint rotations complicate generative modeling. CQF-HMR uses quaternion flow matching conditioned on 2D inputs to generate plausible 3D mesh hypotheses, outperforming related work on Human3.6M and remaining competitive on challenging 3DPW and EMDB.
Problem
Single-view 3D human mesh recovery must handle depth ambiguity and implausible keypoint-based poses, while SMPL rotations are non-additive and difficult for vanilla flow models.
Method
CQF-HMR learns smooth quaternion flows with optimal-transport slerp, mapping randomly sampled poses to plausible 3D hypotheses from heatmap-conditioned 2D inputs.
Results
CQF-HMR performs better on Human3.6M, particularly for ambiguous poses, while remaining competitive on 3DPW and EMDB.
Takeaways & Limitations
Quaternion flows with slerp provide advantages over traditional 3D rotation representations for generating probabilistic 3D human mesh hypotheses.
Takeaways & Limitations
CQF-HMR quality is bounded by the HRNet 2D detector, especially for ambiguity in human scale.
Abstract
from arXiv · showhide
Recovering 3D digital humans from a single 2D image is an ill-posed computer vision problem due to the loss of depth information. Probabilistic 3D human pose estimation compensates for this by estimating a set of 3D hypotheses from a prior distribution via generative models. However, most prior work focuses only on 3D keypoints, which often leads to implausible poses that are difficult to apply to downstream tasks, e.g. animation or digital humans. SMPL-based methods are more scalable thanks to the explicit body priors, but it requires more complex modeling of the generation process due to the non-additive nature of the joint rotations. In this work, we propose a novel approach for probabilistic 3D humans using quaternion-constrained continuous normalizing flows conditioned on 2D pose estimations. Our proposed quaternion flows show significant advantages over approaches using other rotation representations. Experiments demonstrate state-of-the-art results of our method on Human3.6M, particularly in ambiguous settings, and comparable pose estimation accuracy on challenging 3DPW and EMDB benchmarks.
1. Introduction
Single-image 3D human recovery is ill-posed because different 3D poses can produce the same 2D observation. CQF-HMR addresses this probabilistic recovery problem with quaternion-based flows that preserve valid rotation trajectories while modeling SMPL pose and shape.
- Single-view 3D recovery is ill-posed because multiple valid 3D poses can project to the same 2D observation.
- Generative approaches estimate a distribution or multiple 3D pose hypotheses conditioned on 2D inputs.
- Keypoint-based methods can produce unnatural poses, whereas SMPL-based methods impose explicit body kinematics constraints.
- Vanilla flow models can introduce discontinuities for non-additive joint rotations, causing incorrect 3D poses during generation.
- Unit quaternions avoid singularities and provide globally defined differential equations for smooth rotation trajectories.
- CQF-HMR uses Hamilton-product quaternion integration to preserve the unit quaternion on S3 without re-normalization.
2. Related work
Monocular 3D estimation commonly lifts 2D cues, but video-based methods require synchronized inputs unavailable in single-image settings. Probabilistic methods instead generate multiple hypotheses, while CQF-HMR combines this strategy with parallel quaternion and shape flows.
- Monocular 3D HPE primarily uses direct image estimation or lifting from 2D cues.
- 2D-to-3D lifting often improves accuracy but many methods require synchronized video inputs, limiting their use for single-image captures.
- Single-image lifting methods can use graph networks, but traditional approaches often predict only 3D keypoints without full body constraints.
- Probabilistic 3D HPE uses generative models to produce multiple hypotheses approximating the posterior distribution of 3D poses.
- CQF-HMR runs quaternion pose flows and shape normalizing flows in parallel, then converts predicted pose and shape into a 3D mesh through SMPL.
3. Method
CQF-HMR models 3D human mesh recovery with quaternion-constrained continuous flows conditioned on 2D pose evidence. It combines spherical quaternion transport, SMPL body priors, shape flows, numerical integration, and end-to-end training to generate plausible meshes.
- Quaternion flow matching: CQF-HMR learns quaternion-valued continuous normalizing flows that map prior joint rotations to ground-truth poses using 2D-derived conditioning.The angular velocity field is parameterized by an MLP and conditioned on a vector extracted from 2D inputs.
- Quaternion differential equation: Hamilton-product integration preserves the unit-quaternion constraint on S3 without re-normalization, producing smooth continuous trajectories for 3D joint rotations.Quaternion integration is required because ordinary addition does not preserve unit length.
- Quaternion flow matching: Spherical linear interpolation defines the optimal-transport path between sampled and ground-truth quaternions, yielding a constant target angular velocity for regression.The method replaces linear interpolation with slerp to respect quaternion geometry during flow matching.
- Pose initialization: The pose prior can use VPoser latent samples rather than Gaussian axis-angle samples to avoid highly implausible initial poses caused by human kinematic-chain constraints.The VPoser latent embedding z ∈ R32 is sampled from a Gaussian distribution and decoded into a human-body prior.
- 2D conditioning: A graph network converts top-k HRNet 2D heatmap candidates into a conditioning vector, which is combined with quaternion state and time for the QDE.Candidate poses are normalized and randomly permuted during generation before graph processing.
- Shape estimation and recovery: Shape generation uses vanilla flow matching with linear interpolation for the 10-dimensional SMPL shape parameter, while RK2 numerically solves both pose and shape flows.The predicted pose and shape are iteratively solved and passed through SMPL forward kinematics to obtain a mesh and regressed skeleton.
4. Experiments
CQF-HMR is evaluated on Human3.6M, 3DPW, and EMDB using standard and ambiguity-focused protocols. It improves over probabilistic baselines on Human3.6M, especially ambiguous poses, while remaining competitive on 3DPW and EMDB; quaternion slerp yields the strongest rotation-representation results.
- Evaluation protocol: The experiments report MPJPE and PA-MPJPE in millimeters, with probabilistic methods evaluated using the minimum error among sampled hypotheses.Human3.6M includes regular and ambiguous splits, while 3DPW and EMDB test outdoor, occluded, or moving-camera conditions.
- Comparisons to related work: On 3DPW, CQF-HMR (N) improves MPJPE by 2.3% (1.5mm) over ScoreHypo but has 1.5% (0.6mm) worse PA-MPJPE, while CQF-HMR (D) performs worse on both metrics.The authors attribute this setting’s difficulty to greater occlusion and uncertainty.
- Comparisons to related work: On EMDB, CQF-HMR (N) is competitive with HuManiFlow and ScoreHypo, with differences below 1% on MPJPE and 3% on PA-MPJPE.EMDB contains poses captured with a moving camera and occlusion, and CQF-HMR is evaluated without extra finetuning.
- Comparison to different 3D rotations: Quaternion slerp outperforms axis-angle and Euler-angle flows in deterministic MPJPE, improving by 47.6% and 8.5% with Gaussian initialization and by 33.9% and 8.5% with VPoser.The authors link weaker alternatives to integration limitations such as discontinuity and gimbal lock; quaternion slerp with VPoser is strongest across all metrics.
5. Conclusion
CQF-HMR uses quaternion flow matching with optimal-transport slerp for probabilistic 3D human mesh recovery. It performs better on Human3.6M and remains competitive on challenging 3DPW and EMDB settings, while current quality depends on 2D detector performance.
- CQF-HMR studies quaternion flow matching with optimal-transport slerp for probabilistic 3D human mesh recovery.
- CQF-HMR shows better performance on Human3.6M while remaining competitive with comparable methods on challenging 3DPW and EMDB settings.
- Its quality is bounded by the HRNet 2D detector, particularly for human-scale ambiguity caused by similar projections of far large and close small people.
6. Additional experiments
Additional experiments compare conditioning architectures and the computational cost of rotation representations. GCN offers the best complexity–error trade-off, while quaternion slerp achieves the lowest pose error with a modest speed penalty from representation conversions.
- The GCN conditioning module achieves the best trade-off between complexity and pose prediction error.The transformer has slightly lower MPJPE in the Gaussian setting but nearly twice the processing time for 100 hypotheses.
- Quaternion slerp achieves the lowest pose error, while inference takes approximately 10–20 ms longer than lerp or common representations.The main bottleneck is conversion between SMPL axis-angle and quaternion representations.
- The quaternion slerp speed penalty remains an open optimization challenge because of repeated axis-angle-to-quaternion conversions.