PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregation

Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Zhao Wang, Kai Han, Shanshe Wang, Siwei Ma, Wen Gao

arXiv:2303.11579v2cs.CV

TL;DR

Probabilistic 3D pose estimation must generate multiple plausible poses while producing a single useful prediction without ground-truth selection. The paper introduces compatible, customizable diffusion-based generation with joint-wise reprojection aggregation, and reports superiority over deterministic and probabilistic approaches on two benchmarks.

  • Problem

    Existing probabilistic methods have limited compatibility or hypothesis control, while pose-level averaging overlooks joint differences and 2D keypoint priors.

  • Method

    D3DP diffuses 3D poses and learns a 2D-conditioned denoiser, while JPMA selects and combines joints from multiple hypotheses using reprojection errors.

  • Results

    D3DP with JPMA surpasses state-of-the-art probabilistic and deterministic 3D human pose estimation approaches on two benchmarks.

  • Takeaways & Limitations

    Joint-level aggregation exploits differences among hypotheses and provides a practical alternative to ground-truth-dependent pose-level selection.

  • Takeaways & Limitations

    JPMA remains below the J-Best upper bound, with 39.48mm versus 33.99mm, leaving considerable room for improvement.

Abstract

from arXiv · show

In this paper, a novel Diffusion-based 3D Pose estimation (D3DP) method with Joint-wise reProjection-based Multi-hypothesis Aggregation (JPMA) is proposed for probabilistic 3D human pose estimation. On the one hand, D3DP generates multiple possible 3D pose hypotheses for a single 2D observation. It gradually diffuses the ground truth 3D poses to a random distribution, and learns a denoiser conditioned on 2D keypoints to recover the uncontaminated 3D poses. The proposed D3DP is compatible with existing 3D pose estimators and supports users to balance efficiency and accuracy during inference through two customizable parameters. On the other hand, JPMA is proposed to assemble multiple hypotheses generated by D3DP into a single 3D pose for practical use. It reprojects 3D pose hypotheses to the 2D camera plane, selects the best hypothesis joint-by-joint based on the reprojection errors, and combines the selected joints into the final pose. The proposed JPMA conducts aggregation at the joint level and makes use of the 2D prior information, both of which have been overlooked by previous approaches. Extensive experiments on Human3.6M and MPI-INF-3DHP datasets show that our method outperforms the state-of-the-art deterministic and probabilistic approaches by 1.5% and 8.9%, respectively. Code is available at https://github.com/paTRICK-swk/D3DP.

1. Introduction

Monocular 3D human pose estimation lifts detected 2D joints into 3D, but probabilistic methods must produce a practical single pose from multiple plausible hypotheses. D3DP addresses compatibility and aggregation limitations with diffusion-based generation and joint-wise reprojection-guided selection.

  • Monocular 3D human pose estimation maps 2D joint locations to corresponding 3D positions after 2D keypoint detection.
  • Probabilistic approaches represent 2D-to-3D lifting as a distribution and generate possible solutions, whereas deterministic approaches produce one pose per image.
  • Existing probabilistic methods face limited compatibility or hypothesis-count control, then commonly average hypotheses at the pose level without modeling joint differences or 2D keypoint priors.
  • D3DP diffuses ground-truth 3D poses, learns a denoiser conditioned on 2D keypoints, and generates multiple hypotheses from Gaussian noise while remaining compatible with existing estimators.
  • JPMA reprojects hypotheses to the camera plane and selects the best hypothesis separately for each joint, combining selected joints into one final prediction.

2. Related Work

Diffusion models motivate probabilistic 3D pose generation because depth ambiguity makes multiple plausible solutions appropriate. Related methods generate hypotheses but often remain difficult to integrate, restrict hypothesis counts, or aggregate without joint-level and 2D-guided reasoning.

  • Diffusion models gradually add noise to observed data and reverse the process to recover the original data; their high-fidelity sampling motivates application to ambiguous 3D pose estimation.
  • Prior probabilistic pose methods use heuristic sampling, mixture density networks, normalizing flows, conditional variational autoencoders, or generative adversarial networks to generate plausible 3D poses.
  • Earlier methods can have poor compatibility with existing estimators or cannot specify the number of hypotheses according to actual needs.
  • Existing aggregation methods often average pose hypotheses or select one pose, ignoring differences between joints and information in 2D keypoints.
  • JPMA uses reprojection errors between hypotheses and input 2D keypoints for joint-level selection, extending reprojection-based reasoning to probabilistic estimation.

3. Method

D3DP learns to denoise corrupted 3D poses conditioned on 2D keypoints, samples multiple hypotheses, and optionally refines them through repeated reverse diffusion. JPMA then reprojects those hypotheses and selects joints independently using reprojection error.

  • D3DP: D3DP uses a forward diffusion chain that perturbs poses with noise and a reverse denoising chain that reconstructs uncontaminated 3D poses.
  • D3DP: During training, Gaussian noise corrupts a ground-truth pose at timestep t, and a denoiser conditioned on 2D keypoints and t reconstructs the clean pose using an MSE loss.
  • D3DP: During inference, H unit-Gaussian samples initialize noisy 3D poses that the denoiser converts into multiple feasible pose hypotheses.
  • D3DP: The reverse process can be iterated K times with DDIM-generated noisy poses, making the number of hypotheses H and refinement iterations K customizable.
  • JPMA: JPMA reprojects hypotheses using camera parameters, measures each joint’s distance to the input 2D keypoint, and combines the minimum-error joint selections.
  • Design properties: D3DP can use existing deterministic pose estimators as denoiser backbones, while JPMA adds negligible computational overhead because it mainly performs reprojection.

4. Experiments

Experiments evaluate D3DP and JPMA on Human3.6M, MPI-INF-3DHP, and 3DPW, showing gains over deterministic and probabilistic baselines and benefits from joint-level aggregation. Ablations further show that increasing hypotheses and iterations can improve performance, with computational trade-offs controlled by H and K.

  • Human3.6M: D3DP achieves 40.0mm MPJPE on Human3.6M with one hypothesis, outperforming DUE by 0.6mm and MixSTE by 1mm.The single-hypothesis result shows improvement even though D3DP is designed to generate multiple pose hypotheses.
  • MPI-INF-3DHP: D3DP reduces MPJPE by 5.2mm over MixSTE on MPI-INF-3DHP with one hypothesis, and JPMA is similar to P-Best and better than P-Agg when H=20.The same performance pattern as Human3.6M is reported, supporting generalization across datasets.
  • Ablation Study: 39.5mm MPJPE is obtained with JPMA after raising H to 20, improving over 39.9mm under pose-level averaging.The ablation attributes the gain to fine-grained joint-level hypothesis selection.
  • Ablation Study: Joint-level aggregation outperforms pose-level aggregation, and JPMA outperforms joint-level MLP scoring by using reprojection-based 2D priors.These comparisons indicate that both aggregation granularity and reprojection guidance contribute to performance.
  • Ablation Study: Increasing H and K raises MACs and lowers FPS roughly linearly, while performance under all settings saturates at K=30.D3DP therefore exposes inference-time controls for trading computational cost against performance.

5. Conclusions

The paper presents D3DP with JPMA, combining diffusion-based probabilistic 3D pose generation with joint-wise reprojection-based aggregation into a single prediction.

  • D3DP generates multiple 3D pose hypotheses, while JPMA aggregates them joint-by-joint using reprojection errors.The method conducts joint-level aggregation rather than pose-level aggregation to exploit the merits of individual joints.

A. Preliminary: Diffusion Model

Diffusion models corrupt clean data with progressively added noise and learn a reverse process to reconstruct it. This paper uses a denoiser that predicts the original 3D pose from noisy inputs.

  • The diffusion process adds Gaussian noise to the original signal across timesteps until the distribution approaches an isotropic Gaussian.The noise variance follows a schedule β_t, and sufficiently large T yields q(y_T) close to N(0, I).
  • The reverse process iteratively denoises samples or predicts clean data directly from a noisy state.The paper chooses the direct clean-data prediction formulation, where f_θ(y_t, t) predicts y_0.
  • The iterative reverse formulation approximates the unknown posterior because the true clean signal is unavailable during inference.A neural network predicts the noise used to approximate the posterior transition.

B. Preliminary: Projective Geometry

JPMA reprojects predicted 3D joints into the camera plane to compare them with 2D observations. The reprojection function supports both ideal and lens-distorted camera models.

  • JPMA treats the original image formation as a first projection and maps predicted 3D hypotheses back to the camera plane through reprojection.The method defines P(·) for pinhole and distorted pinhole cameras.
  • Under the pinhole model, a joint at 3D coordinates (X, Y, Z) is transformed into 2D coordinates (u, v) using camera intrinsics.The intrinsic parameters include focal lengths and the principal point.
  • The distorted pinhole model accounts for radial and tangential lens distortion using additional camera parameters.Radial distortion uses k1, k2, k3, while tangential distortion is approximated using p1 and p2.
  • JPMA uses ground-truth intrinsics for calibrated cameras and estimated parameters otherwise.The latter case uses a network to estimate the camera parameters.

C. Algorithm

D3DP trains a denoiser to recover clean 3D poses from corrupted poses conditioned on 2D keypoints, then generates hypotheses through iterative denoising. JPMA-related aggregation and implementation choices are evaluated through ablations.

  • Training: D3DP training corrupts ground-truth 3D poses with timestep-dependent Gaussian noise and trains a denoiser conditioned on noisy poses, 2D keypoints, and the timestep.The model predicts the clean 3D pose using an MSE objective.
  • Inference: D3DP inference samples H initial 3D poses from a Gaussian distribution and performs K denoising iterations.Each iteration predicts an uncontaminated pose and uses DDIM to construct the next noisy input.
  • Ablation results: 40.0 mm MPJPE is obtained with diffusion-flipping, compared with 40.3 mm for flipping-once and 40.6 mm without augmentation.These values are reported in the D3DP ablation on Human3.6M.
  • Ablation results: The ablations compare timestep-embedding locations, 2D-conditioning mechanisms, and the maximum number of diffusion timesteps.The reported MPJPE values include 40.0 for first-layer or all-layer timestep embeddings and 40.0 for input-fusion conditioning.
  • Data augmentation: Diffusion-flipping averages predictions from original and horizontally flipped inputs during each diffusion iteration.Both 2D keypoints and noisy 3D poses are flipped before denoising, then predictions are flipped back and combined.

D.1. Components of D3DP

The ablations examine D3DP’s regression target, timestep embedding, augmentation, conditioning, timestep count, camera modeling, and hypothesis behavior. They also test compatibility across backbone estimators and show that joint-level aggregation benefits from reprojection-aware selection.

  • Regression target: Predicting the original 3D data y0 outperforms predicting noise ϵt and remains effective in early iterations.This supports producing satisfactory predictions after limited computation when K is fixed and resources are inadequate.
  • Timestep embedding: Timestep embedding is crucial to denoising, while placing it in the first network layer performs the same as placing it in all layers.The first-layer placement is selected for simplicity.
  • Data augmentation: Diffusion-flipping achieves the best augmentation results by averaging original and flipped branches at each timestep, preventing error accumulation.Unlike flipping-once, it applies the flip-denoise-flip process at every timestep.
  • 2D conditioning: Concatenating noisy 3D poses and 2D conditions at the input provides the best fusion approach and a fast way to modify existing estimators.The comparison covers concatenation, addition, and cross attention across input and embedding fusion.
  • Maximum number of timesteps: An appropriate maximum timestep count is necessary because too few or too many timesteps impair diffusion training and denoising.Too few timesteps prevent diffusion to a Gaussian distribution, while too many create excessive pure-noise samples.
  • Compatibility and performance: D3DP gains performance across different backbone networks, supporting its compatibility and versatility.The evaluation uses other 3D estimators with H=K=1.
  • Hypothesis count: As H increases, average-hypothesis error stays around 39.9mm while best-hypothesis performance improves and worst-hypothesis performance worsens.The result indicates increasing variance among hypotheses without materially changing their mean performance.

E. Additional Quantitative Results

Additional experiments compare aggregation settings, analyze hypothesis diversity, and evaluate generalization. Joint-level aggregation improves results over pose-level alternatives, while failures remain possible for difficult or uncommon poses.

  • Aggregation results: Joint-level settings J-Agg and J-Best further enhance performance over conventional pose-level settings P-Agg and P-Best.The results support disentangling hypotheses at the joint level, and the method surpasses competing approaches by a wide margin.
  • Benchmark evaluation: D3DP with JPMA achieves favorable Human3.6M results using detected or ground-truth 2D keypoints and converts an existing deterministic estimator into a probabilistic version.The evaluation reports P-MPJPE with detected keypoints and MPJPE with ground-truth keypoints.
  • Hypothesis diversity: As H increases, joint-level aggregation benefits from more hypotheses while the average hypothesis remains essentially unchanged and the worst hypothesis degrades.The best hypothesis improves as variance rises.
  • In-the-wild evaluation: The method achieves satisfactory performance in most in-the-wild frames and can generalize to animations and monkey poses because it uses 2D keypoints as inputs.Evaluated videos include 3DPW, Penn Action, JHMDB, and YouTube.

F.3. Analysis of Failure Cases

Failure analysis focuses on qualitative behavior across hypothesis counts and diffusion iterations. The method can produce diverse hypotheses, but severe occlusion, fast motion, and rare poses remain practical failure cases.

  • Failure cases: The method may fail under severe occlusions, fast motion, and rare poses.The reported example includes a person walking through woods whose body is mostly occluded and whose pose is reconstructed inaccurately.
  • Hypothesis and iteration effects: Increasing K gradually raises hypothesis variance and improves diversity because DDIM re-adds different noise between iterations.Increasing H also raises variance while leaving the mean essentially unchanged.
  • Qualitative evaluation: Figures 9 and 10 compare qualitative pose hypotheses across H and K and show predicted 3D hypotheses for in-the-wild videos.Figure 10 uses dashed lines for predicted hypotheses, with each color representing an individual hypothesis.
Loading 2303.11579v2…