PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffPose: Toward More Reliable 3D Pose Estimation

Jia Gong, Lin Geng Foo, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, Jun Liu

arXiv:2211.16940v3cs.CV

TL;DR

Monocular 3D pose estimation must handle depth ambiguity, occlusion, and the resulting uncertainty. DiffPose addresses this by modeling estimation as context-conditioned reverse diffusion with pose-specific initialization and GMM-based forward diffusion, achieving state-of-the-art performance on two benchmarks. The authors identify extension to online, few-shot, and other pose-based settings as future work.

  • Problem

    Monocular 3D pose estimation remains challenging because depth ambiguity and occlusion lead to high uncertainty and indeterminacy.

  • Method

    DiffPose initializes a pose-specific uncertainty distribution, uses GMM-based forward diffusion, and performs context-conditioned reverse diffusion with a GCN-based model.

  • Results

    DiffPose achieves state-of-the-art performance on the Human3.6M and MPI-INF-3DHP benchmark datasets.

  • Takeaways & Limitations

    DiffPose provides a diffusion-based framework that handles uncertainty and indeterminacy in monocular 3D pose estimation.

  • Takeaways & Limitations

    Future work includes extending DiffPose to online, few-shot, and other pose-based settings.

Abstract

from arXiv · show

Monocular 3D human pose estimation is quite challenging due to the inherent ambiguity and occlusion, which often lead to high uncertainty and indeterminacy. On the other hand, diffusion models have recently emerged as an effective tool for generating high-quality images from noise. Inspired by their capability, we explore a novel pose estimation framework (DiffPose) that formulates 3D pose estimation as a reverse diffusion process. We incorporate novel designs into our DiffPose to facilitate the diffusion process for 3D pose estimation: a pose-specific initialization of pose uncertainty distributions, a Gaussian Mixture Model-based forward diffusion process, and a context-conditioned reverse diffusion process. Our proposed DiffPose significantly outperforms existing methods on the widely used pose estimation benchmarks Human3.6M and MPI-INF-3DHP. Project page: https://gongjia0208.github.io/Diffpose/.

1. Introduction

DiffPose frames monocular 3D pose estimation as reverse diffusion to manage pose uncertainty and indeterminacy. It introduces pose-specific uncertainty initialization, GMM-based forward diffusion, and context-conditioned denoising, achieving state-of-the-art results on two benchmarks.

  • Monocular 3D pose estimation is difficult because depth ambiguity and occlusion create high indeterminacy and uncertainty.
  • DiffPose initializes HK from extracted heatmaps to capture pose-specific uncertainty before reverse diffusion.
  • A Gaussian Mixture Model models uncertainty during forward diffusion, while context conditioning uses spatial-temporal information from the input video or frame.A GCN-based diffusion model performs progressive denoising using this context.
  • DiffPose achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP.
  • DiffPose formulates 3D pose estimation as a distribution-to-distribution reverse diffusion process from an indeterminate pose distribution to a low-indeterminacy distribution.The reverse process progressively transforms HK into H0, while forward diffusion supplies intermediate supervisory distributions for training.

3. Background on Diffusion Models

Standard diffusion models bridge a source sample and noise through forward corruption and learn the reverse denoising process with intermediate noisy samples. Repeated reverse steps then generate a clean sample from noise.

  • Diffusion models transform Gaussian noise hK into a sample h0 through recurrent denoising, while forward diffusion applies the opposite direction.
  • Forward diffusion generates intermediate noisy samples that bridge the source sample h0 and Gaussian noise hK for reverse-process training.
  • The noise-scaling sequence α1:K controls the noise added at each diffusion step.
  • When αK ≈ 0 after many steps, hK converges to standard Gaussian noise, corrupting the source signal.
  • A trained diffusion model g is used recurrently with reverse steps to generate a high-quality sample h0 during testing.

4. Proposed Method: DiffPose

DiffPose frames monocular 3D pose estimation as reverse diffusion, transforming an uncertain 3D pose distribution into a determinate pose. Its pipeline initializes pose-specific uncertainty, trains with GMM-based forward diffusion, and uses context-conditioned reverse denoising.

  • Framework: DiffPose constructs a determinate 3D pose distribution H0 from an indeterminate distribution HK derived from the input 2D pose.The final 3D pose is sampled from the reconstructed distribution H0.
  • Pose-specific initialization: Heatmaps from an off-the-shelf 2D pose detector provide the x and y distributions for initializing HK, while depth uncertainty requires separate estimation.The heatmaps capture likely image regions for each joint and therefore expose uncertainty in the 2D predictions.
  • Forward pose diffusion: The forward process diffuses H0 through intermediate distributions toward HK, generating supervisory signals for step-by-step reverse denoising.These intermediate distributions correspond to H1 through HK and train the diffusion model to denoise smoothly.
  • Forward pose diffusion: A Gaussian Mixture Model represents the complex uncertainty distribution HK and supplies the target distribution for the K-step forward process.The GMM parameters describe mixture means, covariances, and component probabilities, allowing samples from intermediate distributions to be generated.
  • Reverse diffusion: The reverse process progressively maps HK to H0 using a diffusion model conditioned on spatial-temporal context from the 2D pose sequence.During training, reverse steps start from generated ˆHK; during testing, they start from HK to produce the final prediction.

5. Experiments

DiffPose is evaluated across video- and frame-based 3D pose estimation settings on Human3.6M and MPI-INF-3DHP, where it achieves strong benchmark performance. Ablations examine its diffusion pipeline, GMM design, context encoder, sampling parameters, and inference speed.

  • 5.1. Comparison with State-of-the-art Methods: DiffPose achieves the best MPJPE on video-based Human3.6M with detected 2D poses, outperforming the SOTA method by around 4 mm.It also outperforms previous methods by a large margin when ground-truth 2D poses are used.
  • 5.1. Comparison with State-of-the-art Methods: DiffPose achieves the best performance on video-based MPI-INF-3DHP, demonstrating efficacy in outdoor scenes.MPI-INF-3DHP evaluation reports MPJPE, PCK at a 150 mm threshold, and AUC across PCK thresholds.
  • 5.1. Comparison with State-of-the-art Methods: DiffPose surpasses all existing methods in average MPJPE on frame-based Human3.6M with detected 2D poses and leads by a large margin with ground-truth 2D poses.
  • 5.1. Comparison with State-of-the-art Methods: Qualitative results show progressively narrowed differences between sampled and ground-truth poses, with more reliable solutions for ambiguous body parts.Under occlusion, DiffPose predicts more accurate poses than previous methods.
  • 5.2. Ablation Study: Both single-step and stacked diffusion baselines perform much worse than DiffPose, indicating gains from the designed diffusion pipeline.The ablation compares models optimized for direct 3D pose prediction against DiffPose’s reverse diffusion process.
  • 5.2. Ablation Study: The GMM-based design significantly outperforms Stand-Diff, while five kernels are sufficient to capture the uncertainty distribution.The study varies the number of GMM kernels from 1 to 9.
  • 5.2. Ablation Study: MPJPE drops substantially through K = 50, improves only slightly beyond K = 50, and remains roughly consistent after N = 5 samples.The method therefore uses K = 50 reverse diffusion steps and N = 5 pose samples.
  • 5.2. Ablation Study: DDIM-accelerated DiffPose reaches competitive speed against the current SOTA while achieving better performance, and exceeds 170 FPS without acceleration.

6. Conclusion

DiffPose addresses uncertainty and indeterminacy in monocular 3D pose estimation through a diffusion-based framework. Experiments show state-of-the-art performance on two widely used benchmark datasets.

  • DiffPose is a diffusion-based framework for handling uncertainty and indeterminacy in monocular 3D pose estimation.
  • The framework initializes an indeterminate 3D pose distribution and recurrently sheds indeterminacy to obtain a high-quality pose distribution.
  • DiffPose achieves state-of-the-art performance on two widely used benchmark datasets.

DiffPose: Toward More Reliable 3D Pose Estimation (Supplementary)

The supplementary material identifies the paper's authors and their institutional affiliations. The listed institutions span universities in Singapore, the United States, Australia, and the United Kingdom.

  • The paper lists Jia Gong, Lin Geng Foo, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, and Jun Liu as authors.
  • The author list marks Jia Gong and Lin Geng Foo as equal contributors, Zhipeng Fan as currently at Meta, and Jun Liu as corresponding author.
  • The authors are affiliated with the Singapore University of Technology and Design, New York University, Monash University, and Lancaster University.

1. Additional Details of GMM Forward Diffusion

The supplementary formulation explains how the GMM-based forward diffusion can be implemented step by step. It samples a Gaussian component, centers the pose relative to that component, applies diffusion steps, and then restores the original center.

  • The GMM-based forward diffusion is detailed as a step-wise formulation of the forward process described in the main paper.
  • The formulation samples a Gaussian component according to mixture probabilities and uses its mean and covariance for the diffusion process.
  • After k step-wise updates, the resulting formulation is equivalent to the direct expression based on h0.
  • The pose is centered by subtracting the selected mixture mean before diffusion and de-centered afterward by restoring that mean.

2. Additional Details of Diffusion Network g

The diffusion network receives the current diffusion step through a sinusoidal embedding. Its even and odd embedding coordinates use sine and cosine functions of the step number.

  • The network generates a diffusion step embedding to provide information about the current step number k.
  • The embedding has dimensions J×256 and is constructed with a sinusoidal function.
  • At even indices, the embedding uses sine values, while odd indices use cosine values based on k and the index.

3. More Implementation Details

The implementation uses a scheduled forward diffusion process, GMM fitting on sampled poses, and Adam-based training for the diffusion model.

  • The GMM parameters are optimized by modeling HK from 1000 sampled poses.
  • The Context Encoder φST is pretrained before the diffusion model g is optimized with Adam.
  • Training uses a 1e−4 initial learning rate, 0.9 decay after ten epochs, batch size 4096, and a single GeForce RTX 3090 GPU within 96 hours.

4. Experiment Results on Human3.6M under P-MPJPE (Protocol 2)

DiffPose is evaluated on Human3.6M in video-based and frame-based settings under P-MPJPE, using detected 2D poses. It outperforms existing methods across the reported settings.

  • DiffPose significantly outperforms state-of-the-art methods on Human3.6M under P-MPJPE in both video-based and frame-based settings.
  • The Human3.6M tables report video-based and frame-based results with detected 2D poses under P-MPJPE.
  • The qualitative comparison uses red 3D poses to denote ground truth when comparing Graformer with DiffPose.

5. Additional Results

Additional experiments examine frame-based Human3.6M results and qualitative behavior of the GMM-based forward and context-conditioned reverse diffusion processes.

  • Frame-based evaluation: The additional investigation evaluates DiffPose in the frame-based Human3.6M scenario.
  • 3D Pose visualization: DiffPose predicts more reliable and accurate poses than the state-of-the-art method for novel gestures and occluded body parts.
  • Forward diffusion process visualization: The GMM-based process adds noise using pose-specific heatmap information and data distribution, producing a more constrained uncertainty initialization than standard diffusion.
  • Reverse diffusion process visualization: The context-conditioned reverse process reconstructs more accurate poses than the unconditioned process, although both produce realistic human poses.

6. Future Work

The paper identifies future work in diffusion-network architecture and extension of the framework to online, few-shot, and other pose-based settings.

  • Future work includes further investigation of the diffusion network architecture.
  • The authors plan to extend the framework to online and few-shot settings and other pose-based tasks.
  • The standard and context-conditioned reverse diffusion processes are presented as qualitative comparison targets in the figures.
Loading 2211.16940v3…