Source-linked AI summary
Weakly Supervised Generative Network for Multiple 3D Human Pose Hypotheses
Chen Li, Gim Hee Lee
TL;DR
Monocular 3D pose estimation is ambiguous because an image lacks depth, and earlier multi-hypothesis methods require difficult-to-obtain 2D-to-3D supervision. This paper uses a weakly supervised generative network to model multiple 3D pose hypotheses, reporting feasible poses and better results than existing weakly supervised approaches across its evaluations.
Problem
Missing depth makes monocular 3D pose estimation ambiguous, while prior multi-hypothesis approaches can require difficult-to-obtain 2D-to-3D correspondences.
Source: §1 · pp. 1–2Method
A weakly supervised generative network approximates the multi-modal posterior by minimizing KL divergence, using reprojection and prior losses and sampling hypotheses to estimate the conditional mode.
Source: §1 · p. 2Results
The network generates feasible 3D hypotheses consistent with 2D reprojections and achieves better results than existing weakly supervised approaches, with evaluations on Human3.6M, MPII, and MPI-INF-3DHP.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
Results on MPII and MPI-INF-3DHP support the network’s generalization capacity beyond Human3.6M.
Source: §1 · p. 2, §5 · p. 10
Abstract
from arXiv · showhide
3D human pose estimation from a single image is an inverse problem due to the inherent ambiguity of the missing depth. Several previous works addressed the inverse problem by generating multiple hypotheses. However, these works are strongly supervised and require ground truth 2D-to-3D correspondences which can be difficult to obtain. In this paper, we propose a weakly supervised deep generative network to address the inverse problem and circumvent the need for ground truth 2D-to-3D correspondences. To this end, we design our network to model a proposal distribution which we use to approximate the unknown multi-modal target posterior distribution. We achieve the approximation by minimizing the KL divergence between the proposal and target distributions, and this leads to a 2D reprojection error and a prior loss term that can be weakly supervised. Furthermore, we determine the most probable solution as the conditional mode of the samples using the mean-shift algorithm. We evaluate our method on three benchmark datasets -- Human3.6M, MPII and MPI-INF-3DHP. Experimental results show that our approach is capable of generating multiple feasible hypotheses and achieves state-of-the-art results compared to existing weakly supervised approaches. Our source code is available at the project website.
1 Introduction
Monocular 3D pose estimation is ambiguous because depth is missing, while many methods assume a single solution or require difficult-to-obtain 2D-to-3D supervision. The paper proposes a weakly supervised generative network that models multiple pose hypotheses and reports strong results against weakly supervised approaches.
1 Introduction
- Missing depth makes monocular 3D pose estimation an inverse problem with multiple feasible solutions, while a uni-modal assumption can lead to overfitting and undesirable performance.The paper frames this ambiguity as a reason to model multiple solutions rather than enforce a single one.Source: §1 · p. 1
- Figure 1 depicts a network conditioned on an input 2D pose that draws normal latent codes to generate 3D pose hypotheses for the target multi-modal posterior.The figure summarizes the sampling setup underlying the proposed approach.Source: Figure 1 · p. 2
- Earlier multi-hypothesis methods include optimization-based sampling [14] and a supervised mixture-density network [18], while weakly supervised alternatives still assume a uni-modal posterior.The mixture-density approach requires ground-truth 2D-to-3D correspondences, which can be difficult to obtain.Source: §1 · pp. 1–2
- The proposed weakly supervised generative network approximates the unknown multi-modal posterior by minimizing KL divergence, yielding a 2D reprojection error and a prior loss.The reprojection term keeps generated 3D poses close to the observed 2D pose, while an MMD-based discriminator encourages human-like poses and additional losses encourage diversity.Source: §1 · p. 2
- Given a 2D pose, the network samples multiple 3D hypotheses; mean-shift selects the conditional mode, and an all-zero latent code provides a time-efficient approximation.The paper reports superior performance to state-of-the-art weakly supervised approaches on Human3.6M and evaluates generalization on MPII and MPI-INF-3DHP.Source: §1 · p. 2
Related work
Prior 3D pose estimation methods include uni-modal approaches and supervised multi-modal approaches. This paper situates its weakly supervised generative model among efforts to estimate multiple feasible poses without requiring 2D-to-3D correspondences.
Related work
- Prior methods fall into uni-modal fully or weakly supervised approaches and fully supervised mixture-of-Gaussians approaches.The paper uses these categories to frame its comparison with weakly supervised multi-hypothesis estimation.Source: Related work · p. 3
- Most fully supervised methods require ground-truth 2D-to-3D labels, which are tedious to collect, especially for outdoor environments.Prior work includes end-to-end and two-stage approaches, including voxel-based estimation and transfer learning.Source: Related work · p. 3
- Weakly supervised, unsupervised, and self-supervised approaches reduce reliance on 3D labels, but the described approaches assume a uni-modal posterior and return one pose per 2D input.Weak supervision can use 2D labels and projection, while self-supervision uses transformed-pose consistency and 2D pose discrimination.Source: Related work · p. 3
- Jahangiri and Yullie [14] sample hypotheses from learned joint-angle occupancy regions, whereas Li and Lee [18] use a mixture-density network to model a multi-modal posterior.Li and Lee [18] require ground-truth 2D-to-3D correspondences; the proposed work instead uses a weakly supervised generative model.Source: Related work · p. 3
3 Our Method
The method models multiple plausible 3D poses as a proposal distribution conditioned on a 2D pose, then trains it using reprojection, pose-prior, and diversity objectives. It selects a likely pose either by mean-shift over generated samples or by a faster all-zero latent input.
3 Our Method
- The network models a proposal distribution Q(y | x) for multiple 3D poses, using a pose generator, camera network, discriminator prior, and encoder.The discriminator learns pose priors from 3D poses that need not be ground-truth labels paired with the input 2D poses.Source: §3 · pp. 3–4, §3.1 · p. 5
3.1 Conditional Pose Generator
- Training approximates the target posterior by minimizing KL divergence, yielding a likelihood term for 2D reprojection and a prior term for plausible 3D poses.The Laplace likelihood is chosen for robustness to noisy and outlier 2D joints; under weak perspective, reprojection is π(y) = My.Source: §3.1 · pp. 4–5
3.2 Diverse Pose Hypotheses
- A diversity regularizer encourages pose differences to track latent-code distances, while an encoder reconstructs the latent code to help prevent mode collapse.The encoder targets the many-to-one mapping in which different latent codes produce the same pose.Source: §3.2 · p. 5
3.3 Optimization
- Optimization combines adversarial pose realism with reprojection consistency, diversity, and latent-code reconstruction objectives.The discriminator distinguishes real and generated distributions, and the camera network is trained to satisfy the weak-perspective constraint.Source: §3.3 · pp. 5–6
3.4 Best Pose Selection
- Mean-shift with a Gaussian kernel finds a local mode among generated pose hypotheses; an all-zero latent input is proposed as a faster alternative.The paper states that the zero-code method achieves results comparable to mean-shift in experiments.Source: §3.4 · p. 6
Experiments
The model is trained on Human3.6M and evaluated on Human3.6M, MPI-INF-3DHP, and MPII; the reported metrics include MPJPE for Human3.6M and 3DPCK for MPI-INF-3DHP.
4.1 Quantitative Results on Human3.6M Dataset
- On Human3.6M protocol #2, our method matches supervised [18] and outperforms weakly supervised [29] [30] in MPJPE.Performance under mean-shift and zero-code settings is similar and close to best-hypothesis performance; zero-code inference avoids sampling.Source: Table 1 · p. 7, §4.1 · p. 8
- On Human3.6M protocol #1, our method outperforms weakly supervised [30] but trails supervised [18].The authors attribute the gap to not using camera-frame 2D-to-3D correspondences and to the weak-perspective camera assumption.Source: Table 2 · p. 8, §4.1 · p. 8
4.2 Ablation Studies
- With one or two missing joints, our method outperforms weakly supervised [30] and achieves comparable results to supervised [18].Missing joints are selected from limb joints, and their 2D-loss weights are set to zero.Source: §4.1 · p. 8, Table 3 · p. 9
- The full model generates more diverse hypotheses and lower error than the model without Lreg and Lrec.Five hypotheses selected by FPS have differing diversity, while their 2D reprojections overlap for each input pose.Source: Table 4 · p. 9, §4.2 · p. 9, Figure 3 · p. 10
- Increasing λreg increases pose diversity, but an overly strong weight also increases estimation error, making it an accuracy–diversity trade-off.Source: Table 4 · p. 9, §4.2 · p. 9
4.3 Results on MPI-INF-3DHP and MPII datasets
- On MPI-INF-3DHP, results are slightly worse than [30] under ZC but outperform other approaches under BH; MPII results are qualitative only.The authors report that the MPII results generalize well to outdoor scenes; 3D ground truth is unavailable for that dataset.Source: Table 5 · p. 9, §4.3 · pp. 9–10, Figure 4 · p. 10
5 Conclusion
The proposed weakly supervised generative network produces feasible 3D pose hypotheses consistent with 2D reprojections and outperforms existing weakly supervised approaches. Results on MPII and MPI-INF-3DHP support its generalization capacity.
5 Conclusion
- The network generates feasible 3D pose hypotheses consistent with 2D reprojections and achieves better results than existing weakly supervised approaches.Results on MPII and MPI-INF-3DHP verify the network's generalization capacity.Source: §5 · p. 10