Source-linked AI summary
Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild
Umar Iqbal, Pavlo Molchanov, Jan Kautz
TL;DR
In-the-wild monocular 3D pose estimation is hindered by the difficulty of obtaining unconstrained images with accurate 3D annotations. This paper learns from unlabeled multi-view data using an end-to-end constrained consistency objective, and reports large-margin gains over semi-/weakly-supervised methods on two large-scale datasets.
Problem
Accurate 3D pose annotations for unconstrained images are difficult to acquire, limiting training data for in-the-wild monocular 3D pose estimation.
Source: §1 · p. 1Method
The method trains an end-to-end 2.5D pose estimator using unlabeled multi-view data, separate 2D-labeled images, and a constrained multi-view consistency objective.
Source: §1 · pp. 1–2Results
The approach outperforms existing semi-/weakly-supervised methods by a large margin on two large-scale datasets.
Source: §1 · p. 2Takeaways & Limitations
The approach can use easily collected in-the-wild multi-view data without 3D annotations and demonstrated state-of-the-art performance on two challenging datasets.
Source: §5 · p. 8Training only on MQC was not sufficient for convergence; it had to be combined with a dataset providing more distant viewing angles.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
One major challenge for monocular 3D human pose estimation in-the-wild is the acquisition of training data that contains unconstrained images annotated with accurate 3D poses. In this paper, we address this challenge by proposing a weakly-supervised approach that does not require 3D annotations and learns to estimate 3D poses from unlabeled multi-view data, which can be acquired easily in in-the-wild environments. We propose a novel end-to-end learning framework that enables weakly-supervised training using multi-view consistency. Since multi-view consistency is prone to degenerated solutions, we adopt a 2.5D pose representation and propose a novel objective function that can only be minimized when the predictions of the trained model are consistent and plausible across all camera views. We evaluate our proposed approach on two large scale datasets (Human3.6M and MPII-INF-3DHP) where it achieves state-of-the-art performance among semi-/weakly-supervised methods.
1. Introduction
In-the-wild monocular 3D pose learning is constrained by scarce accurate 3D annotations and the limits of 2D labels for depth. The paper trains an end-to-end model from unlabeled multi-view data and separate 2D-labeled images, and reports strong results on two large-scale datasets.
1. Introduction
- The method uses unlabeled multi-view data without extrinsic camera calibration, alongside an independent collection of images with 2D pose annotations.The authors describe the multi-view data as easy to collect in in-the-wild settings.Source: §1 · p. 1
- Multi-view data can resolve ambiguities from occlusion, foreshortening, or motion blur that 2D annotations alone do not address.The paper notes that 2D labels provide insufficient 3D information when joints are foreshortened or occluded.Source: §1 · p. 1
- A 2.5D pose representation and constrained multi-view objective support end-to-end learning without 3D pose annotations.The objective is designed so it can be minimized only when predictions are correct across camera views.Source: §1 · pp. 1–2
- The approach is reported to be robust to body-part occlusions and truncations in unlabeled multi-view data, and to improve 2D pose predictions.The authors attribute the 2D improvement to exploiting multi-view consistency during training.Source: §1 · p. 2
- The method outperforms semi-/weakly-supervised methods by a large margin on two datasets and can use MannequinChallenge videos to improve generalizability under significant domain gaps.The paper particularly notes this generalizability benefit when training and testing environments differ substantially.Source: §1 · p. 2
2. Related Work
Prior monocular 3D pose methods variously rely on 3D annotations, 2D pose information, or multi-view constraints, with some approaches retaining calibration or 3D-label requirements. This work combines image-based estimation with unlabeled multi-view training and constrains consistency-based learning to avoid degenerate solutions.
2. Related Work
- Fully supervised approaches need accurate 3D annotations, while other methods use weak signals such as 2D poses, geometry, action labels, or temporal consistency.The paper notes that accurate 3D training data is especially difficult to acquire in unconstrained scenarios.Source: §2 · p. 2
- Methods that lift 2D poses to 3D without image information can be vulnerable to reprojection ambiguities and inconsistencies between 2D and 3D poses.These approaches regress from 2D joints or search databases using 2D projections.Source: §2 · p. 2
- The paper combines image-based 3D estimation with training that does not require images annotated with 3D poses.The authors position this as combining benefits of image-based estimation and approaches that avoid 3D-annotated images.Source: §2 · p. 2
- Multi-view consistency compares 3D joint locations across views after rigid alignment, while the proposed objective constrains its solution space against degenerate poses.The figure description also shows unlabeled multi-view data supplying the consistency loss.Source: Figure 1 · p. 3
- Unlike [14], the method jointly optimizes 2D and 3D poses, avoiding fixed reconstructed 3D targets and allowing multi-view data to improve 2D predictions.The paper also contrasts its approach with [29], which requires calibrated camera views, and [34], which requires some 3D-labeled training data.Source: §2 · pp. 2–3
3. Method
The method learns monocular 3D pose from unlabeled multi-view images and a separate set of 2D-annotated images, without paired 2D–3D training data. It predicts a scale-normalized 2.5D pose, reconstructs 3D differentiably, and combines supervision to constrain cross-view predictions.
3.1. 2.5D Pose Representation
- The 2.5D representation combines image-plane joint coordinates with root-relative metric depth, while scale normalization makes it person-scale agnostic.The representation can use in-the-wild 2D pose annotations, though the unnormalized representation does not account for image scale ambiguity.Source: §3.1 · pp. 3–4
3.1.1 Differentiable 3D Reconstruction
- A scale-normalized 2.5D pose yields reconstructed 3D joint locations through an analytical root-depth solution and a fully differentiable reconstruction process.The method uses the neck-to-pelvis distance for scale normalization.Source: §3.1.1 · p. 4
3.2. 2.5D Pose Regression
- The regression network outputs 2D heatmaps and latent depth maps; explicit supervision of heatmaps supports confidence scores used to handle uncertain 2D estimates.Heatmaps are converted to 2D coordinates using spatial softmax and soft-argmax, while relative depth is obtained by weighting latent depth maps with normalized heatmaps.Source: §3.2 · p. 4
3. Method
- Training uses unlabeled views of the same pose and a separate collection of 2D-annotated images, without 3D annotations or known extrinsic camera parameters.Views may come from multiple cameras or a single camera observing a static pose over time.Source: §3 · p. 3, §3.3 · p. 4
3.3. Weakly-Supervised Training
- The weakly supervised objective combines 2D heatmap, multi-view consistency, and limb-length losses; under accurate 2D coordinates and known intrinsics and transforms, consistency minimization requires correct relative depths.The limb-length term encourages plausible scale-normalized limb lengths and was found to speed convergence.Source: §3.3 · pp. 4–5
- Because extrinsic transforms are unavailable, the method estimates cross-view alignment from predicted 3D poses with Procrustes analysis and stops gradients through that optimization to avoid numerical instability.The multi-view consistency gradients can also affect heatmap predictions through root-depth calculation.Source: §3.3 · p. 5
- For constant-background training samples, fixed pseudo-labels from a pretrained 2D pose model add heatmap supervision to prevent repeated degenerate predictions.The authors apply this regularization to Human3.6M and MPII-INF-3DHP images, which were recorded indoors.Source: §3.3 · p. 5
4. Experiments
Experiments evaluate the approach on Human3.6M and MPII-INF-3DHP, with ablations of supervision and training data followed by comparisons with existing methods. The reported results show strong performance in semi- and weakly-supervised settings, while the ablations examine how multi-view data and dataset choice affect performance.
4.1. Datasets
- Evaluation uses H36M and MPII-INF-3DHP, with MQC and MPII also used for weakly-supervised training; the MQC collection contains 11,413 tracklets and 241k images.MQC videos have no ground-truth annotations; people are tracked across frames, and heavily occluded or truncated poses and tracklets shorter than five frames are discarded.Source: §4.1 · p. 6
4.2. Evaluation Metrics
- H36M evaluation reports MPJPE, NMPJPE, and P-MPJPE; 3DHP evaluation also reports PCK and Normalized-PCK.PCK counts joints within 150mm of ground truth; the 3DHP protocol evaluates 14 joints, excluding the pelvis used for pose alignment.Source: §4.2 · p. 6
4.3. Ablation Studies
- On MQC tracks, adding MQC training data yields more consistent aligned 3D pose estimates than training with H36M alone.The figure also notes that the approach can correct 2D pose estimates in unlabeled multi-view data.Source: Figure 2 · p. 7
- Without 2D annotations for multi-view data, H36M errors are 62.3mm with known camera extrinsics and 69.1mm when they are unknown.With those annotations, the corresponding errors are 57.2mm and 59.3mm; multi-view consistency also reduces the MPII-only 2D error from 8.9px to 8.3px.Source: §4.3 · p. 7
- For H36M testing after training on 3DHP, adding MQC reduces error from 106.2mm to 93.6mm, and combining H36M, 3DHP, and MQC reduces it to 67.4mm.The combined-dataset result is also lower than the 69.1mm error when training used only H36M.Source: §4.3 · p. 7
- MQC alone is insufficient for convergence; it must be combined with data providing multi-view images from more distant viewing angles.The authors suggest that most MQC videos do not capture the same person from sufficiently different angles, unlike H36M and 3DHP.Source: §4.3 · p. 7
4.4. Comparison with the State-of-the-Art
- On 3DHP, the proposed approach outperforms existing methods across all three categories; on H36M, it significantly outperforms semi-/weakly-supervised methods.The fully-supervised baseline is better or on par with state of the art, and the weakly-supervised approach is on par with many recent fully-supervised approaches.Source: Table 3 · p. 8, §4.4 · p. 8, Table 2 · p. 8
5. Conclusion
The paper presents a weakly supervised approach that learns 3D human poses from unlabeled multi-view data without 3D annotations. It reports state-of-the-art performance on two challenging datasets.
5. Conclusion
- The approach achieves state-of-the-art performance on two challenging datasets.The conclusion does not name the datasets.Source: §5 · p. 8
- The method learns 3D poses from unlabeled multi-view data without requiring 3D annotations.Source: §5 · p. 8
- A novel end-to-end framework and objective train the model to predict consistent 3D poses across camera views.Source: §5 · p. 8
- The required training data can be collected easily in in-the-wild scenarios.Source: §5 · p. 8
Appendix
The appendix provides implementation details intended to support reproduction of the paper's results.
Appendix
- The appendix provides implementation details to reproduce the paper's results.Source: Appendix · p. 11
A. Implementation Details
The implementation uses an MPII-pretrained 2D pose model, unlabeled multi-view samples, and specified training settings; inference estimates pose scale from mean bone lengths.
A. Implementation Details
- HRNet-w32 [38] is the backbone; 2D pose estimation is pretrained on MPII before weakly supervised losses, while latent-depth weights are not pretrained.Source: A. Implementation Details · p. 11
- Multi-view consistency uses at most four camera views, randomly sampled per epoch when a sample contains more; batches include 128 annotated images and 32 four-view unlabeled samples.Source: A. Implementation Details · p. 11
- Training uses 256 × 256 centered person crops, random scaling of ±20% and rotation of ±30 degrees, and converges after 60k iterations.Source: A. Implementation Details · p. 11
- Adam uses a learning rate of 5e−4, reduced to 5e−5 at 50k iterations; λ=50, ψ=5, α=10, and β=100 set objective weights.Source: A. Implementation Details · p. 11
- Inference approximates absolute-pose scale using training-set mean bone lengths; the experiments use means from the H36M training set.Source: A. Implementation Details · p. 11