Source-linked AI summary
CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wild
Bastian Wandt, Marco Rudolph, Petrissa Zell, Helge Rhodin, Bodo Rosenhahn
TL;DR
Single-image 3D pose estimation depends on labeled data that can be difficult to acquire for outdoor and sports activities. CanonPose learns from unlabeled multi-view images by mixing canonical poses and camera rotations, and reports strong benchmark results alongside promising in-the-wild results. Its optional camera-consistency enhancement is for static-camera sequences.
Problem
Single-image 3D pose estimation relies on labeled 2D-to-3D training data, but outdoor and sports motions can be hard or impossible to capture with motion-capture systems.
Source: §1 · p. 1Method
CanonPose learns a single-image 3D pose estimator from unlabeled multi-view images by mixing poses across views and exploiting multi-view consistency without camera calibration.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The approach is practically applicable to in-the-wild scenarios, including outdoor scenes with moving cameras.
Source: §5 · p. 9The optional camera-consistency enhancement assumes cameras remain static during a sequence; the base method does not require this assumption.
Source: §3.4 · p. 4, §4.4 · p. 7
Abstract
from arXiv · showhide
Human pose estimation from single images is a challenging problem in computer vision that requires large amounts of labeled training data to be solved accurately. Unfortunately, for many human activities (\eg outdoor sports) such training data does not exist and is hard or even impossible to acquire with traditional motion capture systems. We propose a self-supervised approach that learns a single image 3D pose estimator from unlabeled multi-view data. To this end, we exploit multi-view consistency constraints to disentangle the observed 2D pose into the underlying 3D pose and camera rotation. In contrast to most existing methods, we do not require calibrated cameras and can therefore learn from moving cameras. Nevertheless, in the case of a static camera setup, we present an optional extension to include constant relative camera rotations over multiple views into our framework. Key to the success are new, unbiased reconstruction objectives that mix information across views and training samples. The proposed approach is evaluated on two benchmark datasets (Human3.6M and MPII-INF-3DHP) and on the in-the-wild SkiPose dataset.
1. Introduction
CanonPose targets single-image 3D pose estimation where labeled training data is difficult to obtain, especially for outdoor and sports activities. It learns from unlabeled multi-view images by combining canonical 3D poses and camera rotations, without camera calibration.
1. Introduction
- Outdoor and sports motions can be hard or impossible to capture with commercial motion-capture systems, making training-data acquisition a practical challenge.The paper notes that supervised approaches rely heavily on large amounts of annotated data.Source: §1 · p. 1
- Training requires at least two temporally synchronized cameras viewing the person from different angles, but no scene knowledge, camera calibration, or camera intrinsics.The approach also does not require prior knowledge of the scene or 3D skeleton.Source: §1 · pp. 1–2
- The method uses a pretrained 2D joint estimator and avoids laborious preprocessing such as camera calibration or multi-view geometry estimation.The 2D estimator is pretrained on a different dataset with 2D joint annotations [7].Source: §1 · pp. 1–2
- CanonPose sets state of the art in several self-supervised metrics on Human3.6M and MPI-INF-3DHP, is competitive with Martinez et al. [23], and shows promising results on SkiPose.The benchmark comparisons do not assume camera calibration or static cameras.Source: §1 · p. 2
- CanonPose learns a single-image 3D pose estimator from unlabeled multi-view data by mixing canonical poses and camera rotations to enforce reprojection consistency.For static camera setups, an optional loss mixes relative camera rotations between samples; joint confidences also enter the 2D input and reprojection loss.Source: §1 · p. 2
2. Related Work
Prior work spans fully supervised, weakly supervised, and self-supervised approaches, each with constraints on annotations, generalization, or setup. CanonPose is presented as a multi-view alternative that relaxes several such constraints while outperforming comparable works in most experiments.
Full Supervision
- Fully supervised image-based methods can perform exceptionally well on similar image data, but their generalization to other scenes is limited.Methods trained on 2D detections and 3D ground truth, including Martinez et al. [23], also have limited application to in-the-wild scenes because they do not generalize well to unseen poses.Source: §2 · p. 2
Weak Supervision
- Weakly supervised approaches use sparse 3D annotations, transfer learning, known camera positions, or unpaired 2D and 3D poses, but distribution-based methods struggle with completely unseen poses.The cited multi-view embedding approach uses known camera positions [35], while related work also adopts it [33] [28].Source: §2 · pp. 2–3
Self-supervised and Unsupervised Learning without 3D Ground Truth
- Prior self-supervised methods may require scene knowledge, fixed camera setups, sensitive epipolar-geometry preprocessing, or can overfit to a dataset background.Some methods also rely on ground-truth 3D bone lengths [11].Source: §2 · p. 3
- CanonPose avoids scene, camera-position, and anthropometric constraints, using cross-view pose and rotation reprojections; it outperforms comparable works in most experiments.Its modular design allows any 2D pose estimator, making it agnostic to image data.Source: §2 · p. 3, Figure 2 · p. 3
3. Method
CanonPose lifts per-view 2D detections into a learned canonical 3D pose and camera rotation, then uses cross-view reprojection for self-supervision. Its lifting network operates on a single image at inference.
3. Method
- A single lifting network predicts a canonical 3D pose and camera rotation from each view’s 2D detections, enabling reprojection across views and samples.The canonical pose coordinate system is learned during training.Source: §3 · p. 3
3.1. Reprojection
- Root-centering and Frobenius-norm scaling sidestep scale-depth ambiguity, while the scale-independent reprojection loss compares the input and reprojected poses at matching scale.Under weak-perspective projection, scaling the reprojected pose is essential because the global 3D scale is unknown.Source: §3.1 · pp. 3–4
3.2. View-consistency
- Mixing poses and rotations across views yields m^2 reprojection combinations for m cameras, enforcing multi-view consistency without bias toward trivial solutions.The lifting network needs no other inputs at inference.Source: §3.2 · p. 4
3.3. Confidences
- Joint heatmap maxima serve as confidence surrogates, concatenated to the 2D input and used to weight reprojection differences.The confidence for joint i is the maximum value of its heatmap.Source: §3.3 · p. 4
3.4. Camera-consistency
- For static-camera setups, an optional loss permutes relative rotations across batch samples and uses them to reproject canonical poses.The method also works without this enhancement; camera angles may vary between subjects but remain fixed during one subject’s capture session.Source: §3.4 · pp. 4–5
3.5. Network Architecture
- The lifting network maps pose and joint-confidence inputs through fully connected residual blocks to separate 3D-pose and camera-axis-angle outputs.It trains for 100 epochs with Adam, an initial learning rate of 0.0001, and weight decay at epochs 30, 60, and 90.Source: §3.5 · p. 5
4. Experiments
The experiments assess separately trained models on Human3.6M, MPI-INF-3DHP, and SkiPose, using benchmark metrics, ablations, and qualitative results. They report strong comparisons against self-supervised methods and examine performance with moving cameras and canonical-pose disentanglement.
4. Experiments
- The study evaluates Human3.6M, MPI-INF-3DHP, and SkiPose, training one network per dataset without using additional datasets to assess generalizability in single-image pose estimation with unlabeled images for specific activities.Source: §4 · p. 5
4.1. Metrics
- CPS classifies a pose as correct only if every joint is within threshold θ, then integrates the correct-pose percentage over thresholds from 0 to 300 mm.This evaluates whole-pose correctness rather than averaging errors joint by joint.Source: §4.1 · p. 5, Figure 4 · p. 6
4.2. Skeleton Morphing
- Skeleton morphing maps AlphaPose keypoints to each dataset's 2D skeleton using subject 1's ground-truth poses, which are excluded from self-supervised training.The morphing step is unnecessary when the skeleton structures already match.Source: §4.2 · p. 6
4.3. Quantitative Evaluation on Human3.6M and 3DHP
- Human3.6M results beat comparable methods on PMPJPE and approach supervised Martinez, while Iqbal retains MPJPE state of the art using bone-length constraints; on 3DHP, CanonPose beats other self-supervised methods.The reported 3DHP Correct Poses Score is 134.2.Source: §4.3 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
- On Human3.6M, CanonPose outperforms EpipolarPose on CPS by a large margin and is on par with the supervised Martinez approach.EpipolarPose needs a threshold above 80 mm to exceed 1% correct poses, compared with CanonPose's threshold just below 50 mm.Source: §4.3 · pp. 6–7, Figure 5 · p. 7
4.4. Moving cameras
- SkiPose tests rotating, zooming, uncalibrated cameras; CanonPose handles this synchronized multi-view setup, where the static-camera constraint cannot be applied.The Rhodin et al. comparison is only a baseline because it uses sparse supervision and known camera positions.Source: §4.4 · p. 7
4.5. Ablation Studies
- Using detector confidences as network inputs and in reprojection error improves MPJPE by 19.4 mm and PMPJPE by 11.2 mm; the authors also find pose-rotation mixing essential relative to equality constraints.Training with fewer cameras slightly lowers performance, while ground-truth 2D annotations provide a lower-bound experiment.Source: §4.5 · pp. 7–8, Table 4 · p. 8
4.6. Are We Learning a Canonical Pose Basis?
- Canonical poses from four views have aligned hip joints, with test-set standard deviations of 7.9 mm and 7.7 mm for the right and left hip, respectively.The authors interpret this as plausible disentanglement of pose and rotation.Source: §4.6 · p. 8, Figure 7 · p. 8
5. Conclusion
CanonPose trains a single-image 3D human pose estimator from multi-view data without 2D or 3D annotations. It decomposes 2D observations into a canonical 3D pose and camera rotation, and reports state-of-the-art benchmark results alongside promising results on challenging outdoor scenes.
5. Conclusion
- CanonPose trains a single-image 3D human pose estimator from multi-view data without 2D or 3D annotations.The approach uses a pretrained 2D human pose estimator.Source: §5 · pp. 8–9
- The method reports state-of-the-art results on benchmark datasets with fewer prerequisites than other approaches, and promising results on challenging outdoor scenes.It does not require 2D or 3D annotations for the multi-view data.Source: §5 · p. 9
- Multi-view consistency decomposes a 2D observation into a canonical 3D pose and camera rotation, which is used to reproject it after mixing.Each network estimates a 3D pose in a canonical rotation.Source: §5 · pp. 8–9