Source-linked AI summary
Learning Monocular 3D Human Pose Estimation from Multi-view Images
Helge Rhodin, Jörg Spörri, Isinsu Katircioglu, Victor Constantin, Frédéric Meyer, Erich Müller, Mathieu Salzmann, Pascal Fua
TL;DR
The paper addresses 3D pose estimation for unusual motions where annotated training data is difficult to obtain. It uses multi-view weak supervision with supervised and regularization losses, and demonstrates reduced annotation needs and improved robustness on benchmarks and competitive skiing data.
Problem
Unusual motions such as skiing lack sufficient annotated data, while obtaining extensive annotations is tedious, slow, and error-prone.
Method
The approach supplements a small labeled set with synchronized multi-view images, enforcing pose consistency alongside supervised and regularization losses while estimating camera pose when calibration is difficult.
Results
The weakly supervised multi-view strategy drastically cuts the need for annotated data and increases robustness to viewpoint and scale changes across standard benchmarks and competitive skiing scenarios.
Takeaways & Limitations
A small annotated image-and-pose dataset can be supplemented by synchronized multi-camera images with minimal supervision, even when relative camera positions are not exactly known.
Takeaways & Limitations
Using multi-view losses can cause the model to overfit after several epochs by distinguishing labeled examples from unlabeled ones and predicting consistent but wrong poses for the latter.
Abstract
from arXiv · showhide
Accurate 3D human pose estimation from single images is possible with sophisticated deep-net architectures that have been trained on very large datasets. However, this still leaves open the problem of capturing motions for which no such database exists. Manual annotation is tedious, slow, and error-prone. In this paper, we propose to replace most of the annotations by the use of multiple views, at training time only. Specifically, we train the system to predict the same pose in all views. Such a consistency constraint is necessary but not sufficient to predict accurate poses. We therefore complement it with a supervised loss aiming to predict the correct pose in a small set of labeled images, and with a regularization term that penalizes drift from initial predictions. Furthermore, we propose a method to estimate camera pose jointly with human pose, which lets us utilize multi-view footage where calibration is difficult, e.g., for pan-tilt or moving handheld cameras. We demonstrate the effectiveness of our approach on established benchmarks, as well as on a new Ski dataset with rotating cameras and expert ski motion, for which annotations are truly hard to obtain.
1. Introduction
The paper targets monocular 3D pose estimation for unusual motions with scarce annotations, replacing much manual labeling with multi-view weak supervision while retaining limited labeled data and regularization.
- Motivation: Unusual motions such as skiing remain difficult because suitable annotated training data is hard to obtain outside laboratory settings.Manual annotation is tedious, slow, and error-prone, while skiing is important for biomechanical and performance analysis.
- Approach: Multi-view images replace most annotations during training by forcing the network to predict the same 3D pose across views.The approach uses multiple views as weak supervision rather than requiring complete pose labels.
- Approach: A small labeled image set supplies strong supervision and regularizes multi-view predictions against drift from initial pose predictions.View consistency alone can produce a constant but incorrect pose, so labeled examples serve both supervision and regularization roles.
- Approach: A normalized pose distance disentangles pose from scale and maintains accuracy when annotated data is scarce.The distance is used for losses involving poses.
- Results: Experiments on standard benchmarks and competitive alpine skiing show that the method cuts annotation needs and improves robustness to viewpoint and scale changes.The approach is evaluated in scenarios where annotated data is genuinely hard to obtain.
2. Related work
Prior work addresses scarce pose data through dataset creation, synthetic augmentation, and weak supervision, but these strategies retain important annotation, realism, or camera-calibration requirements.
- Existing limitations: Modern monocular 3D pose methods operate in real time but generally depend on sophisticated architectures trained on very large datasets.Motions without such databases remain an open problem.
- Image annotation: Controlled capture systems can estimate pose automatically, but in-field videos often require costly, labor-intensive, and error-prone manual or semi-automated annotation.Marker-based and marker-less approaches do not generalize well to in-field scenarios.
- Data augmentation: Synthetic augmentation expands labeled data, but simulation tools provide limited appearance and motion diversity and are not yet perfectly realistic.Prior approaches modify backgrounds and appearances, use image mosaicing, or generate fully synthetic images.
- Weak supervision: Earlier weak-supervision methods impose geometric or view-consistency constraints, yet some require extensive manual 2D annotation and known external camera matrices.The paper positions its multi-view method as related in spirit but different in task and methodology.
- Weak supervision: The labeled and unlabeled training setup applies supervised loss S(θ, L) to images with poses and view-consistency loss M(θ, U) to synchronized unlabeled views.In the unlabeled examples, columns represent cameras and rows represent shared times.
3. Approach
The approach trains a monocular pose network with multi-view consistency on unlabeled footage, combined with scarce supervision and regularization to avoid degenerate or drifting predictions. It jointly estimates body pose and camera rotations, requiring only camera intrinsics for moving-camera footage.
- 3. Approach: The loss combines multi-view consistency on unlabeled samples, supervised regression on labeled samples, and regularization against drift from initial predictions.The consistency term enforces agreement across views, while the supervised and regularization terms address trivial or incorrect consistent solutions.
- 3. Approach: Multi-view consistency penalizes differences beyond rigid transformations between views of the same person at the same time.Because poses are pelvis-centered, camera translation is ignored and the rigid transformation is modeled as rotation only.
- 3. Approach: The method computes a robust reference pose from agreeing views, identified with a deterministic RANSAC variant, and measures each prediction's distance from that reference.The consensus set contains views with the largest agreement in mean pose.
- 3. Approach: The scale-normalized NSE distance avoids scale sensitivity and the zero-pose degeneracy of squared-error-based consistency, substantially increasing accuracy with very few labeled samples.Its influence is similar to enforcing constant bone length with a geometric constraint.
- 3. Approach: Camera rotations are estimated jointly with pose by alternating network updates and rotation updates, making the method independent of camera rotation and position.The system predicts hip-centered poses and therefore requires camera intrinsics but not full camera calibration.
- 3. Approach: Regularization is needed because consistency and supervision can initially improve accuracy but later cause overfitting to labeled versus unlabeled examples.The regularizer keeps unlabeled predictions near early predictions from a network trained only on scarce labeled data.
4. Results
Experiments on H36M, 3DHP, and alpine skiing show that multi-view weak supervision improves pose estimation with limited labels, including under viewpoint changes and specialized outdoor motions.
- Baseline: 11 ms runtime makes the modified ResNet-50 considerably simpler and faster than several more complex state-of-the-art architectures.The architecture remains representative of state-of-the-art results despite its simpler design.
- Human3.6M: 5 mm in NMPJPE is recovered by multi-view weak supervision, corresponding to 28% and 42% of the gap to full supervision in two H36M splits.The gains are measured against SNSE trained only on the labeled subset.
- Human3.6M: Weak-supervision gains are larger with fewer labeled subjects, while SNSE + MNSE + RNSE remains the best-performing method across label-set sizes.The same behavior holds when NPCK replaces NMPJPE as the error metric.
- Ablation: The regularization term prevents the accuracy degradation observed after excessive training iterations without regularization.The unregularized SNSE + MNSE model requires early stopping and is more difficult to train.
- Initialization: 16.1 mm improvement is obtained from unlabeled subjects with ImageNet initialization, showing that the multi-view approach also works without 2D-pose pretraining.This corresponds to 25.3% of the gap under the reported initialization setting.
- Viewpoint and skiing experiments: Multi-view training improves novel-camera-view predictions on 3DHP and consistently improves alpine-ski results, with a reported NMPJPE change from 7.4 to 3.4 mm for one skier.On skiing, improvements decrease as more labeled data are used, while similar 3–5 mm NMPJPE gains are maintained on H36M and 3DHP.
5. Conclusion
The method supplements a small labeled image-and-pose dataset with synchronized multi-view images, using consistency and regularization even when camera positions are not exactly known. Its main limitation is treating video frames independently, motivating temporal consistency for future work.
- A small annotated dataset can be supplemented effectively with synchronized multi-view images requiring minimal supervision, even without exact relative camera positions.
- Multi-view consistency encourages consistent 3D poses across views, while regularization prevents different behavior on annotated and unannotated images.
- The current approach processes individual images despite using video data, limiting its exploitation of temporal information.
- The proposed next step is temporal consistency for camera motions and predictions during training, since poses need not remain identical across time.