Source-linked AI summary
Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation
Helge Rhodin, Mathieu Salzmann, Pascal Fua
TL;DR
Deep-network 3D human pose estimation is constrained by the scale and diversity of required annotations, while existing weakly supervised methods still need substantial 3D-labeled data. The paper learns a geometry-aware latent body representation from unlabeled synchronized multi-view images with an encoder-decoder, then maps it to pose semi-supervisedly. The resulting representation supports pose estimation and novel view synthesis, with strong pose performance when labeled data are scarce.
Problem
Current monocular 3D pose methods require large annotated datasets, and existing weakly supervised methods still need enough 3D training data to initialize learning.
Method
An encoder-decoder learns a latent representation of pose, shape, and appearance from unlabeled multi-view images, which is then mapped to 3D pose with supervised learning.
Results
The representation is effective for 3D pose estimation and novel view synthesis, and the semi-supervised approach performs much better than state-of-the-art methods with very little annotated data.
Takeaways & Limitations
Geometry-aware representation learning can reduce the supervised data required for 3D human pose estimation while also supporting novel view synthesis.
Takeaways & Limitations
The representation assumes that appearance changes little between paired frames and that image differences are caused by 3D pose changes.
Abstract
from arXiv · showhide
Modern 3D human pose estimation techniques rely on deep networks, which require large amounts of training data. While weakly-supervised methods require less supervision, by utilizing 2D poses or multi-view imagery without annotations, they still need a sufficiently large set of samples with 3D annotations for learning to succeed. In this paper, we propose to overcome this problem by learning a geometry-aware body representation from multi-view images without annotations. To this end, we use an encoder-decoder that predicts an image from one viewpoint given an image from another viewpoint. Because this representation encodes 3D geometry, using it in a semi-supervised setting makes it easier to learn a mapping from it to 3D human pose. As evidenced by our experiments, our approach significantly outperforms fully-supervised methods given the same amount of labeled data, and improves over other semi-supervised methods while using as little as 1% of the labeled data.
1 Introduction
The paper addresses the annotation bottleneck in monocular 3D human pose estimation by learning a geometry-aware latent representation from unlabeled multi-view images. This representation supports simpler supervised pose mapping and improves performance with limited annotations.
- Large-scale annotation is infeasible for covering diverse human body shapes, appearances, and poses.
- Existing weakly supervised methods still require sufficient 3D training data to initialize learning, limiting gains from unlabeled examples.
- The encoder-decoder learns a latent representation by predicting one viewpoint from another using unlabeled multi-view images.
- Because the latent representation captures 3D geometry, mapping it to 3D pose requires a simpler model and fewer supervised examples.
- The representation resembles a volumetric 3D shape, requires no silhouette information, works from monocular test views, and supports novel view synthesis.
- On Human3.6Million, the semi-supervised method drastically outperforms fully supervised methods in 3D pose reconstruction accuracy with few labeled examples.
2 Related work
Related work spans semi-supervised pose estimation, geometry-aware representations, and novel view synthesis. The paper distinguishes its approach by applying unsupervised geometric representation learning to the less constrained problem of human pose.
- Semi-supervised human pose estimation: Semi-supervised pose methods use 2D annotations, multi-view footage, or multi-view geometry to reduce 3D annotation requirements.
- Geometry-aware representations: Renderer-based approaches encode geometry through 3D representations that can be decoded into images, including learned rendering functions.
- Geometry-aware representations: Unsupervised image-to-geometry methods have primarily been demonstrated on constrained problems such as face modeling, leaving generalization unclear.
- Novel view synthesis: Novel view synthesis commonly uses encoder-decoder architectures whose latent code includes viewpoint-change information.
3 Unsupervised Geometry-Aware Latent Representation
The method learns a geometry-aware latent representation from synchronized multi-view images without pose annotations by predicting one viewpoint from another. It explicitly rotates a 3D latent code while separating geometry, appearance, and background information.
- Unsupervised Geometry-Aware Latent Representation: Synchronized multi-view footage trains an encoder-decoder to learn a latent representation without 2D or 3D pose annotations.The setup uses image pairs from calibrated cameras and compares its annotation effort with labeling tens of thousands of poses.
- Factoring appearance and background: The latent space separates L3D for body geometry and pose from Lapp for appearance and B for background.Appearance is swapped across two time frames of the same subject, while background information is supplied directly to the decoder.
- Representation learning: A 3D latent space, rotation operation, and background fusion module extend a conventional autoencoder for geometry-aware representation learning on natural images.Rotation enforces explicit 3D encoding, while background fusion removes the need to encode background information in the latent code.
- Learning to encode multi-view geometry: L3D ∈ R3×N represents N points in 3D space, allowing viewpoint changes through explicit rotation before decoding.The three-channel latent output and corresponding decoder input make view changes proper 3D rotations.
- Combined optimization: Training combines pixel-wise reconstruction with a weighted feature loss from an ImageNet-trained ResNet-18 to produce crisper decodings and improved pose reconstruction.The feature loss is weighted by two relative to the pixel and feature difference averaging scheme.
4 3D Human Pose Estimation
The learned L3D representation is mapped to semantic 3D human joints using limited supervision. Because L3D already encodes pose and shape, the final pose predictor can be comparatively simple.
- 4 3D Human Pose Estimation: The mapping F from L3D to 3D pose is learned with a small labeled set of image pairs and corresponding ground-truth poses.This supervised stage follows unsupervised representation learning.
- 4 3D Human Pose Estimation: Because L3D encodes human pose and shape, F can be implemented as a simple fully-connected neural network in the semi-supervised setup.The encoder-decoder performs much of the lifting from images to a 3D representation, leaving a comparatively easier final mapping.
5 Evaluation
Evaluation on Human3.6M shows that the representation is most advantageous when annotated training data are scarce, while qualitative tests support separate encoding of geometry, appearance, and background.
- Experimental setup: The Human3.6M evaluation compares supervision regimes using a calibrated multi-view dataset with ground-truth poses for all frames.This enables comparisons among unsupervised, semi-supervised, and fully supervised settings.
- Semi-supervised pose estimation: Below 180,000 annotated images, baseline accuracy degrades significantly while OursResnet degrades more gracefully and becomes significantly better.The improvement is consistent across the evaluated metrics.
- Semi-supervised pose estimation: OursUnet improves over Rhodin by 17.4 mm and OursResnet by 30.7 mm on labeled subject S1.These correspond to relative improvements of 11.3% and 20%, respectively.
- Semi-supervised pose estimation: With fewer annotations, the method loses accuracy slowly down to 5,000 images and still performs adequately with 500 images.When more than 180,000 annotated images are available, the baselines outperform the method.
- Ablation study: Removing the geometry-aware 3D latent space causes the largest ablation decrease, 46.1 mm, exceeding the effects of other architectural changes.The ablation therefore associates the geometry-aware latent representation with the strongest accuracy contribution among the tested choices.
- Latent representation evaluation: Qualitative synthesis tests show that geometry remains correct for unseen subjects, appearance can be transferred, and background handling supports background changes.Without the geometry-aware latent space, novel views become blurrier; limited training appearances can produce mixed or incorrect appearance colors.
- Latent representation evaluation: The method faithfully reconstructs a chair’s appearance and 3D position, suggesting applicability beyond human pose estimation.The paper identifies more generic problems as a direction for future work.
6 Conclusion
The paper introduces an unsupervised geometry-aware human-body representation learned from multi-view imagery. Experiments show benefits for both 3D pose estimation and novel view synthesis, especially when annotated data is scarce.
- The representation is learned unsupervised from multi-view imagery and captures geometry relevant to human-body modeling.
- The learned representation serves as an intermediate representation for 3D pose estimation and supports novel view synthesis.
- With very little annotated data, the semi-supervised approach performs much better than state-of-the-art methods for pose estimation.
- Future work will extend the approach to larger multi-view datasets while retaining unsupervised representation learning.