Source-linked AI summary
MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
Grégory Rogez, Cordelia Schmid
TL;DR
The paper addresses limited large-scale training data for 3D human pose estimation in the wild. It generates photorealistic, MoCap-guided synthetic images from real 2D-annotated images, then trains an end-to-end CNN for pose classification. The method outperforms state-of-the-art results in controlled environments and shows promising performance on in-the-wild images.
Problem
Large-scale real images with accurate 3D pose annotations are difficult to obtain, limiting CNN-based 3D pose estimation in the wild.
Method
The method stitches locally matched patches from real 2D-annotated images using MoCap 3D poses, then trains a K-way end-to-end CNN pose classifier.
Results
The method outperforms state-of-the-art 3D pose estimation in controlled environments and shows promising results on in-the-wild images.
Takeaways & Limitations
CNNs trained with the synthetic images generalize well to real images, and combining synthetic with real data improves classifier performance.
Takeaways & Limitations
Failures occur when observed poses are absent from the MoCap training database or when right-left or front-back ambiguities arise.
Abstract
from arXiv · showhide
This paper addresses the problem of 3D human pose estimation in the wild. A significant challenge is the lack of training data, i.e., 2D images of humans annotated with 3D poses. Such data is necessary to train state-of-the-art CNN architectures. Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations. We introduce an image-based synthesis engine that artificially augments a dataset of real images with 2D human pose annotations using 3D Motion Capture (MoCap) data. Given a candidate 3D pose our algorithm selects for each joint an image whose 2D pose locally matches the projected 3D pose. The selected images are then combined to generate a new synthetic image by stitching local image patches in a kinematically constrained manner. The resulting images are used to train an end-to-end CNN for full-body 3D pose estimation. We cluster the training data into a large number of pose classes and tackle pose estimation as a K-way classification problem. Such an approach is viable only with large training sets such as ours. Our method outperforms the state of the art in terms of 3D pose estimation in controlled environments (Human3.6M) and shows promising results for in-the-wild images (LSP). This demonstrates that CNNs trained on artificial images generalize well to real images.
1 Introduction
The paper targets the lack of large-scale, realistic 3D pose annotations for training CNNs on in-the-wild images. It proposes MoCap-guided image synthesis that augments real annotated images with new poses and 3D labels.
- Motivation: 3D pose annotation for large collections of real images is impractical, while existing large-scale MoCap data comes from constrained marker-based environments.These conditions do not match real environments well, limiting end-to-end CNN development for in-the-wild 3D pose estimation.
- Approach: Unlike rendered human models, the proposed image-based approach uses MoCap data to augment real images annotated with 2D poses.This addresses the domain-shift concern associated with synthetic-to-real data transfer.
- Approach: For each candidate MoCap pose, the system combines annotated image regions around individual joints through kinematically constrained copy-pasting.The resulting images contain new pose configurations while retaining corresponding 3D pose annotations.
- Approach: The method trains an end-to-end CNN by clustering 3D poses into K classes and predicting a distribution over pose classes from a human bounding box.The large synthesized dataset makes this K-way classification formulation viable.
- Results: The approach improves state-of-the-art 3D pose estimation in controlled environments and shows promising results on images captured in the wild.Related work also combines 2D pose images with 3D MoCap data, but this method combines them offline into a large training set for end-to-end CNN training.
- Approach: The synthesis engine takes real 2D-annotated images and 3D MoCap poses as inputs, producing 220x220 synthetic images with associated 3D poses.The engine is designed to expand the available training set with labeled synthetic examples.
2 Image-based synthesis engine
The synthesis engine uses MoCap poses to retrieve locally matching annotated images for each joint, then stitches and blends aligned patches into kinematically correct, locally photorealistic mosaics.
- Overview: The engine takes real images with 2D annotations and MoCap data to generate photorealistic synthetic images with associated 3D poses.It augments the original database with images covering more body-pose configurations.
- 2.1 MoCap-guided image mosaicing: For each projected MoCap joint, the method retrieves an annotated image whose aligned 2D pose matches the query pose locally.The distance compares aligned poses while weighting joints according to their distance from the query joint.
- 2.1 MoCap-guided image mosaicing: The selected image patches are combined by assigning each pixel to one retrieved image using interpolated probability maps over a Delaunay triangulation.The resulting index map determines which aligned image supplies each pixel.
- 2.2 Pose-aware image blending: A pose-aware blending stage smooths mosaic boundaries by weighting aligned images according to index histograms in surrounding regions.The final mosaic is computed as a normalized weighted sum of aligned images.
- 2.2 Pose-aware image blending: The procedure produces plausible images that are kinematically correct and locally photorealistic.The blending stage is designed to preserve human-body image information while removing stitching artifacts.
3 CNN for full-body 3D pose estimation
The paper formulates full-body 3D pose estimation as classification over clustered, oriented 3D poses. A CNN returns pose-class scores, while multiple hypotheses can be retained for ambiguous cases.
- CNN pose classifier: The 3D pose space is partitioned into K clusters, and a CNN is trained to output a distribution over the resulting pose classes.Pose estimation is obtained by selecting the highest-scoring class.
- CNN pose classifier: The classifier supports multiple pose hypotheses that can be rescored using additional information such as temporal context.This addresses multimodal outputs in ambiguous cases.
- Training data and pose classes: The system synthesizes 220×220 images for MoCap poses and camera views, then clusters oriented poses after torso-centering and camera alignment.The authors empirically found K=5000 clusters sufficient.
- Comparison model: A holistic pose regressor is also trained to regress jointly to normalized 2D and 3D pose coordinates for comparison with prior work.The 3D coordinates are expressed in meters and normalized to [−1, 1], as are the 2D coordinates.
4 Experiments
Experiments evaluate the method on controlled Human3.6M and in-the-wild LSP data, testing training sources, synthetic-data scale, pose-class count, and CNN architectures. Synthetic and real data together improve Human3.6M classification, while larger synthetic datasets and realistic image sources improve broader performance.
- Human3.6M evaluation: Human3.6M evaluation uses protocols P1 and P2, measuring average 3D pose error across 13 joints with aligned and absolute errors reported.P1 trains on six subjects and tests every 64th frame of S11; P2 tests all frames from S9 and S11.
- Human3.6M evaluation: With 17,000 training images, the regressor outperforms the classifier, but the classifier performs better with larger training sets.The classification approach requires sufficient examples per pose class.
- Human3.6M evaluation: 88.1mm versus 98.3mm: retraining AlexNet from scratch outperforms fine-tuning an ImageNet-pretrained model on 3D error.The comparison concerns the same Human3.6M evaluation setting described by the authors.
- Human3.6M evaluation: The classifier trained on synthetic images performs similarly to one trained on real images, while combining both sources performs much better.The authors interpret this as evidence that synthetic images provide useful and complementary training information.
- Human3.6M evaluation: The best classifier, trained with synthetic and real data, outperforms state-of-the-art single-frame 3D pose results on Human3.6M.The comparison uses two protocols; a method using temporal information reports better performance, and the proposed method estimates absolute pose.
- LSP evaluation: On LSP, in-the-wild image pools improve performance while constrained indoor pools improve Human3.6M performance but reduce LSP performance.Using more poses and 2 million synthetic images improves both test sets; pose error converges at 1.5 million images and beyond K = 5000 classes gives no further improvement.
- LSP evaluation: 2.3 pixels: fine-tuning VGG-16 decreases average normalized 2D pose error on LSP, producing performance comparable to recent 2D pose methods.Errors are measured in pixels on normalized 220 × 220 images.
- LSP evaluation: Qualitative failures occur when poses fall outside the MoCap database or when right-left or front-back ambiguities arise.Alternative top-scoring hypotheses can often resolve the latter ambiguity, including through temporal rescoring in videos.
5 Conclusion
The paper presents synthetic in-the-wild images with 3D pose annotations for training CNNs, achieving strong controlled-environment results and promising LSP performance. Its current estimator remains coarse, and typical failures involve unseen poses and left-right or front-back confusions.
- The approach augments real images with synthetic in-the-wild images and corresponding 3D pose annotations for CNN-based estimation.The system creates new poses using image-based synthesis and trains an end-to-end CNN classifier.
- The method outperforms state-of-the-art 3D pose estimation results in controlled environments and shows promising results on in-the-wild LSP images.
- Typical LSP failures involve unseen poses and right-left or front-back confusions.The qualitative results distinguish correct estimates from these recurring failure cases.
- The estimator returns a coarse 3D pose by averaging the poses in the top-scoring cluster.The authors identify pose re-ranking and refinement as future work.