Source-linked AI summary
Synthesizing Training Images for Boosting Human 3D Pose Estimation
Wenzheng Chen, Huan Wang, Yangyan Li, Hao Su, Zhenhua Wang, Changhe Tu, Dani Lischinski, Daniel Cohen-Or, Baoquan Chen
TL;DR
The paper addresses the limited availability of annotated training images for monocular human 3D pose estimation, a challenging task involving non-rigid, articulated bodies and varied clothing. It synthesizes diverse human models and poses with transferred clothing textures, then uses domain adaptation to align synthetic and real data; CNNs trained on this data improve 3D pose estimation performance.
Problem
Monocular human 3D pose estimation is challenging, while suitable annotated training images are limited and human bodies introduce non-rigid articulation and clothing variation.
Method
The paper samples body types and poses from a statistical model, transfers clothing textures from real images, and applies domain adaptation to align synthetic and real data.
Results
CNNs trained with the synthetic images show consistent and significant improvements over published state-of-the-art results on human 3D pose estimation.
Takeaways & Limitations
Pose-space coverage and clothing-texture richness are particularly important for effective synthetic training data, which can advance 3D human pose estimation.
Takeaways & Limitations
Synthetic-image realism is inevitably compromised by tractable computation and modeling, creating a feature-domain gap that hurts performance on real images.
Abstract
from arXiv · showhide
Human 3D pose estimation from a single image is a challenging task with numerous applications. Convolutional Neural Networks (CNNs) have recently achieved superior performance on the task of 2D pose estimation from a single image, by training on images with 2D annotations collected by crowd sourcing. This suggests that similar success could be achieved for direct estimation of 3D poses. However, 3D poses are much harder to annotate, and the lack of suitable annotated training images hinders attempts towards end-to-end solutions. To address this issue, we opt to automatically synthesize training images with ground truth pose annotations. Our work is a systematic study along this road. We find that pose space coverage and texture diversity are the key ingredients for the effectiveness of synthetic training data. We present a fully automatic, scalable approach that samples the human pose space for guiding the synthesis procedure and extracts clothing textures from real images. Furthermore, we explore domain adaptation for bridging the gap between our synthetic training images and real testing photos. We demonstrate that CNNs trained with our synthetic images out-perform those trained with real photos on 3D pose estimation tasks.
1. Introduction
The paper targets direct human 3D pose estimation from a monocular RGB image, where diverse 3D-annotated training data is difficult to obtain. It therefore synthesizes large-scale training images from sampled poses, real clothing textures, and varied rendering conditions, while using domain adaptation to reduce the synthetic-to-real gap.
- Direct 3D pose estimation avoids the error accumulation of systems that first predict 2D joints and then recover 3D pose.
- 3D ground-truth annotation is difficult because depth is hard to infer from a single image, while MoCap lacks accompanying appearance diversity.
- Synthetic human data must cover varied body types, poses, viewpoints, clothing, and backgrounds because direct CNN estimation faces extensive appearance variation.
- The approach learns pose variation from MoCap and inferred 3D poses, transfers clothing textures from real images, and generates 5,099,405 rendered training images.It produces 10,556 uniquely textured human models and composites rendered bodies over randomly chosen background images.
- Domain adaptation maps synthetic and real data toward a common feature space to address the visual gap between synthetic training images and real testing photos.
- CNNs trained with the synthetic data show consistent and significant improvements over published state-of-the-art results across human 3D pose estimation evaluations.The authors also introduce Human3D+, a richer 3D-annotated image dataset.
2. Related Work
Prior work spans 2D pose estimation, 2D-to-3D recovery, CNNs trained on MoCap datasets, synthetic human imagery, and domain adaptation. The paper positions its contribution against limited dataset diversity, accumulated errors, scalability constraints, and cross-dataset overfitting.
- Human Pose Datasets: Existing human pose datasets provide 2D annotations but remain limited in scale and diversity for task-specific CNN fine-tuning.FLIC, MPI, and LSP contain 5003, 2179, and 2000 fully annotated human bodies, respectively.
- 3D Pose Estimation: Methods that recover 3D pose from 2D joints ignore other image information that may provide cues for inferring depth and pose.
- 3D Pose Estimation: CNNs trained on Human3.6M perform well on held-out subjects from that dataset but drop significantly on other datasets, indicating strong overfit.The paper attributes this possibility to the dataset’s limited variety despite its millions of frames and 3D poses.
- Human Pose Data Synthesis: Earlier human-image synthesis methods are constrained by limited scalability, pose or viewpoint variation, user interaction, or dependence on particular scenes and people.
- Domain Adaptation: Domain-adaptation research includes adversarial networks and discrepancy-based methods that align feature distributions between source and target domains.
3. Training a 3D Human Pose Estimator by Synthetic Data
The approach synthesizes diverse, pose-annotated human images by sampling valid poses, transferring real clothing textures, and rendering varied models over real backgrounds. Domain adaptation then addresses the feature gap between synthetic training images and real images.
- Training-data design: Synthetic training data combines viewing angles, poses, clothing, and backgrounds under alignment and variation principles.Generated images should resemble real human images while covering the space of real-image variation.
- Image synthesis: Textured models are rendered with varied camera viewpoints, lighting, and materials, then composited over real image backgrounds.SCAPE supplies articulated models with varied body types, poses, genders, and fitness levels.
- Body pose space modelling: Existing pose datasets sparsely cover the pose space, motivating generation of unseen but valid poses.The method learns part co-occurrences in a sparse non-parametric Bayesian network and composes compatible substructures.
- Body pose space modelling: The learned Bayesian network combines MoCap and inferred 3D poses, sampling interpolations that cover both input sources.The complementary sources provide broader coverage than either source alone.
- Clothing texture transfer: Real clothing textures are transferred automatically onto 3D human models because clothing appearance is difficult to generate with a parametric model.The transfer matches clothing contours, warps the textures, projects them onto the model, and mirrors them across the body.
- Domain adaptation: Synthetic images inevitably retain a realism gap that separates their CNN features from real-image features and harms real-image performance.A domain adaptation network uses a feature extractor, pose regressor, and domain mixer in a two-stage training scheme to align the domains.
4. Results and Discussion
The evaluations show that synthetic training data benefits 3D pose estimation across varied datasets, especially when it covers poses and textures broadly; domain adaptation further improves performance.
- Datasets: Human3D+ provides 1,574 indoor and outdoor images with 3D annotations, richer appearance and backgrounds, and varied general actions for real-world evaluation.Its scenes include rooms, playgrounds, and parks, with actions such as walking, running, and playing football.
- Evaluation setup: AlexNet and VGG are adapted for direct 3D pose estimation by changing their final layer to output 3D coordinates, adding Euclidean loss, and fine-tuning the networks.The study evaluates data effectiveness rather than proposing a new network architecture.
- 3D pose estimation: Models trained on Human3.6M perform comparably or better on Human3.6M but worse on the more varied Human3D+, indicating over-fitting to the controlled dataset.VGG performs worse than AlexNet on Human3.6M when trained and tested there, consistent with stronger over-fitting from greater learning capacity.
- 3D pose estimation: Synthetic data outperforms Human3.6M training on Human3D+, while combining synthetic and Human3.6M images consistently improves performance on both datasets.The paper attributes the complementarity to broader pose and texture variation in synthetic images and greater realism in Human3.6M images.
- Domain adaptation: Domain adaptation significantly improves 3D pose estimation over vanilla networks and outperforms the compared method by better mixing features from synthetic and real images.The improvement is especially prominent when only a minimal amount of real images is available.
- Parameter analysis: Increasing the synthetic training-set size and the number of clothing textures improves 3D pose estimation performance.The results identify scalability and texture variability as important factors in synthetic-data effectiveness.
- Parameter analysis: On synthetic test images, models trained on Human3.6M show a larger performance gap than models trained on synthetic images, supporting greater variation in the synthetic data.Less-varied training data is described as more likely to perform well on itself but poorly on new data.
- 3D reconstruction: Estimated 3D pose can articulate and align a SCAPE model to the person, producing a feasible single-image 3D reconstruction.More faithful reconstruction additionally requires recovering body shape and gaze, which are also difficult to annotate but available from synthesis.
5. Future Work and Conclusions
The paper concludes that scalable synthetic training data can provide free ground-truth associations for 3D pose estimation, with clothing textures, pose distributions, and domain adaptation central to its effectiveness.
- Conclusions: Synthetic training images associate inputs with 3D ground-truth data at no annotation cost.The synthesis system generates training images from 3D models with corresponding pose annotations.
- Conclusions: Clothing-texture richness and pose-distribution coverage are particularly important for effective synthetic training data.Because realistic clothing synthesis is difficult, the paper transfers clothing textures from real images instead.
- Conclusions: Domain adaptation improves the use of synthetic images, while CNNs trained on the synthetic data advance state-of-the-art 3D human pose estimation performance.The authors plan to release the data and software publicly to encourage further research.