Source-linked AI summary
Learning from Synthetic Humans
Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, Cordelia Schmid
TL;DR
Large-scale manual annotation is impractical for human depth, motion, and part segmentation. SURREAL generates realistic synthetic people from motion-capture sequences with dense ground truth, and CNNs trained on these images perform accurate depth estimation and part segmentation on real RGB images.
Problem
Large-scale manual supervision for human motion, depth, and body-part segmentation is prohibitively expensive, while 2D pose datasets already provide substantial annotated training data.
Method
SURREAL renders realistic human images from 3D motion-capture sequences using SMPL bodies and varied viewpoints, clothing, and lighting, with dense pixel-wise ground truth.
Results
CNNs trained on synthetic data support accurate human depth estimation and human part segmentation in real RGB images.
Takeaways & Limitations
SURREAL demonstrates that realistic synthetic data can support CNN training for multiple complex human pixel-wise prediction tasks.
Takeaways & Limitations
The validation indicates that conclusions from synthetic images still require investigation on real images.
Abstract
from arXiv · showhide
Estimating human pose, shape, and motion from images and videos are fundamental challenges with many applications. Recent advances in 2D human pose estimation use large amounts of manually-labeled training data for learning convolutional neural networks (CNNs). Such data is time consuming to acquire and difficult to extend. Moreover, manual labeling of 3D pose, depth and motion is impractical. In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data. We generate more than 6 million frames together with ground truth pose, depth maps, and segmentation masks. We show that CNNs trained on our synthetic dataset allow for accurate human depth estimation and human part segmentation in real RGB images. Our results and the new dataset open up new possibilities for advancing person analysis using cheap and large-scale synthetic data.
1. Introduction
SURREAL addresses the scarcity of large-scale supervision for human depth, motion, and body-part segmentation by generating realistic synthetic people with rich ground truth. CNNs trained only on this synthetic data generalize sufficiently to real images for depth estimation and part segmentation.
- Manual supervision is sufficient for 2D pose estimation but prohibitively expensive for large-scale human motion, depth, and body-part segmentation.
- Existing 3D motion-capture data and realistic rendering can synthesize variation and ground truth for depth, motion, body-part segmentation, and occlusions.
- SURREAL renders realistic people from 3D motion-capture sequences using the SMPL body model, varied viewpoints, clothing, and lighting.
- More than 6 million frames include ground-truth pose, depth maps, and segmentation masks.
- CNNs trained on synthetic data support accurate human depth estimation and part segmentation in real RGB images.
2. Related work
Related work has used synthetic human imagery for pose, detection, action recognition, and depth-related tasks. SURREAL differs by combining realistic human rendering with rich per-pixel ground truth for pixel-wise and multi-task prediction.
- Prior synthetic-human datasets address 2D and 3D pose estimation, pedestrian detection, and action recognition.
- Existing approaches include game-engine rendering, 2D image deformation, image-based synthesis, and simple body models focused on upright people.
- Other work renders synthetic depth or infrared images for depth and part prediction, or augments real sequences with backgrounds.
- SURREAL provides depth, part-segmentation, and optical-flow ground truth for synthetic RGB frames at large scale.
- Photo-realistic renderings with varied shape, texture, viewpoint, and pose support pixel-wise human labeling tasks.
3. Data generation
SURREAL generates diverse synthetic human sequences by combining motion-capture poses, SMPL bodies, varied appearance and scene parameters, and multiple rendering passes for dense ground truth. The resulting dataset contains 6.5 million frames in 67,582 continuous sequences.
- 3.1. Synthetic humans: The pipeline randomly varies 3D pose, body shape, texture, viewpoint, lighting, and background when rendering human bodies.
- 3.1. Synthetic humans: 6.5 million frames are grouped into 67,582 continuous image sequences with ground truth for depth, optical flow, surface normals, segmentations, and 2D/3D joints.
- 3.1. Synthetic humans: SMPL decomposes body deformation into pose and person-specific shape, while MoSh fits its parameters to raw 3D motion-capture markers.
- 3.1. Synthetic humans: Estimated body shapes are replaced with randomly sampled shapes to increase diversity.
- 3.1. Synthetic humans: Each motion-capture sequence is rendered three times with different random parameters and overlapping 100-frame clips.
- 3.1. Synthetic humans: Blender rendering passes produce pixel-wise part segmentation, optical flow, depth maps, and surface normals.
4. Approach
The approach uses a stacked hourglass CNN to perform human part segmentation and depth estimation as pixel-wise classification from cropped RGB images. Segmentation uses 14 body-part classes plus background, while depth uses 19 quantized depth bins plus background.
- The method addresses human body-part segmentation and depth estimation using synthetic and/or real data.
- The stacked hourglass network repeats contraction and expansion modules with skip and residual connections to model structured spatial relations.
- The network receives 256 × 256 RGB crops fitted to a ground-truth human bounding box.
- Training uses cross-entropy over all pixels and sums losses across 8 stacked modules.
- Segmentation assigns each pixel to 14 human parts or background, while depth is quantized into 19 bins after pelvis-based z-axis alignment.
5. Experiments
Experiments evaluate SURREAL-trained networks on synthetic and real datasets for human-part segmentation and depth estimation. Synthetic pre-training transfers to real images, while fine-tuning and distribution matching improve results, with limitations on challenging poses, viewpoints, and subject diversity.
- Evaluation setup: The experiments evaluate segmentation and depth estimation on synthetic SURREAL data and real Freiburg Sitting People, Human3.6M, and MPII datasets.The study also varies training-data amount, clothing variation, and MoCap source.
- Synthetic test set: 69.13% IOU, 80.61% pixel accuracy, 72.9mm RMSE, and 56.3mm st-RMSE are obtained on 12,528 held-out synthetic test images.Predictions are mostly accurate, but crawling poses, extreme close-ups, and hand details cause errors.
- Freiburg Sitting People: 51.88% pixel accuracy and 40.1% IOU on Freiburg Sitting People show that synthetic pre-training outperforms training from two real subjects.Fine-tuning with those two subjects helps significantly and converges after 200 iterations.
- Human3.6M: On Human3.6M segmentation, real-image training increases IOU by 3.4% and pixel accuracy by 2.14% over synthetic-image training, but generalizes poorly to different clothing.The real baseline can learn skin-color and clothing boundaries rather than body parts; synthetic pre-training remains reasonable despite different pose distributions.
- Human3.6M: Synthetic pre-training fails on some challenging Human3.6M depth poses, while fine-tuning on real data compensates for errors and refines estimates.PoseRMSE evaluates depth at known 2D joints, whereas occluded joints are beyond the paper’s scope.
- MPII Human Pose: On MPII, predictions generalize reasonably well, except for multiple nearby people and extreme viewpoints absent from training; lower-body occlusions and cloth shapes are handled accurately.The MPII evaluation is qualitative because it lacks ground truth for part segmentation and depth.
- Design choices: Human3.6M MoCap pre-training reaches 48.11% IOU and 2.44 RMSE, versus 42.82% and 2.57 for CMU MoCap on real Human3.6M.The result supports using diverse MoCap while matching the target distribution.
6. Conclusions
The study demonstrates successful large-scale CNN training from synthetically generated images, targeting tasks where extensive manual annotation is infeasible. SURREAL provides rich pixel-wise ground truth and may support tasks beyond those evaluated.
- SURREAL enables large-scale CNN training from synthetically generated images of people.
- The study addresses human body-part segmentation and depth estimation, for which large-scale manual annotation is infeasible.
- SURREAL provides rich pixel-wise ground truth and can potentially support tasks beyond body-part segmentation and depth estimation.