Source-linked AI summary
Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images
Valentin Gabeur, Jean-Sebastien Franco, Xavier Martin, Cordelia Schmid, Gregory Rogez
TL;DR
Full 3D human shape estimation from a single image remains challenging because parametric models omit details and volumetric outputs face resolution limits. This paper estimates visible and hidden depth maps and combines them into a 3D shape, with reported improvements on 3D HUMANS and under artificial occlusion.
Problem
Single-image full 3D human shape estimation remains challenging because parametric models limit detail, non-parametric methods can produce physically impossible shapes, and fine surface estimates require high-dimensional outputs and realistic training data.
Source: §1 · pp. 1–2Method
The method predicts visible and hidden depth maps from one image, combines them into a full-surface point cloud, and uses adversarial training to improve accuracy and humanness.
Source: §1 · p. 2Results
On 3D HUMANS, performance is best without SURREAL images and beats an SMPL-fit baseline by a large margin; adversarial training reduces chamfer distance error 7% under artificial occlusions.
Source: §5.2 · p. 7, §5.3 · p. 8Takeaways & Limitations
The approach targets detailed 3D estimates that include clothing, hair, and manipulated objects, beyond the under-cloth shapes produced by parametric methods.
Source: §1 · p. 2Completely occluded limbs can lead to non-human shapes, and adversarial training may hallucinate plausible limbs far from ground truth, increasing error.
Source: §5.2 · p. 7, §5.3 · p. 8
Abstract
from arXiv · showhide
In this paper, we tackle the problem of 3D human shape estimation from single RGB images. While the recent progress in convolutional neural networks has allowed impressive results for 3D human pose estimation, estimating the full 3D shape of a person is still an open issue. Model-based approaches can output precise meshes of naked under-cloth human bodies but fail to estimate details and un-modelled elements such as hair or clothing. On the other hand, non-parametric volumetric approaches can potentially estimate complete shapes but, in practice, they are limited by the resolution of the output grid and cannot produce detailed estimates. In this work, we propose a non-parametric approach that employs a double depth map to represent the 3D shape of a person: a visible depth map and a "hidden" depth map are estimated and combined, to reconstruct the human 3D shape as done with a "mould". This representation through 2D depth maps allows a higher resolution output with a much lower dimension than voxel-based volumetric representations. Additionally, our fully derivable depth-based model allows us to efficiently incorporate a discriminator in an adversarial fashion to improve the accuracy and "humanness" of the 3D output. We train and quantitatively validate our approach on SURREAL and on 3D-HUMANS, a new photorealistic dataset made of semi-synthetic in-house videos annotated with 3D ground truth surfaces.
1. Introduction
Estimating full 3D human shape from images requires representing fine surface detail while addressing the limitations of parametric models, voxel outputs, and demanding training data. The paper proposes paired visible and hidden depth maps and introduces realistic training data for detailed shape estimation.
1. Introduction
- Parametric models limit estimates to their covered details, while non-parametric methods can capture surface detail but may produce physically impossible shapes; voxel resolution also constrains precision.The section also notes that detailed 3D shape estimation requires higher-dimensional outputs and realistic training data.Source: §1 · pp. 1–2
- The method combines visible and hidden depth maps to reconstruct full 3D shape, with output dimensionality O(N^2) rather than O(N^3) for voxel representations.An encoder-decoder predicts both maps from one image; their point cloud is reconstructed using Poisson reconstruction.Source: §1 · p. 2
- The approach aims to estimate detailed human surfaces, including clothing, hair, and manipulated objects, and incorporates a discriminator to improve output accuracy and humanness.The paper states that the differentiable depth-based model allows efficient adversarial training, especially for strong occlusions.Source: §1 · p. 2
- The authors introduce a large-scale dataset of realistic human imagery with ground-truth 3D shape and pose to train and evaluate the method.They identify realistic data as important for general applicability and for avoiding domain-specific adaptations in the network.Source: §1 · p. 2
2. Related Work
Related work spans voxel, point-cloud, surface-element, and depth-map representations, along with parametric body models and monocular or multi-view human reconstruction. The section also identifies a gap in realistic datasets with detailed 3D surface annotations.
2. Related Work
- Voxel, octree, point-cloud, and parametric surface-element approaches represent object shape in different ways; the proposed paired depth maps align with the input image and retain its resolution.The authors characterize their representation as capturing finer details while keeping output dimensionality reasonable.Source: §2 · p. 2
- Most single-image body-shape methods fit parametric models, while newer neural methods predict shape from video, silhouettes, or images and BodyNet predicts voxel occupancy.Existing methods may rely on user-supplied or detected pose estimates, or inertial sensors.Source: §2 · p. 3
- Multi-view methods can provide more complete and precise results when multiple viewpoints are available, and the authors use such capture techniques to create their dataset.These approaches are not designed for monocular input.Source: §2 · p. 3
- Prior projection-based methods predict depth or map image pixels to a 3D surface; unlike these approaches, this method jointly predicts visible and hidden maps across varied poses and camera views.The section also contrasts its clothing, hair, and object detail with model-based methods trained on parametric human models.Source: §2 · p. 3
- Because it is based on SMPL, SURREAL is not realistic in clothing, hair, or object interactions; the authors propose a large-scale realistic-scene dataset with real clothes, real objects, and full 3D mesh and pose ground-truth data.Source: §2 · p. 3
3. Methodology
The method represents full human shape with visible and hidden depth maps, predicts them from an image, and uses adversarial training to guide realistic outputs.
3.1. “Mould” representation
- The mould combines image-aligned visible and hidden depth maps into a complete 3D shape, using ray-traced nearest and furthest surface intersections from a shared origin.The depth values are centered on the mesh’s center of mass before their point clouds are merged.Source: §3.1 · pp. 3–4
- The fixed-size depth-map target supports image-resolution output with O(N^2) dimensions, compared with O(N^3) for voxel representations.N is the size of the bounding box framing the human in the input image.Source: §3.1 · p. 4
- For manageable representation sizes, the mould captures more detail than voxel grids, while its reconstruction error bottoms out where rays intersect the surface more than twice.The paper notes that voxel grids could reach perfect results with infinitely many voxels; Poisson reconstruction can smooth the mould’s difficult regions.Source: Figure 3 · p. 4, §3.1 · p. 4
3.2. Architecture
- A stacked hourglass network predicts both depth maps pixel by pixel, processing features at low resolution and upsampling them to the output while skip connections provide intermediate supervision.The described architecture uses two stacks and outputs maps aligned with the cropped RGB image.Source: §3.2 · p. 4
- The LL1 loss averages depth error across both maps; unlike LL2, it avoids letting background-assignment outliers dominate and also encourages accurate per-pixel depth.The loss is defined over all pixels in the visible and hidden maps.Source: §3.2 · p. 5
3.3. Adversarial training
- Adversarial training pairs depth-map fitting with a discriminator that distinguishes ground-truth from generated maps, guiding sharper, more realistic predictions during learning.The discriminator guides the generator during training but is not used at test time.Source: §3.3 · p. 5
4. Dataset generation
3D HUMANS combines detailed captured human meshes with rendered backgrounds to produce images and ground-truth 3D annotations. Its capture spans varied garments, movements, and activities.
4. Dataset generation
- The dataset contains more than 150,000 reconstructed meshes from 19 subjects captured in varied garments, movements, and activities.Thirteen subjects were captured performing basic movements in four garment types, and six were captured performing everyday activities.Source: §4 · p. 5
- Captured clothed subjects manipulating objects were rendered against real-world backgrounds to generate ground-truth visible and hidden depth maps.A separate test set used meshes rendered in realistic 3D environments.Source: Figure 4 · p. 6
- Videos were rendered at 320 x 240 resolution and 100 frames; annotations used four Human3.6M virtual-MoCap views and LCR-Net++ to estimate 2D/3D poses, then multi-view reconstruction initialized SMPL fitting.Source: §4 · p. 6
5. Experiments
Experiments compare the approach on SURREAL and 3D HUMANS, assess occlusion and generalization, and report that dataset choice and adversarial training affect performance.
5. Experiments
- Evaluation uses SURREAL and 3D HUMANS, which respectively cover broader motion and more shape detail; images are cropped, resized to 256x256, and assessed on 128x128 depth maps.Performance measures the distance between each ground-truth foreground pixel and its corresponding predicted depth-map pixel.Source: §5 · p. 6
5.1. SURREAL
- On SURREAL, image-domain depth estimates substantially outperform BodyNet; regression yields 5% more pixels with correct depth than classification at 30 mm.The comparison evaluates depth maps, where BodyNet’s performance drops; the authors report that their estimates align with the image and better recover details.Source: Figure 5 · p. 6, §5.1 · pp. 6–7
5.2. 3D HUMANS
- On 3D-HUMANS, 50,000 images suffice, more add little, and training without SURREAL performs best; the method beats the fitted-SMPL upper-bound baseline.At 50 mm, with/without clothing is 83.30%/85.43%, and with/without objects is 79.11%/84.55%; visible depth is estimated better than hidden depth.Source: §5.2 · p. 7
5.3. GAN
- Under severe occlusion, adversarial training drops chamfer-distance error by 7% and qualitatively improves reconstructions, though plausible hallucinated limbs can score worse.The metric does not assess the overall plausibility of the estimate.Source: Figure 7 · p. 7, §5.3 · p. 8
5.4. Generalisation
- Mixed SURREAL and 3D-HUMANS training performs best on realistic-background renders; qualitative LSP and DeepFashion examples show generalization to unseen poses and detailed clothed shapes.The conclusion describes combining visible and hidden depth maps into a full-surface point cloud and incorporating a discriminator to improve accuracy and humanness.Source: §5.4 · p. 8, §6 · p. 8, Figure 9 · p. 8
7. Architecture details
The generator predicts two 128×128 depth maps using a modified stacked-hourglass architecture, while a CNN discriminator scores whether the maps resemble ground truth.
7.1. Generator
- The generator outputs two 128×128 depth maps, rather than 64×64 joint heatmaps, using two stacked hourglasses with five skip connections and group normalization.It also omits max pooling after layer1 and projects the hourglass result to two channels; group normalization performs better on small training batches.Source: §7.1 · p. 10
7.2. Discriminator
- The discriminator takes the two 128×128 depth maps and outputs a scalar indicating whether it judges them to be ground-truth or generator-produced.Scores near 1 indicate ground-truth maps; scores near 0 indicate generated maps.Source: §7.2 · p. 10