Source-linked AI summary
3DPeople: Modeling the Geometry of Dressed Humans
Albert Pumarola, Jordi Sanchez, Gary P. T. Choi, Alberto Sanfeliu, Francesc Moreno-Noguer
TL;DR
Low-rank parametric models describe naked-body shape well but do not represent clothing geometry. The paper combines a dressed-human dataset, geometry-image parameterization, and an end-to-end image-to-shape network; it reports promising results, while noting that non-parametric reconstructions tend to be less smooth than parametric fits.
Problem
Low-rank parametric models represent naked-body geometry well but are not appropriate for modeling clothing geometry.
Source: §1 · p. 2Method
The paper represents clothed bodies as geometry images and maps single RGB images to them with an end-to-end network, using semantically aligned geometry images across dataset meshes.
Source: §4 · p. 4, §5.3 · p. 5Results
Takeaways & Limitations
The paper contributes a large-scale dataset with explicit cloth geometry, an algorithm for spherical parameterization of elongated body parts, and a single-image network for estimating body and clothing shape without parametric models.
Source: §8 · p. 8Non-parametric reconstructions tend to be less smooth than parametric fittings.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Recent advances in 3D human shape estimation build upon parametric representations that model very well the shape of the naked body, but are not appropriate to represent the clothing geometry. In this paper, we present an approach to model dressed humans and predict their geometry from single images. We contribute in three fundamental aspects of the problem, namely, a new dataset, a novel shape parameterization algorithm and an end-to-end deep generative network for predicting shape. First, we present 3DPeople, a large-scale synthetic dataset with 2.5 Million photo-realistic images of 80 subjects performing 70 activities and wearing diverse outfits. Besides providing textured 3D meshes for clothes and body, we annotate the dataset with segmentation masks, skeletons, depth, normal maps and optical flow. All this together makes 3DPeople suitable for a plethora of tasks. We then represent the 3D shapes using 2D geometry images. To build these images we propose a novel spherical area-preserving parameterization algorithm based on the optimal mass transportation method. We show this approach to improve existing spherical maps which tend to shrink the elongated parts of the full body models such as the arms and legs, making the geometry images incomplete. Finally, we design a multi-resolution deep generative network that, given an input image of a dressed human, predicts his/her geometry image (and thus the clothed body shape) in an end-to-end manner. We obtain very promising results in jointly capturing body pose and clothing shape, both for synthetic validation and on the wild images.
1. Introduction
Existing parametric models describe naked-body geometry well but do not capture clothing, while prior alternatives retain constraints on clothing or input conditions. The paper addresses dressed-human reconstruction through a dataset, a spherical parameterization method, and an end-to-end image-to-shape network.
1. Introduction
- Low-rank parametric models accurately describe naked bodies but are not appropriate for capturing clothed-human shape.The paper notes that a silhouette-and-visual-hull alternative requires a frontal image, no background, and relatively simple poses.Source: §1 · p. 2
- 3DPeople provides 2.5 million photorealistic synthetic images with separate geometry for clothes and body, plus skeleton, depth, normal, optical-flow, and segmentation annotations.Unlike SURREAL, its clothing has its own geometry rather than being mapped as a texture onto the naked body.Source: §1 · p. 2
- The proposed spherical area-preserving parameterization combines and extends FLASH and optimal mass transportation to address inaccurate mappings of elongated body parts.Existing algorithms were found to shrink arms and legs, leaving geometry images incomplete.Source: §1 · p. 2
- An end-to-end generative network maps dressed-human RGB images to geometry images using a coarse-to-fine strategy and geometry-aware losses.The paper reports promising results on synthetic and real images.Source: §1 · p. 2
2. Related work
Prior work has emphasized 3D joint localization, while body-shape estimation has relied heavily on low-dimensional models of naked bodies. Other representations and synthetic datasets broaden the approaches available, but the cited work identifies limits in clothing geometry and input conditions.
2. Related work
- Compared with extensive work on 3D joint localization, 3D body-shape estimation has received relatively little attention, with existing datasets uniquely annotated with skeleton joints.The authors suggest this may reflect the availability of established pose datasets.Source: §2 · p. 2
- Shape-estimation methods commonly fit low-dimensional SCAPE or SMPL models, which describe naked-body geometry but are unsuitable for clothing geometry.Prior approaches include CNN prediction of shape parameters, differential-rendering losses, and an adversarial realism loss.Source: §2 · p. 2
- Non-parametric alternatives include voxels, octrees, point clouds, geometry images, and single-image methods for nonrigid hands or human bodies.The cited visual-hull approach predicts silhouettes from a frontal image before reconstructing 3D shape.Source: §2 · p. 2
- Prior GAN applications include generating people under arbitrary poses, learning SMPL-parameter distributions, and editing or generating talking faces.The cited examples span body and face applications.Source: §2 · pp. 2–3
- SURREAL provides more than 6 million annotated frames, but its clothes are textures projected onto naked SMPL shapes rather than explicitly modeled geometry.3DPeople is presented to fill this clothing-geometry gap.Source: §2 · p. 3
3. 3DPeople dataset
3DPeople is a synthetic dataset of dressed people in motion, designed with explicit clothing geometry and varied subjects, outfits, actions, and rendered scenes. Its frames carry geometry, pose, appearance, and image-formation annotations.
3. 3DPeople dataset
- 3DPeople is introduced as the first dataset of dressed humans with a specific geometry representation for clothes.The dataset contains 2.5 million photorealistic 640 × 480 images of 80 people—40 male and 40 female—performing 70 actions.Source: §3 · p. 3
- Each frame includes textured meshes of the naked and dressed body, a 3D skeleton, body-part and clothing segmentation, depth, optical flow, and camera parameters.Clothing labels align with the dressed human, while body-part labels align with the naked body.Source: §3 · pp. 3–4
- The 80 textured body models span varied body shapes, skin tones, and hair, and outfits combine tight and loose garments with accessories.Rigged body-and-clothing meshes contain approximately 20K vertices.Source: §3 · p. 3
- The dataset uses 70 motion-capture sequences, ranging from small movements to complex actions, with a mean sequence length of 110 frames.The authors believe the motion expressivity also makes the dataset appropriate for exploring action recognition.Source: §3 · p. 3
- Figure 3 traces the reference mesh through spherical and octahedral parameterization to a planar geometry image and reconstructed mesh; colored edges mark symmetry used by the regressor.Source: Figure 3 · p. 4
- SAPP is the only compared mapping reported to effectively recover feet and hands, relative to FLASH [12] and [7].Source: Figure 4 · p. 4
- Each sequence is rendered from four approximately orthogonal camera views, with randomized clothing textures, lighting direction, and static background images.The projective camera uses an 800 mm focal length and 640 × 480 resolution.Source: §3 · p. 4
4. Problem formulation
The task is to estimate a dressed person’s 3D shape from one image by predicting a geometry image that represents the clothed body.
4. Problem formulation
- Given a single image of a person wearing an arbitrary outfit, the model aims to estimate the clothed body’s 3D shape.The geometry is represented by a mesh with N^2 vertices encoded in an N × N × 3 geometry image.Source: §4 · p. 4
- The geometry image maps the 3D mesh to a regular 2D grid that preserves neighborhood relations for CNN locality.Its resolution can be uniformly increased or reduced by upsampling or downsampling.Source: §4 · p. 4
- The pipeline constructs geometry images and then uses a deep generative model to predict 3D shape.The network is trained on pairs of input images and their corresponding geometry images.Source: §4 · p. 4
5. Geometry image for dressed humans
The section describes constructing geometry images for dressed-human meshes, from a reference mesh and spherical parameterization to mappings for arbitrary poses. The area-preserving parameterization addresses limb shrinkage, while the arbitrary-mesh procedure yields semantically aligned geometry images.
5. Geometry image for dressed humans
- Training pairs use input images and corresponding geometry images, constructed either for a reference t-pose mesh or for another dataset mesh.A dataset subject in t-pose is selected as the reference mesh.Source: §5 · p. 4, §5.1 · p. 4
5.1. Geometry image for a reference mesh
- A reference mesh is repaired, mapped to a sphere, and unfolded through an octahedron into a flat geometry image.The repair heuristic voxelizes the mesh, selects the largest connected α-shape region, and fills holes using a medial-axis approach.Source: §5.1 · pp. 4–5
5.2. Spherical area-Preserving parameterization
- The spherical area-preserving parameterization combines FLASH and optimal mass transportation to handle elongated full-body parts that existing schemes shrink.It begins with a rescaled planar parameterization, reduces area distortion through quasi-conformal composition, then uses optimal mass transportation and inverse stereographic projection.Source: §5.2 · p. 5
5.3. Geometry image for arbitrary meshes
- For an arbitrary-pose mesh, known vertex correspondence and nonrigid ICP transfer its t-pose mesh to the reference before mapping it into a geometry image.Figure 5 shows the resulting mesh reconstructed from the geometry image as a very good approximation of the input mesh.Source: §5.3 · p. 5, Figure 5 · p. 5
- After the t-pose correspondence is computed once per subject, a new mesh’s geometry image takes seconds to create and is semantically aligned across the dataset.Approximately corresponding UV entries represent the same semantic model part, which the paper says alleviates network learning.Source: §5.3 · p. 5
6. GimNet
GimNet predicts a dressed person’s geometry image from an RGB image and 2D joint locations, then reconstructs the 3D shape progressively. Its architecture combines multiscale regression with discriminators that assess mesh consistency.
6. GimNet
- An RGB image and extracted 2D joint locations feed GimNet’s regressor to predict a geometry image representing the dressed person’s shape.The regressor progressively reconstructs meshes at higher resolution, while two discriminators further encourage anthropomorphic shapes.Source: §6 · pp. 5–6
6.1. Model architecture
- The regressor uses coarse-to-fine prediction, moving from basic shape configuration to finer details, and a differentiable layer enforces geometry-image edge symmetry.The progressive approach is described as more stable than directly learning the full mapping.Source: §6.1 · p. 6
- Two same-architecture discriminators operate at different scales: one evaluates whole-shape coherence and the other enforces local surface-face consistency.The global discriminator uses a large receptive field; the local discriminator focuses on small patches.Source: §6.1 · p. 6
6.2. Learning the model
- Learning combines multiscale L1 reconstruction and projection losses, weighting errors across scales and encouraging the mesh to project correctly onto the input image.The authors report that L1 loss yielded better reconstructions than mean squared error in initial experimentation.Source: §6.2 · p. 6
- Feature matching penalizes higher-level discriminator features, while adversarial training uses two discriminators and a gradient penalty on true data to encourage anthropomorphic shapes.The discriminator-gradient penalty addresses training instability associated with non-overlapping supports of the real and generated distributions.Source: §6.2 · pp. 6–7
6.3. Implementation details
- Implementation extends a U-Net regressor to multiple scales and uses PatchGan discriminators at different mesh resolutions, trained on 170,000 synthetic images for 60 epochs.The geometry images are 128 × 128 × 3, and the model uses four scales.Source: §6.3 · p. 7
7. Experimental evaluation
The evaluation measures mesh error on synthetic test images and presents qualitative reconstructions on synthetic and real fashion images. GimNet’s reported errors are 15–35 mm, while qualitative examples include long dresses and some spiky meshes.
Synthetic Results
- Evaluation uses 25,000 test images from eight held-out subjects, with ground-truth 2D poses corrupted by Gaussian noise with 2-pixel standard deviation.The proposed bidirectional nearest-neighbor metric accommodates meshes without one-to-one vertex correspondence.Source: §7 · p. 7
- GimNet’s mean error distance is 15–35 mm, while ground-truth GIM error is below 5 mm; is indicative only and not directly comparable.The results were not retrained on 3DPeople and used rigid ICP alignment.Source: §7 · p. 8
Qualitative Results
- Qualitative examples include long-dress reconstructions, but some meshes show spikes and are less smooth than parametric fits.Figure 8 shows synthetic and real fashion images, with two views and mesh colors encoding xyz vertex position.Source: §7 · p. 8, Figure 8 · p. 8
8. Conclusions
The paper contributes a dataset with explicitly modeled clothing geometry, a spherical parameterization algorithm for elongated body parts, and a single-image network for estimating body and clothing shape. It also identifies video, geometry-image regularization, and combining segmentation with 3D reconstruction as open directions.
8. Conclusions
- The dataset is the first large-scale collection of 3D humans in action with explicitly modeled cloth geometry.The paper presents it as a contribution to reconstructing dressed-human shape.Source: §8 · p. 8
- A new algorithm parameterizes elongated body parts spherically for representing rigged human meshes as geometry images.Source: §8 · p. 8
- An end-to-end network estimates human body and clothing shape from single images without relying on parametric models.Source: §8 · p. 8
- The paper identifies video, new geometry-image regularization, and combining segmentation with 3D reconstruction as open directions.It states these directions can benefit from the 3DPeople dataset.Source: §8 · p. 8