Source-linked AI summary
A Generative Model of People in Clothing
Christoph Lassner, Gerard Pons-Moll, Peter V. Gehler
TL;DR
Generating realistic full-body clothed people is difficult because pose, shape, and appearance vary widely, while graphics pipelines and 3D clothed-person data are costly or scarce. ClothNet learns directly from images by generating a dressed-person segmentation and then synthesizing its appearance, with optional pose-and-shape conditioning. The model produces novel clothed people, and evaluations report both substantial segmentation transfer and human confusion between generated and real images.
Problem
Generating full-body images of clothed people must handle high variation in pose, shape, and appearance without relying on complex graphics pipelines or scarce 3D clothed-person data.
Method
ClothNet stacks a variational sketch module that generates dressed-person semantic segmentations with a portray module that synthesizes detailed images, optionally conditioned on a 3D body-model sketch.
Results
85% of the segmentation performance achieved on real data was reached on artificial data, and participants mistook generated images for real in 24.7% of cases.
Takeaways & Limitations
The results suggest that modern machine-learning models can sidestep traditional graphics-pipeline design and 3D data acquisition for generating people with realistic appearance.
Takeaways & Limitations
Generated segmentations can contain bulky limbs, overly smooth fine structures, and less realistic faces caused by differing facial-landmark statistics.
Abstract
from arXiv · showhide
We present the first image-based generative model of people in clothing for the full body. We sidestep the commonly used complex graphics rendering pipeline and the need for high-quality 3D scans of dressed people. Instead, we learn generative models from a large image database. The main challenge is to cope with the high variance in human pose, shape and appearance. For this reason, pure image-based approaches have not been considered so far. We show that this challenge can be overcome by splitting the generating process in two parts. First, we learn to generate a semantic segmentation of the body and clothing. Second, we learn a conditional model on the resulting segments that creates realistic images. The full model is differentiable and can be conditioned on pose, shape or color. The result are samples of people in different clothing items and styles. The proposed model can generate entirely new people with realistic clothing. In several experiments we present encouraging results that suggest an entirely data-driven approach to people generation is possible.
1. Introduction
The paper replaces complex, labor-intensive graphics pipelines with ClothNet, an image-based generative model that learns to create clothed people. Its two variants provide random generation or pose-and-shape control, while experiments show that generated images can appear realistic to human observers.
- Rendering clothed people traditionally requires 3D avatars, garment patterns, physical draping simulation, and texture mapping.These components are costly and complex, particularly because clothing parameters and body–garment interactions are difficult to model.
- ClothNet-full randomly generates clothed people from a learned latent space, while ClothNet-body generates people with similar pose and shape in varied clothing styles.The conditional model takes a synthetic silhouette from a projected 3D body model as input.
- The image-based model learns clothing-to-body statistics from large photo collections and automatically adds plausible clothing and accessories.Examples of accessories include bags, sunglasses, and scarves.
- 24.7% or more of generated images, depending on the ClothNet variant, were mistaken for real by human participants.The paper uses representative samples, latent-space exploration, and a user study to assess generative quality.
2. Related Work
Prior work generates or modifies virtual humans through graphics rendering, pixel warping, or 3D-based systems. These approaches often require manual input, limited subjects, or clothing-free body models, while variational models and GANs provide relevant generative foundations.
- Photo-reshaping methods can require multi-view data or substantial user interaction, and they modify existing photographs rather than generating new ones.One cited morphing approach used only 11 subjects because it required multi-view camera data.
- Rendering systems often use body models without clothing geometry, limiting the quality of their generated results.A cited exception models only the projected 2D cloth contour.
- Variational autoencoders provide a principled foundation for generative modeling and are a key component of ClothNet.
- GANs use a discriminator during training to distinguish generated from training data, and ClothNet uses this strategy to increase image detail.
3. Chictopia made SMPL
The authors augment Chictopia10K with pose, shape, facial, and cleaned segmentation annotations to support supervised generative-model training. SMPL fits are generally reliable but retain recurring head and foot orientation errors, while face matching and mask processing address annotation weaknesses.
- The dataset lacked pose and shape annotations, so the authors fit SMPL using SMPLify and 2D keypoints from DeeperCut.
- Chictopia10K contains 17,706 fashion-blog images with fine-grained segmentation across 18 classes, including clothing categories, background, hair, and skin.
- SMPL fitting performs well across the dataset, but wrong head and foot orientations are the most frequent failures.The authors retain the automatically fitted results rather than manually curating them.
- A single face label produced nearly blank facial regions, motivating facial-landmark augmentation through a face-shape matcher.
- Morphological closing and blackhat operations fill spurious holes in segmentation masks while preserving small structures.A kernel size of 7 pixels fixed most annotation mistakes according to the authors.
4. ClothNet
ClothNet separates clothing-shape generation from textured image synthesis, using variational sketch modules to model multimodality and an image-to-image portray module for detail. Its conditional variant incorporates a body-model silhouette to control pose and shape, while the concatenated models remain differentiable.
- Motivation: Direct image-to-image translation struggles with multiple clothing completions for one pose, whereas variational autoencoders provide latent sampling for high variation.This motivates separating sketch generation from image synthesis.
- Model design: The two-part model generates a semantic segmentation sketch with a variational module, then converts that sketch into a detailed person image with an image-to-image translation network.The portray stage uses the sketch to generate texture and can use skip connections to retain detail.
- The Latent Sketch Module: The latent sketch module is a variational autoencoder that samples random sketches of people from a learned latent space.For 256 × 256 sketches, the model encodes images into a 512-D latent space and uses an image-to-image translation architecture.
- The Latent Sketch Module: The VAE uses a Gaussian encoder distribution and a KL-divergence term that regularizes it toward a zero-mean identity-covariance Gaussian prior.The reparameterization trick makes sampling differentiable, enabling test-time decoding from zi ∼ N(0, I).
- The Conditional Sketch Module: The conditional sketch module controls pose and shape by conditioning generation on a six-part 3D body-model silhouette encoded as y.Its decoder reconstructs sketches using both the deterministic conditioning representation y and the random latent variable z.
- ClothNet-full and ClothNet-body: ClothNet-full concatenates the latent sketch and portray modules, while ClothNet-body concatenates the conditional sketch and portray modules; all stages are differentiable.The sketch and portray modules are trained separately for technical convenience, although gradients can propagate through the full model.
5. Experiments
The experiments evaluate reconstruction, latent-space variation, conditional sketch generation, color conditioning, and realism of ClothNet outputs. Results show high reconstruction accuracy, controllable variation, and perceptually realistic images, while synthetic outputs retain limitations in fine structures and faces.
- 5.1. The Latent Sketch Module: More than 0.95 overall reconstruction accuracy is achieved in all tested settings.The other metrics are affected by small parts, especially facial features, and the conditional sketch module overfits faster.
- 5.1. The Latent Sketch Module: The latent space is explored by traversing the highest-variance PCA direction from -1STD to 1STD using equidistant steps.The PCA is built from 512-dimensional latent predictions of test-set images.
- 5.2. The Conditional Sketch Module: Conditioning on the same six-part SMPL sketch produces people with similar pose and shape but varied clothing, hair, style, and accessories.At test time, different samples from the latent prior generate different clothed sketches for the same conditioning input.
- 5.3. Color Conditioning: Color conditioning generates different clothing appearances from selected color combinations while retaining nonuniform clothing detail.The portray module is retrained using colored segment inputs and can condition individual sketch regions such as dresses, jackets, and hats.
- 5.4.1. Generating an Artificial Dataset: Fully synthetic training loses at most 5.3 accuracy points relative to real-data training, while IoU suffers from missing fine structures such as sunglasses and belts.Models trained and tested on the same data source perform best, and the real-data model reaches the highest performance.
6. Conclusion
The paper presents a data-driven, two-stage model for generating people in clothing and reports evidence of realistic outputs. It also supports controllable generation and may provide an inexpensive source of synthetic training data.
- The authors conclude that machine learning can sidestep traditional graphics-pipeline design and 3D data acquisition for generating people with accurate appearance.
- ClothNet uses a two-stage model that predicts semantic segmentation before texture, providing a modular approach to structured, high-resolution image generation.
- The generated data reached 85% of real-data segmentation performance, while participants mistook generated images for real in 24.7% of trials.
- The method can generate large amounts of training data at low computational and infrastructural cost while conditioning images on pose, shape, or color.