Source-linked AI summary
Be Your Own Prada: Fashion Synthesis with Structural Coherence
Shizhan Zhu, Sanja Fidler, Raquel Urtasun, Dahua Lin, Chen Change Loy
TL;DR
FashionGAN tackles text-guided outfit replacement while preserving a wearer’s body structure and pose, a setting where standard GANs do not ensure global coherence. It separates shape and texture generation into two conditional stages, and the authors report precise description-consistent regions, preserved pose and body shape, and results outperforming baselines; the model is limited by training images dominated by plain backgrounds.
Problem
The paper addresses generating a wearer’s new outfit from an input photo and text while preserving body shape, pose, description-consistent regions, and coherent body-part visibility.
Method
FashionGAN uses a shape generator with an input-derived spatial constraint, followed by an image generator that uses the generated segmentation map and text for compositional region-specific rendering.
Results
The generated images contain precise regions consistent with descriptions while keeping body shape and pose unchanged, and quantitative and qualitative results outperform baselines.
Takeaways & Limitations
The approach demonstrates text-based redressing of existing photographs while retaining the wearer’s structural appearance and pose.
Takeaways & Limitations
The results are limited by a training database dominated by plain-background online-shopping images, biasing the model toward that background distribution.
Abstract
from arXiv · showhide
We present a novel and effective approach for generating new clothing on a wearer through generative adversarial learning. Given an input image of a person and a sentence describing a different outfit, our model "redresses" the person as desired, while at the same time keeping the wearer and her/his pose unchanged. Generating new outfits with precise regions conforming to a language description while retaining wearer's body structure is a new challenging task. Existing generative adversarial networks are not ideal in ensuring global coherence of structure given both the input photograph and language description as conditions. We address this challenge by decomposing the complex generative process into two conditional stages. In the first stage, we generate a plausible semantic segmentation map that obeys the wearer's pose as a latent spatial arrangement. An effective spatial constraint is formulated to guide the generation of this semantic segmentation map. In the second stage, a generative model with a newly proposed compositional mapping layer is used to render the final image with precise regions and textures conditioned on this map. We extended the DeepFashion dataset [8] by collecting sentence descriptions for 79K images. We demonstrate the effectiveness of our approach through both quantitative and qualitative evaluations. A user study is also conducted. The codes and the data are available at http://mmlab.ie.cuhk. edu.hk/projects/FashionGAN/.
1. Introduction
FashionGAN redresses a wearer from an input photograph using textual outfit descriptions while preserving body structure and pose. It addresses this challenge with a two-stage GAN that separates shape generation from texture rendering.
- Problem: The task is to generate new outfits whose regions and textures follow language descriptions while retaining the wearer’s body shape, pose, and coherent body-part visibility.A single input view makes body-shape recovery difficult, especially when the new garment reveals arms or skin hidden by the original clothing.
- Limitations of prior methods: Existing 2D and 3D methods impose structural constraints but require explicit human intervention and offer no obvious way to incorporate textual descriptions.Their interventions include physical measurements, body-shape sliders, joint positions, or rough silhouette sketches.
- Limitations of prior methods: Standard DCGANs condition on text but directly generate pixels without enforcing input-image structure, often producing artifacts such as blurry boundaries.The paper identifies pixel averaging as a source of these artifacts.
- Approach: FashionGAN uses two conditional stages: a segmentation generator creates a pose-consistent semantic map, then an image generator renders region-specific textures from that map and the description.The first stage handles shape and visibility; the second performs compositional rendering.
- Approach: The model uses an input-derived spatial constraint and a compositional mapping layer to preserve structural coherence and generate coherent visibility with region-specific textures.The spatial constraint is designed not to contradict the textual condition, while the mapping layer specializes texture generation by image region.
- Evaluation: The study extends DeepFashion with sentence descriptions and human annotations for 79K upper-body images, and evaluates the approach quantitatively, qualitatively, and through a user study.Comparisons include existing GAN baselines and 2D nonparametric approaches.
2. Related Work
The related work spans conditional GANs, image-to-image translation, and stacked generation. FashionGAN differs by conditioning outfit synthesis on input-image spatial configuration and by changing a wearer’s outfit from text rather than generating a fixed-view clothing item.
- Conditional generation: Prior GANs condition image generation on category labels, text, or encoded feature vectors, whereas FashionGAN conditions on the spatial configuration of the input image.The paper positions spatial configuration as its distinctive conditioning signal.
- Image transformation: Earlier image-transfer methods transform images across resolution, domains, viewing angles, or styles, generally applying global transformations.The cited examples include super-resolution, edge-to-image, day-to-night, viewpoint change, and neural style transfer.
- Stacked generation: Stacked GAN approaches commonly use an initial stage for basic composition and a later stage for refinement, but FashionGAN’s first stage generates a wearer-conditioned clothing configuration.This contrasts with S2GAN, whose first stage synthesizes a surface map from a random vector.
- Clothing synthesis: Unlike clothing-generation work that produces a garment from a dressed-person image in a fixed view angle, FashionGAN redesigns the wearer’s outfit from a textual description under unconstrained poses.The paper identifies unconstrained human poses as an additional difficulty.
3. Methodology
FashionGAN decomposes outfit synthesis into segmentation-map generation and texture rendering, using the wearer’s spatial structure and textual design coding as conditions. A downsampled spatial constraint preserves body shape and pose while allowing clothing changes, and compositional mapping renders region-specific textures.
- Overview: FashionGAN formulates outfit synthesis from a wearer image and outfit sentence while preserving the wearer’s pose and body shape.The design coding combines extracted wearer attributes and a text representation of the desired outfit.
- Overview: The framework separates generation into a shape stage that produces a human segmentation map and an image stage that renders textures.The first stage generates the target outfit’s semantic regions; the second renders garments onto the photograph.
- Texture Rendering (Gimage): The second generator uses the generated segmentation map and design coding, with specialized channels combined by semantic region to render coherent textures.The compositional mapping layer is intended to improve region-specific texture details and maintain visibility of body parts.
- Segmentation Map Generation (Gshape): The first generator conditions on a downsampled segmentation map, design coding, and Gaussian noise to generate a new semantic segmentation map.The output uses seven labels: background, hair, face, upper-clothes, pants/shorts, legs, and arms.
- Segmentation Map Generation (Gshape): Downsampling and merging the original segmentation map into four categories makes the spatial constraint clothing-agnostic while retaining body information.The four categories are background, hair, face, and rest; this weakens contradictions between the original clothing partition and the textual design.
- Segmentation Map Generation (Gshape): Different design codings and random vectors yield varied sleeves, upper-clothes lengths, and pants or leg labels while maintaining pose and body shape.The shape generator uses noise and design coding to introduce outfit variation without changing the wearer’s underlying structure.
4. Experiments
FashionGAN is evaluated quantitatively, qualitatively, and through user studies for attribute accuracy, structural coherence, visual quality, and relevance to text descriptions.
- Quantitative Evaluation: FashionGAN more accurately predicts five structure-relevant attributes than the evaluated baselines and performs close to the upper bound.The comparison includes one-step GANs and a two-stage GAN without compositional mapping.
- Qualitative Evaluation: With the same wearer and different descriptions, generated samples vary in sleeve shape, textures, and colors while retaining body shape and pose.The process first generates a plausible segmentation map, then renders textures guided by that map.
- Qualitative Evaluation: With different wearers and the same description, FashionGAN consistently generates outputs that respect the sentence despite varying body shapes and poses.This experiment tests whether textual conditioning remains consistent across different wearers.
- Qualitative Evaluation: Interpolating first-stage inputs changes clothing shapes, second-stage inputs change textures, and interpolating both produces smooth transfers between samples.The interpolations use a concatenation of Gaussian noise and text encoding.
- Qualitative Evaluation: Compared with one-step and non-compositional baselines, FashionGAN produces better visual quality, clearer clothing regions, fewer artifacts, and more consistent human shape.Compared with the 2D non-parametric baseline, exemplar shape inconsistency makes rendering results unsatisfactory.
- User Study: 42% of segmentation-map test cases were misclassified by 50 volunteers, indicating that the generated maps fooled most participants.The study used 8,979 test cases and compared generated maps with original-photo segmentation maps.
- User Study: Most high user-study ranks went to FashionGAN, indicating the best visual quality and relevance to the text input among the compared approaches.Participants ranked FashionGAN against three GAN baselines and a 2D non-parametric method.
5. Conclusion
The paper presents two task-specific GANs with a spatial constraint for generating text-consistent clothing while preserving the wearer’s body shape and pose. Generated images outperform baselines quantitatively and qualitatively, although results remain limited by the training database’s predominantly plain backgrounds.
- The approach uses separate shape and image generators, including a spatial constraint in the shape generator.The framework decomposes clothing synthesis into shape generation and image generation.
- Generated images contain precise regions consistent with descriptions while keeping the wearer’s body shape and pose unchanged.
- The generated results are limited by a training database dominated by images with plain backgrounds.The model is biased toward that background distribution because no background constraints or postprocessing are assumed.