Source-linked AI summary
The Conditional Analogy GAN: Swapping Fashion Articles on People Images
Nikolay Jetchev, Urs Bergmann
TL;DR
The paper addresses image analogy for fashion images: learning how a standalone clothing article appears on a human and applying that relation to new articles. It introduces CAGAN, an adversarial, end-to-end trainable model that implicitly learns segmentation, producing plausible masks and often convincing clothing swaps while remaining limited on complex textures and unrestricted photorealistic generation.
Problem
Fashion image analogy requires rendering a new clothing article on a human despite occlusion, illumination, 3D rotation, and deformation, while fully new photorealistic humans are difficult to generate.
Method
CAGAN uses adversarial training with convolutional networks to learn the relation between paired human-and-article images and implicitly localize and segment clothing end to end.
Results
The model generally preserves the human's appearance while swapping the relevant article, produces sometimes high-quality implicit masks, and can combine humans and articles beyond the training pairs.
Takeaways & Limitations
Using available fashion images allows precise article specification without expensive supervised segmentation data and supports visually appealing clothing swaps across humans and articles.
Takeaways & Limitations
The method transfers colors and rough clothing structures but is inaccurate with complex fine textures, and its generalization assumes a photoshoot style consistent with the training data.
Abstract
from arXiv · showhide
We present a novel method to solve image analogy problems : it allows to learn the relation between paired images present in training data, and then generalize and generate images that correspond to the relation, but were never seen in the training set. Therefore, we call the method Conditional Analogy Generative Adversarial Network (CAGAN), as it is based on adversarial training and employs deep convolutional neural networks. An especially interesting application of that technique is automatic swapping of clothing on fashion model photos. Our work has the following contributions. First, the definition of the end-to-end trainable CAGAN architecture, which implicitly learns segmentation masks without expensive supervised labeling data. Second, experimental results show plausible segmentation masks and often convincing swapped images, given the target article. Finally, we discuss the next steps for that technique: neural network architecture improvements and more advanced applications.
1. Introduction
Fashion businesses face costly, slow production of model photographs at large scale, while virtual try-on remains technically unresolved. The paper motivates exploiting existing 2D fashion-image collections with generative image-to-image translation methods.
- Producing model photographs for thousands of new fashion articles is slow and expensive, creating a scaling bottleneck for fashion businesses.Reusing available images of human models and products is presented as a potential alternative.
- Virtual try-on could let customers preview different products on their own images, but the problem has not been solved convincingly.
- Modeling humans and articles as 3D objects is expensive because high-quality 3D scans create engineering and computational challenges.The paper instead points to rich 2D fashion-image collections as data for generative deep learning.
- Image-to-image translation modifies an input image into an output image, while conventional CNNs may produce blurry results under unsuitable losses.GANs adapt their learned losses to the data and are therefore suited to generating sharp images resembling training examples.
- Conditional GANs generate images from conditioning information, whereas unpaired translation methods address cases without explicitly paired examples.The paper situates its fashion application within this broader image-to-image translation literature.
2. The CAGAN model
CAGAN learns from paired human-and-article images to swap clothing on a given person without ground-truth images for every human–article combination. Its generator and discriminator use adversarial, identity, and cycle-based constraints to produce plausible, localized edits.
- Image analogy formulation: CAGAN learns the relation between a human image xi wearing article yi and a standalone image yi, then applies that relation to a new article.The mapping accounts for occlusion, illumination, 3D rotation, and deformation, while using the whole dataset to infer the most plausible relation.
- Image analogy formulation: The model swaps article yi with yj on human image xi even though the desired swapped image is absent from the training data.The target is inferred indirectly from paired examples, with the architecture encouraging changes only to clothing-related regions.
- Adversarial training: The adversarial objective asks whether an image looks like a human from the training distribution and whether the supplied article is correctly worn.Negative examples pair a human xi with a different article yj, forcing the discriminator to assess consistency between the human and article rather than realism alone.
- Generator: The generator produces a blending mask and a color image, combining them with the original human image to form the final swapped result.The mask values are constrained to [0, 1] with a sigmoid, enabling the network to preserve unchanged regions while painting the replacement article.
- Regularization: An identity regularizer limits changes to the original human image, while a cycle loss penalizes deviations after swapping to yj and back to yi.Together, these terms encourage the generator to modify the relevant clothing region while leaving other parts of the image unchanged.
- Network architecture: The architecture uses an encoder-decoder generator with skip connections and a PatchGAN discriminator that outputs spatially local classification values.The discriminator marginalizes over spatial positions, providing local consistency judgments rather than a single image-level output.
3. Experiments
Experiments use paired human and upper-body garment images at two resolutions to evaluate CAGAN’s clothing swaps and implicitly learned masks. The results generally preserve the person while replacing the relevant garment, although cluttered backgrounds and complex textures remain challenging.
- 3.1. Setup: CAGAN uses an encoder-decoder generator with skip connections and preserved conditioning channels to improve convergence and image quality.The discriminator used four layers with a 63x63-pixel receptive field for local consistency.
- 3.1. Setup: Training used 15,000 frontal human images paired with upper-body garments, evaluated at 128x96 and 256x192 pixels.The data came from Zalando and included pullovers and hoodies in RGB color space.
- 3.2. Generation results: images of human models with swapped clothes: Generated images generally preserve the human’s appearance while swapping only the relevant article, with in-place color changes easier than texture or geometric deformations.Figure 4 reports results on three randomly chosen models at 128x96 resolution.
- 3.2. Generation results: images of human models with swapped clothes: The implicitly learned alpha masks can be of good quality, supporting accurate repainting at the higher 256x192 resolution.The segmentation is learned through the CAGAN objective rather than supervised mask labels.
- 3.2. Generation results: images of human models with swapped clothes: Background clutter makes segmentation more complex than in the consistent-background Zalando images.The paper identifies street-view imagery as a likely more difficult setting.
4. Discussion
The discussion positions CAGAN as a way to specify exact fashion articles using readily available fashion images rather than requiring semantic segmentation data. The authors identify broader garment coverage and architectural improvements as next steps, especially for complex textures.
- 4.1. Related methods: Unlike a related method that generates unspecified garments within segmented regions, CAGAN can precisely specify the article to paint.It uses available fashion images instead of requiring supervised semantic segmentation data.
- 4.2. Future work: The authors demonstrated a GAN that swaps clothes on humans and supports additional image-manipulation possibilities for fashion.They plan to improve both the model architecture and its applications.
- 4.2. Future work: Future work includes swapping all fashion categories, including upper- and lower-body garments, shoes, and accessories, toward a fuller virtual try-on experience.The authors also plan to study whether better human-image segmentation improves results.
- 4.2. Future work: The current model is inaccurate with complex textile textures, motivating tests of texture descriptors and improved conditioning embeddings.Other planned changes include evaluating Lab color space and helping early layers localize the original garment.