Source-linked AI summary
Pixel-Level Domain Transfer
Donggeun Yoo, Namil Kim, Sunggyun Park, Anthony S. Paek, In So Kweon
TL;DR
The paper addresses image-conditioned generation across semantically related domains when a source image has many valid pixel-level targets. It introduces an encoder-decoder converter with real/fake and domain discriminators, and demonstrates decent person-to-clothing generation on the LookBook dataset.
Problem
Pixel-level domain transfer must generate realistic target images that remain semantically related to a source image even though the target is not unique.
Method
An encoder-decoder converter produces target images, while real/fake and domain discriminators supervise realism and source-target association.
Results
The model generates decent clothing target images from dressed-person inputs, and qualitative and user-study results outperform the baselines.
Takeaways & Limitations
Pixel-level transfer can connect semantically related image domains using adversarial supervision that accommodates multiple valid targets.
Takeaways & Limitations
MSE-based pixel supervision is constrained by non-unique targets and can force one target, while also producing blurry images for natural-image distributions.
Abstract
from arXiv · showhide
We present an image-conditional image generation model. The model transfers an input domain to a target domain in semantic level, and generates the target image in pixel level. To generate realistic target images, we employ the real/fake-discriminator as in Generative Adversarial Nets, but also introduce a novel domain-discriminator to make the generated image relevant to the input image. We verify our model through a challenging task of generating a piece of clothing from an input image of a dressed person. We present a high quality clothing dataset containing the two domains, and succeed in demonstrating decent results.
1 Introduction
The paper introduces pixel-level domain transfer: an image-conditioned model that transforms a source-domain image into a realistic, semantically related target image. It addresses nondeterministic targets with a domain discriminator and evaluates the approach on person-to-clothing generation using the LookBook dataset.
- Problem and novelty: Image-conditioned generation is distinguished from prior image generation because the input image conditions the target image in another domain.The source and target domains are connected by semantic meaning, such as a dressed person and that person’s clothing.
- Problem and novelty: The task transfers a source image into a target-domain image while preserving semantic meaning at the pixel level.The model takes an image from one domain and directly produces a target image in another domain, rather than adapting only feature-space parameters.
- Problem and novelty: The target is nondeterministic: one source image can correspond to unlimited valid target images, making direct converter training difficult.Possible targets may vary in clothing shape and viewpoint, so a single pixel-level target is not uniquely determined.
- Method: The proposed framework combines an encoder-decoder converter with real/fake and domain discriminators.The domain discriminator evaluates whether a source-target pair is associated, while adversarial training also promotes realistic outputs.
- Evaluation: The study contributes a two-domain clothing dataset and demonstrates decent person-to-top-product image generation.LookBook contains 84k images, including 75k human images associated with 10k top product images.
2 Related Work
The paper relates its approach to generative image models and transfer learning, while distinguishing it from methods that adapt parameters or features rather than generate cross-domain images. Its key methodological distinction is an additional domain discriminator for image-conditioned transfer with nondeterministic targets.
- Image generation: The method belongs to image-generative modeling but replaces a conventional generator with an image-conditioned converter.Unlike class-conditional or attribute-conditional models, the converter uses an input image to produce an image in another domain.
- Image generation: Mean square loss is unsuitable for this transfer setting because target images are nondeterministic.The paper therefore introduces a domain discriminator instead of using the strong pixel-level supervision applied in related conditional generation work.
- Domain adaptation: Transfer-learning and domain-adaptation methods generally transfer model parameters or learn domain-invariant features rather than generate target-domain images.The paper distinguishes its goal as cross-domain image generation, not only recognition or retrieval across clothing-related domains.
3 Review of Generative Adversarial Nets
The paper reviews GANs as adversarially trained generator-discriminator systems for producing realistic images. It uses this framework because realism can be learned with weak real/fake supervision, then extends it to connect distinct image domains.
- GAN framework: GANs consist of a generator and discriminator, with the generator mapping a low-dimensional random vector to a pixel-level image.The discriminator distinguishes real images from images drawn by the generator.
- GAN framework: GAN training alternates discriminator updates on real and generated images with generator updates that aim to make generated images appear real.These procedures are repeated until convergence under a minimax objective.
- GAN framework: The discriminator produces a scalar probability that is high for real inputs and low for fake inputs.Its loss is defined using binary cross entropy.
- Motivation: GANs demonstrate that realistic images can be generated under low-level real-or-fake supervision without direct image-matching losses.This motivates asking whether weak supervision can also learn connections between distinct image domains.
- Motivation: The paper answers that question by transforming an image from one domain into a realistic image in another domain.This cross-domain transformation is presented as the paper’s method beyond the standard GAN setting.
4 Pixel-Level Domain Transfer
The method transfers source images into pixel-level target images using an encoder-decoder converter trained with adversarial real/fake and domain-discriminator objectives. The domain discriminator accommodates diverse valid targets while enforcing source-target relevance.
- Converter Network: The converter encodes a source image into a semantic 64-dimensional code and decodes it into a relevant pixel-level target image.It uses five convolutional encoding layers and five fractional-strided decoding layers in an end-to-end trainable network.
- Pixel-Level Domain Transfer: MSE is unsuitable because each source can have infinitely many valid pixel-level targets with varied shapes and viewpoints.MSE can produce blurry images and forces the converter toward one target, penalizing small geometric misalignments and target diversity.
- Discriminator Networks: The domain discriminator classifies source-target pairs as associated or irrelevant, providing pairwise supervision tolerant of target diversity.It receives a source and candidate target, assigns high probability only to ground-truth associated pairs, and is adversarially optimized with the converter.
- Discriminator Networks: The real/fake discriminator penalizes unrealistic targets, while the domain discriminator penalizes targets irrelevant to their source.Both discriminators act as distinct adversarial loss signals for optimizing the converter.
- Adversarial Training: Adversarial training jointly optimizes the converter, real/fake discriminator, and domain discriminator using their respective loss functions.The training procedure follows adversarial optimization with the real/fake loss and domain-discriminator loss.
R Update the domain discriminator DA:
The training procedure first updates the discriminators using source and target batches, then freezes them while optimizing the converter against both discriminator losses.
- R Update the domain discriminator DA:: The procedure begins by training the real/fake discriminator on the target batch.The target batch is used to reduce the real/fake discriminator loss.
- R Update the domain discriminator DA:: The domain discriminator is trained using both source and target batches.Its loss is reduced before the discriminator parameters are frozen.
- R Update the domain discriminator DA:: After discriminator updates, their parameters are frozen while the converter is optimized to increase both discriminator losses.The converter loss is represented separately in the training procedure, with random target selection defined by sel(·).
5 Evaluation
The evaluation uses the LookBook clothing dataset and compares the proposed converter with baselines through qualitative inspection, user studies, pixel-level metrics, supervision ablations, and generalization tests.
- 5.1 LookBook Dataset: The dataset separates fashion-model images from top-product images photographed against clean backgrounds.The two domains are manually associated so each product image corresponds accurately to images of models fitting that product.
- 5.1 LookBook Dataset: LookBook contains 84,748 images: 9,732 top products associated with 75,016 fashion-model images.Each product has around eight associated fashion-model images on average, with pairs manually connected through the same product.
- 5.3 Qualitative evaluation: Qualitatively, the proposed method produces more source-relevant and realistic products than the baselines, with sharp boundaries and generally preserved stripes and patterns.C+RF produces realistic but source-irrelevant outputs, whereas C+MSE outputs are blurry.
- 5.3 Qualitative evaluation: Across varying backgrounds, viewpoints, poses, and self-occlusions, the converter generates less-varying targets while reflecting clothing attributes and categories.These results indicate that the encoder summarizes source information at a semantic level despite substantial input variation.
- 5.4 Quantitative evaluation by user study: In the user study, the proposed method outperforms the baselines on realism, attribute capture, and category consistency.The study used 100 test sources, three generated targets per source, three criteria, and 25 users.
- 5.5 Quantitative evaluation by pixel-level (dis)similarity: For pixel-level similarity, the proposed method achieves the best SSIM among the baselines, while C+MSE has the lowest RMSE.RMSE and color SSIM were measured between generated images and target images over the test set.
- 5.5 Quantitative evaluation by pixel-level (dis)similarity: The proposed method outperforms retrieval by DD-score, supporting its ability to draw unseen items rather than only copy similar training items.The comparison tests generalization against retrieved real items that could otherwise benefit a memorizing model.
6 Conclusion
The paper presents pixel-level domain transfer with a domain discriminator and reports decent target-image generation. It also contributes a large dataset and illustrates qualitative results for clothing transfer and product-to-human generation.
- 6 Conclusion: The proposed framework performs pixel-level domain transfer using a Generative Adversarial Nets framework.Its domain discriminator trains the semantic relation between source and target domains.
- 6 Conclusion: The framework generates decent target images while retaining semantic relations between the two domains.
- 6 Conclusion: The LookBook dataset is presented as a contribution to domain adaptation research, and the framework is intended for transfer problems from low-level processing to high-level synthesis.
- 6 Conclusion: Figure 5 compares source images, “C+RF” results, “C+MSE” results, and the authors’ results at 64×64×3 resolution.
- 6 Conclusion: Figure 6 presents 100 chosen “product to human” results, with each image shown at 64×64×3 resolution.