Source-linked AI summary
RelGAN: Multi-Domain Image-to-Image Translation via Relative Attributes
Po-Wei Wu, Yu-Jing Lin, Che-Han Chang, Edward Y. Chang, Shih-Wei Liao
TL;DR
Existing methods struggle with continuous attribute interpolation and require complete target-attribute specifications, including unchanged attributes. RelGAN conditions a generator on relative attributes and uses three discriminators to support realistic, attribute-matched translation and interpolation. The paper reports superior visual quality and interpolation performance over state-of-the-art methods on facial attribute tasks.
Problem
Existing methods assume binary attributes, limiting fine-grained interpolation, and require users to specify unchanged attributes whose values may be unknown.
Method
RelGAN maps an input image and relative attribute vector to an output, using one generator with realness, matching-aware, and interpolation discriminators.
Results
RelGAN achieves superior performance over state-of-the-art methods in both visual quality and interpolation on facial attribute transfer and interpolation tasks.
Takeaways & Limitations
Relative attributes let RelGAN modify selected attributes continuously while preserving other unchanged attributes.
Takeaways & Limitations
The training data are unpaired, with input images and comparison images having different identities and attributes.
Abstract
from arXiv · showhide
Multi-domain image-to-image translation has gained increasing attention recently. Previous methods take an image and some target attributes as inputs and generate an output image with the desired attributes. However, such methods have two limitations. First, these methods assume binary-valued attributes and thus cannot yield satisfactory results for fine-grained control. Second, these methods require specifying the entire set of target attributes, even if most of the attributes would not be changed. To address these limitations, we propose RelGAN, a new method for multi-domain image-to-image translation. The key idea is to use relative attributes, which describes the desired change on selected attributes. Our method is capable of modifying images by changing particular attributes of interest in a continuous manner while preserving the other attributes. Experimental results demonstrate both the quantitative and qualitative effectiveness of our method on the tasks of facial attribute transfer and interpolation.
1. Introduction
RelGAN addresses two limitations of target-attribute-based translation: binary attributes hinder interpolation, and complete target specifications burden users with unchanged attributes. It instead uses relative attributes to control selected changes continuously while preserving unchanged attributes.
- Motivation: Existing multi-domain methods assume binary attributes, producing unsatisfactory interpolation when real-valued attributes are supplied to models trained on binary labels.This limits fine-grained control over attribute strength, such as hair-color mixtures or smile degree.
- Illustration: Figure 1 illustrates that RelGAN changes hair color while preserving smile and supports realistic interpolation through continuous relative-attribute adjustment.The comparison contrasts RelGAN with StarGAN, which strengthens the smile while changing hair color.
- Motivation: Target-attribute methods require users to specify unchanged attributes whose underlying values may be unknown.Users must identify the values of all unchanged attributes in addition to setting the attributes they want to edit.
- Approach: RelGAN represents the desired change as relative attributes v = ˆa − a, with nonzero values for edited attributes and zero values for unchanged ones.The generator maps an input image and relative attributes to an output image.
- Approach: RelGAN combines one generator with realness, matching-aware, and interpolation discriminators that guide realism, attribute accuracy, and interpolation quality.The matching-aware discriminator checks whether input-output pairs match the relative attributes, while the interpolation discriminator predicts interpolation degree.
2. Related Work
Related work frames image translation as conditional generation and surveys representative facial attribute-transfer methods. RelGAN differs by using relative attributes and a matching-aware discriminator rather than target attributes and auxiliary classifiers.
- Conditional generation: Conditional GANs extend GANs by conditioning generators and discriminators on additional information such as text or images.Image-to-image translation can therefore be formulated as a conditional GAN problem.
- Image translation: Unpaired image-to-image translation methods address settings where paired training data are unavailable.CycleGAN is described as learning two generative models regularized by cycle consistency.
- Facial attribute transfer: IcGAN, StarGAN, and AttGAN formulate facial attribute transfer as unpaired multi-domain translation using target attributes.StarGAN uses a single generator, whereas AttGAN uses an encoder-decoder architecture.
- RelGAN: RelGAN uses relative attributes and a matching-aware discriminator, unlike StarGAN and AttGAN, which use target attributes and auxiliary classifiers.The comparison identifies formulation and discriminator design as key differences.
3. Method
RelGAN maps an input image and relative attribute changes to a translated image, using adversarial and reconstruction objectives to enforce realism, attribute matching, preservation, and smooth interpolation.
- Relative Attributes: RelGAN maps an image x and relative attribute vector v to an output y, where v specifies desired attribute changes and zero entries preserve attributes.Relative attributes are defined from the difference between target and original attribute vectors; nonzero values mark attributes of interest.
- Adversarial Loss: The generator G is trained with unconditional adversarial loss so generated images resemble real images.The unconditional discriminator DReal distinguishes real images from generated images.
- Conditional Adversarial Loss: The matching-aware discriminator DMatch checks whether the input, relative attributes, and output form a matched triplet.It distinguishes real matched triplets from generated and deliberately mismatched triplets, using unpaired training images.
- Reconstruction Loss: Cycle- and self-reconstruction losses regularize G so edited images preserve non-target content and unchanged inputs are reconstructed.For a zero relative attribute vector, G(x, 0) should remain close to x; both reconstruction losses use the L1 norm.
- Interpolation Loss: RelGAN performs interpolation by applying G(x, αv), while DInterp encourages interpolated images to appear realistic and predicts their interpolation degree.The interpolation degree is defined as ˆα = min(α, 1 − α), with 0 indicating no interpolation and 0.5 maximum interpolation.
- Overall Objective: The full objectives combine the three discriminator losses and generator losses, with hyper-parameters controlling their relative importance.Orthogonal regularization is added to stabilize training.
4. Experiments
RelGAN is evaluated on facial attribute transfer, reconstruction, interpolation, and user preference using CelebA-HQ and related datasets. Across these experiments, it generally improves visual quality, attribute preservation, reconstruction, interpolation smoothness, and perceived image quality over StarGAN and AttGAN.
- Experimental settings: RelGAN is compared with StarGAN and AttGAN on CelebA, CelebA-HQ, and cross-dataset FFHQ evaluation settings.The models are trained using the authors’ released code and default hyper-parameters for StarGAN and AttGAN.
- Facial attribute transfer: RelGAN consistently achieves better FID scores than StarGAN and AttGAN across the three training sets and on FFHQ generalization.FID measures visual quality, with lower scores indicating better performance.
- Facial attribute transfer: RelGAN obtains the highest average classification accuracy and ranks first for 3 of the 7 evaluated attributes.Accuracy on real CelebA-HQ images serves as an upper bound for generated-image performance.
- Facial attribute transfer: RelGAN preserves unchanged attributes more effectively, while StarGAN produces artifacts and AttGAN produces blurrier, less detailed results.For hair-color editing, RelGAN preserves the smile attribute whereas the other methods cause the mouth to open.
- Facial image reconstruction: RelGAN without LCycle already outperforms StarGAN and AttGAN on L1, L2, and SSIM reconstruction metrics, with further improvement from the complete loss.The ablation study also reports that removing LCycle+LSelf harms identity preservation, while removing LMatch causes input reconstruction.
- Facial attribute interpolation: The interpolation discriminator reduces the standard deviation of adjacent-image SSIM scores, producing more realistic and smoothly varying interpolations.With m = 10, a smaller standard deviation indicates better interpolation quality; visualizations show that LInterp distributes appearance changes more evenly.
- User study: In a user study with 302 participants and 10 tasks, RelGAN receives the majority of votes in every task except smile.Users compared outputs from StarGAN, AttGAN, and RelGAN across attribute transfer, reconstruction, and interpolation tasks.
5. Conclusion
RelGAN is a relative-attribute-based model that edits selected attributes while preserving unchanged ones. It achieves superior visual quality and interpolation performance over state-of-the-art methods.
- RelGAN uses relative attributes to modify selected image attributes while preserving unchanged ones.
- The model achieves superior performance over state-of-the-art methods in visual quality and interpolation.
- Future work includes advanced adversarial learning methods and mask mechanisms.
Appendix A. Network Architecture
RelGAN uses a generator with three weight-sharing discriminators for realism, attribute matching, and interpolation. Its generator and discriminator contain approximately 8M and 53M trainable parameters, respectively.
- RelGAN's generator is guided by DReal, DMatch, and DInterp, whose feature layers share weights.
- The generator architecture uses switchable normalization in all layers except the final output layer and has about 8M trainable parameters.
- The discriminator architecture uses Leaky ReLU with a negative slope of 0.01 and has about 53M trainable parameters.
Appendix B. Additional Results
Additional experiments compare RelGAN with StarGAN and AttGAN on facial attribute transfer and show single-attribute transfer and interpolation results. The comparisons emphasize preserving attributes while varying selected attributes continuously.
- Hair-color comparisons use residual heat maps to visualize differences between input and output images.
- RelGAN preserves the smile attribute during hair-color transfer, whereas StarGAN and AttGAN make the woman look unhappy.
- Single-attribute transfer results cover hair color, gender, mustache, skin tone, smiling, bangs, glasses, and age.
- RelGAN generates different levels of age, gender, mustache, and smile transfer by varying α.