Source-linked AI summary
ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes
Taihong Xiao, Jiapeng Hong, Jinwen Ma
TL;DR
Face attribute transfer methods often lack exemplar control, simultaneous multi-attribute manipulation, or high visual quality. ELEGANT exchanges disentangled latent encoding parts between two images and uses residual learning with multi-scale discriminators. On CelebA, the authors report that it overcomes these three common limitations, while noting that FID comparisons are only for reference.
Problem
Existing face attribute transfer methods commonly lack exemplar-based generation, simultaneous manipulation of multiple attributes, or visually satisfying high-quality outputs.
Method
ELEGANT exchanges selected parts of disentangled latent encodings between images and combines residual learning with multi-scale discriminators.
Results
On the CelebA face database, ELEGANT successfully overcomes the three common limitations reported for other face attribute transfer methods.
Takeaways & Limitations
The model supports exemplar-based transfer of multiple face attributes while targeting finer details and fewer artifacts.
Takeaways & Limitations
FID scores are only for reference because exemplar-based generation is more general and difficult to compare fairly, and suitable GAN quality measures remain undetermined.
Abstract
from arXiv · showhide
Recent studies on face attribute transfer have achieved great success. A lot of models are able to transfer face attributes with an input image. However, they suffer from three limitations: (1) incapability of generating image by exemplars; (2) being unable to transfer multiple face attributes simultaneously; (3) low quality of generated images, such as low-resolution or artifacts. To address these limitations, we propose a novel model which receives two images of opposite attributes as inputs. Our model can transfer exactly the same type of attributes from one image to another by exchanging certain part of their encodings. All the attributes are encoded in a disentangled manner in the latent space, which enables us to manipulate several attributes simultaneously. Besides, our model learns the residual images so as to facilitate training on higher resolution images. With the help of multi-scale discriminators for adversarial training, it can even generate high-quality images with finer details and less artifacts. We demonstrate the effectiveness of our model on overcoming the above three limitations by comparing with other methods on the CelebA face database. A pytorch implementation is available at https://github.com/Prinsphield/ELEGANT.
1 Introduction
Face attribute transfer modifies a source face while preserving identity, but existing approaches often cannot use exemplars, combine attributes, or produce visually satisfying images.
- Face attribute transfer modifies a source face while preserving person identity, including when bangs are added or removed.
- Linear feature-space methods use a universal attribute vector, but generate the same bangs style despite substantial style variation.The formulation adds v_bangs to source features before inverse mapping to image space.
- Reference-pair methods increase diversity but require difficult-to-obtain paired images, such as male and female images of the same person.
- Cycle-based domain mappings face intrinsic-dimension issues, while autoencoder variants bypass them but generally manipulate only one attribute at a time.
- Many conditional generation methods transfer multiple attributes from labels but cannot reproduce exactly the attribute style of a reference exemplar.Label conditions provide less information than the diversity of images sharing that label.
- Existing methods also include approaches that cannot generate exemplar-based images, require identity labels, struggle with high-resolution training, or produce low-resolution images and artifacts.
2 Purpose and Intuition
ELEGANT addresses exemplar generation, simultaneous multi-attribute transfer, and image quality by using reference encodings, disentangled latent parts, residual learning, and multi-scale discriminators.
- ELEGANT is proposed to overcome the three limitations of exemplar generation, simultaneous multi-attribute transfer, and low-quality generated images.
- Reference-image latent encodings guide generation toward exactly the same attribute style as the exemplar.
- Disentangled latent parts encode individual attributes, so exchanging selected parts enables simultaneous transfer of several attributes.
- Residual learning modifies only local image regions, helping alleviate training difficulty for face attributes and higher-resolution generation.
- Multi-scale discriminators capture holistic content and local details useful for improving generated-image quality.
3 Our Method
ELEGANT transfers face attributes by exchanging designated parts of latent encodings from unpaired positive and negative image sets, then decoding residual images. Iterative attribute-wise training, U-Net shortcuts, conditional multi-scale discriminators, and reconstruction plus adversarial losses support disentangled manipulation and detailed outputs.
- Disentangled Attribute Learning: Iterative training over opposite-attribute image pairs learns disentangled representations for all predefined attributes.Each iteration trains on one attribute, then repeats across attributes; the positive and negative sets need not be paired.
- Latent Encoding Exchange: ELEGANT encodes positive and negative images into attribute-partitioned latent representations, then exchanges the selected attribute part to create novel encodings.For attribute i, zA and zB are split into n channel-wise parts and the i-th parts are swapped to obtain zC and zD.
- Residual Decoding: The decoder reconstructs inputs and generates novel attributes by producing residual images added to the original inputs.Concatenated latent encodings produce residuals for reconstructed or attribute-transferred images, while U-Net shortcuts provide context for seamless results.
- Multi-Scale Adversarial Training: The encoder and decoder form the generator, while two conditional discriminators operate at different image scales.D1 guides finer details with a smaller receptive field, whereas D2 handles holistic content and helps avoid grimaces.
- Loss Functions: The generator combines reconstruction loss with adversarial loss to preserve input reconstruction while improving generated-image realism.Reconstruction loss measures recovery of original inputs; adversarial loss measures how realistic generated images are.
4 Experiments
Experiments on CelebA evaluate ELEGANT for exemplar-based transfer, simultaneous manipulation of multiple attributes, and generated-image quality. Results show exemplar-specific styles, finer details than DNA-GAN, competitive FID scores, and design choices addressing training and preservation challenges.
- Dataset and setup: Experiments use CelebA face images, aligned and cropped to 256 × 256, with all evaluations performed at this scale.CelebA contains 202599 images from 10177 identities, annotated with 40 attributes and 5 landmark locations.
- Face image generation by exemplars: ELEGANT generates exemplar-specific attribute styles, whereas UNIT, CycleGAN, and StarGAN generate a common attribute style across input images.In Fig. 8, ELEGANT produces different face images with exactly the same attribute style as the reference images.
- Face image generation by exemplars: ELEGANT focuses on the manipulated attribute without requiring input labels at testing and learns subtle differences between bangs styles.These properties are attributed to exchanging latent encodings and are illustrated by interpolation results in Fig. 7.
- Dealing with Multiple Attributes Simultaneously: Compared with DNA-GAN, ELEGANT produces visually finer details when manipulating multiple face attributes.The reported improvement is mainly attributed to residual learning and multi-scale discriminators.
- Dealing with Multiple Attributes Simultaneously: ELEGANT avoids DNA-GAN's additional face-identity and background encoding part and annihilating operation while learning residual images that automatically preserve identity and background.The paper presents this as making the framework more elegant and easier to understand.
- High-quality Generated Images: ELEGANT achieves competitive FID results against other methods across five attributes, but the authors treat FID only as a reference measure.The authors cite the difficulty of exemplar-based generation and the lack of a determined reasonable qualitative GAN measure as reasons for this qualification.
5 Conclusions
ELEGANT addresses multiple face-attribute transfer limitations through disentangled latent-code exchange, residual learning, U-Net design, and multi-scale discriminators. Experiments on CelebA demonstrate that it overcomes three common limitations of other methods.
- ELEGANT transfers multiple face attributes by encoding different attributes into disentangled latent parts and exchanging selected parts.This latent-space design enables images with novel attributes to be generated through targeted code exchange.
- Residual learning leverages the fact that attribute transfer modifies only local image regions, facilitating training on high-resolution images.
- A U-Net structure and multi-scale discriminators improve the quality of generated face images.
- Experiments on the CelebA face database show that ELEGANT overcomes three common limitations found in most other methods.