PaperlayerView exact revision on arXiv

Source-linked AI summary

AttGAN: Facial Attribute Editing by Only Changing What You Want

Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, Xilin Chen

arXiv:1711.10678v3cs.CVstat.ML

TL;DR

Facial attribute editing must change selected facial properties while preserving other details, but attribute-independent latent constraints can restrict representation capacity and lose information. AttGAN instead constrains generated images with attribute classification, reconstruction, and adversarial learning, and reports realistic editing with preserved facial details, alongside intensity control and style manipulation.

  • Problem

    Existing attribute-independent constraints on latent representations can restrict representation ability and cause information loss, while facial editing requires changing selected attributes without losing other details.

  • Method

    AttGAN combines generated-image attribute classification, reconstruction learning, and adversarial learning in a unified encoder-decoder framework.

  • Results

    AttGAN outperforms state-of-the-art methods on realistic facial attribute editing with facial details well preserved.

  • Takeaways & Limitations

    The method supports multiple facial attribute editing with one model and is directly applicable to attribute intensity control and attribute style manipulation.

Abstract

from arXiv · show

Facial attribute editing aims to manipulate single or multiple attributes of a face image, i.e., to generate a new face with desired attributes while preserving other details. Recently, generative adversarial net (GAN) and encoder-decoder architecture are usually incorporated to handle this task with promising results. Based on the encoder-decoder architecture, facial attribute editing is achieved by decoding the latent representation of the given face conditioned on the desired attributes. Some existing methods attempt to establish an attribute-independent latent representation for further attribute editing. However, such attribute-independent constraint on the latent representation is excessive because it restricts the capacity of the latent representation and may result in information loss, leading to over-smooth and distorted generation. Instead of imposing constraints on the latent representation, in this work we apply an attribute classification constraint to the generated image to just guarantee the correct change of desired attributes, i.e., to "change what you want". Meanwhile, the reconstruction learning is introduced to preserve attribute-excluding details, in other words, to "only change what you want". Besides, the adversarial learning is employed for visually realistic editing. These three components cooperate with each other forming an effective framework for high quality facial attribute editing, referred as AttGAN. Furthermore, our method is also directly applicable for attribute intensity control and can be naturally extended for attribute style manipulation. Experiments on CelebA dataset show that our method outperforms the state-of-the-arts on realistic attribute editing with facial details well preserved.

I. INTRODUCTION

AttGAN addresses facial attribute editing by using one encoder-decoder framework that changes desired attributes while preserving other facial details and realism. It replaces restrictive latent attribute-independence with generated-image classification, reconstruction, and adversarial objectives.

  • Motivation: Facial attribute editing seeks to manipulate selected attributes while preserving the remaining details of a face image.Supervised learning is generally inapplicable because labeled images of the same person with varying attributes are difficult to collect.
  • Motivation: A single encoder-decoder model can handle multiple attribute-editing subtasks instead of requiring separate models for different attributes.The authors focus on encoder-decoder architectures to support high-quality editing with one model.
  • Design rationale: Attribute-independent latent constraints can restrict representation ability and cause information loss that harms attribute editing.The paper argues that facial attributes and latent representations are closely dependent, making simple invariance excessive.
  • Method: AttGAN applies attribute classification to generated images, reconstruction learning to preserve attribute-excluding details, and adversarial learning for visual realism.Attribute-excluding details include identity, illumination, and background.
  • Results: AttGAN reports better perceptual quality than state-of-the-art methods and supports multiple-attribute editing, attribute intensity control, and attribute style manipulation.The method is presented as producing visually pleasing results with fine facial details.

II. RELATED WORK

Related work spans optimization-based and learning-based facial attribute editing, including methods that use separate models, latent manipulation, attribute-independent representations, or cycle consistency. AttGAN builds on encoder-decoder approaches while targeting their latent-representation limitations.

  • Facial Attribute Editing: Facial attribute editing methods include optimization-based approaches that minimize attribute-related losses with respect to the input face.CNAI defines an attribute loss from CNN feature differences, while DFI is another optimization-based method.
  • Facial Attribute Editing: Several learning-based methods train different models for different attributes, limiting convenient handling of multiple editing subtasks.Other methods transfer, remove, swap, or separately add and remove specific attributes.
  • Facial Attribute Editing: Multiple-attribute methods include VAE/GAN, IcGAN, StarGAN, and related approaches using latent modification, attribute-independent codes, or classification and cycle-consistency losses.These methods provide the main encoder-decoder and conditional-generation context for AttGAN.
  • Facial Attribute Editing: AttGAN is a learning-based method motivated mainly by VAE/GAN, IcGAN, and Fader Networks to address how latent representations relate to attributes.Its contribution is framed as a novel solution to disadvantages in those encoder-decoder-based methods.
  • Generative Adversarial Networks: GANs learn generators through adversarial competition between a generator and discriminator, but training can be unstable and suffer from mode collapse.The paper adopts WGAN-GP for adversarial learning, using gradient penalties rather than weight clipping for the Lipschitz constraint.
  • Generative Adversarial Networks: AttGAN maps edited face images back to target attributes through an attribute classification constraint, drawing on conditional-generation methods such as AC-GAN and InfoGAN.The paper distinguishes this use from those methods by applying the mapping to edited face images.

III. ATTRIBUTE GAN (ATTGAN)

AttGAN edits binary facial attributes with an encoder-decoder, classifier, and discriminator. The encoder produces a latent representation from an input face, and the decoder generates an edited image conditioned on target attributes.

  • Architecture: AttGAN comprises an encoder, decoder, attribute classifier, and discriminator for editing binary facial attributes.The encoder and decoder form the basic editing subnetworks, while the classifier and discriminator support training objectives.
  • Testing Formulation: Given an input face with binary attributes, the encoder maps the face to a latent representation.The input attributes are represented as a vector of n binary values.
  • Testing Formulation: The decoder generates an edited face by decoding the latent representation conditioned on a target attribute vector.The output is expected to possess the target attributes.
  • Testing Formulation: The complete editing process is formulated through the encoder-decoder composition that maps the input face and desired attributes to the edited image.The paper notes that the ground-truth edited image is unavailable, making this learning problem unsupervised.

B. Training Formulation

AttGAN trains an encoder-decoder to produce realistic images with desired attributes while preserving attribute-excluding details. Its objective combines attribute classification, reconstruction, and adversarial losses.

  • Adversarial Loss: Adversarial learning encourages the generated image x̂b to be visually realistic.The adversarial losses are optimized via WGAN-GP.
  • Reconstruction Loss: Reconstruction learning makes the generated image conditioned on original attributes approximate the input image, preserving attribute-excluding details.The reconstruction objective uses an ℓ1 loss rather than ℓ2 to suppress blurriness.
  • Attribute Classification Constraint: Attribute classification constrains the generated image to possess the desired attributes b.The classifier prediction C(x̂b) is trained toward b using binary cross-entropy across attributes.
  • Overall Objective: The overall encoder-decoder objective combines reconstruction, attribute classification, and adversarial losses.The discriminator and attribute classifier have a corresponding objective, with λ1, λ2, and λ3 balancing the losses.

C. Why are attribute-excluding details preserved?

AttGAN’s detail preservation is attributed to jointly learning face reconstruction and attribute editing. Because both tasks share the encoder-decoder and are learned simultaneously, reconstruction ability is not flushed by editing learning.

  • Shared Learning: AttGAN transfers detail-preservation ability from face reconstruction to attribute editing through a shared encoder-decoder network.The paper describes the two tasks as multi-task learning with shared parameters.
  • Shared Learning: Simultaneous learning dynamically preserves facial detail reconstruction while attribute editing is learned.The paper states that editing learning does not flush the reconstruction ability.

D. Extension for Attribute Style Manipulation

AttGAN is extended with style controllers and a style predictor so attribute styles can be manipulated without labeled style data. Mutual-information maximization correlates each controller with its corresponding output attribute style.

  • Style Controllers: Style controllers θ are introduced and bound to corresponding attributes to enable manipulation of attribute styles such as different eyeglass types.The extension addresses settings where labeled attribute-style data is unavailable.
  • Extended Framework: The extended framework adds style controllers θ and a style predictor Q to the original AttGAN.The generated image is conditioned on the encoded face, target attributes, and style controls.
  • Style Control Objective: Mutual-information maximization correlates style controllers with output images, allowing unsupervised manipulation of attributes in different styles.Each controller is sampled from a categorical distribution with a predefined number of styles for its attribute.
  • Architectural Configurations: The architecture tables describe AttGAN configurations for 128+2 and 642 image settings.The implementation notation defines convolutional and transposed convolutional layers by dimension, kernel size, and stride.

IV. IMPLEMENTATION DETAILS

AttGAN uses a convolutional encoder-decoder with shared discriminator-classifier convolutional layers and U-Net-like skip connections. Training uses Adam with specified batch size, learning rate, and loss weights.

  • Implementation: The AttGAN implementation is provided in TensorFlow, with source code publicly available.The paper directs readers to the project website for additional implementation details.
  • Network Architecture: The encoder and decoder use convolutional and transposed convolutional stacks connected by U-Net-like symmetric skip connections.The skip connections are intended to support high-quality image translation.
  • Network Architecture: The discriminator is convolutional followed by fully connected layers, while the classifier shares all convolutional layers with it.This shared architecture couples adversarial discrimination and attribute classification.
  • Training Details: Training uses Adam with batch size 32, learning rate 0.0002, β1 = 0.5, and β2 = 0.999.The loss coefficients are λ1 = 100, λ2 = 10, and λ3 = 1.

V. EXPERIMENTS

AttGAN is evaluated on CelebA using thirteen visually impactful binary facial attributes, with training and validation data used for learning and the test set reserved for evaluation. Comparisons include single-model multi-attribute methods and methods requiring one model per attribute.

  • Dataset: CelebA contains 200,000 images annotated with 40 binary attributes, and thirteen attributes with strong visual impact are selected for evaluation.The selected attributes include hair, facial hair, gender, mouth openness, skin tone, and age.
  • Dataset: The model trains on CelebA’s training and validation sets together and evaluates on the testing set.
  • Methods: AttGAN, VAE/GAN, IcGAN, and StarGAN are trained to edit thirteen attributes with a single model under the same experimental settings.
  • Methods: Additional comparisons include Fader Networks, Shen et al., and CycleGAN, while Shen et al. and CycleGAN handle only one attribute with one model.

A. Visual Analysis

Visual analyses show AttGAN editing specified attributes while preserving unrelated facial details and producing realistic results. It also supports multiple-attribute editing, continuous intensity control, and flexible attribute-style manipulation, although large appearance changes can fail.

  • Single Facial Attribute Editing: AttGAN accurately edits local and global attributes while preserving face identity, illumination, and background better than VAE/GAN and IcGAN.VAE/GAN can change correlated attributes, whereas IcGAN can distort identity and other attribute-excluding details.
  • Single Facial Attribute Editing: AttGAN produces more natural and realistic results than StarGAN, whose accurately edited outputs contain artifacts.
  • Single Facial Attribute Editing: AttGAN naturally adds mustaches while retaining female characteristics, unlike Shen et al. and CycleGAN, which can make female faces appear male.
  • Multiple Facial Attribute Editing: AttGAN remains effective for simultaneously editing two or three attributes, whereas VAE/GAN introduces undesired changes and IcGAN increasingly distorts or smooths facial details.The comparison covers multiple-attribute editing among AttGAN, VAE/GAN, and IcGAN.
  • Attribute Intensity Control: Continuous inputs in [0, 1] produce smooth and natural gradual changes despite AttGAN being trained with binary attribute values.
  • Attribute Style Manipulation: The extended AttGAN exposes flexible attribute styles, such as left-, right-, or middle-parted bangs, rather than a single fixed style.
  • High Quality Results and Failures: Failures often occur when editing requires large appearance modifications, such as changing a face with plentiful hair to bald.

B. Quantitative Analysis

Quantitative comparisons evaluate attribute editing accuracy and preservation error using an independent CelebA-trained classifier. AttGAN generally matches strong editing accuracy while preserving other attributes better than several baselines, though StarGAN has slightly lower preservation error and more artifacts visually.

  • Evaluation Metrics: The independent attribute classifier used for evaluation achieves 90.89% average accuracy per attribute on the CelebA testing set.A generated image is counted as correct when the classifier predicts the desired attribute.
  • Single-Model Comparisons: AttGAN and StarGAN achieve much higher editing accuracy than VAE/GAN and IcGAN, especially for No Beard, Pale Skin, and Age.
  • Single-Model Comparisons: AttGAN and StarGAN have much lower preservation errors than VAE/GAN and IcGAN, while AttGAN’s preservation error is slightly higher than StarGAN’s.
  • Single-Model Comparisons: Despite comparable accuracy, AttGAN’s generated images are reported as more natural and realistic than StarGAN’s.
  • Per-Attribute Comparisons: Fader Networks, Shen et al., and CycleGAN achieve attribute accuracy comparable to AttGAN but have higher preservation errors for other attributes.

C. Ablation Study: Effect of Each Component

The ablation study shows that AttGAN’s classification, adversarial, and reconstruction components work together: classification enables correct edits, adversarial learning supports that constraint, and reconstruction preserves other facial details. The attribute-independent latent constraint degrades outputs, while style translation remains difficult for large variations.

  • Removing attribute classification produces reconstruction images because no signal forces the network to generate the desired attributes.
  • Without adversarial loss, classification can be fooled by noisy adversarial examples, so the classification constraint does not work reliably alone.
  • Removing reconstruction loss preserves correct attributes but changes face identities substantially and introduces many artifacts.
  • Combining attribute-independent constraints with AttGAN losses correctly edits attributes but causes artifacts and changes attribute-excluding details.
  • Style translation works acceptably on seasons but yields artifacts and blurriness for paintings with larger texture and color variation.
  • AttGAN combines attribute classification, reconstruction, and adversarial learning into a framework for high-quality facial attribute editing.
Loading 1711.10678v3…