PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Identity-aware Transfer of Facial Attributes

Mu Li, Wangmeng Zuo, David Zhang

arXiv:1610.05586v2cs.CV

TL;DR

Facial attribute transfer must change a desired attribute without unnecessarily changing source identity, but transfer ground truth is generally unavailable. DIAT addresses this with jointly learned masking and transformation networks plus attribute, identity, and perceptual objectives, and reports favorable results across local, global, and identity-related transfers.

  • Problem

    Facial attribute transfer seeks the desired attribute while preserving source identity, despite generally unavailable ground-truth transfer results.

  • Method

    DIAT combines a mask network and attribute transform network with adversarial attribute, adaptive perceptual, identity, perceptual, and attribute-ratio objectives.

  • Results

    Experiments report satisfying local and global transfer, visually impressive gender transfer with minor identity-feature modification, and favorable face hallucination extension.

  • Takeaways & Limitations

    DIAT provides a unified facial attribute-transfer solution spanning expression, accessory removal, age progression, and gender transfer.

Abstract

from arXiv · show

This paper presents a Deep convolutional network model for Identity-Aware Transfer (DIAT) of facial attributes. Given the source input image and the reference attribute, DIAT aims to generate a facial image that owns the reference attribute as well as keeps the same or similar identity to the input image. In general, our model consists of a mask network and an attribute transform network which work in synergy to generate a photo-realistic facial image with the reference attribute. Considering that the reference attribute may be only related to some parts of the image, the mask network is introduced to avoid the incorrect editing on attribute irrelevant region. Then the estimated mask is adopted to combine the input and transformed image for producing the transfer result. For joint training of transform network and mask network, we incorporate the adversarial attribute loss, identity-aware adaptive perceptual loss, and VGG-FACE based identity loss. Furthermore, a denoising network is presented to serve for perceptual regularization to suppress the artifacts in transfer result, while an attribute ratio regularization is introduced to constrain the size of attribute relevant region. Our DIAT can provide a unified solution for several representative facial attribute transfer tasks, e.g., expression transfer, accessory removal, age progression, and gender transfer, and can be extended for other face enhancement tasks such as face hallucination. The experimental results validate the effectiveness of the proposed method. Even for the identity-related attribute (e.g., gender), our DIAT can obtain visually impressive results by changing the attribute while retaining most identity-aware features.

I. INTRODUCTION

DIAT addresses facial attribute transfer when changing attributes can compromise source identity, using a unified identity-aware model across local, global, and identity-related tasks.

  • I. INTRODUCTION: DIAT unifies facial attribute transfer for expression, accessory removal, age progression, and gender while aiming to preserve source identity.The model is intended for local, global, and identity-related attributes.
  • I. INTRODUCTION: A mask network limits editing to attribute-relevant regions, while an attribute transform network generates the desired facial attribute.The two subnetworks collaborate to produce the final transfer result.
  • I. INTRODUCTION: Because transfer ground truth is difficult or impossible to obtain, DIAT uses adversarial attribute and identity-aware perceptual losses for unsupervised training.Training uses binary attribute labels and attribute-guided image sets from CelebA.
  • I. INTRODUCTION: Identity preservation is further encouraged with VGG-Face identity loss, while perceptual and attribute-ratio regularization suppress artifacts and constrain mask size.A denoising network provides perceptual regularization for the generated image.
  • I. INTRODUCTION: Experiments report favorable transfer with minor or no identity modification, including visually impressive gender transfer, and testing throughput above one hundred images per second.The method is evaluated on CelebA and real images from iStock.

II. RELATED WORK

Related work spans CNN-based image generation, perceptual-loss methods, GANs, and facial attribute transfer, motivating DIAT’s combined architecture and objectives.

  • II. RELATED WORK: CNN image-generation research includes convolutional autoencoders, PixelCNN and PixelRNN, fully convolutional networks, generative CNNs, and DC-IGN.These models address image generation or transformations such as pose and lighting changes.
  • II. RELATED WORK: Perceptual losses transfer content from deep representations and motivate DIAT’s identity-aware adaptive perceptual loss and perceptual regularization.These losses are presented as adaptations of perceptual-loss approaches for facial attribute transfer.
  • II. RELATED WORK: Prior facial attribute-transfer methods include VAE/GAN and other CNN-based approaches, but reported limitations include transfer performance, runtime, and visible artifacts.GANs are commonly used because labeled transfer data are impracticable to collect.
  • II. RELATED WORK: DIAT combines a mask network and an attribute transform network with adversarial, identity, perceptual, and mask-related objectives.The schematic describes this combination for glasses removal.

III. DEEP CNNS FOR IDENTITY-AWARE ATTRIBUTE TRANSFER

DIAT uses jointly learned mask and transformation subnetworks so attribute edits can be localized while preserving irrelevant facial content.

  • III. DEEP CNNS FOR IDENTITY-AWARE ATTRIBUTE TRANSFER: Localizing edits is important because expressions and accessories are local, while age and gender can still leave regions such as the background unchanged.Manual region specification would restrict universality and adaptivity.
  • III. DEEP CNNS FOR IDENTITY-AWARE ATTRIBUTE TRANSFER: The model predicts an attribute-relevant mask and a transformed image, then combines them elementwise with the source image.The two subnetworks take the source image as input and are learned end to end.
  • III. DEEP CNNS FOR IDENTITY-AWARE ATTRIBUTE TRANSFER: The attribute transform network uses a 10-layer U-Net with encoder-decoder structure, skip connections, pooling, and multi-scale information.The architecture contains five convolutional encoding layers and five decoding layers.
  • III. DEEP CNNS FOR IDENTITY-AWARE ATTRIBUTE TRANSFER: The mask network produces a 32 × 32 binary mask, upsamples it by replication, and smooths it with a Gaussian filter to obtain the final mask.The fifth convolution uses sigmoid activation followed by binarization.
  • III. DEEP CNNS FOR IDENTITY-AWARE ATTRIBUTE TRANSFER: A straight-through gradient proxy allows layers before the binarizer to receive updates during training.The proxy addresses the binarizer’s zero gradient almost everywhere.

B. Model objective

DIAT learns identity-aware facial attribute transfer by combining attribute supervision with identity, artifact-suppression, and mask-size constraints. Its objective jointly trains the transfer model and discriminator despite unavailable, non-unique ground-truth transfer images.

  • Because transfer ground truth is unavailable or non-unique, training uses separate source and guided image sets without requiring identity matches.The guided set supplies images with the desired attribute, while the source set contains input images without the reference attribute.
  • The adversarial attribute loss targets the desired attribute, while adaptive perceptual and identity losses preserve similarity to the input identity.Adaptive perceptual loss compares discriminator feature representations rather than enforcing pixel-level equality.
  • Adaptive perceptual loss uses discriminator features that are learned during training, avoiding separate off-the-shelf CNN computation and providing hidden-layer supervision.The authors report improved training efficiency because forward and backward calculations need only use the discriminator.
  • A denoising network supplies perceptual regularization that suppresses artifacts by encouraging the transfer result toward a clean facial image.The network predicts a residual, and minimizing its response on F(x) suppresses noise and artifacts; a threshold preserves small details.
  • Attribute-ratio regularization constrains the mask size according to whether the target attribute is local or global.The method uses smaller expected ratios for local attributes and larger ratios for global attributes.
  • DIAT combines adversarial attribute, adaptive perceptual, perceptual, and attribute-ratio losses to train the transfer model and discriminator.The transfer model is learned with F(x) = M(x)◦T(x)+(1−M(x))◦x.

C. Learning algorithm

DIAT training uses a two-stage procedure that first initializes the transform network and discriminator, then alternates updates of the full transfer model and discriminator.

  • C. Learning algorithm: DIAT first pre-trains the transform network and discriminator, then alternates updates of the complete transfer model F and discriminator D.The transform network is initialized using an auto-encoder reconstruction objective, while the discriminator uses source and guided images as negative and positive samples.
  • C. Learning algorithm: The discriminator is initialized with source images as negatives and guided-set images as positives.
  • C. Learning algorithm: Assigning y_i = 1 to positive images and y_i = −1 to negative images provides an initialization that benefits DIAT convergence and training stability.
  • C. Learning algorithm: During joint training, the full transfer model alternates between minimizing the transfer objective and the denoising regularization objective.The transfer network and discriminator are optimized with RMSProp at a learning rate of 5 × 10^-5.

D. Extension to face hallucination

The paper extends DIAT to 8× face hallucination by treating it as global transfer, removing the mask-related components, and adding pixel-wise reconstruction guidance from high-resolution images.

  • D. Extension to face hallucination: For 8× face hallucination, DIAT removes the mask network and attribute-ratio regularization, using the transform network alone.The bicubic-upsampled low-resolution image is supplied to the transfer network at high-resolution size.
  • D. Extension to face hallucination: The face-hallucination training procedure retains alternating optimization of the transfer network and discriminator.The provided algorithm initializes both networks, repeatedly updates the discriminator, and then updates the transfer network with its transfer and smoothing objectives.
  • D. Extension to face hallucination: Ground-truth high-resolution images guide training through an added pixel-wise reconstruction loss.The loss compares the network output with the ground-truth high-resolution image after bicubic interpolation of the low-resolution input.

IV. EXPERIMENTAL RESULTS

The experiments evaluate DIAT across local and global attribute transfer, quantitative and real-image settings, regularization effects, and face hallucination.

  • IV. EXPERIMENTAL RESULTS: The experimental section covers local and global attribute transfer, quantitative metrics, real-image results, regularization analyses, and face hallucination.

A. Experimental settings

Experiments use CelebA-based training and held-out testing, compare DIAT with several available-code methods, and report parameter settings for different attribute scales.

  • A. Experimental settings: DIAT is trained on aligned CelebA images resized to 128 × 128, using reference-attribute images as the guided set and 10,000 non-reference images as the source set.
  • A. Experimental settings: Global-transfer visual results are organized with input facial images in the left column and transfer results in the right column.
  • A. Experimental settings: After training, 2,000 held-out images and real images from iStock are used for assessment.
  • A. Experimental settings: The comparisons include CNIA, IcGAN, and VAE/GAN, with outputs resized when necessary and manual glasses-region detection available for glasses removal.
  • A. Experimental settings: The attribute-ratio parameter is set to p = 0.16 for small local attributes, p = 0.32 for large local attributes, and p = 0.62 for global attributes.

B. Local attribute transfer

DIAT transfers local facial attributes while preserving attribute-irrelevant identity details, and outperforms competing methods in visual quality and runtime.

  • DIAT preserves more identity-aware and attribute-irrelevant details during mouth and eyeglasses transfer.The mask network supports localized editing while retaining other facial information.
  • DIAT can be trained as separate models for reverse local tasks such as mouth opening and mouth closing when training data are sufficient.
  • DIAT produces more visually pleasing results than CNIA across all three evaluated local attribute-transfer tasks.The comparison covers mouth open, mouth close, and eyeglasses removal.
  • 0.0045 s per image: DIAT is substantially faster than CNIA's approximately 30 s runtime.Both runtimes are reported for processing one image.

C. Global attribute transfer

DIAT handles global attribute transfer while retaining identity-aware features, achieves the desired attributes quantitatively, and maintains image quality under reverse transfer.

  • DIAT remains effective for gender and age transfer, retaining most identity-aware features even when gender changes identity-related characteristics.The reported evaluations cover male-to-female gender transfer and older-to-younger age transfer.
  • > 0.70 attribute-classification accuracy: DIAT achieves the desired attribute across five evaluated transfer tasks.The tasks are mouth open, mouth close, glasses removal, gender transfer, and age transfer.
  • DIAT preserves identity-aware features for local attribute transfer under OpenFace2 verification.Identity verification accuracy is reported for mouth opening, mouth closing, and glasses removal.
  • 33.27 dB average PSNR: reverse mouth transfer indicates effective local attribute transfer.The score is computed after applying mouth-open and mouth-close transfers sequentially to 2,000 test images.

E. Results on other real facial images

Models trained on CelebA generalize to aligned real facial images from iStock for both local mouth-opening and global gender transfer.

  • DIAT generalizes to 15 iStock images per task for mouth opening and gender transfer.Each external image is aligned using five facial landmarks before inference.
  • The iStock results demonstrate transfer beyond the CelebA training-domain images.

G. Results of the learnt mask

DIAT's mask network localizes edits according to attribute scope, while the full model supports perceptual quality and face hallucination beyond attribute transfer.

  • Local-task masks cover the relevant facial part, whereas gender-transfer masks cover most of the face while excluding the background.This demonstrates task-dependent spatial localization by the mask network.
  • DIAT produces hallucinated faces with richer textures and sharper edges than Unet despite lower PSNR and SSIM.The comparison evaluates 8× face hallucination against bicubic interpolation and Unet.
  • The model combines mask and attribute-transform subnetworks with adversarial, adaptive perceptual, identity, and attribute-ratio regularization.These components jointly target attribute transfer while constraining localized editing and perceptual quality.
  • Gender-transfer results modify identity-related attributes while making only minor changes to identity-related features.
  • DIAT extends beyond facial attribute transfer to face hallucination and recovers facial details favorably.
Loading 1610.05586v2…