Source-linked AI summary
Learning Residual Images for Face Attribute Manipulation
Wei Shen, Rujie Liu
TL;DR
Face attribute manipulation seeks to modify a face toward a desired attribute while preserving irrelevant and identity-related details. The paper learns attribute-focused residual images using two inverse transformation networks, a discriminator, and dual learning. Experiments show successful manipulation with most attribute-irrelevant details retained.
Problem
Face attribute manipulation asks how to modify a face image toward a desired attribute value while preserving identity-related and other irrelevant details.
Method
The method learns residual images with two CNNs for inverse manipulations, a shared discriminative network, and dual learning.
Results
The method successfully manipulates face images while retaining most details in attribute-irrelevant areas.
Takeaways & Limitations
Residual and dual learning allow transformation networks to focus on attribute-specific areas and learn from each other.
Takeaways & Limitations
The evaluation may include identity overlaps between training and test sets, although differing poses, illumination, and backgrounds are reported to limit information leakage.
Abstract
from arXiv · showhide
Face attributes are interesting due to their detailed description of human faces. Unlike prior researches working on attribute prediction, we address an inverse and more challenging problem called face attribute manipulation which aims at modifying a face image according to a given attribute value. Instead of manipulating the whole image, we propose to learn the corresponding residual image defined as the difference between images before and after the manipulation. In this way, the manipulation can be operated efficiently with modest pixel modification. The framework of our approach is based on the Generative Adversarial Network. It consists of two image transformation networks and a discriminative network. The transformation networks are responsible for the attribute manipulation and its dual operation and the discriminative network is used to distinguish the generated images from real images. We also apply dual learning to allow transformation networks to learn from each other. Experiments show that residual images can be effectively learned and used for attribute manipulations. The generated images remain most of the details in attribute-irrelevant areas.
1. Introduction
Face attribute manipulation asks whether a face image can be changed toward a desired attribute value while preserving identity-related and other irrelevant details. The paper learns residual images with dual transformations to make this targeted manipulation efficient.
- Motivation: Face attribute manipulation reverses the usual prediction task by modifying an image toward a desired attribute value.Examples include glasses, mouth-open, and no-beard manipulations.
- Motivation: Directly using GANs or VAEs is difficult because reconstruction or generation can harm identity-related details.GANs generate realistic images but do not encode inputs, while encoder-decoder pipelines can degrade reconstruction quality.
- Motivation: A feed-forward CNN must alter attribute-specific regions while keeping irrelevant areas unchanged, making targeted transformation challenging.The paper addresses this constraint rather than transforming the whole face indiscriminately.
- Method: Residual image learning models manipulation as the difference between the original and desired images, concentrating on essential attribute-specific information.This avoids redundant attribute-irrelevant content and enables modest pixel modification.
- Method: Two CNNs learn inverse manipulations simultaneously through dual learning, such as removing glasses and adding glasses.The shared learning process is reported to help generate high-quality images.
- Results: Experiments successfully manipulate faces while preserving most details in attribute-irrelevant areas.For glasses removal, landmark detection accuracy gain is used as a quantitative effectiveness measure.
2. Related Work
Prior work developed generative models for image synthesis and representation learning, while this paper addresses facial attribute manipulation with a distinct residual-learning and joint inverse-manipulation design.
- Generative Models: GANs, VAEs, and related models have been used for image generation, representation learning, disentanglement, and editable embeddings.Examples include DC-GANs, information-theoretic GANs, and VAE-GAN combinations.
- Facial Attribute Manipulation: Compared with identity-aware facial attribute transfer, this method uses residual images, jointly models inverse manipulations with one discriminator, and requires no post-processing.The cited prior method treats each manipulation independently and requires post-processing.
3. Learning the Residual Image
The method learns face attribute manipulations as sparse residual images, using paired transformation networks, adversarial discrimination, perceptual preservation, and dual learning. Each transformed image is formed by adding a learned residual to its input.
- Image Transformation Networks: The architecture contains two transformation networks, G0 and G1, plus a discriminative network D for each face attribute manipulation.G0 and G1 model primal and dual operations, while D classifies reference and generated images.
- Image Transformation Networks: Residual learning concentrates on attribute-specific changes instead of redundantly modeling attribute-irrelevant image information.The residual is defined as the difference between the original and desired manipulated images.
- Image Transformation Networks: The transformed output is computed as ˜x_i = x_i + r_i = x_i + G_i(x_i), combining each input image with its predicted residual.G0 and G1 receive images with negative and positive attribute values and produce residuals r0 and r1.
- Loss Function: An L-1 pixel regularization term encourages the learned residual images to be sparse.This regularization supports modest modification of the input image.
- The Discriminative Network: The discriminator treats transformed images as an extra category alongside real images with positive and negative attribute labels.This creates a three-category discrimination task for adversarial training.
- Dual Learning: Dual learning feeds an image transformed by one network through the other, expecting the reconstructed result to retain the original attribute value.The two inverse operations provide feedback through a closed loop, and the authors state that this helps generate high-quality images.
4. Datasets
Experiments use CelebA for training and evaluation across six binary face attributes, while LFW tests generalization. Training lacks ground-truth manipulated images.
- Datasets: CelebA contains more than 200K celebrity images with 40 binary attributes; the study evaluates glasses, mouth open, smile, no beard, young, and male.Aligned face images are cropped and scaled to 128×128.
- Datasets: The authors balance each CelebA attribute dataset by selecting 1,000 images from each attribute class.The balancing addresses highly biased attribute labels.
- Datasets: LFW is used only to test the generalization of the proposed method.
- Datasets: CelebA provides no ground-truth manipulated images for training the transformation networks.
5. Implementation Details
The implementation uses attribute-dependent regularization weights, standard Normal initialization, and Adam optimization for training.
- Implementation Details: The method sets β = 0.1α, with α=5e-4 for local attributes and α=1e-6 for global attributes.Local attributes include glasses, no beard, mouth open, and smile; global attributes include male and young.
- Implementation Details: All network weights are initialized from a zero-centered Normal distribution with standard deviation 0.02.
- Implementation Details: The Adam optimizer is used during training for the transformation and discriminative networks.
6. Experiments
Experiments evaluate local and global attribute manipulation on CelebA and LFW, compare against VAE-GAN, and test residual and dual learning components. The proposed method preserves irrelevant facial details while producing attribute-specific changes, with glasses removal also improving eye-landmark detection.
- Local and Global Attribute Manipulation: On CelebA, the proposed method preserves most original details, unlike VAE-GAN, which changes hair, skin color, and background objects.Residual responses for local attributes concentrate mainly in the manipulated regions.
- Local and Global Attribute Manipulation: For local attributes, residuals focus on target regions, such as sunglasses or eyes during glasses addition and removal.The residual images visualize the areas modified by the manipulation.
- Local and Global Attribute Manipulation: Mouth-open and mouth-close manipulations capture chin movement by lowering and lifting the chin, respectively.These transformations demonstrate that the networks learn spatial changes beyond a single fixed local feature.
- Local and Global Attribute Manipulation: Global young and male manipulations are more challenging because they require subtle, image-wide changes involving wrinkles, hair color, beard, lips, and eyes.Residual responses for these attributes are scattered across the entire image rather than restricted to local areas.
- Ablation Study: Ablation experiments separately remove identity mapping or the data-feed loop to validate residual image learning and dual learning.The modified models otherwise retain the proposed model’s network settings.
- Landmark Detection with Glasses Removal: Glasses removal lowers eye-landmark detection error on D1m relative to D1, while rest-landmark errors remain almost unchanged.This indicates improved eye-landmark detection without substantial changes to other facial regions.
- Landmark Detection with Glasses Removal: Glasses-removal evaluation compares landmark errors on D0, D1, and processed set D1m using average normalized distance error.The table distinguishes eye landmarks from rest landmarks.
7. Discussion
Ablation visualizations compare the proposed model with variants lacking residual image learning or dual learning, while landmark detection evaluates glasses-removal effectiveness and large poses produce failure cases.
- Residual and dual learning: Figure 6 compares the proposed glasses-manipulation model with variants without residual image learning and without dual learning.The rows show original inputs, proposed results, results without residual image learning, and results without dual learning.
- Visual feature decorrelation: Figure 7 compares VAE-GAN reconstructions and manipulations with manipulated images from the proposed model for the no-beard attribute.The rows distinguish original images, VAE-GAN reconstructions, VAE-GAN manipulations, and proposed-model manipulations.
- Glasses-removal evaluation: Landmark detection accuracy gain is used to quantitatively evaluate the effectiveness of glasses removal.The evaluation is motivated by the difficulty of assessing manipulated images directly.
- Glasses-removal evaluation: Figure 8 distinguishes ground-truth landmarks from detections before and after glasses removal using green, red, and yellow points.The figure encodes ground truth in green, pre-removal detections in red, and post-removal detections in yellow.
- Failure cases: Figure 9 presents glasses-removal failure cases attributed mainly to large poses.It shows original inputs, manipulated images, and residual images in separate rows.
8. Conclusion
The paper presents a GAN-based method for face attribute manipulation that combines residual image learning with dual learning. Experiments show successful manipulation while preserving most details in attribute-irrelevant areas.
- Conclusion: The method uses a GAN-based framework with residual image learning and dual learning for face attribute manipulation.These strategies focus transformation networks on attribute-specific areas and allow them to learn from each other.