Source-linked AI summary
PSGAN: Pose and Expression Robust Spatial-Aware GAN for Customizable Makeup Transfer
Wentao Jiang, Si Liu, Chen Gao, Jie Cao, Ran He, Jiashi Feng, Shuicheng Yan
TL;DR
PSGAN addresses makeup transfer when source and reference images differ in pose or expression and when users need region-specific or shade-controlled editing. It distills reference makeup into spatial-aware matrices, morphs them with attention, and applies them to source features. The paper reports state-of-the-art transfer on frontal and varied-pose images, with partial and shade-adjustable transfer, and introduces the Makeup-Wild dataset.
Problem
Existing makeup-transfer methods are limited by pose and expression differences and cannot provide partial or shade-controllable transfer.
Method
PSGAN distills reference makeup into two spatial-aware matrices, morphs them with AMM using source-reference attention, and applies them through MANet.
Results
PSGAN achieves state-of-the-art transfer on frontal and varied-pose facial images while supporting partial and shade-adjustable makeup transfer.
Takeaways & Limitations
Spatial-aware makeup matrices broaden makeup transfer by enabling region-specific editing and adjustment of transferred makeup shade.
Takeaways & Limitations
The formulation assumes unpaired source and reference domains, with source and reference images having different identities.
Abstract
from arXiv · showhide
In this paper, we address the makeup transfer task, which aims to transfer the makeup from a reference image to a source image. Existing methods have achieved promising progress in constrained scenarios, but transferring between images with large pose and expression differences is still challenging. Besides, they cannot realize customizable transfer that allows a controllable shade of makeup or specifies the part to transfer, which limits their applications. To address these issues, we propose Pose and expression robust Spatial-aware GAN (PSGAN). It first utilizes Makeup Distill Network to disentangle the makeup of the reference image as two spatial-aware makeup matrices. Then, Attentive Makeup Morphing module is introduced to specify how the makeup of a pixel in the source image is morphed from the reference image. With the makeup matrices and the source image, Makeup Apply Network is used to perform makeup transfer. Our PSGAN not only achieves state-of-the-art results even when large pose and expression differences exist but also is able to perform partial and shade-controllable makeup transfer. We also collected a dataset containing facial images with various poses and expressions for evaluations.
1. Introduction
PSGAN targets makeup transfer under pose and expression differences while supporting partial and shade-controllable editing. It combines spatial-aware makeup representations, attentive morphing, and a new diverse evaluation dataset.
- Existing methods struggle with non-frontal faces and neutral expressions because they lack misalignment handling and overfit frontal images.
- PSGAN uses MDNet, AMM, and MANet to distill, adapt, and apply makeup through spatial-aware feature transformations.
- Spatial-aware makeup matrices enable pixel-wise partial transfer and shade control by masking facial regions or scaling matrix weights within [0, 1].
- The method reports pose- and expression-robust results, including consistent transfer when applied to every frame of facial videos.
- PSGAN is presented as the first method to combine partial, shade-controllable, and pose- and expression-robust makeup transfer.
- The Makeup-Wild dataset contains facial images with diverse poses and expressions for evaluating real-world makeup transfer.
2. Related Work
Prior makeup-transfer and style-transfer methods established GAN, normalization, and attention-based foundations. PSGAN extends these ideas with spatial-aware, pixel-level makeup transfer using visual appearance and location jointly.
- Earlier makeup-transfer methods commonly use GANs, face parsing or landmarks, and unpaired non-makeup and makeup image sets.
- AdaIN enabled arbitrary style transfer by aligning content-feature statistics with style-feature statistics.
- PSGAN transfers spatial-aware makeup for each pixel rather than a general style from the reference image.
- Its AMM module computes weighted feature aggregation using both visual appearances and relative locations.
3. PSGAN
PSGAN learns an unsupervised makeup-transfer function for unpaired source and reference images, using spatial-aware makeup representations and attentive morphing to handle pose and expression differences. Its objective combines adversarial, cycle-consistency, perceptual, and makeup losses to generate realistic transfers while preserving source identity and matching reference makeup.
- PSGAN maps unpaired source and reference images to a transferred result through G(x, y), without requiring paired datasets.The source and reference images may have different identities and are sampled from separate domains.
- The framework comprises MDNet, AMM, and MANet, which distill reference makeup, morph it for source geometry, and apply it to source features.MDNet represents makeup as spatial-aware matrices γ and β; AMM aligns them across pose and expression differences before MANet applies them.
- AMM computes pixel affinities from visual appearance, normalized relative positions, and facial-region agreement to select semantically corresponding reference pixels.The relative-position features use distances to 68 facial landmarks, while the indicator function restricts attention to matching facial regions.
- AMM reshapes attention rows into maps that concentrate around semantically similar facial locations, supporting pose- and expression-robust makeup morphing.The source-point attention example shows large values around the corresponding nose-corner region, with softmax producing more concentrated attention.
- MANet applies morphed makeup tensors to the source feature map and decodes the updated representation into the transferred image.The morphed tensors Γ′ and B′ are expanded across channels before being applied in MANet's bottleneck.
- Training combines adversarial loss for realistic outputs, cycle consistency for unsupervised reconstruction, perceptual loss for identity preservation, and makeup loss for regional color guidance.The makeup loss uses region-wise histogram matching as coarse guidance, while the perceptual loss compares VGG-16 hidden-layer activations.
4. Experiments
Experiments evaluate PSGAN on standard and wild makeup-transfer data, ablate its attention and morphing components, and test partial, interpolated, comparative, and video transfer. The results show improved handling of pose and expression differences alongside controllable makeup composition and stable video transfer.
- Experimental setting: The Makeup-Wild dataset contains facial images with varied poses, expressions, and complex backgrounds for real-world evaluation.Images were collected online, filtered to remove frontal or neutral faces, and resized without alignment.
- Ablation studies: Without AMM, pose and expression differences cause poor transfers, whereas AMM alleviates misalignment between source and reference images.The ablation shows incorrect transfer regions without AMM; the module morphs distilled makeup matrices to account for these differences.
- Ablation studies: A visual-feature weight of 0.01 focuses attention on skin while avoiding nostrils and background regions.Using no visual features produces Gaussian-like maps that can cross facial boundaries, while larger weights create scattered maps.
- Partial and interpolated makeup transfer: Spatial-aware makeup matrices enable partial transfer by assigning different makeup parameters to pixels selected through face-parsing masks.Experiments mix lipstick from one reference with other makeup from another and can leave nonselected regions unchanged.
- Partial and interpolated makeup transfer: PSGAN supports shade adjustment with one reference and smooth makeup-style interpolation between two references.The method can also combine partial transfer and interpolation by using face-parsing maps together with interpolation coefficients.
- Comparison: Compared with baseline methods, PSGAN generates vivid images matching reference makeup more closely and produces nice, stable results when applied frame-by-frame to videos.Other methods show issues including wrong-region transfer, blurry results, unnatural backgrounds, or weaker reference-style preservation.
5. Conclusion
PSGAN combines spatial-aware makeup matrices with Attentive Makeup Morphing to enable accurate, customizable makeup transfer. Experiments show state-of-the-art results across frontal and varied-pose facial images, including partial and shade-adjustable transfer.
- PSGAN distills reference makeup into spatial-aware matrices and uses Attentive Makeup Morphing for accurate transfer.
- Experiments demonstrate state-of-the-art transfer results on frontal facial images and images with varied poses and expressions.
- Spatial-aware makeup matrices enable partial makeup transfer and adjustable transfer shade.
- The framework may also support other conditional image-synthesis problems requiring customizable and precise synthesis.