PaperlayerView exact revision on arXiv

Source-linked AI summary

LADN: Local Adversarial Disentangling Network for Facial Makeup and De-Makeup

Qiao Gu, Guanzhi Wang, Mang Tik Chiu, Yu-Wing Tai, Chi-Keung Tang

arXiv:1904.11272v2cs.CVcs.AIcs.LG

TL;DR

Existing techniques are limited on dramatic makeup whose details extend beyond defined facial regions. LADN combines overlapping local discriminators, content-style disentanglement, and asymmetric losses, achieving state-of-the-art performance on light and typical makeup while handling extreme styles.

  • Problem

    Existing methods mainly demonstrate conventional or simple styles, while region-specific transfer is limited when makeup color and texture extend beyond defined facial regions.

    Source: §1 · p. 2

  • Method

    LADN combines multiple overlapping local adversarial discriminators with content-style disentanglement and asymmetric losses for makeup transfer and removal.

    Source: §1 · p. 1

  • Results

    LADN achieves state-of-the-art performance on light/typical makeup and transfers or removes extreme styles with high-frequency details across large facial areas.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The paper plans to release an unpaired before-and-after makeup dataset for noncommercial use upon acceptance.

    Source: §1 · p. 2

    The network struggles with extreme styles whose local regions are internally consistent but sharply different across patches, especially when few high-frequency details leave the smooth loss ineffective.

    Source: §5 · p. 8

Abstract

from arXiv · show

We propose a local adversarial disentangling network (LADN) for facial makeup and de-makeup. Central to our method are multiple and overlapping local adversarial discriminators in a content-style disentangling network for achieving local detail transfer between facial images, with the use of asymmetric loss functions for dramatic makeup styles with high-frequency details. Existing techniques do not demonstrate or fail to transfer high-frequency details in a global adversarial setting, or train a single local discriminator only to ensure image structure consistency and thus work only for relatively simple styles. Unlike others, our proposed local adversarial discriminators can distinguish whether the generated local image details are consistent with the corresponding regions in the given reference image in cross-image style transfer in an unsupervised setting. Incorporating these technical contributions, we achieve not only state-of-the-art results on conventional styles but also novel results involving complex and dramatic styles with high-frequency details covering large areas across multiple facial features. A carefully designed dataset of unpaired before and after makeup images is released.

1. Introduction

Existing techniques are limited on dramatic makeup with details beyond conventional facial regions. LADN combines overlapping local discriminators and content-style disentanglement with asymmetric losses to transfer and remove complex styles.

1. Introduction

  • Asymmetric losses on the transfer and removal branches encourage makeup-latent disentanglement and mostly preserve facial identity.The paper also plans to release an unpaired before- and after-makeup dataset for noncommercial use upon acceptance.Source: §1 · pp. 1–2
  • Existing deep-learning methods mainly demonstrate conventional or relatively simple makeup styles and often fail to transfer or remove dramatic makeup.The introduction notes that dramatic makeup may be the main use case before users physically apply makeup that can take hours.Source: §1 · p. 2
  • Region-specific transfer handles defined areas such as eyes and mouth but limits transfer of dramatic color and texture beyond those features.The paper identifies content-style separation, or extracting the makeup-only latent variable, as a central problem.Source: §1 · p. 2
  • Multiple overlapping local discriminators in a content-style disentangling network transfer or remove complex dramatic makeup details.The introduction says increasing the number and overlap of discriminators supports high-frequency detail transfer while preserving facial identity and structure.Source: §1 · pp. 1–2

2. Related Work

Prior image-translation and makeup methods address unpaired translation, defined facial regions, or separate transfer and removal tasks, but have limits on dramatic styles. LADN uses multiple style discriminators for facial patches to compare generated makeup with a reference.

2. Related Work

  • DRIT disentangles domain-invariant content from domain-specific attributes for unpaired translation, but the paper reports it handles only light makeup styles.The authors say this limitation follows from the makeup transfer being formulable as image-to-image translation only for light styles.Source: §2 · p. 2
  • Earlier makeup transfer methods use paired aligned images, layered decomposition, reflectance models, or local facial-component style transfer.The cited approaches include Tong et al., Guo et al., Li et al., and Liu et al. [22].Source: §2 · p. 2
  • Prior removal methods treat transfer and removal separately or crop eyes and mouth as local paths, limiting treatment of other facial regions.The paper contrasts these methods with Li et al. and says it is the first to achieve makeup-latent disentanglement and transfer and removal on extreme dramatic styles.Source: §2 · p. 3
  • Unlike prior work using a single local discriminator on sampled patches, LADN uses multiple style discriminators specialized for landmark-defined facial patches.These discriminators assess whether generated makeup style matches the reference; Table 1 frames the comparison around local and global discriminators and overlapping regions.Source: Table 1 · p. 2, §2 · p. 3

3. LADN

LADN frames makeup transfer and removal as unsupervised translation between unpaired face domains, separating makeup style from other facial content. It combines content-style encoders with overlapping local discriminators and asymmetric losses to handle local makeup details and de-makeup smoothness.

3.1. Problem Formulation

  • Without paired before- and after-makeup images, the model transfers a reference’s makeup style to a source identity and removes makeup without reference conditioning.The transfer and removal mappings are formulated as unsupervised cross-domain image translation.Source: §3.1 · p. 3, §3 · p. 3

3.2. Network Architecture

  • Content and style encoders separate non-makeup features from makeup attributes, which generators recombine; U-Net skip connections help retain source identity details.Domain discriminators distinguish generated images from real samples to support realistic outputs.Source: §3.2 · p. 4

3.3. Local Style Discriminator

  • Multiple overlapping, landmark-defined local discriminators assess whether generated patches match the reference’s makeup style, capturing details that may extend beyond fixed facial regions.For 512 × 512 images, each discriminator considers a 102 × 102 patch; warped reference patches provide guidance but are not ground truth and may contain artifacts.Source: §3.3 · pp. 4–5

3.4. Asymmetric Losses

  • Asymmetric losses encourage high-frequency detail in makeup transfer but smoothness in de-makeup, with lower eye-region smoothing and stronger smoothing on cheeks and nose.The high-order loss uses Laplacian-filtered warped patches to guide detail transfer, while the smooth loss reflects the assumption that skin color beneath makeup is generally smooth.Source: §3.4 · p. 5

3.5. Other Loss Functions

  • Self- and cross-cycle reconstruction losses encourage consistency, while a KL loss regularizes makeup-style attributes toward a prior Gaussian distribution.The cross-cycle reconstruction loss is scaled by an additional factor of 8 to encourage makeup style in the transfer result.Source: §3.5 · p. 5

4. Experiments

The experiments focus on complex and dramatic makeup; conventional/light transfer, removal, and user-study comparisons are deferred to supplemental materials. The section evaluates local discriminator configurations, makeup transfer and removal, and attribute interpolation.

4.1. Data Collection

  • The team collected frontal, unoccluded web images and classified 333 before-makeup and 302 after-makeup faces, adding 115 manually selected extreme-makeup images.The extreme images varied in makeup color, style, and regional coverage.Source: §4.1 · p. 6

4.4. Makeup Transfer Results

  • LADN transferred styles from highly dramatic makeup to eye-and-mouth-only looks, though stars disappeared and eyeshadow color could be harsh.Without local high-order loss, fine details blurred, especially in edge-rich styles.Source: Figure 5 · p. 6, §4.4 · p. 7

4.2. Training Details

  • Training proceeded for 700 epochs without smooth or high-order losses, then 2,000 epochs with both losses set to 20.The setup used 512 × 512 images, batch size 1, and K = 12 local discriminators.Source: §4.2 · pp. 6–7

4.3. Local Discriminators

  • A single global discriminator missed the reference style, while increasing overlapping local discriminators from K = 3 to 12 improved makeup coverage and resemblance.The reported comparisons show stronger nose-texture resemblance at K = 12 than K = 6.Source: §4.3 · p. 7

4.5. Makeup Removal Results

  • For dramatic removal, asymmetric losses separated makeup from facial texture, while the generator hallucinated a reasonable skin color where the reference concealed it.Smooth loss mattered most for edge-spanning styles; light-style removal also restored normal face color without significant changes to other areas.Source: §4.5 · p. 7, Figure 6 · p. 7

4.7. Interpolated Makeup Styles

  • Interpolating two makeup attribute vectors produced smooth transitions between styles without affecting the source face’s identity or structure.The interpolation supports adjusting makeup heaviness and mixing styles.Source: Figure 7 · p. 8, §4.7 · p. 8

4.6. Qualitative Comparison

  • In extreme-makeup comparisons, LADN required only roughly correct landmarks, whereas required highly accurate face-geometry alignment and [20]’s refinement blurred makeup details.The comparison also reports boundary discontinuity and makeup-color fading for.Source: §4.6 · pp. 7–8, Figure 8 · p. 8

5. Limitations and Conclusion

LADN combines local style discriminators, disentangled representation, and asymmetric losses for makeup transfer and removal. It reports strong results on light/typical styles and transfer of extreme high-frequency styles, but struggles with certain sharply contrasting patchwise styles.

  • When colors are consistent within local regions but vary sharply between patches, few high-frequency details leave the smooth loss ineffective and the result plausible behind each half of the makeup.Source: §5 · p. 8
  • The smooth loss can also remove a mole that may be part of dramatic makeup.Source: §5 · p. 8
  • LADN transfers extreme makeup with high-frequency color changes and details across large facial areas, while achieving state-of-the-art performance on light/typical styles.Source: §5 · p. 8
  • LADN incorporates local style discriminators, disentangled representation, and asymmetric loss functions into a cross-domain image translation network.Source: §5 · p. 8
Loading 1904.11272v2…