Source-linked AI summary
Protecting Facial Privacy: Generating Adversarial Identity Masks via Style-robust Makeup Transfer
Shengshan Hu, Xiaogeng Liu, Yechao Zhang, Minghui Li, Leo Yu Zhang, Hai Jin, Libing Wu
TL;DR
Deep face recognition creates privacy risks through surveillance of public face images. AMT-GAN generates makeup-transferred adversarial faces using regularization and joint training, achieving strong black-box attacks while preserving visual quality.
Problem
Deep face recognition systems can identify social-media profiles and track user relationships, creating a need to protect face images from unauthorized recognition.
Method
AMT-GAN organizes adversarial perturbations through makeup transfer, using feature-disentanglement regularization and joint training to support natural appearance and adversarial attack simultaneously.
Results
AMT-GAN achieves a balance between adversarial attack strength and visual quality across multiple datasets, open-source models, and commercial APIs, outperforming state-of-the-art methods by about 4% ∼60%.
Takeaways & Limitations
The method provides a practical face-protection approach with black-box attack performance and makeup styles transferable from chosen reference images.
Takeaways & Limitations
AMT-GAN is stronger and visually better on female images because the makeup-transfer training dataset has imbalanced gender representation.
Abstract
from arXiv · showhide
While deep face recognition (FR) systems have shown amazing performance in identification and verification, they also arouse privacy concerns for their excessive surveillance on users, especially for public face images widely spread on social networks. Recently, some studies adopt adversarial examples to protect photos from being identified by unauthorized face recognition systems. However, existing methods of generating adversarial face images suffer from many limitations, such as awkward visual, white-box setting, weak transferability, making them difficult to be applied to protect face privacy in reality. In this paper, we propose adversarial makeup transfer GAN (AMT-GAN), a novel face protection method aiming at constructing adversarial face images that preserve stronger black-box transferability and better visual quality simultaneously. AMT-GAN leverages generative adversarial networks (GAN) to synthesize adversarial face images with makeup transferred from reference images. In particular, we introduce a new regularization module along with a joint training strategy to reconcile the conflicts between the adversarial noises and the cycle consistence loss in makeup transfer, achieving a desirable balance between the attack strength and visual changes. Extensive experiments verify that compared with state of the arts, AMT-GAN can not only preserve a comfortable visual quality, but also achieve a higher attack success rate over commercial FR APIs, including Face++, Aliyun, and Microsoft.
1. Introduction
AMT-GAN addresses privacy risks from unauthorized face recognition by generating adversarial images that combine stronger black-box transferability with more natural makeup-based visual changes. Its regularization and joint-training design supports simultaneous makeup transfer and adversarial attack, outperforming prior methods across commercial APIs.
- Unauthorized face recognition can identify social-media profiles and track user relationships, creating an urgent need for privacy protection.
- Existing adversarial face-protection methods are difficult to deploy because many require white-box access, show weak transferability, or produce awkward visual modifications.
- AMT-GAN organizes extensive perturbations as transferred makeup to preserve natural appearance while maintaining strong black-box attack ability.
- Its regularization module and joint training strategy reconcile adversarial noise with cycle consistency, enabling makeup transfer and adversarial attack simultaneously.
2. Related Works
Prior privacy attacks face deployment and visual-quality limitations, while style and makeup transfer provide a framework for recombining image content and style. The paper positions AMT-GAN against these limitations by targeting natural black-box adversarial faces while addressing cycle-consistency failure.
- 2.1. Adversarial Attacks on Face Recognition: White-box and query-based black-box attacks require access to target models, making transferability-based black-box attacks more suitable for real-world privacy protection.
- 2.1. Adversarial Attacks on Face Recognition: Optimization-based transferability attacks can be time-consuming, overfit, and generate perceptible noise, while GAN-based attacks face stability challenges on high-definition adversarial data.
- 2.1. Adversarial Attacks on Face Recognition: Latent-feature and patch-based attacks can alter facial attributes dramatically, making them unsuitable for users seeking natural social-media images.
- 2.2. Style Transfer and Makeup Transfer: Style transfer separates and recombines image content and style, while makeup transfer transfers reference makeup while preserving face-recognition-related content.
- 2.2. Style Transfer and Makeup Transfer: Existing adversarial makeup-transfer methods have shown low attack success and abnormal, noticeable modifications, especially when reference and source styles differ.
- 2.2. Style Transfer and Makeup Transfer: Adversarial noise can disrupt cycle consistency, making natural adversarial makeup transfer with high black-box attack success difficult.
3. Adversarial Makeup Transfer GAN (AMT-GAN)
AMT-GAN formulates adversarial makeup transfer as targeted attacks that combine face-identity impersonation with makeup-style translation. Its generator, regularization module, discriminators, and ensemble of local recognition models are jointly trained to preserve content and visual quality while improving black-box transferability.
- Problem formulation: Targeted adversarial makeup transfer generates an image recognized as a specified target identity while transferring reference makeup and preserving the source visual identity.The adversarial objective compares source outputs with a target image through a face-recognition feature extractor, while the transfer mapping changes makeup style between domains.
- Generator and discriminators: The GAN generator learns bidirectional source-reference mappings with adversarial, cycle-consistency, and discriminator objectives.The cycle loss supports unsupervised mappings between the two image domains, while the discriminators distinguish generated images from real domain distributions.
- Regularization module: The regularization module H converts generated adversarial images toward the clean domain while retaining their content, style, and dimensions.Its encoder-decoder and residual-in-residual dense blocks are intended to recover content and texture while discarding adversarial perturbations; regularized cycle and GAN losses train this behavior.
- Transferability enhancement: Ensemble training with K local face-recognition models and input diversity is used to imitate inaccessible target decision boundaries and enhance black-box attack transferability.The transformations include image resizing and Gaussian noise, which are used during adversarial training.
- Training procedure: The training procedure alternates updates for the discriminators, generator, and regularization module using Adam over randomly selected source and reference images.The algorithm returns parameters for G, H, DX, and DY after iterative optimization.
- Auxiliary objectives: Histogram matching and self-reconstruction objectives support makeup similarity, content preservation, structural integrity, and reduced distortion of face attributes.Histogram matching targets lips, eye shadows, and face regions, while self-reconstruction uses the source image as its own reference.
4. Experiments
Experiments evaluate AMT-GAN against adversarial-attack and makeup-transfer baselines across black-box models, commercial APIs, image-quality metrics, and ablations. Results indicate strong black-box attack performance, improved visual naturalness, and robustness to makeup-style changes, while regularization supports image quality.
- Experimental Setting: Experiments compare AMT-GAN with PGD, MI-FGSM, TI-DIM, TIP-IM, and Adv-Makeup on black-box face-recognition tasks.The evaluation targets four pretrained models and three commercial APIs, using ASR for black-box models and API confidence scores for commercial services.
- Comparison Study: AMT-GAN produces more natural visual appearances than TIP-IM according to FID, despite worse PSNR(dB) and SSIM.The authors interpret this as more natural appearance with more information changed, consistent with arranging rather than simply restricting perturbations.
- Comparison Study: AMT-GAN yields smoother, more realistic makeup details than Adv-Makeup, whose eye-region patches create sharp margins and weak black-box attack strength.The visual comparison links Adv-Makeup’s quantitative image-quality behavior with insufficient attack performance and unresolved margin artifacts.
- Comparison Study: AMT-GAN achieves stronger attack performance than competitors across Face++ and Aliyun test datasets.The study averages confidence scores from many adversarial examples and reports AMT-GAN outperforming competitors for both APIs.
- Ablation Studies: Removing the regularization module worsens image quality, whereas regularized cycle consistency makes generator outputs more natural.The ablation attributes degradation to adversarial toxicity damaging the cycle reconstruction path and style-domain mappings.
- Comparison Study: 64.58 versus 61.99 is AMT-GAN’s mean Face++ confidence score compared with the runner-up TIP-IM.For Aliyun, AMT-GAN reaches 53.53 versus 32.37 for the runner-up TI-DIM, with reported gains of about 4%–60%.
- Ablation Studies: Changes in makeup references have weak impact on attack strength while preserving a balance between source content and reference makeup style.The style-robustness evaluation uses ten references from the MT and LADN datasets.
5. Limitations and Future Work
AMT-GAN has several reported limitations involving gender imbalance, structural alignment, and evaluation or visual-quality constraints. The authors identify broader training data, alignment improvements, and better quality metrics as future directions.
- AMT-GAN performs better on female images because the makeup-transfer training dataset has imbalanced gender representation.The authors suggest a more general and comprehensive training dataset could address this limitation.
- Structural information can become slightly unaligned despite objective functions designed to alleviate the problem.The authors note that this issue may worsen when generating adversarial examples.
- Existing image-quality metrics are unsuitable for evaluating naturalness in makeup transfer.The authors call for new metrics that can assess whether one face image appears more natural than another.
- Future work should further improve adversarial-face visual quality and restrict makeup transfer to smaller regions while preserving high attack success rates.The authors specifically mention areas such as the eyes as a possible restricted transfer region.
6. Conclusion
The paper proposes AMT-GAN for protecting facial privacy against malicious face-recognition models by combining stronger black-box attacks with better visual quality. Experiments across multiple datasets and target models report effectiveness on open-source models and commercial APIs while balancing attack strength and appearance.
- AMT-GAN constructs adversarial face images with stronger black-box attack ability while maintaining better visual quality.It transfers makeup from any reference image.
- Experiments across multiple datasets and target models show effectiveness against open-source face-recognition models and commercial APIs.The conclusion reports a balance between adversarial-face visual quality and attack strength.