Source-linked AI summary
AdvHat: Real-world adversarial attack on ArcFace Face ID system
Stepan Komkov, Aleksandr Petiushko
TL;DR
The paper addresses the limited practicality and robustness of earlier physical face-recognition attacks. It introduces AdvHat, a printed rectangular sticker attack using off-plane transformations, and reports effective attacks against ArcFace across shooting conditions with transferability to other models.
Problem
Earlier physical face-recognition attacks required complex-shaped objects and fixed shooting conditions, limiting their practical evaluation.
Method
AdvHat prints a rectangular sticker and optimizes its off-plane transformation and projection onto face images to imitate placement on a hat.
Results
The attack efficiently targets the ArcFace system in different shooting conditions and transfers to other Face ID models.
Takeaways & Limitations
A color-printed rectangular sticker on a hat provides an easily reproducible real-world attack against a state-of-the-art public Face ID system.
Takeaways & Limitations
The study concentrates on real-world attacks; detailed evaluation in the digital domain is outside its scope.
Abstract
from arXiv · showhide
In this paper we propose a novel easily reproducible technique to attack the best public Face ID system ArcFace in different shooting conditions. To create an attack, we print the rectangular paper sticker on a common color printer and put it on the hat. The adversarial sticker is prepared with a novel algorithm for off-plane transformations of the image which imitates sticker location on the hat. Such an approach confuses the state-of-the-art public Face ID model LResNet100E-IR, ArcFace@ms1m-refine-v2 and is transferable to other Face ID models.
1. INTRODUCTION
AdvHat addresses limitations of prior real-world face-recognition attacks with a reproducible hat sticker that targets ArcFace under varied shooting conditions. The attack reduces similarity to the ground-truth identity and transfers to other Face ID models.
- Prior face-recognition attacks required cutting complex-shaped objects and used fixed lighting, face angles, and backgrounds.
- Figure 1 reports ground-truth similarity decreases of 0.592 for the left pair and 0.429 for the right pair.
- AdvHat attacks the public LResNet100E-IR, ArcFace@ms1m-refine-v2 system using a sticker placed on a hat.
- The attack is easily reproducible because it requires printing only a color rectangle.
- One attack works under different shooting conditions, including changes in lighting, face angle, and background.
- The proposed sticker-projection technique imitates sticker placement on the hat, and the attack transfers to other Face ID models.
2. BACKGROUND AND RELATED WORKS
Adversarial attacks modify inputs to fool neural networks, with transferability across models and distinctions between white-box, black-box, input-aware, and universal settings. Real-world attacks address transformations introduced by physical capture, with EOT emerging as a robustness strategy and prior work applying patches, stickers, eyeglasses, and detector attacks.
- 2.1. Adversarial attacks: Adversarial attacks slightly modify classifier inputs to change the recognized class, using perturbations constrained by objectives such as L2, L∞, or L0 norms.Targeted attacks seek a predefined class, whereas untargeted attacks seek an incorrect class; in face recognition these correspond to impersonation and dodging.
- 2.1. Adversarial attacks: Transferability allows an attack prepared for one neural network to confuse another network with a different architecture and training dataset.This property is identified as an important characteristic of adversarial attacks.
- 2.1. Adversarial attacks: Attacks differ by model access and input scope: white-box methods use model architecture or weights, while black-box methods do not, and universal perturbations apply across images.The paper positions its initial work as a white-box, input-aware attack.
- 2.2. Attacks in physical world: Physical attacks remain less certain than digital pixel-level attacks, although printed adversarial images have fooled classifiers when captured by camera phones.Prior physical attacks also include traffic-sign detectors, person detectors, and face-recognition systems using patches, EOT, NPS, TV loss, or adversarial eyeglasses.
- 2.2. Attacks in physical world: Expectation Over Transformation (EOT) models real-world changes such as scaling, jittering, brightness, contrast, rotations, and translations to construct attacks robust to transformations.Adversarial Patch applied EOT to universal ImageNet patches, while later physical attacks combined EOT with L0-limited perturbations, TV loss, or NPS.
- 2.2. Attacks in physical world: AdvHat’s pipeline reshapes a sticker, projects it onto face images with varied parameters, converts them to ArcFace templates, and updates the sticker using cosine similarity and TV-loss gradients.The pipeline is designed for a practical sticker-based attack under physical shooting conditions.
3. PROPOSED METHOD
The method optimizes a printable sticker whose simulated bending, rotation, projection, and ArcFace preprocessing reduce similarity to a person’s ground-truth class. It uses differentiable transformations and a combined embedding-similarity and total-variation objective, with parameter variation for robustness.
- Attack objective: The attack aims to lower predicted similarity to the ground-truth class below the Face ID decision threshold.Face recognition treats a person as recognized when top-1 similarity exceeds a predefined threshold.
- Sticker transformation: A novel off-plane transformation imitates how the rectangular sticker bends and rotates when placed on a hat.The transformation separates off-plane bending from pitch rotation, modeling bending with a parabolic cylinder and rotation with a 3D affine transformation.
- Robustness: Transformation and projection parameters are varied during the attack to improve robustness to uncertain sticker placement and geometry.The method changes the parabola rate, rotation angle, and projection parameters slightly while optimizing over batches of transformed images.
- Image projection and preprocessing: The sticker is projected onto a high-quality face image before conversion to ArcFace’s standard input template.Projecting onto a high-quality image preserves realistic boundary interpolation because neighboring face pixels influence preprocessing.
- Loss function: The optimization minimizes cosine similarity between obtained and anchor embeddings together with a total-variation loss.The final objective is a weighted sum, with the TV-loss weight set to 1e-4 in the experiments.
4. EXPERIMENTS AND RESULTS
The experiments evaluate AdvHat's attack process, fixed-condition effectiveness, robustness across shooting conditions, and transferability across Face ID models. Printed stickers substantially reduce similarity to the ground-truth identity, remain effective under tested variations, and transfer beyond the preparation model.
- Attack method: The sticker is projected through differentiable transformations, optimized with average gradients and iterative FGSM with momentum.The pipeline uses batch images with shared stickers and varying transformation parameters, followed by a two-stage attack schedule.
- Sticker localization: Sticker placement is optimized above the eyes, with gradients moving it downward toward the eyes because lower placement gives better validation values.The experiments place the hat and sticker at the lowest possible position to achieve the best results.
- Scope and limitations: The study focuses on real-world attacks, leaving detailed digital-domain analyses outside its scope.The authors also caution that the varied-condition testing set is crucially small.
- Fixed conditions: In fixed conditions, adversarial stickers significantly reduce ground-truth similarity, usually by more than 0.5; only one attack remains above 0.2.The evaluation uses baseline similarity, final similarity, their difference, and top-1 similarity to the first 1000 CASIA classes.
- Various shooting conditions: The robustness test uses 22 extra photos forming 11 paired comparisons across head tilts and lighting conditions, reusing stickers without new attacks.Four people from the initial group are evaluated under eight pose combinations and three lighting conditions.
- Various shooting conditions: Final similarity increases under varied conditions, but the attack still works; the authors cautiously associate robustness with rotations that keep the sticker visible.When the head leans forward, final similarity remains less than 0.2 and increases as the head rises.
5. CONCLUSION AND FUTURE WORK
AdvHat is presented as an easily reproducible method that efficiently attacks the best public Face ID model under different shooting conditions. Experiments verify its robustness against ArcFace, while future work targets state-of-the-art face detectors.
- AdvHat attacks the Face ID system using a reproducible method.
- The method efficiently attacks the best public Face ID model in different shooting conditions.
- Experimental results verify the attack's robustness against the state-of-the-art ArcFace system.
- Future work will apply the model to state-of-the-art face detectors.