Source-linked AI summary
Adversarial Sticker: A Stealthy Attack Method in the Physical World
Xingxing Wei, Ying Guo, Jie Yu
TL;DR
The paper addresses how to construct stealthy, physically robust adversarial patches under black-box access and limited information. It proposes Meaningful Adversarial Stickers using real stickers and RHDE to search placement parameters, with experiments spanning three tasks and reporting up to 98.46% attacked video frames under changing face postures.
Problem
Existing physical adversarial patches face printing distortion, high computation costs, unnatural patterns, and dependence on detailed white-box model information.
Method
The method uses real stickers in a query-based black-box attack, manipulating pasting positions and rotation angles and optimizing them with RHDE.
Results
The method was effective across face recognition, image retrieval, and traffic sign recognition, with at most 98.46% of video frames successfully attacked during changing face postures.
Takeaways & Limitations
Real sticker placement parameters provide a physically feasible and stealthy attack form that generalizes across the evaluated vision tasks.
Takeaways & Limitations
The approach is evaluated under a black-box setting with limited information rather than full threat-model details.
Abstract
from arXiv · showhide
To assess the vulnerability of deep learning in the physical world, recent works introduce adversarial patches and apply them on different tasks. In this paper, we propose another kind of adversarial patch: the Meaningful Adversarial Sticker, a physically feasible and stealthy attack method by using real stickers existing in our life. Unlike the previous adversarial patches by designing perturbations, our method manipulates the sticker's pasting position and rotation angle on the objects to perform physical attacks. Because the position and rotation angle are less affected by the printing loss and color distortion, adversarial stickers can keep good attacking performance in the physical world. Besides, to make adversarial stickers more practical in real scenes, we conduct attacks in the black-box setting with the limited information rather than the white-box setting with all the details of threat models. To effectively solve for the sticker's parameters, we design the Region based Heuristic Differential Evolution Algorithm, which utilizes the new-found regional aggregation of effective solutions and the adaptive adjustment strategy of the evaluation criteria. Our method is comprehensively verified in the face recognition and then extended to the image retrieval and traffic sign recognition. Extensive experiments show the proposed method is effective and efficient in complex physical conditions and has a good generalization for different tasks.
1 INTRODUCTION
The paper targets physically robust, stealthy adversarial attacks when only limited black-box information is available. It uses real stickers and searches their placement parameters, reporting results across three vision tasks.
- DNN-based vision systems can produce wrong identity judgments after small malicious perturbations, motivating physical-world attack research.
- Existing adversarial patches face digital-to-physical distortion, computation costs, unnatural appearance, and reliance on white-box model information.These limitations involve transformations such as EOT and printing-related distortion, as well as requiring detailed target-model parameters.
- The Meaningful Adversarial Sticker uses real stickers and manipulates their pasting positions and rotation angles instead of designing perturbation patterns.The approach is framed as a query-based black-box attack using limited information.
- RHDE improves parameter search by exploiting regional aggregation of effective sticker locations and adaptively adjusting the evolution direction and evaluation criteria.
- 98.46% of video frames were successfully attacked during continuously changing face postures, while image retrieval and traffic sign recognition showed good generalization.The experiments covered face recognition, image retrieval, and traffic sign recognition in digital and physical settings.
2 RELATED WORK
Prior work develops digital and physical adversarial attacks across multiple computer-vision tasks, but commonly uses perturbation patterns or white-box access. This paper instead uses existing stickers and black-box parameter manipulation.
- 2.1 Digital Attacks: Digital attacks include optimization-based, one-step, multi-step, and localized-patch methods, generally conducted with access to model structures and weights.
- 2.1 Digital Attacks: Black-box attacks avoid detailed model parameters and include transfer-based, score-based, and decision-based settings.
- 2.2 Physical Attacks: Physical attacks have been demonstrated through camera-captured perturbations and patches robust to multiple physical transformations.
- 2.2 Physical Attacks: Adversarial patches have been applied to automatic driving, face recognition, object detection, wearable accessories, graffiti, posters, and clothing.
- 2.2 Physical Attacks: The proposed stickers use real-life stickers and change their pasting parameters rather than generating or printing perturbation patterns.The paper presents this as physically feasible and stealthy.
3 METHODOLOGY
The methodology studies face-recognition attacks by analyzing effective parameter regions, designing RHDE, and calculating sticker deformation for varying positions.
- The face-recognition methodology first investigates regional aggregation of effective parameters.
- It then details the proposed Region based Differential Evolution algorithm for solving sticker parameters.
- The methodology finally introduces sticker-deformation calculation during parameter solving.
3.1 Problem formulation
The face-recognition attack forms an adversarial image by combining a clean face with a masked sticker region, using an existing sticker rather than generating perturbations. The sticker’s shape is fixed after selection, while its pasting position is optimized to induce an incorrect identity prediction.
- The attack aims to make a face-recognition model predict the wrong identity for an adversarial face image.
- The adversarial image combines the clean image and sticker region through a mask using a Hadamard product.The mask constrains the patch shape and pasting position, with value 1 in the pasted area.
- Unlike prior methods that optimize perturbations with a fixed mask, this method uses an existing life-like sticker as the perturbation.
- After choosing a sticker, its shape is fixed and the attack searches for an optimal pasting position on the face.
3.2 Regional aggregation
Successful sticker positions are regionally clustered rather than dispersed across faces. This pattern supports heuristic searches that continue around previously successful positions to improve attack-parameter efficiency.
- 96.5% of successfully attacked LFW faces and 97.8% of successfully attacked CelebA faces have clustered sticker positions.Among these faces, one area contains the clusters for 58.7% and 61.2%, while multiple clustered areas account for 37.8% and 36.6%, respectively.
- In a small region around the highest wrong-label probability, wrong-label probability decreases with distance while ground-truth probability increases.
- The position study queries FaceNet exhaustively across every valid position for 1,000 images each from LFW and CelebA.
- Regional clustering motivates searching around previously successful positions instead of exploring positions independently.
3.3 Region based Heuristic Differential Evolution
The paper formulates sticker placement and rotation as a score-based black-box optimization problem and solves it with RHDE, which exploits clustered effective positions while preserving search diversity.
- Attack setting: The attack parameters include sticker pasting position and rotation angle, optimized for dodging or impersonation objectives.Dodging minimizes the ground-truth probability, while impersonation maximizes the target identity probability.
- Attack setting: Score-based black-box queries provide predicted labels and probabilities when the recognition model’s internal parameters are unavailable.Gradient-based optimization is unsuitable because invalid positions create discontinuous objective functions with unknown smoothness.
- Strategies for the offspring’s generation: RHDE accelerates evolutionary search by generating offspring from both random crossover and regions surrounding superior individuals.This combines regional exploitation with broader exploration to reduce the risk of local optima.
- Region based Heuristic Differential Evolution: Each generation compares candidate and current populations, retains better individuals, and stops after attack success or T iterations.Individuals encode pasting position, rotation angle, and other attack parameters.
- Strategies for the offspring’s generation: The inbreeding operation searches position parameters around superior solutions and retains nearby individuals that minimize the loss function.It applies the regional search in r directions around selected superior individuals.
- Adaptive adjustment of the evaluation criteria: For dodging attacks, the evaluation criterion initially reduces ground-truth probability and later promotes the top-2 class when the population reaches a good state.The transition occurs when the top-1 and top-2 probabilities for the best individual differ by less than δ.
3.4 The generation of adversarial stickers
The sticker-generation procedure bends and rotates a real sticker to fit a face’s 3D geometry, then reconstructs its image pattern through interpolation and backward mapping.
- Bending and rotation: A 3D morphable model generates face geometry and identifies the highest point of the sticker’s pasting area for deformation.Bending uses the X-Z plane, while rotation uses the Y-Z plane.
- Bending transformation: The bending transformation leaves Y coordinates unchanged and models the sticker’s X-Z projection as a parabola.The bent sticker’s width is selected so its arc length equals the original sticker width.
- Bending transformation: A matrix MA stores the 3D coordinates of all pixels in the bent sticker.The resulting representation has dimensions h × wn.
- Pattern reconstruction: Bilinear interpolation with backward mapping reconstructs each pixel value in the deformed sticker pattern.The interpolation function determines pixel values at transformed positions.
- Rotation transformation: The bent sticker is rotated in 3D according to the angle derived from Y-Z-plane geometry.The rotation angle is estimated from the corresponding Δy and Δz lengths.
3.5 Implementation in the physical world
After solving attack parameters digitally, the physical procedure applies real stickers to subjects’ faces at the calculated positions and orientations without printing custom perturbations.
- Physical deployment: Physical attacks paste real stickers on subjects’ faces according to parameters obtained in the digital environment.The digital process first solves the parameters corresponding to each subject’s face.
- Physical deployment: Because the method does not require printing or making perturbation patterns, it avoids NPS and TV losses and their associated computation costs.
- Physical deployment: The implementation also omits EOT in the physical attack procedure.
4 EXTENSIONS TO OTHER APPLICATIONS
The paper extends meaningful adversarial stickers to image retrieval and traffic sign recognition, adapting the attack objective and detection alignment to each task.
- Image retrieval: Image retrieval attacks use a relevance-based score because black-box systems typically expose ranked results rather than similarity scores.The method formulates the attack using the returned top-k list and a relevance-based objective.
- Image retrieval: When k is small, changing the original top-k results may still return images relevant to the adversarial query.The evaluation only checks overlap between the original and attacked top-k lists, while secondary similar images can remain relevant.
- Image retrieval: The iterative query method expands candidates by recursively retrieving top-k images for each image initially returned.It can collect at most k+k^2 candidate images before evaluating the attack.
- Traffic sign recognition: Traffic-sign attacks target object detectors through classification and disappearance objectives rather than single-image classification.The detector must handle multiple signs, bounding boxes, and labels within a scene.
- Traffic sign recognition: Each traffic-sign attack iteration aligns the target using the detected box with the largest IoU, then evaluates its predicted-label probability.A box is retained as the target when its IoU exceeds threshold u; disappearance is achieved when no valid box remains.
5 EXPERIMENTS AND RESULTS
Experiments evaluate meaningful adversarial stickers across face-recognition models, datasets, stickers, physical conditions, and defenses. Results show effective attacks with hundreds of queries, physical robustness, and improved search efficiency from RHDE.
- Face classification: 81.78% and 51.11% are the highest fooling rates for dodging and impersonation attacks, respectively, in face classification.Experiments use FaceNet, SphereFace, and CosFace on LFW and CelebA with fooling rate and query count metrics.
- Ablation study: RHDE substantially improves fooling rates and reduces query counts when its adaptive evaluation and region-based offspring strategies are combined.The comparison uses the same maximum iteration budget across direct DE, adaptive-DE, region-DE, and their combination.
- Ablation study: The exhaustive upper bound exceeds the proposed method by only 3.26%, 3.23%, and 4.07%, while requiring 11162, 12499, and 12320 more queries on three models.This comparison indicates that the method finds successful solutions much more efficiently than traversal of the search space.
- Physical face attacks: 98.46% of video frames can be successfully attacked while face postures continuously change in the physical environment.The method maintains attack effects under varying physical conditions and face postures.
- Robustness: Using a different sticker for attack than for adversarial training changes fooling rates by at most 3.06% and queries by 17.Training on the same sticker causes a larger fooling-rate decrease, whereas unseen sticker types remain comparatively effective.
- Face identification: Commercial API attacks reach fooling rates of 46.17% and 40.25%, while open-source-model attacks reach at most 80.03% and 58.82%.The reported pairs correspond to dodging and impersonation performance in the commercial API and the broader face-identification experiments.
6 EXTENSIONS TO OTHER APPLICATIONS
The method is extended to clothing image retrieval and traffic-sign detection in digital and physical settings. Reported results show lower retrieval recall, irrelevant clothing returns, and successful sign misclassification or disappearance.
- Image retrieval: Image retrieval attacks lower Recall@K by nearly 0.6 using 343 queries on the In-shop clothes retrieval task.The adversarial query image obtains lower recall than the original query image for the same retrieval system.
- Image retrieval: After stickers are pasted on clothing, retrieval can return weakly correlated items such as jackets for a query containing jeans.The original query returns similar clothing, whereas the adversarial query produces unrelated or weakly related results.
- Traffic sign recognition: In digital traffic-sign experiments, YOLO misclassifies adversarial signs and can fail to detect the sign region entirely.The classification and disappearance attacks are illustrated on original and adversarial images.
- Traffic sign recognition: Physical traffic-sign attacks test precomputed sticker positions across distances of 1 m and 3 m and angles from 0° to 20°.The captured images are evaluated by YOLO under varied physical conditions.
7 CONCLUSION
The paper concludes that meaningful adversarial stickers provide physically feasible, stealthy black-box attacks by optimizing real-sticker placement and rotation across multiple vision tasks.
- Conclusion: Extensive digital and physical experiments on face recognition, image retrieval, and traffic sign recognition demonstrate the method’s effectiveness.The conclusion frames the method as applicable when model information is unknown.
- Conclusion: RHDE efficiently solves sticker parameters using regional aggregation of effective solutions and adaptive evaluation-criterion adjustment.The optimized parameters include pasting positions, rotation angles, and other sticker parameters.