Source-linked AI summary
Adversarial Texture for Fooling Person Detectors in the Physical World
Zhanhao Hu, Siyuan Huang, Xiaopei Zhu, Fuchun Sun, Bo Zhang, Xiaolin Hu
TL;DR
Physical adversarial patches can evade person detectors but may lose attack success as viewing angles change. The paper introduces expandable AdvTexture and the TC-EGA generative method to provide adversarial effectiveness across clothing areas. Printed garments made with AdvTexture successfully attacked person detectors in physical experiments, though transferability across detectors was limited.
Problem
Existing person-detector attacks can lose attack success when viewing angles increase because captured image regions may omit parts of the adversarial patches.
Method
The paper uses expandable AdvTexture generated by TC-EGA, which optimizes a local latent pattern and tiles it to produce larger textures for clothing.
Results
Printed T-shirts, skirts, and dresses made from AdvTexture were effective against person detectors in physical-world experiments across changing poses and viewing angles.
Takeaways & Limitations
AdvTexture provides a physical clothing-based attack designed to maintain detector-evasion effectiveness when garments change orientation or posture.
Takeaways & Limitations
Transferability from a texture targeting one detector to another detector was not very good.
Abstract
from arXiv · showhide
Nowadays, cameras equipped with AI systems can capture and analyze images to detect people automatically. However, the AI system can make mistakes when receiving deliberately designed patterns in the real world, i.e., physical adversarial examples. Prior works have shown that it is possible to print adversarial patches on clothes to evade DNN-based person detectors. However, these adversarial examples could have catastrophic drops in the attack success rate when the viewing angle (i.e., the camera's angle towards the object) changes. To perform a multi-angle attack, we propose Adversarial Texture (AdvTexture). AdvTexture can cover clothes with arbitrary shapes so that people wearing such clothes can hide from person detectors from different viewing angles. We propose a generative method, named Toroidal-Cropping-based Expandable Generative Attack (TC-EGA), to craft AdvTexture with repetitive structures. We printed several pieces of cloth with AdvTexure and then made T-shirts, skirts, and dresses in the physical world. Experiments showed that these clothes could fool person detectors in the physical world.
1. Introduction
Prior physical attacks on person detectors can fail when viewing angles change because cameras capture only fragments of adversarial patches. The paper proposes expandable AdvTexture and TC-EGA to maintain adversarial effectiveness across local clothing areas, and reports successful physical attacks with printed garments.
- Motivation: Physical adversarial patches can evade person detectors but may lose effectiveness when clothing is viewed from different angles.A single patch can suffer from the segment-missing problem when the camera captures only a deformed segment.
- Motivation: 3D rendering approaches also face difficulties modeling non-rigid clothing deformation, with attack success rate decreasing on unseen meshes.The cited example renders logos on flat areas of 3D human meshes.
- AdvTexture: AdvTexture can be generated in arbitrary sizes so every local clothing area captured by the camera retains adversarial effectiveness.This design directly targets the segment-missing problem rather than relying on one camera-facing patch.
- TC-EGA: TC-EGA uses a fully convolutional generator and toroidal cropping to optimize a local latent pattern that can be tiled into larger textures.The expandable latent representation lets the generator produce textures in multiple sizes.
- Physical evaluation: Printed AdvTexture garments, including dresses, T-shirts, and skirts, lowered detection performance and caused example persons to evade YOLOv2.The experiments realized the proposed textures in the physical world and evaluated multiple garment forms.
2. Related Work
Earlier work established physical attacks against classifiers and person detectors, but viewing-angle changes can reduce attack success when patches are not captured reliably. This motivates methods that address viewpoint variation in real-world deployments.
- Adversarial attack settings: Digital adversarial examples add small image perturbations, while physical attacks use manufactured objects to affect deployed vision systems.The related work describes both digital and physical attack settings.
- Prior methods: Prior physical attacks targeted face recognition, image classification, road-sign classification, and person detection using glasses, objects, patches, and clothing.Examples include robust 3D objects, neighboring patches, and adversarial T-shirts.
- Viewpoint limitation: Several person-detector attacks reported drops in attack success rate as viewing angles increased.The cited explanation is that parts of adversarial patches may not be captured when the camera rotates substantially.
- Viewpoint limitation: Uncaptured patch regions can underestimate the threat because cameras may be placed at arbitrary locations in real-world scenarios.This limitation makes viewpoint robustness relevant to practical deployment settings.
3. Methods
The method trains a generative adversarial-patch model and extends it into arbitrary-sized AdvTextures using a fully convolutional generator and toroidal cropping. Its objectives reduce detector confidence while encouraging smoothness and latent-variable diversity.
- Adversarial Patch Generator: AdvTexture generation begins by modeling adversarial patches with an energy-based distribution and approximating it using a parameterized generator Gφ.The target distribution assigns higher probability to patches with stronger adversarial effectiveness, while Gφ maps standard-normal latent variables to generated patches.
- Adversarial Objective Function: The adversary objective minimizes expected detector confidence after physical transformations and random patch placement on detected persons.Transformations include changes in scale, contrast, brightness, added noise, and thin-plate-spline deformation.
- Adversarial Objective Function: A total-variation term is combined with the detector objective to encourage smoother textures, with coefficients α and β controlling the combined energy.The energy is formed as 1/β(U_obj + αU_TV), and its components are minimized together.
- Adversarial Objective Function: An auxiliary network Tω maximizes mutual information between latent variables and generated patches, encouraging different latent variables to produce different patches.The information objective estimates joint and resampled latent-patch pairs during training.
- TC-EGA: TC-EGA uses a two-stage process: an FCN learns expandable adversarial patches, then latent representations are optimized to produce effective textures.The FCN uses convolutional layers with zero padding, while the second stage searches for a strong latent pattern.
- TC-EGA: Toroidal Cropping optimizes a local latent pattern that can be tiled into arbitrary shapes while preserving continuity across crop junctions.The local pattern is treated as an unfolded two-dimensional torus, enabling recursive crops for arbitrary-sized latent variables.
4. Experiment settings
The experiments evaluate AdvTexture and competing repeated textures on pedestrian detection, including physical tests with tailored garments worn in indoor and outdoor settings.
- Datasets and Physical Evaluation: Three subjects wore adversarial clothes and slowly turned in front of a camera indoors and outdoors to construct the physical test set.The subjects had a mean age of 24.0 years, ranged from 21–26, and included two males and one female.
- Datasets and Physical Evaluation: The study used the Inria Person dataset, with 614 training images and 288 test images, and evaluated patch-based attacks on its test set.Physical evaluation used separately collected videos and manually labeled frames.
- Datasets and Physical Evaluation: The physical videos yielded 192 manually labeled frames for each adversarial piece of clothing.Six videos were recorded from three subjects across two environments, with 32 frames extracted from each video.
- Compared Textures: The comparison set included AdvPatch, AdvTshirt, their tiled variants AdvPatchTile and AdvTshirtTile, and a repetitive random-color texture.The original adversarial patterns were copied from prior papers before tiling.
- Ablation Variants: TC-EGA was assessed through three component variants: EGA, TCA, and RCA, with RCA evaluated at 300 × 300 and 900 × 900 pixels.EGA omits latent optimization, TCA directly optimizes a toroidally cropped local texture, and RCA optimizes a fixed large patch.
- Target Detectors: AdvTexture was crafted primarily against YOLOv2, YOLOv3, Faster R-CNN, and Mask R-CNN pretrained on MS COCO.Detector outputs were filtered to the person class.
- Physical Implementation: The texture was digitally printed on polyester cloth and tailored into T-shirts, skirts, and dresses for physical implementation.The resulting garments were used in the physical evaluation.
5. Results
Digital and physical evaluations show that TC-EGA produces effective, expandable adversarial textures, with robustness across cropped positions and viewing angles. Physical effectiveness varies with viewing angle, clothing coverage, detector, and camera distance.
- Patch-Based Attack in the Digital World: 0.3521 AP was obtained by AdvPatch, but its non-expandable design limits attacks across multiple viewing angles.AdvPatch was slightly stronger digitally than TC-EGA, yet cannot generate arbitrary-size textures.
- Patch-Based Attack in the Digital World: TC-EGA achieved the lowest AP among resampled patches, reaching 0.362 on YOLOv2.Lower AP indicates a stronger attack.
- Patch-Based Attack in the Digital World: TC-EGA remained robust when cropped patches shifted, whereas AdvPatch weakened and tiled patches remained problematic.This evaluation directly examines the segment-missing problem.
- Attack in the Physical World: TC-EGA produced the lowest recall-confidence curve and AP among the evaluated physical attacks on YOLOv2.Lower recall at a fixed confidence threshold indicates stronger adversarial effectiveness.
- Attack in the Physical World: At most viewing angles, TC-EGA outperformed other methods; its mASR was approximately 1.0 at 0° and 180° but lower near 90° and 270°.The lower effectiveness near 90° and 270° was associated with the smaller area captured by the camera.
- Attack in the Physical World: Physical attack effectiveness increased with larger clothes, remained comparable indoors and outdoors, and dropped as camera distance increased.The larger-clothing effect was attributed to more texture area being captured by the camera.
6. Conclusions
The paper proposes AdvTexture for physical attacks on person detectors and reports effectiveness across turning and posture changes. It also acknowledges limited transferability across detectors and potential camera-security misuse.
- AdvTexture is physically implemented as T-shirts, skirts, and dresses for attacking person detection systems.The textures were printed on large cloth and used to make different garments.
- The garments remained effective when wearers turned around or changed postures in physical-world experiments.
- Transferability from a texture targeting one detector to another detector was not very good.The authors suggest model ensembles as a possible way to improve transferability.
- Adversarial research may lead to unwanted real-world applications, including camera security issues.The paper also notes that defense methods based on exposed vulnerabilities can improve security.
A.1. Proof of Theorem 1
The proof reformulates the generator objective by decomposing a KL divergence into mutual-information and energy terms. It then replaces mutual information with a Jensen-Shannon estimator and derives the resulting optimization objective.
- The KL divergence is decomposed into conditional-entropy and marginal-entropy terms, introducing mutual information into the objective.The derivation expresses the decomposition as −Hφ(τ̃|z) + Hφ(τ̃).
- Because τ̃ is determined by z through Gφ, the conditional entropy Hφ(τ̃|z) is constant and can be ignored.The supplied proof notes a distribution-dependent caveat for this constant.
- The remaining KL objective combines negative mutual information with the expected energy Eτ̃∼qφ(τ̃)[U(τ̃)].The partition-function term is constant with respect to the generator parameters.
- Mutual information is estimated with a Jensen-Shannon estimator involving a scalar neural-network function Tω, optimized jointly with φ.
A.2. Proof of Theorem 2
The proof shows that two generators with equivalent outputs and identical latent-variable distributions induce identical output distributions. It establishes this by equating their cumulative distribution functions.
- Equivalent generators G1 and G2 have outputs τ1 and τ2 with the same dimension K.
- Identical latent distributions for Z1 and Z2 allow corresponding output probability events to be represented by equal integrals.
- Equality of the output cumulative distribution functions proves that T1 and T2 are identically distributed.
A.3. Proof of Corollary 2.1
The proof formalizes how fully convolutional generators preserve local, translation-consistent computation across layers. It concludes that appropriately padded convolutions support expandable texture generation, with supplementary figures illustrating resulting textures and clothes.
- The FCN maps latent input z=v(0) to output texture τ=v(L) through convolution and element-wise activation at each layer.The proof defines the convolutional kernel, activation, feature map, and layerwise recurrence.
- A local rectangular region at one layer depends on the corresponding expanded region in the preceding layer through convolution.The preceding region sizes satisfy w(l−1)=w(l)+a(l)−1 and h(l−1)=h(l)+b(l)−1.
- Translation-equivalent local regions remain functionally equivalent through the convolutional and element-wise activation operations.
- Supplementary figures show additional adversarial textures and clothes, with physical-attack results generally obtained using adversarial T-shirts.
C. Results of attacking different detectors in the digital world
TC-EGA-generated AdvTextures reduced detector performance on digital Inria images across multiple detectors. The supplementary results also compare the resulting real-world clothes with clothes produced by other methods.
- AP on clean Inria test images was 1.0 for each evaluated detector, establishing the digital-world baseline.
- AdvTextures lowered the AP of evaluated detectors, although their effect was weaker than the YOLOv2 result with AP 0.362.
- The attacks lowered clean-image AP by half for the reported detectors.
D. Comparison between Indoor and Outdoor Conditions
The supplementary experiments examine adversarial clothing across indoor and outdoor scenes, camera distances, detector architectures, and transfer settings. Results indicate comparable indoor and outdoor effectiveness, distance-dependent measurements, and selective transfer across models.
- Indoor and Outdoor Conditions: Indoor mASR was comparable to outdoor mASR for each tested adversarial clothing item.
- Camera Distance: YOLOv2 T-shirt attacks were evaluated across seven camera distances ranging from 1.6 m to 7.0 m.The evaluation collected 192 frames at each distance across three persons and two scenes.
- YOLOv3 Branches: YOLOv3 uses three relatively independent prediction branches, so attacks can become biased toward one branch; input scaling was examined to address this issue.
- Transferability: Adversarial clothes made for YOLOv2 and YOLOv3 remained effective against each other, but generally had low mASRs against other models except RetinaNet.