Source-linked AI summary
Physical Adversarial Examples for Person Detectors in Thermal Images Based on 3D Modeling
Xiaopei Zhu, Siyuan Huang, Zhanhao Hu, Jianmin Li, Jun Zhu, Xiaolin Hu
TL;DR
The paper addresses limited security research on physical adversarial examples for thermal infrared imaging and the simulation gap between 2D designs and physical clothing. It develops infrared 3D human and clothing models, optimizes black-patch layouts, and manufactures aerogel-based adversarial clothing for multiangle detector attacks.
Problem
Physical adversarial examples in thermal infrared imaging have attracted little attention, while directly converting RGB 3D models creates a large gap from real infrared images.
Method
The method builds infrared 3D human and clothing models from real infrared photos, optimizes black-patch positions and orientations, and implements the pattern with aerogel insulation patches.
Results
The paper presents a 3D-model-based infrared physical attack method and demonstrates adversarial clothing that can hide from infrared detectors at multiple angles.
Takeaways & Limitations
The proposed approach combines more realistic infrared modeling with physically implementable clothing to study attacks on infrared person detectors.
Takeaways & Limitations
Attack effectiveness decreases when clothing is far from the camera or when the adversarial pattern is severely occluded.
Abstract
from arXiv · showhide
Thermal Infrared detection is widely used in autonomous driving, medical AI, etc., but its security has only attracted attention recently. We propose infrared adversarial clothing designed to evade thermal person detectors in real-world scenarios. The design of the adversarial clothing is based on 3D modeling, which makes it easier to simulate multiangle scenes near the real world compared to 2D modeling. We optimized the black patch layout pattern of 3D clothing based on the adversarial example technique and made physical adversarial clothing using the aerogel. The idea is to paste a set of square aerogel patches, which display black squares in thermal images, in the inner side of clothing at specific locations with specific orientations. To enhance realism, we propose a method to build infrared 3D models with real infrared photos and develop texture maps for 3D models to simulate varied infrared characteristics over time and location. In physical attacks, we achieved an attack success rate of 80.11\% indoors and 76.85\% outdoors against YOLOv9. In contrast, randomly placed patches yielded much lower success rates (26.53\% indoors and 23.03\% outdoors). The adversarial clothing also showed good transferability to unknown detectors with an ensemble attack method, demonstrating the effectiveness of our approach.
1 INTRODUCTION
The paper addresses limited attention to physical adversarial examples in thermal infrared detection and the restricted viewing angles of prior 2D designs. It proposes 3D infrared clothing models and optimized black-patch patterns to better simulate and attack detectors across viewpoints.
- Motivation: Thermal infrared adversarial-example research has received limited attention despite infrared imaging’s use in autonomous driving, medical diagnosis, and temperature measurement.Infrared imaging provides temperature information, tolerates occlusion, and works without external light sources.
- Problem: Prior 2D patches do not cover an object’s full 3D surface and may fail outside narrow viewpoints, such as when a person is viewed from the side.The cited adversarial T-shirt is described as effective only from the front.
- Approach: 3D modeling enables adversarial pattern design from different angles and more realistic simulation of physical attacks than 2D images.The paper contrasts 3D models’ resemblance to real-world objects with 2D representations.
- Novelty: The study differs from earlier 3D work by using differentiable rendering, attacking across 0-360 degrees, and targeting object detection rather than classification.Earlier studies used nondifferentiable rendering, narrower angular ranges, or classifier attacks.
- Contribution: The paper builds infrared 3D models from real infrared photos and varied texture maps to reduce the gap between digital simulation and physical implementation.The models represent changing infrared clothing characteristics across times and places.
2 RELATED WORKS
Related work covers physical adversarial attacks in visible, infrared, and LiDAR domains, including printed patterns, thermal devices, insulation materials, and 3D attacks. The review identifies full-view 3D infrared attacks as an underexplored area.
- Physical attacks: Physical attacks extend adversarial examples into real-world settings, while infrared physical attacks have attracted attention only in recent years.Most physical-attack research has focused on visible light.
- 2D physical attacks: Visible-light physical attacks use printed RGB patterns on papers, stickers, or clothing to disrupt recognition or detection.Examples include adversarial papers, stop-sign stickers, and adversarial clothing.
- Infrared attacks: Infrared attacks use modeled thermal characteristics and physical implementations based on heating or cooling devices and thermal-insulation materials.Examples include bulbs, carbon heaters, warming or cooling paste, low-E films, and infrared patches.
- 3D attacks: 2D adversarial attacks usually work within narrow angular ranges, whereas 3D attacks can cover the full 0-360° viewpoint range.The review presents angular coverage as a key distinction between 2D and 3D attacks.
- Research gap: Existing 3D adversarial attacks mainly target visible and LiDAR domains, leaving 3D infrared attacks insufficiently explored.Prior examples include 3D shoes, cars, logos, vehicle camouflage, point-cloud perturbations, and geometric attacks.
- Stealth materials: Traditional infrared stealth materials include iron, zinc, and aluminum composites, whose low emissivity is linked to electrical conductivity but whose performance can depend on surface topography.The review notes later efforts to address metal-composite limitations with semiconductor materials.
3 METHODS
The method builds infrared 3D human and clothing models from real infrared photos, simulates changing conditions, and optimizes multiangle black-patch patterns for physical implementation. Its pipeline connects realistic digital rendering with physical adversarial clothing construction and testing.
- 3.1 Building 3D infrared models with real infrared photos: The pipeline combines digital optimization with physical testing, using 3D human and clothing meshes to simulate people wearing adversarial clothing.The clothing mesh provides the optimization target, while the human model and rendered clothing are combined for scene generation.
- 3.1 Building 3D infrared models with real infrared photos: Real infrared photos are mapped onto unfolded 3D mesh faces to create textured clothing models and reduce the digital-to-real infrared domain gap.The process uses faces maps, region-specific cropped images, texture-map assembly, and Pytorch3D rendering.
- 3.3 3D infrared adversarial clothing model: Black patches represent thermal insulation areas whose positions and orientations define the adversarial pattern on the 3D clothing surface.Lower infrared pixel values represent lower temperatures, so insulated regions appear black in thermal images.
- 3.3 3D infrared adversarial clothing model: Infrared texture maps are captured across day/night and indoor/outdoor conditions, then randomly changed during optimization and testing to model environmental variation.This procedure represents changing infrared clothing characteristics across times and places.
- 3.5 Optimization of adversarial pattern in 3D infrared adversarial clothing: Realistic physical variation is modeled by perturbing patch coordinates, orientations, and black-patch pixel values during optimization.These perturbations simulate movement, ambient-temperature changes, and diverse real-clothing textures.
- 3.5 Optimization of adversarial pattern in 3D infrared adversarial clothing: The optimization loss increases average pairwise patch distance within a bounded range, reducing overlap while preventing patches from spreading excessively.Patch centers are constrained to valid mapped regions on the clothing model.
4 EXPERIMENTS
The experiments use FLIR ADAS 1.3 infrared imagery, captured with a FLIR Tau2 camera, for training and testing object detection.
- 10,228 infrared images comprise the FLIR ADAS 1.3 dataset, split into 7,160 training and 3,068 test images.Images were captured on highways and streets in Santa Barbara, USA, from November to May.
4.2 Target detectors
YOLOv9 is the primary target detector, fine-tuned on FLIR ADAS 1.3, with transferability evaluated on multiple unseen detectors.
- YOLOv9 is the primary target detector and was fine-tuned from its official pre-trained model on FLIR ADAS 1.3.The fine-tuned detector achieved person-category average precision of 0.98 on training data and 0.95 on test data.
- Transferability was evaluated against DETR, DINO, DDQ, DAB-DETR, YOLOv7, YOLOv8, YOLOv11, and other unseen detectors.
4.3 Evaluation Metrics
Attack success rate is evaluated using person annotations as ground truth and intersection over union to assess detection accuracy.
- Attack success rate (ASR) is the evaluation metric for digital and physical attacks.
- Ground truth consists of the annotated 3D person model for digital attacks or real persons for physical attacks.
- Intersection over union (IOU) is used to calculate detection accuracy while accounting for other persons in background images.
4.4 Simulation of physical attacks
The study simulates multiangle physical attacks by rendering optimized black-patch clothing on 3D people across varied scenes, distances, and infrared conditions. The optimized design outperforms clean and random controls and generalizes across datasets.
- Simulation setup: 36 black patches were distributed across the clothing front, back, and sides for the main simulation experiments.The layout used 12 patches on the front, 12 on the back, and 6 on each side.
- Simulation setup: 0–360° viewing angles and 2.5–5 m camera distances were simulated by rendering clothed 3D people onto randomly selected FLIR backgrounds.
- Digital attack results: The optimized pattern achieved 86.38% ASR against YOLOv9, compared with 38.19% for random patches and 4.65% for clean clothing.
- Ablation studies: ASR increased and then decreased as patch count rose from 6 to 60, with 36 patches producing the best attack effect.
- Ablation studies: ASR increased and then decreased with patch size, making 6 cm × 6 cm the best tested size.
- Threshold analysis: ASR increased with detection threshold, while adversarial patterns remained higher than random and clean patterns across thresholds.The authors identify approximately 0.7–0.75 as typical threshold settings, while noting recall and false-detection trade-offs.
- Cross-dataset evaluation: The method achieved similar ASRs on the KAIST and FLIR ADAS v2 datasets, indicating generalizability across datasets.
4.5 Evaluation of attacks in the physical world
Physical tests show that aerogel-based adversarial clothing substantially reduces YOLOv9 person detection across indoor, outdoor, angular, distance, environmental, temporal, and posture conditions.
- Overall physical attack: 80.11% indoor and 76.85% outdoor ASR against YOLOv9 exceeded random-pattern clothing at 26.53% and 23.03%, respectively.Ordinary clothing produced 7.12% indoor and 8.38% outdoor ASR.
- Angle robustness: ASR remained above 77% across camera angles from 0-360 degrees.Front and back views outperformed side views, attributed to the smaller adversarial area on the clothing side.
- Distance robustness: ASR exceeded 78% at distances from 3-8 meters but declined beyond 8 meters.
- Weather and landscape robustness: ASR exceeded 74% across sunny, cloudy, foggy, urban-road, hillside, and riverside conditions.Training simulations included noise, temperature variation, clothing-texture changes, and background switching.
- Temporal stability: 77.83% ASR remained after five hours despite a slight decline over time.The authors associate this stability with aerogel’s stable thermal insulation properties.
- Posture robustness: Standing poses outperformed sitting poses, while crossed arms caused a 20.27-21.32% ASR drop compared with aligned postures.Lifted or spread arms caused smaller 2.21-7.84% decreases.
4.6 Ensemble attacks
Single-model optimization transferred poorly to unknown detectors, whereas ensemble optimization produced substantially higher digital and physical-world transferability.
- Single-model transferability: 29.75%-58.36% ASRs showed limited transfer from a YOLOv9-optimized pattern to nine other detectors in the digital world.
- Ensemble transferability: 81.03%, 70.49%, 75.20%, 73.42%, 78.13%, 80.28%, 76.01%, 72.57%, and 70.35% ASRs were achieved across nine detectors by the ensemble-optimized pattern.The ensemble integrated Deformable DETR, YOLOv9, and Faster-RCNN.
- Physical-world validation: Physical ASRs against all evaluated detectors were higher for ensemble optimization than for single-model YOLOv9 optimization.
- Rationale: Ensemble optimization can reduce overfitting to one detector and capture disruptive features shared across models.The authors attribute this to differences in architecture, training data, and loss functions across detectors.
4.7 Adversarial defense methods
The evaluated defenses provided limited protection against the proposed digital 3D infrared attack.
- Defense evaluation: Seven categories of defense methods were evaluated, including detector-specific defenses, augmentation, adversarial training, masking, squeezing, smoothing, and TVM.
- Defense evaluation: 11.28% was the largest ASR reduction, achieved by adversarial training from 86.38% to 75.10% on YOLOv9.The attack still retained 75.10% ASR after this defense.
5 CONCLUSION
The paper develops infrared 3D models and aerogel-based adversarial clothing that hide people from infrared detectors across multiple physical conditions and improve transferability through model ensembles.
- Method: The method builds infrared human and clothing models from real infrared photos and varied texture maps, then optimizes black-patch positions and orientations.
- Physical attack: Aerogel-based clothing can appear ordinary in visible light while hiding from infrared detectors at multiple angles and scenes.
- Transferability: Model ensembles improve attack transferability to unknown detectors.
6 LIMITATION
The adversarial clothing has reduced effectiveness when viewed from far away or when its pattern is severely occluded. These limitations arise from small-scale image blurring and mismatch between physical occlusion and digitally optimized patterns.
- At small image sizes, clothing placed too far from the camera is difficult to use against infrared detectors.The adversarial images become blurry when scaled to very small sizes.
- Severe occlusion substantially decreases the physical adversarial pattern’s attack effect.Occlusion causes the physical pattern to deviate greatly from the pattern optimized digitally.