PaperlayerView exact revision on arXiv

Source-linked AI summary

Fooling thermal infrared pedestrian detectors in real world using small bulbs

Xiaopei Zhu, Xiao Li, Jianmin Li, Zheyao Wang, Xiaolin Hu

arXiv:2101.08154v2cs.CV

TL;DR

Thermal infrared detector security is underexplored, and physical attacks cannot be realized by simply printing digital infrared images. The paper uses small-bulb boards to attack thermal infrared pedestrian detectors, demonstrating real-world attacks on YOLOv3 and transfer to unseen detectors.

  • Problem

    Thermal infrared detection systems are widely used, but their security has received little research, and digital adversarial infrared images cannot simply be printed for physical attacks.

    Source: Introduction · pp. 1–2

  • Method

    The paper designs small-bulb boards and Gaussian-function patches to create physical infrared attacks against pedestrian detectors.

    Source: Introduction · p. 2, The Gaussian Functions Patch · p. 4

  • Results

    The physical board successfully fooled the YOLOv3-based infrared pedestrian detector, and ensemble training produced a board that fooled detectors unknown to the authors.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The work demonstrates that real-world physical attacks on thermal infrared pedestrian detection are possible.

    Source: Conclusion · p. 7

    The pixel-level patch contained numerous grayscale noise pixels that were difficult to realize physically, while the Gaussian patch was less effective but easier to realize.

    Source: Pixel-Level Patch Attack · p. 5, Gaussian Functions Patch Attack · p. 5

Abstract

from arXiv · show

Thermal infrared detection systems play an important role in many areas such as night security, autonomous driving, and body temperature detection. They have the unique advantages of passive imaging, temperature sensitivity and penetration. But the security of these systems themselves has not been fully explored, which poses risks in applying these systems. We propose a physical attack method with small bulbs on a board against the state of-the-art pedestrian detectors. Our goal is to make infrared pedestrian detectors unable to detect real-world pedestrians. Towards this goal, we first showed that it is possible to use two kinds of patches to attack the infrared pedestrian detector based on YOLOv3. The average precision (AP) dropped by 64.12% in the digital world, while a blank board with the same size caused the AP to drop by 29.69% only. After that, we designed and manufactured a physical board and successfully attacked YOLOv3 in the real world. In recorded videos, the physical board caused AP of the target detector to drop by 34.48%, while a blank board with the same size caused the AP to drop by 14.91% only. With the ensemble attack techniques, the designed physical board had good transferability to unseen detectors. We also proposed the first physical multispectral (infrared and visible) attack. By using a combination method, we successfully hide from the visible light and infrared object detection systems at the same time.

Introduction

Thermal infrared detectors offer useful capabilities, but their security is underexplored and physical attacks are difficult to realize by printing. The paper introduces a small-bulb board for attacking real-world thermal infrared pedestrian detectors.

Introduction

  • Prior physical attacks fooled DNNs in object and person detection, raising risks for widely deployed deep-learning security systems (Athalye et al. 2018; Simen et al. 2019; Xu et al. 2019).Thermal infrared detection has also progressed with deep learning.Source: Introduction · pp. 1–2
  • Thermal infrared detection safety is underexplored despite its night operation, obstacle penetration, and temperature information.The paper notes that infrared systems are used in surveillance, autonomous driving, and body-temperature detection.Source: Introduction · p. 1
  • Infrared images have one grayscale channel and less texture than visible images, and digital adversarial infrared images cannot simply be printed.This makes physical infrared attacks harder to realize than printing adversarial patterns for visible images.Source: Introduction · p. 2
  • A hand-held cardboard board decorated with small bulbs realizes a physical attack on thermal infrared pedestrian detectors; the setup costs less than 5 US dollars.The board uses eight 4.5V batteries, and the paper presents this as the first physical attack on a thermal infrared pedestrian detector.Source: Introduction · p. 2

Related Work

Related work covers digital adversarial perturbations and physical attacks, while the paper situates its target among object detectors and thermal pedestrian-detection studies.

Digital Attacks in Visible Light Images

Physical Attacks in Visible Light Images

  • Prior work described wearable glasses attacking facial recognition, perturbed road signs fooling recognition in practical driving, 3D adversarial examples, an invisible mask attacking face recognition, and a patch hiding a person from a person detector.Source: Physical Attacks in Visible Light Images · p. 2

Object Detection

  • Object detection spans one-stage YOLOv3 and RetinaNet, two-stage Faster R-CNN, and multi-stage Cascade R-CNN; thermal studies compare standard detectors (Kristo, Ivasic-Kos, and Pobar 2020).YOLOv3 is described as fast and accurate, while Cascade R-CNN uses detectors trained with increasing IOU thresholds.Source: Object Detection · p. 2

Generating Infrared Adversarial Patches

The section formulates an infrared patch attack around reducing detector objectness, then moves from a pixel-level digital patch to Gaussian-shaped bulb patterns and an adjustable physical board. It also incorporates patch transformations and an ensemble objective to target varied pedestrians and detectors.

Problem Formulation

  • The attack minimizes detector objectness across pedestrians and patch transformations, with total variation encouraging smoother patches and an ensemble loss targeting multiple detectors.The stated loss combines objectness and total variation using an empirically determined weight.Source: Problem Formulation · pp. 2–3

The Pixel-Level Patch

  • The pixel-level experiment adapts a visible-light patch method by converting RGB images to grayscale before optimization.The passage poses physical realizability as the question motivating this experiment, but gives no result here.Source: The Pixel-Level Patch · p. 3

The Gaussian Functions Patch

  • A single bulb’s temperature profile was fitted with a Gaussian, and multiple bulbs produce a patch composed of 2D Gaussian spots on a constant background.Temperatures along other measured sections were nearly the same as along the horizontal section.Source: The Gaussian Functions Patch · pp. 3–4, Figure 3 · p. 4
  • Gaussian bulb patterns reduce optimization parameters nearly 1000-fold by optimizing spot centers while fixing amplitude and standard deviation to measured values.The measured Gaussian fit had RMSE 0.1511, amplitude 10.62, and standard deviation 70.07; pixel values were reported not to change with distance.Source: The Gaussian Functions Patch · pp. 3–4

Physical Board Attack

  • Magnets make bulb positions adjustable on the board, while parallel circuits with individual switches and rheostats let bulbs operate independently.The board’s measured total power did not exceed 22 W.Source: Physical Board Attack · p. 4

Experiments and Results

Experiments use a filtered FLIR thermal-pedestrian dataset and a YOLOv3 target detector to evaluate digital patches. The Gaussian-functions patch provides a physically realizable alternative to a pixel-level patch, with attack performance assessed against control patches.

Preparing the Data

  • The authors selected 1,011 FLIR person select images containing pedestrians taller than 120 pixels, splitting them into 710 training and 301 test images.The source data were manually annotated thermal images filtered to retain people.Source: Preparing the Data · p. 4

Target Detector

  • YOLOv3 was selected as the target detector for its speed and performance; after fine-tuning, its test-set AP was 0.8522.The model used 416 × 416 inputs and official YOLO pretraining weights.Source: Target Detector · pp. 4–5

Pixel-Level Patch Attack

  • 74.57%: the pixel-level patch reduced YOLOv3 accuracy, but its numerous grayscale noise pixels were difficult to realize physically, so the authors abandoned it.The patch followed the process described by Simen et al. (2019).Source: Pixel-Level Patch Attack · p. 5

Gaussian Functions Patch Attack

  • Gaussian-function patches were trained with random noise, rotation, translation, brightness, and contrast transformations to improve robustness to physical-world perturbations.The patch was placed on pedestrians’ upper bodies at one-fifth of bounding-box height; stochastic gradient descent optimized Gaussian parameters.Source: Gaussian Functions Patch Attack · p. 5
  • 64.12%: the Gaussian-functions patch reduced YOLOv3 AP, versus 25.05% for random noise and 29.69% for a blank patch.The study evaluated the optimized patch on the test set with transformations, using clean-image detections as ground truth.Source: Gaussian Functions Patch Attack · p. 5, Figure 7 · p. 6
  • The Gaussian-functions patch with 22 spots provided a good attack effect while retaining a small number of parameters.The authors compared patches with different numbers of Gaussian functions in Table 1.Source: Gaussian Functions Patch Attack · p. 5, Table 1 · p. 5

The Effects of Patch Size

Patch size affected the digital attack, while physical experiments tested the bulb-board design against YOLOv3. In recorded videos, the board reduced AP more than a same-sized blank board.

The Effects of Patch Size

  • 95.42%: doubling the patch size caused YOLOv3 AP to drop; halving the original size substantially weakened the attack and marked its size limit.The original patch was 300 × 300 pixels, and five sizes were tested.Source: The Effects of Patch Size · p. 6, Figure 8 · p. 6

Evaluation of Attacks in the Real World

Ensemble Attack

The authors trained a Gaussian patch using an ensemble of YOLOv3, Faster-RCNN, and Mask-RCNN. The ensemble patch reduced AP for Cascade-RCNN and RetinaNet in both digital and physical tests.

Ensemble Attack

  • Ensemble training combined YOLOv3, Faster-RCNN, and Mask-RCNN to produce a Gaussian patch.The resulting patch is shown in Figure 12.Source: Ensemble Attack · p. 7, Figure 12 · p. 7
  • A patch trained on YOLOv3 alone reduced Cascade-RCNN and RetinaNet AP by 11.60% and 25.86%, respectively.These were the initial transfer results before ensemble training.Source: Ensemble Attack · p. 7
  • In physical tests, the ensemble patch reduced AP by 21.67% for Cascade RCNN and 25.17% for RetinaNet.In digital tests, the corresponding AP drops were 35.28% and 46.95%.Source: Ensemble Attack · p. 7

Conclusion

The paper demonstrates real-world attacks on infrared pedestrian detectors, including attacks on detectors unknown to the authors using an ensemble technique.

Conclusion

  • The physical bulb-decorated cardboard attacked YOLOv3, while ensemble training produced a board capable of fooling detectors unknown to the authors.The paper presents this work as practically significant given thermal infrared detection's use in night security, automatic driving, and body-temperature detection.Source: Conclusion · p. 7

Supplementary Material:

The supplied passage is only a fragment of the paper title, not substantive supplementary material.

Supplementary Material:

Details of the pixel-level adversarial patch

The pixel-level patch is optimized as a grayscale image and trained with transformations intended to improve robustness. On the FLIR person select test set, it reduced detector AP more than random-noise and blank patches.

Details of the pixel-level adversarial patch

  • The patch used a 300×300 grayscale image with each pixel optimized as a variable, initialized uniformly to 0.5.The authors report better network convergence with this initialization than with random initialization.Source: attack in the digital world · p. 10
  • Training placed the patch on pedestrians' upper bodies at one-fifth of bounding-box height and used noise, rotation, translation, brightness, and contrast transformations.Rotations were within 20 degrees; the authors intended the transformations to simulate physical-world perturbations.Source: attack in the digital world · p. 10
  • 74.57%: the pixel-level patch's AP drop on the FLIR person select test set, versus 25.30% for random-noise and 29.27% for blank patches.The evaluation and control experiments are illustrated in Figures S2 and S3.Source: attack in the digital world · p. 10, Figure S2 · p. 10, Figure S3 · p. 10

Influence of the pixel value of the blank patch

Blank patches with pixel values from 0.1 to 0.9 caused similar YOLOv3 AP drops, so later experiments used a value of 0.75.

Influence of the pixel value of the blank patch

  • 30% ± 5%: blank patches caused YOLOv3 AP to drop across the tested pixel values, with no significant variation between values.The tested values were 0.1, 0.25, 0.5, 0.75, and 0.9; Figure S4 shows the evaluation.Source: to the detection performance · p. 11, Figure S4 · p. 11
  • The blank patches used a single pixel value throughout, varied from 0 to 1 in the study.The five selected values were 0.1, 0.25, 0.5, 0.75, and 0.9.Source: to the detection performance · p. 11
  • The authors chose 0.75 as the typical blank-patch pixel value for subsequent experiments.This choice followed the finding that detection performance did not vary significantly across the tested values.Source: to the detection performance · p. 11

Attack the visible light and infrared object

A printed adversarial patch placed over a small-bulb board successfully attacked YOLOv3 on visible-light and thermal-infrared images in the same test. The authors describe this as the first physical multispectral attack.

detection systems at the same time

  • The physical setup placed a 29.8 cm × 29.8 cm printed adversarial patch on a board with small bulbs.The bulbs were covered by the printed paper in the physical-board image.Source: detection systems at the same time · p. 11, Figure S5 · p. 11
  • The test successfully attacked YOLOv3’s visible-light and infrared object detection systems at the same time, which the authors identify as the first physical multispectral attack.Figure S6 provides example images; visible-light faces are blurred for privacy.Source: detection systems at the same time · p. 11, Figure S6 · p. 11
  • Participants were imaged with a visible-light camera and a thermal infrared camera under the same conditions, using an adversarial board, a blank board, or no board.The resulting images were input to YOLOv3.Source: detection systems at the same time · p. 11

Related papers

Loading 2101.08154v2…