PaperlayerView exact revision on arXiv

Source-linked AI summary

T-SEA: Transfer-based Self-Ensemble Attack on Object Detection

Hao Huang, Ziyan Chen, Huanran Chen, Yongtao Wang, Kevin Zhang

arXiv:2211.09773v1cs.CVcs.AI

TL;DR

Transfer-based black-box detector attacks avoid target-model queries, but existing approaches often rely on costly ensembles of suitable models. T-SEA optimizes a patch using one accessible detector and self-ensembles data, model, and patch. The authors report improved black-box transferability across multiple detectors and improved white-box performance.

  • Problem

    Existing transfer-based detector attacks often rely on costly ensembles of models, while suitable same-task models can be difficult to obtain.

    Source: §1 · p. 2

  • Method

    T-SEA adjusts an existing attack baseline and self-ensembles the input data, attacked model, and adversarial patch using one accessible detector.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    T-SEA improves both white-box and black-box performance over AdvPatch on eight detectors, with black-box average mAP around 10 for some white-box detectors such as YOLO v2 and Faster R-CNN.

    Source: §4.2.1 · p. 7

  • Takeaways & Limitations

    The experiments indicate that an attacker with only one model could still achieve a high-transferability black-box attack on multiple popular detectors.

    Source: §6 · pp. 8–9

    The evaluation focuses on CNN-based detectors, and model ShakeDrop cannot be directly applied to transformer-based detectors.

    Source: §5 · p. 8

Abstract

from arXiv · show

Compared to query-based black-box attacks, transfer-based black-box attacks do not require any information of the attacked models, which ensures their secrecy. However, most existing transfer-based approaches rely on ensembling multiple models to boost the attack transferability, which is time- and resource-intensive, not to mention the difficulty of obtaining diverse models on the same task. To address this limitation, in this work, we focus on the single-model transfer-based black-box attack on object detection, utilizing only one model to achieve a high-transferability adversarial attack on multiple black-box detectors. Specifically, we first make observations on the patch optimization process of the existing method and propose an enhanced attack framework by slightly adjusting its training strategies. Then, we analogize patch optimization with regular model optimization, proposing a series of self-ensemble approaches on the input data, the attacked model, and the adversarial patch to efficiently make use of the limited information and prevent the patch from overfitting. The experimental results show that the proposed framework can be applied with multiple classical base attack methods (e.g., PGD and MIM) to greatly improve the black-box transferability of the well-optimized patch on multiple mainstream detectors, meanwhile boosting white-box performance. Our code is available at https://github.com/VDIGPKU/T-SEA.

1. Introduction

Transfer-based attacks avoid querying target detectors, but existing methods often require costly ensembles of suitable models. T-SEA uses one accessible detector and self-ensembles the data, model, and patch to improve patch transferability.

1. Introduction

  • The paper motivates detector-attack research by noting that adversarial examples can disrupt detectors in digital and physical domains, threatening detector-based applications.It argues that further study of adversarial examples on detectors can help improve robustness.Source: §1 · p. 1
  • The paper frames patch optimization as analogous to regular model optimization and applies generalization-inspired self-ensembling to the input, attacked model, and patch.The proposed strategies are constrained data augmentation, model ShakeDrop, and patch cutout to address overfitting on particular models or images.Source: Figure 1 · p. 1, §1 · p. 2
  • T-SEA targets high-transferability black-box attacks on object detectors using only one attacked model.Existing model-ensemble approaches are laborious and costly, and suitable models for the same task can be difficult to find.Source: §1 · p. 2
  • Slight changes to the existing method’s training strategies form an enhanced baseline.The paper says this adjustment increases the baseline’s performance.Source: §1 · p. 2
  • T-SEA reduces mAP on multiple widely used black-box detectors compared with previous methods and works with multiple base attack methods.The abstract names PGD and MIM as examples of base attack methods.Source: §1 · p. 2

2. Related Work

The related work distinguishes query-based and transfer-based black-box attacks, surveys object-detector families and prior detector attacks, and positions this work as a single-detector transfer approach.

2.1. Black-box Adversarial Attack

  • Transfer-based attacks optimize on a substitute model without target-model knowledge, whereas query-based attacks use target outputs and may disclose attack behavior through frequent queries.The paper identifies limited transfer performance and reliance on multiple models as challenges for prior transfer approaches.Source: §2.1 · p. 2

2.2. Object Detectors

  • The paper selects eight mainstream one-stage and two-stage detectors to evaluate T-SEA, including YOLO, Faster R-CNN, and SSD variants.One-stage detectors directly predict object positions and classes; two-stage detectors first generate proposals with an RPN.Source: §2.2 · p. 3

2.3. Attacks on Object Detector

  • Prior detector attacks include iterative gradient-based methods, physical-world adversarial patches, and ensembles attacking multiple detectors; this work instead uses one detector for transfer attacks.The cited examples are [36], [19] [33], [7] [31], and [16] [34] [40], respectively.Source: §2.3 · p. 3

3. Method

The method section proceeds from problem formulation and the overall framework to an enhanced baseline and self-ensemble strategies.

3. Method

  • The method is organized around problem formulation, the overall framework, an enhanced baseline, and self-ensemble strategies.These topics are assigned to Sections 3.1, 3.2, 3.3, and 3.4, respectively.Source: §3 · p. 3

3.1. Problem Formulation

The formulation optimizes a universal adversarial patch using one white-box detector and input images, with the objective evaluated over self-ensembled data, model, and patch.

3.1. Problem Formulation

  • The objective seeks a universal patch that disrupts detection using one pre-trained white-box detector and training images.Its loss measures detector corruption over self-ensembled data, model, and patch; the regularizer is total variation of the patch.Source: §3.1 · p. 3

3.2. Overall Framework

T-SEA trains a patch through constrained image augmentation, target-object placement, patch cutout, and ShakeDrop model variants, then applies the optimized patch to test images against black-box detectors.

3.2. Overall Framework

  • Training first augments input images, locates target objects with the white-box detector, and places patches at their centers with patch cutout.The augmented, patched images then pass through ShakeDrop models while object confidence is minimized until training ends.Source: §3.2 · p. 3
  • The training pipeline optimizes the adversarial patch on ShakeDrop model variants until the maximum epoch is reached.The optimization minimizes object confidence on patched images.Source: §3.2 · pp. 3–4
  • At attack time, the optimized patch is applied to test images to disrupt multiple black-box detectors.Source: §3.2 · p. 4

3.3. Enhanced Baseline

The enhanced baseline iteratively places a patch on detected target objects, evaluates object confidence, and updates the patch, with revised learning-rate scheduling and a smaller training patch scale.

3.3. Enhanced Baseline

  • Each epoch batches training images, detects objects with the white-box detector, applies the patch, measures adversarial object confidence, and updates the patch.The baseline uses an attack loss and adjusts the learning rate after each epoch.Source: §3.3 · p. 4
  • A plateau-based scheduler can sharply lower the learning rate at a saddle point, so the baseline adjusts its decline strategy for more stable decreases.The scheduler uses the decay factor μ, mean loss ℓ_t, and thresholds ε1 and ε2 to control updates.Source: §3.3 · p. 4
  • Reducing training patch scale helps the optimized patch learn more global patterns that are less disrupted when the patch is scaled.Patch scale is the ratio of patch length to the target object's bounding-box length.Source: §3.3 · p. 4

3.4. Self-Ensemble Strategies

T-SEA adapts model-generalization strategies to patch optimization, virtually expanding its training inputs through constrained data and model variation while regularizing the patch with cutout.

3.4.1 Theoretical Analysis

  • Patch optimization treats the detector as an input, shifting the relevant distribution to data-model pairs; more images and models could improve generalization, but are costly or impractical to obtain.The authors also propose patch cutout to reduce patch capacity and alleviate overfitting.Source: §3.4.1 · pp. 4–5

3.4.2 Data: Constrained Data Augmentation

  • Constrained data augmentation virtually expands the training set with mild resizing and cropping, small color changes, and limited random rotations to avoid unnatural scenes.Source: §3.4.2 · p. 5

3.4.3 Model: ShakeDrop

  • ShakeDrop generates attacked-model variants by linearly combining a residual block's stacked-layer output with its identity path.In forward propagation, the combination is controlled by sampled β and α values.Source: §3.4.3 · p. 5

3.4.4 Patch: Cutout

  • Patch cutout randomly masks a square region of the adversarial patch during training, reducing reliance on any one patch area and helping prevent overfitting.The mask is applied with probability ϕc and covers an ηH × ηW area centered on a sampled point.Source: §3.4.4 · p. 5

4. Experiment

The experiments evaluate T-SEA against multiple detectors, base attack methods, and existing detection attacks, alongside implementation details and ablations. Across these comparisons, T-SEA improves attack performance in white-box and black-box settings.

4.1. Implementation Details

  • Experiments train and test patches on INRIA person images, then use COCO-person and CCTV-person as additional test sets.INRIA has 614 training and 288 test images; COCO-person has 1,684 images and CCTV-person has 559.Source: §4.1 · p. 6
  • The evaluation measures attack capability with mAP, treating clean-data detector predictions as ground truth; lower mAP indicates a stronger attack.The clean-data reference is assigned mAP = 1.Source: §4.1 · p. 6

4.2.1 Results on Different Attack Models

  • T-SEA significantly improves white-box and black-box performance over AdvPatch across eight detectors, with black-box average mAP around 10 for some white-box detectors.The comparisons attack each detector separately and use the other seven as black-box detectors.Source: Table 1 · p. 6, §4.2.1 · p. 7

4.2.2 Results of Different Attack Methods

  • Across Adam, SGD, MIM, BIM, and PGD on YOLO v5, T-SEA improves white-box and black-box performance over AdvPatch.The reported gains are not limited to one base attack method.Source: Table 2 · p. 6, §4.2.2 · p. 7

4. Experiment

  • The ablation adds each self-ensemble strategy to the E-baseline on YOLO v5; data augmentation alone lowers white-box performance, while all strategies improve black-box performance.Combining all strategies yields results on both white-box and black-box settings.Source: Table 4 · p. 7

4.3. Comparison with SOTA Methods

  • Under the same perturbed-area inference setting, E-baseline and T-SEA achieve the highest performance among the compared adversarial approaches.The comparison also includes gray, random-noise, and white-patch controls.Source: §4.3 · p. 7, Table 3 · p. 7

4.4. Ablation Study

The ablations examine training adjustments and self-ensemble strategies, then test cross-dataset transfer and a physical demonstration. The results report black-box gains across strategies and datasets, plus a limited physical example.

4.4.1 Training Strategies Adjustment

  • Reducing training patch scale accelerates test-set mAP reduction, while the adjusted scheduler further decreases detection loss; combining adjustments improves results.A suitable smaller scale is associated with learning more global patterns that are less disrupted by scaling.Source: §4.4.1 · p. 7, Figure 4 · p. 8

4.4.2 Self-Ensemble Strategies

  • Each self-ensemble strategy improves black-box results; data augmentation alone slightly reduces white-box performance, while combining the strategies achieves the best performance.The strategies augment input data, the attacked model, and the training patch.Source: §4.4.2 · p. 8

4.5. Cross-Dataset Verification

  • On COCO-person and CCTV-person, the INRIA-trained T-SEA patch has comparable white-box results to AdvPatch but stronger black-box attack capability.The two test sets cover varied scenes and security-camera footage, respectively.Source: §4.5 · p. 8, Table 5 · p. 8

4.6. Physical Verification

  • In a simple physical demonstration, an iPad-displayed patch prevents YOLO v5 and SSD from detecting a person without disrupting detection of other objects.The authors state that physical-world attack is not the work’s main goal.Source: §4.6 · p. 8, Figure 5 · p. 8

5. Future Work

The future work is limited by the study’s focus on CNN-based detectors and the incompatibility of model ShakeDrop with transformers.

5. Future Work

  • The strategies are studied on CNN-based detectors; model ShakeDrop cannot directly be applied to transformers, although the other strategies can.Future work proposes designing a new approach for transformer-based detectors.Source: §5 · p. 8

6. Conclusion

The paper concludes that T-SEA improves transfer-based black-box attacks using one attacked model, with gains across base attack methods and popular detectors.

6. Conclusion

  • T-SEA combines slight training-strategy adjustments with self-ensemble of input data, the attacked model, and the training patch to improve patch transferability.The conclusion reports that an attacker with only one model could still succeed in a high-transferability black-box attack.Source: §6 · pp. 8–9

Related papers

Loading 2211.09773v1…