PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Attack Difficulty Be Characterized Before Optimization? A Study of Pre-optimization Difficulty in Person-Vanishing Attacks

Jingyao Xu, Dongdong Wang, Siyang Lu

arXiv:2609.34501v1cs.CV

TL;DR

Person-vanishing attack difficulty is usually observed only after optimization, leaving open whether relative difficulty can be characterized beforehand. The paper proposes Quad-CLEVER to estimate it from clean inputs and use it for adaptive budget allocation. Across attack algorithms, the estimate correlates with observed optimization cost, and the framework reports higher efficiency with competitive performance on two benchmarks.

  • Problem

    Person-vanishing studies largely treat attack difficulty as an optimization outcome, leaving comparatively little attention to whether relative difficulty can be characterized from clean inputs beforehand.

    Source: Introduction · p. 1

  • Method

    Quad-CLEVER estimates relative difficulty from clean inputs using a quadratic approximation of the local person-vanishing margin, and the framework adaptively allocates optimization budgets.

    Source: Introduction · pp. 1–2

  • Results

    Across multiple attack algorithms, Quad-CLEVER consistently correlates with observed optimization cost, and the framework achieves higher attack efficiency with competitive performance on two benchmarks.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The results indicate that person-vanishing attack difficulty is not solely an outcome of a particular optimization process.

    Source: Conclusion · p. 7

    Evaluation uses a white-box setting with an ℓ2 perturbation constraint and judges success by suppressing matched person detections after the complete detection pipeline.

    Source: Experiment Setup · p. 5

Abstract

from arXiv · show

Adversarial attacks against object detectors are traditionally studied from an optimization perspective, where attack difficulty is regarded as an outcome observed only after adversarial optimization. This raises a fundamental question: \emph{can the relative attack difficulty of different inputs be characterized before optimization begins?} In this paper, we investigate this question for person-vanishing attacks by introducing the concept of pre-optimization attack difficulty, which captures intrinsic differences in optimization effort across input images. To estimate this latent difficulty before optimization, we propose Quad-CLEVER, an efficient geometry-based estimator derived from a quadratic approximation of the local person-vanishing margin along the most attack-relevant direction. Extensive experiments across multiple attack algorithms demonstrate that Quad-CLEVER consistently correlates with the observed optimization cost, providing empirical evidence that attack difficulty exhibits a predictable pre-optimization structure. Building upon this finding, we further propose a difficulty-aware attack framework that leverages the estimated difficulty to adaptively allocate optimization budgets for a base attack under a fixed computational budget. On BDD100K, the proposed framework improves the image-level attack success rate by up to 5.78$\%$ while reducing the average optimization cost by up to 11.42 iterations. On the more challenging EventPed dataset, it saves 2.25 optimization iterations while maintaining comparable attack performance. These results demonstrate that attack difficulty can be meaningfully estimated before optimization and that exploiting such estimates enables more computationally efficient adversarial attacks.

Introduction

Because pedestrian detection is safety-critical, the paper asks whether person-vanishing attack difficulty can be predicted from clean inputs before optimization. It proposes Quad-CLEVER and difficulty-aware budget allocation as ways to estimate and use that difficulty.

Introduction

  • Prior person-vanishing studies emphasize stronger attack objectives and optimization, while treating difficulty as an outcome; whether clean inputs reveal relative difficulty remains comparatively underexplored.The paper poses whether difficulty emerges solely from optimization or also has an observable input-dependent component.Source: Introduction · p. 1
  • If difficulty contains an observable input-dependent component, relative image difficulty should be at least partly preserved across attack algorithms and estimable from clean inputs.The authors argue that establishing these properties would shift attention toward understanding attack-difficulty structure, beyond designing stronger optimizers.Source: Introduction · p. 1
  • Quad-CLEVER consistently correlates with observed optimization cost across attack algorithms, and its framework achieves higher attack efficiency with competitive performance on two benchmarks.The framework uses estimated difficulty to allocate optimization budgets adaptively.Source: Introduction · p. 2
  • Quad-CLEVER estimates relative attack difficulty from clean inputs before adversarial optimization, using a quadratic approximation of the local person-vanishing margin.The paper evaluates its consistency with observed difficulty across attack algorithms and its use in adaptive budget routing.Source: Introduction · pp. 1–2
  • The difficulty-aware framework uses estimated difficulty to adaptively allocate optimization budgets for a base attack under a fixed computational budget.The proposed approach aims to avoid redundant computation on easy images while giving hard images more optimization effort.Source: Introduction · p. 2, Figure 1 · p. 2

Related Work

Related work studies person-vanishing attacks, sample-level adversarial vulnerability, local robustness geometry, and adaptive optimization. The paper distinguishes its goal of estimating detector-attack effort before optimization from these prior targets and approaches.

Person-vanishing attacks on object detectors

  • Person-vanishing attacks have advanced attack capability, but their difficulty is usually assessed retrospectively through optimization outcomes; pre-optimization characterization has received comparatively limited attention.Prior examples include attacks on non-maximum suppression, detector robustness evaluation, and learnable attention-based attacks.Source: Person-vanishing attacks on object… · p. 2

Sample-level adversarial vulnerability

  • Prior sample-level vulnerability work examines attack success, transferability, required perturbation, attackability, or adversarial-training difficulty, whereas this work targets optimization effort for detector attacks.Quad-CLEVER is analytically derived from clean detector responses and does not require an auxiliary predictor trained on attack-generated difficulty labels.Source: Sample-level adversarial vulnerability · p. 2

Robustness estimation

  • Local-geometry methods use first- and second-order information to estimate vulnerability, while Quad-CLEVER uses that perspective as a proxy for relative computational effort, not a certified robustness radius.Examples include DeepFool, decision-boundary analyses, and CLEVER's local Lipschitz and curvature-based extensions.Source: Robustness estimation · pp. 2–3

Adaptive adversarial optimization

Preliminaries

The paper defines attack difficulty as expected optimization effort across an attack family and treats intrinsic difficulty as latent before optimization. It motivates using clean-input and detector geometry as observable evidence, drawing on CLEVER's pre-optimization formulation.

Attack Difficulty

  • Attack difficulty is an input's expected optimization cost across an attack family, where cost may be iterations, time, queries, or another measure of effort.A larger expected cost indicates that the input requires more effort, on average, to attack successfully.Source: Attack Difficulty · p. 3

Bayesian Perspective of Attack Difficulty

  • Intrinsic attack difficulty is modeled as a latent variable, inferred from observable pre-optimization evidence derived from the clean input and target model.The paper proposes local decision geometry around the clean input as informative evidence of subsequent optimization difficulty.Source: Bayesian Perspective of Attack… · p. 3

CLEVER

  • CLEVER estimates a lower bound on minimum adversarial perturbation from local decision-boundary geometry without adversarial optimization; this paper uses its geometric formulation to estimate difficulty, not robustness.Source: CLEVER · p. 3

Methodology

The method estimates input-dependent person-vanishing attack difficulty before optimization with Quad-CLEVER, then uses the estimate to allocate a base attack’s optimization budget under a fixed total budget.

Methodology

  • The method instantiates geometry-based evidence with Quad-CLEVER to infer latent attack difficulty, then uses that estimate to adapt a base attack’s optimization budget.The allocation is performed under a fixed computational budget.Source: Methodology · p. 3

Quad-CLEVER for Difficulty Estimation

  • Quad-CLEVER approximates each target’s local detector-margin geometry to estimate the margin-crossing distance along the steepest margin-decrease direction.It uses the gradient magnitude and directional curvature in a second-order approximation, with a stable limit that reduces to the first-order estimate as curvature approaches zero.Source: Quad-CLEVER for Difficulty Estimation · pp. 3–4
  • Quad-CLEVER requires one gradient evaluation and one Hessian-vector product, eliminating the sampling overhead of local Lipschitz estimation used by CLEVER.A smaller score indicates that the margin reaches the decision boundary under a smaller local perturbation and is expected to require less optimization effort.Source: Quad-CLEVER for Difficulty Estimation · p. 4
  • For images with multiple detected target persons, the image-level difficulty score is the minimum object-level estimate because an attack succeeds once any target disappears.The minimum therefore represents the difficulty of the attack objective.Source: Quad-CLEVER for Difficulty Estimation · p. 4

Difficulty-Aware Attack

  • The framework estimates difficulty with Quad-CLEVER, converts it into an image-level budget, and runs a base attack with that budget under a fixed total budget.It allocates more computation to difficult inputs while avoiding unnecessary optimization on easier ones.Source: Difficulty-Aware Attack · p. 4
  • The framework offers discrete budget routing across three difficulty levels and continuous routing that maps estimated difficulty to budget through a monotonic function.The discrete policy uses calibration-set thresholds; the continuous policy applies logarithmic normalization and percentile clipping to limit heavy-tail and extreme-score effects.Source: Discrete Budget Routing · p. 4, Continuous Budget Routing · p. 4

Experiments

Experiments test whether Quad-CLEVER predicts relative optimization difficulty before attacks and whether using it to route budgets improves efficiency under a fixed computational budget.

Experiment Setup

  • Experiments use white-box PGD and TOG against fine-tuned YOLOv8s on BDD100K and EventPed, measuring image-level ASR and average iterations under an ℓ2 perturbation constraint.The evaluation includes a maximum budget of 500 iterations; first-success iteration measures optimization effort in RQ1.Source: Experiment Setup · p. 5, RQ1: Can Attack Difficulty Be… · p. 6

RQ1: Can Attack Difficulty Be Estimated Before Optimization?

  • Quad-CLEVER predicts first-success effort: PGD correlations are ρ = 0.975 on BDD100K and ρ = 0.922 on EventPed, while TOG correlations are ρ = 0.607 and ρ = 0.701.The score–iteration relationship is weaker for TOG, whose trajectory can also reflect detector-hypothesis interactions, confidence filtering, and NMS regeneration.Source: Figure 2 · p. 5, RQ1: Can Attack Difficulty Be… · p. 6

RQ2: Can Pre-optimization Difficulty Improve Attack Efficiency?

  • At matched ASR, Log continuous routing saves 11.42 iterations per image for PGD and 2.36 for TOG on BDD100K, and 2.25 for PGD and 2.13 for TOG on EventPed.The ASR–budget comparison is shown for PGD and TOG across both datasets.Source: Figure 3 · p. 6, RQ2: Can Pre-optimization Difficulty… · pp. 6–7
  • On BDD100K, Log continuous routing improves over fixed-K, uniform, and random-matched attacks at the same average computational budget for both PGD and TOG.The random-matching comparison supports the role of assigning larger budgets to more difficult images, rather than merely changing the overall budget distribution.Source: RQ2: Can Pre-optimization Difficulty… · p. 6, Figure 3 · p. 6

Ablation study

  • Routing generally beats fixed-K at similar average budgets, but excessive γ compresses budgets toward Kmin and can erase the ASR advantage.Easy images receive fewer iterations while saved computation is redirected to harder ones.Source: Figure 4 · p. 6, Ablation study · p. 7
  • Quad-CLEVER’s difficult-sample rankings agree with PGD and TOG above random reference, while PGD correlations remain strong across person counts and TOG varies more by dataset.TOG correlation declines as person count increases on BDD100K, but remains stable or slightly improves on EventPed.Source: Ablation study · p. 7, Table 2 · p. 7, Table 3 · p. 7

Conclusion

The conclusion argues that person-vanishing attack difficulty has an input-dependent structure that can be estimated from clean inputs before optimization. It reports that difficulty-aware routing improves the trade-off between attack success and optimization cost across multiple attacks and datasets.

Conclusion

  • Difficulty-aware routing improves the trade-off between attack success and optimization cost across multiple attacks and datasets.The strategies allocate fewer iterations to easier images and more computation to harder ones.Source: Conclusion · p. 7
  • Quad-CLEVER uses a directional second-order approximation of the detector margin to estimate relative optimization effort before attacks run.The conclusion reports that its estimates consistently predict effort across different attack algorithms.Source: Conclusion · p. 7
  • The findings indicate that person-vanishing attack difficulty is not solely an outcome of a particular optimization process.Source: Conclusion · p. 7
Loading 2609.34501v1…