PaperlayerView exact revision on arXiv

Source-linked AI summary

InfraPatch: Cross-Task Targeted Grayscale Patch Attacks on Infrared-Adapted Vision-Language Models

Chengyin Hu, Dingyi Lu, Jiaju Han, Xiang Chen, Weiwen Shi, Jiahuan Long, Yiwei Wei, Jiujiang Guo

arXiv:2609.02233v1cs.CVcs.AI

TL;DR

InfraPatch studies whether localized grayscale perturbations can inject chosen semantics into infrared-adapted VLMs, addressing limited cross-task evidence. It uses white-box per-instance proxy-guided patch optimization with task-specific objectives and clean-conditioned parsers, achieving 86.00%–100% targeted ASR across ten variants. The findings support effective digital semantic injection within the tested synthetic setting while leaving physical and transfer robustness unresolved.

  • Problem

    Prior work does not systematically test whether localized grayscale perturbations can inject one attacker-chosen semantic across heterogeneous infrared-adapted VLM outputs.

  • Method

    InfraPatch uses proxy-guided placement and task-adaptive semantic objectives to optimize one compact grayscale patch for classification, generation, and binary VQA.

  • Results

    86.00%–100% targeted ASR was achieved across ten infrared-adapted variants under clean-conditioned evaluation.

  • Takeaways & Limitations

    Localized grayscale injection is effective across heterogeneous IR-VLM outputs within the tested synthetic, digital, white-box scope.

  • Takeaways & Limitations

    The study is limited to one synthetic DiffV2IR-translated COCO set, one target semantic, and white-box per-instance digital attacks, without establishing transfer, sensor robustness, or physical realizability.

Abstract

from arXiv · show

Infrared vision-language models (IR-VLMs) have emerged as a promising paradigm for multimodal perception under low-visibility conditions, yet their robustness to targeted adversarial attacks remains poorly understood. Existing adversarial patch methods mainly study RGB-based models or a single downstream task and do not characterize whether localized perturbations can induce an intended semantic target in IR-VLMs. We propose InfraPatch, a white-box, per-instance framework for targeted digital grayscale patch attacks against IR-VLMs. InfraPatch optimizes a compact single-channel patch within an approximately 5% local-area budget, combines proxy-guided placement with task-adaptive semantic objectives, and induces target behaviors in image classification, image captioning, and binary visual question answering. We evaluate ten infrared-adapted model variants on 300 synthetic infrared-style images generated by applying DiffV2IR to a fixed 30-category COCO subset, using clean-conditioned targeted success criteria. InfraPatch achieves targeted attack success rates from 86.00% to 100% across the ten variants. On CLIP and BLIP-2, proxy location search improves success by 6.67 and 10.33 percentage points over optimized random placement, respectively; LLaVA-1.5 remains saturated near 100% under both settings. Patch-area and objective ablations further expose substantial differences in vulnerability across architectures and task formats. These results show that small grayscale patches can inject chosen target semantics across IR-VLM families under a controlled digital threat model, motivating stronger robustness evaluation for infrared multimodal systems.

1 Introduction

InfraPatch addresses the lack of systematic evidence on localized grayscale attacks that inject one chosen semantic across heterogeneous infrared-adapted VLM outputs. It unifies task-specific objectives and clean-conditioned success criteria within a compact digital patch framework.

  • Infrared semantic manipulation matters because model outputs can guide downstream perception pipelines or human decision-makers.
  • Prior studies separately examine infrared classifiers and detectors, targeted RGB-VLM attacks, or illumination-based RGB-VLM sensitivity.
  • InfraPatch targets classification, conditional generation, and binary VQA with one cross-task formulation that avoids conflating their success notions.
  • The framework uses a white-box, per-instance single-channel grayscale patch with a nominal 5% area budget and task-adaptive semantic objectives.
  • The evaluation spans ten model variants from the CLIP, BLIP, OpenFlamingo, and LLaVA families, using clean-output exclusion and task-specific success parsing.

2 Related Work

Prior work establishes localized adversarial patches across visual and infrared detection settings, but InfraPatch studies digital grayscale semantic injection across heterogeneous infrared-adapted VLM outputs. Its scope is distinct from physical, transformation-aware attacks and existing cross-modal transfer studies.

  • Universal and physical-world patch studies optimize localized patterns for classifier or detector failures under transformations such as viewpoint, distance, illumination, and printing.
  • Infrared attack research includes simulated-infrared examples and thermal-material patches targeting pedestrian and vehicle detectors.
  • InfraPatch instead evaluates a digital single-channel intensity patch across classification, conditional generation, and binary VQA without claiming physical realizability.
  • Prior VLM attack studies address surrogate transfer, set-level cross-modal guidance, autonomous-driving perception, or physical-world spatial-spectral perturbations, rather than one clean-conditioned target semantic across infrared-adapted tasks.
  • DiffV2IR is used only to generate synthetic infrared-style inputs while preserving visible-benchmark semantic content for downstream robustness evaluation.

3 Threat Model and Problem Formulation

InfraPatch assumes white-box access to a frozen IR-VLM and restricts attacks to one per-instance grayscale patch near a 5% area budget. Success is evaluated conditionally on clean outputs with task-specific parsers and a shared cross-task metric.

  • Threat Model and Problem Formulation: The evaluator applies a task-specific parser to clean and adversarial outputs for classification, generation, and binary VQA.
  • Digital Threat Model: The attacker may choose one square location and grayscale patch but cannot alter model parameters, prompts, outside-mask pixels, or the output parser.
  • Digital Threat Model: The default per-instance digital attack uses an approximately 5% area budget and makes no claims about universal transfer, black-box transfer, sensor robustness, or physical realization.
  • Clean-Conditioned Target Semantics: Classification samples are valid only when the clean top-1 prediction is correct and non-target, and success requires a target top-1 prediction with positive margin.
  • Clean-Conditioned Target Semantics: Generation excludes clean target-containing text, while binary VQA requires a clean “no” and an unambiguous adversarial “yes”; outputs containing both tokens are rejected.
  • Evaluation Metric: The evaluation reports one cross-task primary metric for the task-conditioned attack outcomes.
  • Evaluation Metric: Clean target hits enter the denominator but never count as attack successes, alongside recording realized area, patch TV, and target margin where applicable.

4 InfraPatch

InfraPatch jointly selects a patch location and optimizes a single-channel patch using a shared objective structure with task-specific semantic losses and total-variation regularization. Early-exit proxy search aligns placement selection with final attack evaluation across classification, generation, and VQA.

  • Method Overview: InfraPatch jointly optimizes a single-channel patch and placement mask, with grayscale values broadcast across input channels and projected to the valid image range.
  • Method Overview: The complete objective combines a task-specific loss with total-variation regularization, while classification uses target, alignment, competitor-suppression, and clean-class-suppression terms.
  • Method Overview: Figure 1 depicts a DiffV2IR input and fixed target entering proxy placement search before task-specific patch refinement and parser-based evaluation.
  • Target-Aware Patch Placement: InfraPatch samples K=5 legal coordinates, optimizes each candidate for T_proxy=10 steps, and ranks candidates by the lowest complete objective during that trajectory.
  • Target-Aware Patch Placement: A candidate search terminates when the targeted-success criterion is met; otherwise the best proxy candidate continues for at most 290 additional steps.
  • Target-Aware Patch Placement: The worst-case per-image search cost is 5 × 10 + 290 = 340 gradient updates, although continuation can terminate early.
  • Task Objectives: CLIP optimization combines target probability, target-text alignment, competitor suppression, and clean-prediction suppression, with decoded top-1 label and margin determining success.
  • Task Objectives: Generation uses teacher-forced target and clean-output NLL terms, but candidate and final success are determined by free-running generated text; LLaVA requires an unambiguous generated “yes”.

5 Experimental Setup

The evaluation covers ten infrared-adapted model variants across three output formats, using a fixed synthetic infrared-style image set and task-specific attack comparisons. Experiments also test placement, patch area, objective, and total-variation design choices under controlled digital settings.

  • Models and tasks: Ten infrared-adapted variants from four model families are evaluated across classification, captioning, and binary VQA.The models include CLIP-family classifiers, BLIP-family and OpenFlamingo captioning models, and LLaVA VQA models.
  • Data and preprocessing: The evaluation uses 300 synthetic infrared-style images from a fixed 30-category MS COCO subset, with ten images per category.DiffV2IR generates the infrared-style digital inputs rather than sensor measurements.
  • Patch configuration: Fixed-resolution pipelines use patches realizing approximately 4.98% area, while LLaVA-1.6 retains its any-resolution path and records the ratio per image.The fixed-resolution settings use 50 × 50 patches at 224^2 and 75 × 75 patches at 336^2.
  • Baselines: Baseline comparisons preserve alternative patch representations and derivative-free DE/PSO searches while replacing detector-disappearance fitness with task-targeted fitness.The comparison is area-controlled rather than query-matched.
  • Ablations: Component analysis compares InfraPatch with optimized random placement, unoptimized random grayscale patches, and unoptimized center patches.Area settings near 3%, 5%, and 7% and one-factor objective/TV ablations use the same image pool, target, initialization, budget, and parser.
  • Optimization protocol: InfraPatch uses Adam with learning rate 0.03, five proxy candidates, ten proxy updates per candidate, and at most 290 continuation updates.The attack is purely digital, uses no EOT, and all comparisons apply clean-conditioned definitions.

6 Results and Analysis

InfraPatch achieves high clean-conditioned targeted success across ten IR-VLM variants and three task formats, while ablations reveal architecture- and task-dependent vulnerability patterns. Proxy-guided placement improves over optimized random placement on CLIP and BLIP-2, but comparisons are bounded by unequal compute and synthetic digital evaluation.

  • Main Results across Models and Tasks: 86.00%–100% targeted ASR was achieved across ten IR-VLM variants under the clean-conditioned protocol.OpenCLIP and both LLaVA variants reached 100%, while BLIP-2 recorded 86.00%.
  • Main Results across Models and Tasks: 100% targeted ASR was reached by both binary-VQA variants, producing reliable clean-conditioned no-to-yes transitions.The result corresponds to 286/286 valid LLaVA-1.5 inputs and 300/300 LLaVA-1.6 inputs.
  • Comparison with Adapted Structured Infrared Attacks: 95.15% versus 9.70% was achieved by InfraPatch and AdvIC-Adapted on CLIP, while BLIP-2 reached 86.00% versus 7.00%.Across the representative task pairs, adapted structured attacks remained between 1.33% and 14.69%.
  • Comparison with Internal Controls: 0% targeted ASR was produced by equal-area unoptimized random and center grayscale controls on all three representative variants.This indicates that learned patch values, rather than equal-area occlusion alone, were required for the chosen target.
  • Comparison with Internal Controls: 6.67 and 10.33 percentage points were added by proxy search over optimized random placement on CLIP and BLIP-2, respectively.LLaVA-1.5 was already near saturation: optimized random placement averaged 99.88% and InfraPatch reached 100%.
  • Ablation Studies: 2.42 points were lost by removing target-embedding alignment from CLIP, while removing TV reduced BLIP-2 by 4.00 points and CLIP by 0.60 points.Rival and clean-class suppression increased CLIP ASR under the tested target and sample pool, while LLaVA-1.5 remained saturated.
  • Ablation Studies: 76.67% to 89.67% was BLIP-2’s ASR increase when realized patch area grew from approximately 3% to 7%.CLIP rose from 92.73% to 96.97%, whereas LLaVA-1.5 remained at 100% throughout.
  • Main Results across Models and Tasks: 86.00%–96.33% ASR was obtained across conditional-generation variants, with success evaluated from free-running decoded text.The lexical parser, rather than teacher-forced loss alone, determines success.

7 Limitations, Responsible Use, and Conclusion

The study is limited to authorized, white-box, per-instance digital attacks on one synthetic infrared dataset and does not establish physical or transfer robustness. Within that scope, InfraPatch achieved high targeted success across ten variants, while the authors recommend controlled robustness evaluation and safeguards.

  • Limitations: The study is bounded to white-box, per-instance digital attacks on one synthetic DiffV2IR-translated COCO set and one target semantic.It does not establish universal or black-box transfer, sensor robustness, or physical realizability.
  • Limitations: The 300-image, 30-category benchmark does not measure target or longtail variation, and the shared adaptation recipe cannot separate pre-training from LoRA-induced vulnerability.
  • Responsible Use: InfraPatch is intended for authorized digital robustness evaluation rather than covert physical deployment.The proposed release and mitigation practices include provenance, fixed parsers, model cards, patch-aware augmentation, and output-consistency checks.
  • Conclusion: InfraPatch reached 86.00%–100% targeted ASR across ten infrared-adapted variants under the clean-conditioned protocol.Area and objective ablations showed architecture-dependent vulnerability and component utility, while LLaVA exhibited a ceiling effect.

A Additional Implementation Details

The supplementary implementation section includes a table of prompts, target semantics, and additional attack settings.

  • Table 5 lists prompts, target semantics, and supplementary attack settings.

B Adapted Structured Infrared Baseline Details

The adapted structured infrared baselines use derivative-free optimization to search constrained geometric patch parameters, with final success evaluated using InfraPatch’s discrete task rules. Realized area may be smaller than nominal area because of overlap or clipping.

  • AdvIB searches seven rotatable square cold blocks using differential evolution over 21 normalized position-and-angle parameters.The search uses a population of 100, ten generations, and at most 1,100 fitness evaluations.
  • AdvIC-Adapted renders two black quadratic Bézier curves specified by 12 normalized control-point parameters.
  • Final ASR for the structured baselines uses the same discrete success rules as InfraPatch, despite continuous derivative-free fitness optimization.
  • Overlap or clipping can make the realized union area smaller than the nominal budget.Each run retains the best fitness, parameters, realized mask, query count, decoded output, and success status.
Loading 2609.02233v1…