PaperlayerView exact revision on arXiv

Source-linked AI summary

Backdoor as Probe: Test-Time Adversarial Defense for CLIP

Zhongqi Wang, Jie Zhang, Nie Sen, Zhiyu Chen, Shiguang Shan, Xilin Chen

arXiv:2609.34641v1cs.CV

TL;DR

Existing test-time defenses treat adversarial activation shifts as distortions, while backdoor-based defenses have been class-specific and difficult to extend to open-vocabulary CLIP. BaP implants a defender-controlled probe using a closed-form model edit, then detects and selectively rectifies adversarial inputs. Across 16 zero-shot image-classification datasets, it raises average robust accuracy from 1.0% to 52.3% while retaining clean accuracy.

  • Problem

    Existing test-time defenses suppress adversarial activation shifts, and prior backdoor-based defenses are class-specific, limiting their direct extension to open-vocabulary CLIP.

    Source: §1 · pp. 1–2

  • Method

    BaP uses a closed-form edit to implant a defender-controlled, class-agnostic probe that detects adversarial inputs through target-direction responses and selectively rectifies suspicious inputs.

    Source: §1 · p. 2

  • Results

    Average robust accuracy improves from 1.0% to 52.3% across 16 zero-shot image-classification datasets while BaP retains clean accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    BaP demonstrates that adversarial activation shifts can serve as test-time defense signals in CLIP.

    Source: §5 · p. 10

    The threat model assumes the defender has white-box model access and may edit the model.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Test-time adversarial defense improves the robustness of vision-language foundation models such as CLIP without retraining. However, adversarial activation shifts are typically treated as distortions to suppress, rather than signals to exploit. We turn these shifts into defense signals by repurposing the trigger-to-target mechanism of backdoors. The key is to implant a defender-controlled backdoor as a probe that is weakly activated by clean inputs but strongly activated by adversarial shifts. Based on this insight, we propose \emph{Backdoor as Probe} (BaP), a test-time adversarial defense for CLIP. BaP constructs the probe through a closed-form model edit to a selected MLP layer. It projects the average adversarial activation shift and a defender-specified semantic direction onto the layer's low-energy input and output activation subspaces to obtain the trigger and target directions, respectively. At inference time, adversarial inputs produce measurable responses along the target direction for detection. BaP then selectively rectifies detected inputs by optimizing a small perturbation that steers their representations away from adversarial shifts and toward the clean subspace. Experiments across 16 benchmarks show that BaP improves average robust accuracy from 1.0\% to 52.3\% while retaining clean accuracy, achieving performance comparable to state-of-the-art methods with up to a \(5.7\times\) inference speedup. BaP further shows the generalization to adversarial attacks on large vision-language models. Project page: https://robin-wzq.github.io/Backdoor-as-Probe/

1 INTRODUCTION

BaP reframes adversarial activation shifts as signals for a defender-controlled backdoor probe, addressing the class-specificity, clean-input activation, and implantation-cost challenges of adapting backdoors to open-vocabulary CLIP. Across 16 zero-shot image-classification datasets, it improves average robust accuracy while retaining clean accuracy.

1 INTRODUCTION

  • Adapting backdoors to open-vocabulary CLIP requires a class-agnostic probe that stays inactive on clean inputs, responds strongly to adversarial inputs, and avoids costly implantation.Earlier backdoor-based defenses are class-specific, while CLIP backdoor methods commonly rely on large-scale fine-tuning.Source: §1 · p. 2
  • BaP uses a closed-form edit to implant a class-agnostic probe in a selected MLP layer, deriving its trigger from adversarial activation shifts and its target from a defender-specified semantic direction.Both directions are projected onto low-energy activation subspaces; target-response magnitude supports detection, followed by selective input rectification.Source: §1 · p. 2
  • Average robust accuracy rises from 1.0% to 52.3% across 16 zero-shot image-classification datasets while BaP retains clean accuracy.BaP also achieves performance comparable to state-of-the-art methods with up to a 5.7× inference speedup.Source: §1 · p. 2
  • BaP repurposes the trigger-to-target mapping of a backdoor as a defender-controlled diagnostic probe rather than an attacker-controlled mechanism.The paper presents this as the first backdoor-based adversarial defense for open-vocabulary models, to the authors’ knowledge.Source: §1 · pp. 1–2

2 PRELIMINARIES AND RELATED WORK

This section introduces CLIP’s zero-shot image–text classification setting, describes adversarial attacks and defense families, and positions BaP against prior backdoor-based defenses. It highlights that earlier backdoor defenses depend on specific class sets, limiting direct application to open-vocabulary CLIP.

Zero-Shot Classification with CLIP

  • CLIP performs open-vocabulary zero-shot classification using image and text encoders with class-specific prompts.The supplied setup introduces an image and a set of class-specific text prompts.Source: §2 · p. 2

Adversarial Attacks on CLIP

  • Adversarial attacks add a constrained perturbation to a clean image to corrupt zero-shot predictions, with PGD and AutoAttack optimizing input gradients within an ℓ∞-norm ball.The perturbation is bounded by ε, and the attack objective uses cross-entropy over zero-shot classification logits.Source: §2 · p. 3

Adversarial Defenses for CLIP

  • CLIP defenses include adversarial fine-tuning, prompt tuning, test-time adaptation, and input transformation or purification approaches.BaP differs from transformation-based defenses by using adversarial activation shifts as trigger signals associated with an observable target.Source: §2 · p. 3

Backdoors for Adversarial Defense

  • Trapdoor and AI-Shielder use class-specific defensive backdoors, limiting their applicability to open-vocabulary CLIP.Trapdoor uses activation signatures for detection; AI-Shielder uses a source-to-target class mapping to recover predictions.Source: §2 · p. 3

3 METHOD

BaP edits a selected CLIP MLP to install an attack-sensitive probe, then uses its response to detect suspicious inputs and selectively rectify them toward clean activations.

3.1 THREAT MODEL AND METHOD OVERVIEW

  • The threat model gives the defender white-box access and permission to edit CLIP, with the goal of detecting and rectifying adversarial samples while preserving clean performance.The attacker perturbs a clean image within an ℓ∞ budget.Source: §3.1 · pp. 3–4

3.2 DEFENDER-CONTROLLED PROBE IMPLANTATION

  • BaP projects the average paired attack shift and defender-specified semantic direction into low-energy activation subspaces, then binds trigger and target through a closed-form edit.The low-energy subspaces capture directions with low clean activation energy, helping reduce interference with clean representations.Source: §3.2 · pp. 4–5

3.3 PROBE-BASED ADVERSARIAL SAMPLE DETECTION

  • At inference, BaP reads the response along the unit target direction; clean responses stay near zero while adversarial responses increase, and threshold-flagged inputs proceed to rectification.The response separation is shown for 1,000 clean and 1,000 adversarial STL-10 images using CLIP ViT-B/16.Source: §3.3 · p. 5

3.4 ADVERSARIAL SAMPLE RECTIFICATION

  • Rectification uses an escape stage followed by repair: two projected-gradient ascent steps move away from the adversarial state, then repair guides representations toward the clean activation distribution.The repair loss reduces distance from the clean subspace and aligns the residual with the clean direction.Source: §3.4 · p. 6

4 EXPERIMENTS

Across 16 zero-shot classification datasets, BaP substantially improves robust accuracy while retaining clean accuracy, and the experiments test its efficiency, transfer, ablations, and adaptive-attack robustness.

4.1 EXPERIMENTAL SETUP

  • The benchmark spans 16 datasets, compares state-of-the-art test-time defenses, and uses CLIP ViT-B/16 as the default backbone.The evaluation also includes ViT-B/32, ViT-L/14, and ViT-L/14@336; experiments use a fixed detection threshold of τ=1.93.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • 52.3% average robust accuracy, versus 1.0% for original CLIP, with 61.1% clean accuracy; BaP also exceeds R-TPT in robust accuracy.Table 2 reports zero-shot accuracy under 10-step PGD at ℓ∞=1/255.Source: §4.2 · p. 7, Table 2 · p. 7
  • 30.56 ms average latency makes BaP 5.7× faster than R-TPT and 6× faster than HD.Clean inputs take 3.75 ms, while adversarial inputs take 57.37 ms; the 53.94 ms rectification stage runs only for suspicious inputs, and probe implantation takes 2.70 ms.Source: §4.2 · p. 7, Table 3 · p. 7, §4.1 · p. 7
  • At ℓ∞=4/255, BaP achieves the highest robust accuracy across all seven tested attack-objective configurations.The probe was trained using ImageNet PGD at ℓ∞=1/255, while Table 4 evaluates 50-step attacks at ℓ∞=4/255.Source: §4.2 · p. 7, Table 4 · p. 7
  • Under stronger ℓ∞=4/255 attacks, BaP leads on General and Domain benchmarks with PGD, but R-TPT is stronger on three categories under AutoAttack.With PGD, BaP reaches 57.2% on General and 29.1% on Domain benchmarks.Source: §4.2 · p. 8, Table 5 · p. 8
  • BaP transfers across CLIP backbones and to LLaVA image captioning, where it improves robust scores under both M-Attack and FOA-Attack.On ViT-L/14, it reaches 40.7% robustness on Scene recognition; for LLaVA, robust scores rise from 21.1 to 24.6 under M-Attack and from 19.1 to 23.6 under FOA-Attack.Source: §4.2 · pp. 8–9, Table 6 · p. 8, Table 9 · p. 8

4.3 ABLATION STUDY

  • The escape loss raises rectification robust accuracy to 52.3%, while Layer 6 gives the highest robustness and performance is stable across response scales 10–80.Adding clean-direction alignment alone raises robust accuracy from 21.3% to 22.3%; adding the escape loss reaches 52.3% with clean accuracy decreasing from 62.3% to 61.1%. The default configuration uses Layer 6, probe norm 5, and response scale 40.Source: §4.3 · p. 9, Table 7 · p. 9, Figure 5 · p. 9

4.4 ROBUSTNESS TO AN ADAPTIVE ATTACK

  • Against an adaptive attack targeting the probe response, BaP retains 27.2% robust accuracy at β=0.1, compared with 0.3% without BaP.At β=0, BaP’s robust accuracy is 39.6%; at β=1 it is 34.7%, while the attack also weakens as β increases.Source: §4.4 · p. 9, Table 8 · p. 9

5 CONCLUSION

BaP uses a closed-form edit to turn a defender-controlled backdoor into a CLIP probe that detects and rectifies adversarial inputs. The paper reports robustness, efficiency, and transferability across backbones and attacks.

5 CONCLUSION

  • BaP’s closed-form probe detects and rectifies adversarial inputs while largely preserving clean performance, with robustness, efficiency, and transferability across backbones and attacks.The conclusion frames the method as repurposing a controlled backdoor as an internal probe for CLIP.Source: §5 · p. 10

ETHICS STATEMENT

  • The probe is intended for adversarial defense and should be applied only to models for which the defender has authorization.Source: ETHICS STATEMENT · p. 10

A ALGORITHM OF BAP

  • BaP’s algorithm constructs low-energy activation subspaces, derives trigger and target directions, and implants the probe through a weight edit.At inference, it detects inputs using the probe response and rectifies detected inputs through iterative updates.Source: A ALGORITHM OF BAP · p. 14

B DETAILS OF THE WEIGHT EDIT

  • The weight edit seeks to map the input-side probe direction to a target shift while changing the original weight as little as possible.The resulting solution is the unique minimum-norm solution and satisfies the mapping exactly.Source: B DETAILS OF THE WEIGHT EDIT · pp. 14–15
  • The output-side low-energy construction suppresses the original weight’s response on clean inputs, while the attack-aligned input direction strengthens the edit’s response on adversarial inputs.Source: B DETAILS OF THE WEIGHT EDIT · p. 15

C DETAILS OF DIFFERENT ATTACK OBJECTIVES

The evaluation considers cross-modal, targeted, and label-free attacks against BaP. The objectives target visual-text alignment, a predefined class, or consistency between clean and adversarial visual representations.

C DETAILS OF DIFFERENT ATTACK OBJECTIVES

Cross-modal Attacks

  • Cross-modal attacks minimize cosine similarity between an adversarial visual feature and its ground-truth text feature, using PGD or AutoAttack.The threat model uses a 4/255 perturbation budget and 50 iterations, with adversarial examples generated against the editing model.Source: C DETAILS OF DIFFERENT ATTACK OBJECTIVES · p. 15

Targeted Attacks

Label-free Attacks

  • Label-free attacks maximize negative cosine similarity between clean and adversarial visual features without using ground-truth labels or text prompts.The objective is optimized using PGD or APGD.Source: C DETAILS OF DIFFERENT ATTACK OBJECTIVES · p. 16

D.1 ABLATION STUDY ON DETECTION THRESHOLD τ

The detection threshold controls which inputs are rectified, trading clean accuracy against robust accuracy as the threshold changes.

D.1 ABLATION STUDY ON DETECTION THRESHOLD τ

  • Clean accuracy increases while robust accuracy gradually decreases as threshold τ rises; the authors set τ to the 95th percentile of clean calibration scores, q95 = 1.93.This threshold is estimated on held-out ImageNet calibration data and fixed across downstream datasets and attacks.Source: Figure 6 · p. 16, §D.1 · p. 16

D.2 ABLATION STUDY ON TARGET PROMPT p

BaP’s results are consistent across four semantically distinct target prompts. The appendix also describes GPTScore prompting for captioning evaluation.

D.2 ABLATION STUDY ON TARGET PROMPT p

  • Clean accuracy and robustness remain highly consistent across the tested prompts, indicating that performance does not rely on the target’s specific semantic content.Source: §D.2 · p. 16
  • Table 10 examines BaP with four semantically distinct target prompts.Source: Table 10 · p. 16, §D.2 · p. 16

E DETAILED PROMPT FOR GPTSCORE

F DETAILED RESULTS ON CLIP-B/32 AND CLIP-L/14

Tables 11 and 12 report per-dataset results across 16 datasets for CLIP-B/32 and CLIP-L/14, extending the category-level comparison and demonstrating BaP’s effectiveness across backbones.

F DETAILED RESULTS ON CLIP-B/32 AND CLIP-L/14

G DETECTION ROC OF BAP

BaP’s ROC results show consistently high AUC values across backbones and attacks, indicating that its probe distinguishes adversarial inputs from clean ones. The section also provides ROC curves for four CLIP configurations and comparison tables for two backbones.

G DETECTION ROC OF BAP

Loading 2609.34641v1…