PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Backdoor Defense in CLIP

Junhao Kuang, Siyuan Liang, Jiawei Liang, Kuanrong Liu, Xiaochun Cao

arXiv:2409.15968v1cs.CV

TL;DR

CLIP is vulnerable to backdoor attacks, while conventional augmentation-based defenses may fail to capture backdoor-specific features. ABD uses adversarial examples that approximate backdoor samples as augmentation; it reduces attack success rates across three attacks versus CleanCLIP with a small reported clean-accuracy decrease.

  • Problem

    Conventional augmentation-based multimodal defenses can fail to capture backdoor-specific features, while CLIP is vulnerable to backdoor attacks.

    Source: §I · p. 1

  • Method

    ABD uses crafted adversarial examples that approximate backdoor samples in the feature space as data augmentation during fine-tuning.

    Source: §I · pp. 1–2

  • Results

    8.66%, 10.52%, and 53.64%: ABD reduces ASR versus CleanCLIP for BadNet, Blended, and BadCLIP, respectively, with a 1.73% average decrease in clean accuracy.

    Source: §I · p. 2, B. Main Results · p. 4

  • Takeaways & Limitations

    ABD provides defense against traditional uni-modal and multimodal backdoor attacks targeting CLIP in the reported experiments.

    Source: §I · p. 2, B. Main Results · p. 4

Abstract

from arXiv · show

Multimodal contrastive pretraining, exemplified by models like CLIP, has been found to be vulnerable to backdoor attacks. While current backdoor defense methods primarily employ conventional data augmentation to create augmented samples aimed at feature alignment, these methods fail to capture the distinct features of backdoor samples, resulting in suboptimal defense performance. Observations reveal that adversarial examples and backdoor samples exhibit similarities in the feature space within the compromised models. Building on this insight, we propose Adversarial Backdoor Defense (ABD), a novel data augmentation strategy that aligns features with meticulously crafted adversarial examples. This approach effectively disrupts the backdoor association. Our experiments demonstrate that ABD provides robust defense against both traditional uni-modal and multimodal backdoor attacks targeting CLIP. Compared to the current state-of-the-art defense method, CleanCLIP, ABD reduces the attack success rate by 8.66% for BadNet, 10.52% for Blended, and 53.64% for BadCLIP, while maintaining a minimal average decrease of just 1.73% in clean accuracy.

I. INTRODUCTION

CLIP and related multimodal contrastive models learn joint image-text representations, but CLIP is vulnerable to backdoor attacks. ABD uses adversarial examples to approximate backdoor features, addressing a gap in conventional augmentation-based defenses.

I. INTRODUCTION

  • CLIP backdoor attacks can use a poisoning ratio as low as 0.01%, reflecting their low cost and posing a threat to real-world deployment.Attackers embed image triggers and pair them with template sentences containing a target label.Source: §I · p. 1
  • Existing multimodal defenses use simple augmentation for feature alignment, which can fail to capture backdoor-specific features and yield suboptimal defense performance.RoCLIP augments images and pairs them with similar captions, while CleanCLIP aligns original samples with augmented versions; RoCLIP's iterative process is resource-intensive.Source: §I · p. 1
  • ABD uses crafted adversarial examples that approximate backdoor samples as data augmentation during fine-tuning to align features with backdoor features.The approach builds on reported feature-space similarities between adversarial examples and backdoor samples in compromised models.Source: §I · pp. 1–2
  • 8.66%, 10.52%, and 53.64%: ABD reduces ASR versus CleanCLIP for BadNet, Blended, and BadCLIP, respectively, with a 1.73% average decrease in clean accuracy.The reported experiments use the ImageNet 1K validation dataset.Source: §I · p. 2

A. Threat Model

The threat model assumes an attacker poisons a public dataset and fine-tunes official CLIP weights on the poisoned data. The attack pairs triggered images with captions containing the target label.

A. Threat Model

  • The poisoned dataset P comprises image-text pairs that include triggered images xT and textual descriptions c containing the target label.The attack fine-tunes downloaded official CLIP pre-trained weights on data injected into a public dataset.Source: A. Threat Model · p. 2

B. Adversarial Examples Design

ABD crafts adversarial examples to approximate backdoor samples in a poisoned model’s feature space. These examples capture backdoor-related features while retaining similarity to original captions, supporting their use in defense.

B. Adversarial Examples Design

  • ABD adds a backdoor loss to AdvCLIP-generated adversarial examples so they incorporate backdoor features.AdvCLIP uses GANs to create adversarial examples in image-text pairs.Source: B. Adversarial Examples Design · p. 2
  • InfoNCE optimization increases similarity between normal-model image features and backdoor features from the poisoned model.The loss uses features from both the normal and poisoned visual encoders.Source: B. Adversarial Examples Design · p. 2
  • 28.49 for backdoor image-caption pairs exceeds 25.02 for benign pairs, while designed adversarial images score 16.04 with backdoor captions.The figure compares original, backdoor, and adversarial images paired with different captions.Source: Figure 2 · p. 3, B. Adversarial Examples Design · p. 3
  • Adversarial images score 19.82 with original captions and 5.04 with unrelated target captions, supporting their use to approximate poisoned samples.The authors report that this augmentation does not significantly impact clean-sample accuracy.Source: B. Adversarial Examples Design · p. 3

C. Data Augmentation Defense

ABD builds a fine-tuning dataset by augmenting images with adversarial examples and captions with Easy Data Augmentation.

C. Data Augmentation Defense

A. Experimental Settings

Experiments use CLIP with ResNet-50 and Transformer encoders, poisoning setups for three attack methods, and ABD fine-tuning on CC3M data. Evaluation measures clean accuracy and attack success rate on ImageNet-1K.

A. Experimental Settings

  • Experiments use a 1M CC3M subset for training, LAION-400M-pretrained CLIP weights, a ResNet-50 visual encoder, and a Transformer text encoder.The authors state that medium-scale CC3M data suffices to train the model while reducing storage and compute needs.Source: A. Experimental Settings · p. 3
  • BadNet, Blended, and BadCLIP poison CLIP using a 500K CC3M subset at a 0.3% poisoning rate.BadNet and BadCLIP use a fixed noise patch of size 16; Blended uses global blending noise.Source: A. Experimental Settings · p. 3
  • ABD trains AdvCLIP on 250K CC3M examples, then fine-tunes the poisoned model for 10 epochs using the same data.Defense fine-tuning uses adversarial image perturbations and EDA captions; the reported defense phase takes about 14 A100 GPU hours.Source: A. Experimental Settings · p. 3
  • ImageNet-1K validation evaluation measures clean accuracy and attack success rate, targeting lower ASR with minimal CA impact.ASR is the proportion of triggered images classified as the target class.Source: A. Experimental Settings · p. 3

B. Main Results

On ImageNet-1K, ABD reduces attack success across BadNet, Blended, and BadCLIP while limiting clean-accuracy loss. Against CleanCLIP, the reported ASR reductions are largest for BadCLIP.

B. Main Results

  • The evaluation compares ABD with RoCLIP and CleanCLIP across traditional uni-modal BadNet and Blended attacks and multimodal BadCLIP.Source: B. Main Results · pp. 3–4
  • 8.66% for BadNet, 10.52% for Blended, and 53.64% for BadCLIP are ABD’s ASR reductions versus CleanCLIP, with a 1.73% average CA decrease.Table I reports method performance on ImageNet-1K.Source: Table I · p. 4, B. Main Results · p. 4
  • 65.57% average ASR reduction versus the victim model accompanies a 4.91% average CA drop under ABD.Source: B. Main Results · p. 4

C. Ablation Study

The ablation finds that adversarial image enhancement drives defense gains, while EDA alone has negligible impact on ASR and complements image enhancement. Adding backdoor loss to AdvCLIP also improves defense.

C. Ablation Study

  • Adversarial image enhancement reduces ASR by 39.97% on average and increases CA by 0.27%, while EDA alone has negligible impact on ASR.Fine-tuning with 250K clean samples alone does not significantly improve defense against BadNet, Blended, or BadCLIP.Source: C. Ablation Study · p. 4
  • 8.54% lower ASR and only 0.3% lower CA result from adding EDA to adversarial image enhancement, compared with adversarial image enhancement alone.The ablation identifies adversarial examples as key to defense and EDA as a complement.Source: C. Ablation Study · p. 4
  • ASR decreases by 7.5% and CA increases by 0.15% when AdvCLIP incorporates backdoor loss, compared with the original UAP method.UAP denotes adversarial examples from original AdvCLIP; UAP(Lbd) denotes those from the improved model with backdoor loss.Source: C. Ablation Study · p. 4, Table III · p. 4

IV. CONCLUSIONS

ABD uses adversarial examples approximating backdoor samples in feature space to defend CLIP against backdoor attacks. Experiments report better defense than RoCLIP and CleanCLIP, with reduced attack success and minimal clean-accuracy impact.

IV. CONCLUSIONS

  • ABD generates adversarial examples that approximate backdoor samples in feature space and uses them as data augmentation for CLIP backdoor defense.The paper reports subtle connections between adversarial and backdoor samples in compromised models.Source: §IV · p. 4
  • Across ImageNet-1K validation experiments, ABD consistently outperforms RoCLIP and CleanCLIP while significantly reducing attack success with minimal impact on clean accuracy.The experiments are described as extensive.Source: §IV · p. 4
  • The paper presents adversarial examples as a source of insight into the correlation between adversarial and backdoor samples in compromised models.Source: §IV · p. 4

Related papers

Loading 2409.15968v1…