Source-linked AI summary
BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning
Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, Ee-Chien Chang
TL;DR
Backdoor attacks on multimodal contrastive learning can be countered by specialized defenses, raising concerns for practical use. BadCLIP uses dual-embedding guidance to target detection and clean fine-tuning defenses, and reports improved attack results against these defenses. The authors identify stronger MCL backdoor defenses and complex-task attacks as future needs.
Problem
Existing backdoor attacks on MCL can be mitigated by defenses, leaving the practical threat of attacks that persist after detection and clean fine-tuning to be examined.
Source: §1 · pp. 1–2Method
BadCLIP guides visual triggers toward target text semantics and target vision features to make the attack harder to detect and unlearn.
Source: §1 · p. 2Results
+45.3% ASR against fine-tuning baselines is reported for BadCLIP, which also achieves 92.50% ASR after FT and 89.60% after CleanClip.
Source: §1 · p. 2, §5.2 · p. 7Takeaways & Limitations
The authors aim to raise awareness of MCL backdoor threats and promote advanced backdoor defense research.
Source: §6 · p. 8The paper identifies attacks on complex MCL tasks and more robust backdoor detection and mitigation as limitations for future exploration.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Studying backdoor attacks is valuable for model copyright protection and enhancing defenses. While existing backdoor attacks have successfully infected multimodal contrastive learning models such as CLIP, they can be easily countered by specialized backdoor defenses for MCL models. This paper reveals the threats in this practical scenario that backdoor attacks can remain effective even after defenses and introduces the \emph{\toolns} attack, which is resistant to backdoor detection and model fine-tuning defenses. To achieve this, we draw motivations from the perspective of the Bayesian rule and propose a dual-embedding guided framework for backdoor attacks. Specifically, we ensure that visual trigger patterns approximate the textual target semantics in the embedding space, making it challenging to detect the subtle parameter variations induced by backdoor learning on such natural trigger patterns. Additionally, we optimize the visual trigger patterns to align the poisoned samples with target vision features in order to hinder the backdoor unlearning through clean fine-tuning. Extensive experiments demonstrate that our attack significantly outperforms state-of-the-art baselines (+45.3% ASR) in the presence of SoTA backdoor defenses, rendering these mitigation and detection strategies virtually ineffective. Furthermore, our approach effectively attacks some more rigorous scenarios like downstream tasks. We believe that this paper raises awareness regarding the potential threats associated with the practical application of multimodal contrastive learning and encourages the development of more robust defense mechanisms.
1. Introduction
The paper examines whether backdoors in multimodal contrastive learning can withstand defenses used after users obtain pretrained encoders. It proposes BadCLIP, a dual-embedding-guided attack, and reports stronger results than baselines against detection and fine-tuning defenses.
1. Introduction
- MCL models such as CLIP learn shared image–text representations from large-scale data and support downstream use through fine-tuning publicly available encoders.The paper situates its threat study in this practical usage scenario.Source: §1 · p. 1
- BadCLIP draws on two observations: small poisoned-versus-clean parameter deviations may evade detection, while a poisoned dataset resembling clean fine-tuning data may resist unlearning.The paper frames these observations from the perspective of the Bayesian rule.Source: §1 · p. 2
- +0.082 PL1-norm scores in backdoor detection and +45.3% ASR against fine-tuning are reported for BadCLIP versus other baselines.The authors also report that the attack bypasses state-of-the-art detection and fine-tuning defenses on CLIP models.Source: §1 · p. 2
- Its textual embedding consistency optimization brings visual triggers toward target-label text semantics, while visual embedding resistance optimization aligns poisoned samples with target vision features.The authors state that these choices aim to make the attack difficult to detect and unlearn through clean fine-tuning.Source: §1 · p. 2
2. Related Work
Related work covers multimodal contrastive learning models, especially CLIP, and prior backdoor attacks and defenses. The paper positions its work against MCL attacks that existing defenses can largely mitigate.
2.1. Multimodal Contrastive Learning
- CLIP learns image–text matching from large-scale paired data and is the paper’s selected target because of its broad impact.The paper describes CLIP as using 400M image–text pairs and notes its influence on subsequent MCL research.Source: §2.1 · p. 2
2.2. Backdoor Attacks and Defences
- Prior MCL backdoor work demonstrated attacks on CLIP, including successful poisoning with 0.01% of the data, while other studies examined modal attacks and SSL settings.The passage also names supervised-learning backdoor methods and SSL attack studies.Source: §2.2 · p. 2
- CleanCLIP mitigates backdoors through self-supervised multimodal augmentation and clean-data fine-tuning; other defenses assume access to poisoned data or only the poisoned model.The paper states that these defenses can largely reduce backdoor effects on MCL or SSL models, motivating a stronger attack.Source: §2.2 · pp. 2–3
3. Threat Model
The threat model targets a pretrained CLIP model and assumes an attacker can control its training process. The attacker poisons a small fraction of similar-distribution data, aiming to preserve normal benign behavior while retaining triggered behavior through detection and clean fine-tuning defenses.
3. Threat Model
- CLIP optimizes its visual and textual encoder parameters with InfoNCE, increasing similarity for positive image–text pairs and decreasing it for negative pairs.The model thereby maps similar pairs nearby in embedding space and dissimilar pairs farther apart.Source: §3 · p. 3
- The attack aims for normal behavior on benign inputs and incorrect embedded features on triggered inputs, while evading detection and sustaining efficacy after clean-image fine-tuning.The attacker collects data similar in distribution to the pretraining data and modifies a small fraction of clean samples.Source: §3 · p. 3
- The poisoned pairs include visual and text triggers, and the attacker fine-tunes the pretrained model on that poisoned dataset.The stated attacker pathway also allows users to download the poisoned dataset and train their own model without changing the standard CLIP paradigm.Source: §3 · p. 3
- The attacker is assumed to control training and have access to the pretraining data and CLIP’s white-box structure and parameters.The paper describes the attacker as injecting a backdoor into a clean pretrained CLIP model for efficiency.Source: §3 · p. 3
4. Approach
BadCLIP frames poisoning and clean fine-tuning as competing updates, then designs a dual-embedding attack to make backdoors harder to detect and remove. Its approach aligns visual triggers with target text and real target-image features, and selects varied poisoned samples.
4.1. Attack Motivation
- The analysis identifies two defense-evasion targets: keep poisoned-model parameter changes subtle to avoid detection and make D1 resemble D2 to resist clean fine-tuning.The Bayesian framing treats pretraining, poisoning, and clean fine-tuning as posterior updates.Source: §4.1 · pp. 3–4
4.2. BadCLIP Attack Design
- BadCLIP combines textual-embedding consistency and visual-embedding resistance optimization to target detection and fine-tuning defenses.The framework is motivated by the need for small parameter changes and less conflict between poisoned and clean fine-tuning data.Source: §4.2 · p. 4, §4.1 · p. 4
4.2.1 Textual Embedding Consistency Optimization
- Natural target-label descriptions combine text triggers with target text, while optimized visual triggers approach target-text semantics to reduce required encoder changes.This makes the visual trigger closer to target-label features before backdoor learning.Source: §4.2.1 · pp. 4–5
4.2.2 Visual Embedding Resistance Optimization
- Because D2 is inaccessible, the authors use D0 as its proxy and align triggered-image embeddings with authentic target images while penalizing other-category images.The poisoned data is intended to resemble authentic target representations and thereby resist clean fine-tuning.Source: §4.2.2 · p. 5
4.2.3 Overall Poisoning Process
- The final poisoning objective optimizes patch-based triggers with textual and visual losses; λ1 balances those contributions, while λ2 and η balance negative-sample distance.The visual terms encourage similarity to real target images and use other-category images as negatives.Source: §4.2.3 · pp. 5–6, §4.2.2 · p. 5
- Poisoned-pair construction combines boundary, farthest, and random samples at a 1:1:1 ratio, then adds triggers and target descriptions to the selected images.Boundary samples rank second-highest for the target class; farthest samples have low target-class predictions.Source: §4.2.3 · p. 6
5. Experiments
BadCLIP maintained high attack success across fine-tuning defenses, detection, and downstream and more rigorous settings. The experiments also examine its setup, component choices, and the limits of ABL defense in MCL.
5.1. Experiment Setup
- Experiments used OpenAI CLIP, 500K CC3M pairs with 1,500 poisoned banana samples, and zero-shot ImageNet-1K classification and Linear Probe evaluation.The setup also evaluated DECREE detection, FT, CleanCLIP, and the more rigorous ABL defense.Source: §5.1 · p. 6
5.2. Main Results
- BadCLIP reached 98.81% ASR without defense and retained 92.50% after FT and 89.60% after CleanCLIP.The paper reports the highest no-defense ASR among compared attacks and high ASR after both fine-tuning defenses.Source: Table 1 · p. 6, §5.2 · p. 7
- DECREE could not determine whether BadCLIP was infected: its L1 and PL1-norms were high, and its reversed triggers were distributed like the clean encoder rather than clustered.Figure 3 visualizes reversed triggers and reports L1 and PL1-norm values.Source: §5.2 · p. 7, Figure 3 · p. 7
5.3. Attacks on the Linear Probe Task
- BadCLIP achieved 99.14% ASR on Linear Probe and 66.40% against CleanCLIP, while most compared attacks had ASRs below 0.1%.Clean accuracies of the compared methods were mostly around 64%.Source: §5.3 · p. 7, Table 2 · p. 7
5.4. Attacks on More Rigorous Scenarios
- With cross-domain SBU data used for CleanCLIP fine-tuning, BadCLIP maintained 87.21% ASR while baseline ASRs dropped by up to 84%.Clean accuracy for both clean and infected models decreased, mostly below 50%, under this setting.Source: Table 3 · p. 7, §5.4 · p. 8
- During ABL, BadCLIP’s poisoned samples were distributed more like clean samples than those of BadNet or SSBA; after removing 2,000 samples and fine-tuning, its reported value was 89.03.The corresponding reported values were 70.01 for BadNet and 25.42 for SSBA.Source: §5.4 · p. 8, Figure 4 · p. 8
- The authors report that ABL has limited performance defending against backdoor attacks in MCL, motivating further unlearning strategies for MCL.This limitation is stated in the context of the ABL-based strategy.Source: §5.4 · p. 8
5.5. Analysis
- L + PPS gave the strongest CleanCLIP resistance at 89.6% ASR; ASR stabilized above 16 × 16 patches and peaked at 1,500 poisoned samples.Clean accuracy remained comparatively stable as the number of poisoned samples increased.Source: §5.5 · p. 8, Table 4 · p. 8, Figure 5 · p. 8
6. Conclusions and Future Work
The paper concludes that BadCLIP poses a strong threat in MCL because it remains effective under advanced backdoor defenses. Future work includes complex MCL tasks and more robust detection and mitigation methods.
6. Conclusions and Future Work
- BadCLIP remains effective under advanced backdoor defense methods and poses a strong threat in the MCL usage scenario.The authors aim to raise awareness of backdoor threats in MCL and promote further defense research.Source: §6 · p. 8
- The authors identify backdoor attacks for complex MCL tasks as a limitation to explore.Source: §6 · p. 8
- The authors also identify more robust backdoor detection and mitigation methods as future work.Source: §6 · p. 8