CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning

1. Introduction

• Downloaded pre-trained models may carry unknown security risks, making lightweight defenses applied during fine-tuning important.Existing defenses primarily target pretraining, while users often download and fine-tune models for personalized applications.Source: §1 · p. 1
• CleanCLIP’s self-supervised defenses can fail against BadCLIP, which exploits hidden toxic features in the clean feature space.Traditional attacks create detectable pseudo-semantic clusters, whereas BadCLIP targets true feature regions without creating them.Source: §1 · p. 2
• CleanerCLIP augments fine-tuning with positive and negative subtexts to disrupt the semantic link between a trigger and its target.Negative subtexts alter semantic components, while positive subtexts preserve essential meaning for processing clean data.Source: §1 · p. 2
• Across six attack techniques, the authors report reduced attack success rates while maintaining benign accuracy and usability.The paper presents CleanerCLIP as a lightweight fine-tuning defense strategy.Source: §1 · p. 2

2. Related work and preliminaries

• CLIP maps images and text into a shared feature space and learns by bringing matching pairs closer while separating mismatched pairs.Its image and text encoders are trained on large-scale internet image-text pairs.Source: §2 · pp. 2–3
• Backdoor attacks associate trigger patterns with malicious behavior, and CLIP attacks can poison image-text pairs to establish that association.Prior work reported a successful CLIP attack using 0.01% poisoned data.Source: §2 · p. 3
• Fine-tuning defenses address attacks on existing pretrained models when defenders cannot access the original poisoned training set.The paper contrasts this setting with full retraining using augmented data, as in RoCLIP.Source: §2 · p. 3
• CleanCLIP fine-tunes on clean data using a self-supervised loss based on multimodal data augmentation to reduce backdoor effects.The paper identifies CleanCLIP as a representative defense for the fine-tuning phase.Source: §2 · p. 3

3.1. Threat model

• The threat model allows an attacker to inject a small number of poisoned samples or fine-tune and distribute a poisoned pretrained model, while the defender has clean fine-tuning data.The attacker is assumed to know the model structure, training algorithm, and victim hyperparameters but cannot directly modify the training process.Source: §3.1 · p. 3

3.2. Counterfactual Text Augmentation

• CleanerCLIP generates factual positive and counterfactual negative sub-captions, randomly selecting a counterfactual strategy and augmenting only a small sample subset each epoch.The selective augmentation is described as limiting computational overhead.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
• Factual sub-captions retain key nouns and relational verbs while simplifying or omitting adjectives to preserve the original caption’s core meaning.For very simple captions, the method may retain only the most critical word.Source: §3.2 · p. 4
• Counterfactual sub-captions randomly replace subject or object adjectives, relational verbs, or nouns to disrupt trigger-target binding.The method selects one of these replacement types for each sample.Source: §3.2 · p. 4

3.3. CleanerCLIP: Fine-grained Counterfactual Semantic Finetuning

• Augmenting only K randomly selected samples is intended to preserve clean image-text alignment and downstream generalization while reducing added computation.The paper avoids applying fine-grained augmentation to the whole dataset because it may lower downstream zero-shot accuracy.Source: §3.3 · p. 5
• The positive-negative loss uses image-to-text and text-to-image terms to jointly optimize consistency in the multimodal embedding space.Positive and negative sample temperatures affect similarity-score sensitivity, and the complete objective weights this loss alongside the CleanCLIP loss.Source: §3.3 · p. 5

4.1. Setup

• Experiments use OpenAI CLIP fine-tuned on CC500K; six poisoned models are evaluated with Top-k BA/ASR on ImageNet1K zero-shot classification and linear probing.The paper notes that prior CLIP backdoor defenses had not reported Top-k metrics.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.2. CleanerCLIP performance

• 71.75% lower BadCLIP ASR relative to the victim model at Top-10; CleanCLIP’s Top-10 ASR is 91.96%.CleanerCLIP also achieves near-zero Top-1 ASR against BadNet and WaNet: 0.45% and 0.15%, respectively.Source: Table 1 · p. 6, §4.2 · p. 7
• CleanerCLIP reduces Top-1 ASR against BadNet from 91.28% to 0.45% and against WaNet from 91.83% to 0.15%.The reported zero-shot results compare CleanerCLIP with the victim model and evaluate six attack methods using Top-k BA and ASR.Source: §4.2 · pp. 6–7
• CleanerCLIP retains almost the same benign accuracies as CleanCLIP and produces comparable linear-probe results across datasets.The authors report reduced ASR without compromising performance and transferability in the linear-probe evaluation.Source: §4.2 · p. 7
• CleanerCLIP often reduces ASR within the first defense epoch and converges more steadily than CleanCLIP on BadNet, SIG, and BadCLIP.The authors attribute the faster defense to fine-tuning with a few thousand additional samples.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3. Analysis

• Adding positive and negative sub-samples through lossp−n improves defense over CleanCLIP without compromising clean-sample performance; increasing β/α further improves defense.The weight ablation sets α to 1 and varies β.Source: §4.3 · p. 7
• When tp exceeds tn, negative samples receive less attention and defense worsens; the study adopts tp = tn = 0.3 as its default.The authors report minimal impact on BA when tp = 0.3.Source: §4.3 · p. 8
• Fifty samples suffice for BadNet, while 2,000–3,000 are needed for BadCLIP; increasing negative samples reduces ASR without notably affecting BA.These sample counts are small relative to the 500K fine-tuning dataset.Source: §4.3 · p. 8

• CleanerCLIP reduces attack success rates across datasets and attack scenarios while maintaining benign accuracy.Source: §5 · p. 9
• The defense primarily addresses image-modality attacks, leaving its performance against text-modality attacks unknown.Source: §5 · p. 9