DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors

Tuo Chen; Jie Gui; Minjing Dong; Lanting Fang; Ju Jia; Benlei Cui; Jian Liu

Source-linked AI summary

DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors

Tuo Chen, Jie Gui, Minjing Dong, Lanting Fang, Ju Jia, Benlei Cui, Jian Liu

arXiv:2608.25851v1Aug 26, 2026cs.CV

TL;DR

Backdoored SSL encoders are difficult to detect without assumptions about the encoder, training data, or attack, motivating a general defense. DEFUSE reconstructs encoder representations with a conditional diffusion model and evaluates semantic consistency rather than exact pixels. Experiments report strong, consistent detection across visual and vision-language encoders and diverse attacks, while the threat model remains black-box and assumes access to public generative models, encoders, and internet datasets.

Problem
Existing SSL backdoor defenses are often paradigm-specific or depend on prior knowledge, clean data, or pseudo-labels, limiting detection when the encoder or attack is unknown.
Method
DEFUSE conditions a fine-tuned diffusion model on suspect encoder representations, reconstructs images, and measures semantic consistency in a reference representation space.
Results
DEFUSE achieves the best performance across all evaluated attacks, maintains AUPRC above 0.8 on SSLBKD and DRUPE, and generalizes across visual SSL and vision-language encoders.
Takeaways & Limitations
DEFUSE detects anomalous representations without prior knowledge of the victim encoder or attack strategy and demonstrates strong performance across diverse evaluation settings.
Takeaways & Limitations
The black-box threat model assumes access to publicly available generative models, encoders, and large-scale internet datasets.

Abstract

from arXiv · show

Self-supervised learning (SSL) encoders are vulnerable to backdoor attacks, posing threats to both visual SSL encoders and vision-language encoders. Existing defenses are typically designed for only one of these paradigms and rely on restrictive assumptions such as access to uninfected in-distribution data or precomputed pseudo-labels, which are difficult to satisfy in practice. To address these limitations, we propose DEFUSE, a generalizable backdoor detection framework for SSL encoders. Inspired by Bayesian posterior inference, we reformulate backdoor detection as a representation-conditioned image likelihood estimation problem parameterized by a conditional diffusion generative model. Uninfected representations tend to yield semantically consistent reconstructions, whereas backdoored ones are more likely to be mapped to the attacker's target class or semantically meaningless images, deviating from the original semantics and thereby exposing the backdoor. However, we find that the exact likelihood is intractable, because highly abstracted representations discard the low-level information necessary for pixel-faithful reconstruction. We therefore relax the objective to semantic reconstruction and evaluate it in a well-separated representation space provided by a reference encoder. Rather than training from scratch, we fine-tune a pretrained diffusion model, leveraging its generative prior to map data onto the natural image manifold while preserving semantic content. Extensive experiments demonstrate that DEFUSE substantially outperforms existing detectors across diverse attack settings, generalizing to both visual SSL and vision-language encoders. Notably, our method greatly reduces the reliance on prior knowledge about the victim encoder or the attack strategy. The source code is available at https://github.com/jsrdcht/DEFUSE .

1 Introduction

SSL encoders can be backdoored across visual and vision-language paradigms, while existing defenses often depend on paradigm-specific assumptions or prior knowledge. DEFUSE detects anomalous representations by reconstructing them into images and measuring semantic consistency, achieving broad effectiveness across attack settings.

Threat: Backdoor attacks can make triggered inputs map near an attacker-designated target representation while clean inputs remain normal.Attacks may poison pretraining data or directly manipulate encoder parameters.
Limitations of prior defenses: Existing defenses are limited by SSL-paradigm specificity and dependencies on prior knowledge, clean data, or pseudo-labels.Uni-paradigm methods may fail to transfer across visual and multimodal encoders.
DEFUSE: DEFUSE reformulates backdoor detection as representation-conditioned image likelihood estimation using a conditional diffusion model.Its pipeline conditions diffusion generation on suspect encoder representations and evaluates reconstructed images for semantic consistency.
DEFUSE: Because abstract representations discard spatial details, DEFUSE relaxes pixel-faithful reconstruction to semantic alignment in a reference representation space.Generative priors constrain outputs toward the natural image manifold while preserving semantic content.
Results: DEFUSE is agnostic to SSL paradigm and victim or attack knowledge, and experiments show strong performance across diverse attacks and encoder types.The method is reported to remain robust under partial poisoning and adaptive white-box attacks.

3 Preliminaries

SSL encoders learn abstract representations through contrastive objectives, but backdoor training can redirect triggered inputs toward attacker-selected targets. Conditional diffusion models provide a way to generate images from external representation or text conditions.

SSL encoders: SSL encoders map images to abstract representations and train positive image or image-text pairs to have similar features.Contrastive learning typically uses an InfoNCE objective and cosine similarity.
Backdoor behavior: Backdoored encoders map triggered inputs close to attacker-selected target-class representations in latent space.A compromised CLIP model could associate a trigger with a target prompt such as “a photo of a banana.”
SSL interpretation: Self-supervised encoders are described as density-ratio-based mutual-information estimators.The proportionality relation is defined up to a multiplicative constant.
Diffusion models: Diffusion models learn a forward noising chain and a neural network that predicts noise to reverse the process during generation.The training objective uses the discrepancy between added and predicted noise.
Conditional diffusion: Conditional diffusion models incorporate an external condition such as a class label, text prompt, or pre-extracted feature vector.At sampling time, the reverse chain generates images from the conditional distribution p(x(0) | y).
Motivation: Figure 2 contrasts reconstruction performance across CLIP encoders with the unreliability of low-level distances for semantic consistency.Semantically similar images can have high pixel-space distances.

4 Method

DEFUSE detects backdoors by reconstructing images from suspect encoder representations and evaluating semantic consistency rather than exact pixel recovery. It operates in a black-box setting, adapting a pretrained conditional diffusion model and comparing originals with reconstructions in a reference representation space.

Problem formulation: The defender identifies backdoored inputs from a suspicious encoder without prior knowledge of training data or attack strategy, using publicly available generative models and encoders.The considered setting is black-box detection with access to resources such as Stable Diffusion, DINOv2, and large-scale Internet datasets.
Problem formulation: DEFUSE reformulates backdoor detection as representation-conditioned image likelihood estimation using a conditional diffusion model.The Bayesian reformulation assesses how well an image is explained by a generative model conditioned on its encoder representation.
Semantic reconstruction: Because global representations discard spatial details, DEFUSE relaxes pixel-faithful reconstruction to semantic reconstruction.Exact image likelihood estimation is unreliable when pooled or CLS-token features lack low-level information needed for pixel-accurate generation.
Conditional generative model: The method fine-tunes a pretrained conditional diffusion model to reconstruct images from encoder representations, projecting each representation into four image-prompt tokens.A learned linear layer produces the conditioning tokens, while standard diffusion noise-prediction loss trains the reconstruction model.
Conditional generative model: DEFUSE replaces original diffusion prompts with image prompts and uses deterministic DDIM sampling to obtain the final reconstructed image.The reconstructed image is produced after marginalizing the diffusion path from the noisy state to x(t=0).
Semantic consistency: Semantic consistency is measured by cosine similarity between the original and reconstructed images in a well-separated reference encoder space.The reference encoder can be DINOv2, and this high-level metric addresses semantic similarity despite viewpoint or layout differences.

5 Experiments

Experiments evaluate DEFUSE across encoders, attacks, metrics, generative models, triggers, adaptive poisoning, and inference budgets. DEFUSE achieves the best performance across attacks, while high-level semantic metrics, suitable fine-tuning, and generative priors support its effectiveness.

Experimental Setup: The evaluation uses CLIP ViT-B and SimSiam ResNet18 encoders across ImageNet-based datasets, seven backdoor attacks, and standard detection metrics.The attacks include four data-poisoning and three model-manipulation methods; metrics include Recall, Precision, TPR, FPR, AUROC, and AUPRC.
Main Results: DEFUSE achieves the best performance across all evaluated attacks, whereas DBCL is effective mainly against CLIP Backdoor and DeDe degrades at 224×224 resolution.The reported degradation of DeDe is attributed to poorer image generation quality and less reliable pixel-space distances at higher resolution.
Main Results: A linear model captures most optimal threshold points across datasets, with R2 = 0.67, enabling threshold estimation from DINOv2 similarity.Optimal thresholds are selected by maximizing Youden’s J statistic.
Ablation Study: High-level consistency metrics, especially DINOv2 similarity, outperform low-level metrics, supporting semantic rather than direct pixel-space evaluation.L2 acts as a proxy for P(x|z), and its weaker performance supports the impracticality of directly estimating the likelihood.
Ablation Study: SDXL outperforms SD1.5 and PixelFlow in convergence speed and generation quality, while unfreezing cross-attention parameters except q-projection generally performs best.Detection performance also improves with increasing feature similarity.
Ablation Study: Fine-tuning improves AUROC by 0.09–0.17 over training-free methods, and 20 inference steps suffice for the best performance.DEFUSE remains effective across HTBA, Blended, Watermark, and SIG triggers, and adaptive poisoning with up to approximately 3.33% malicious samples does not significantly affect it.
Adaptive Attack: Adversarial noise increases reconstruction similarity and reduces the separability of backdoor images under the evaluated adaptive attack.The experiment uses 20 PGD steps with step size 2/255 and examines 50 randomly selected images.

6 Conclusion

The conclusion presents DEFUSE as a generalizable framework that detects anomalous representations by reconstructing images and evaluating semantic consistency. Experiments across visual SSL and vision-language encoders support its effectiveness under diverse settings without prior knowledge of the victim encoder or attack strategy.

6 Conclusion: DEFUSE projects suspicious representations into image space and evaluates semantic consistency to identify anomalous representations.The framework is designed without prior knowledge about the victim encoder or attack strategy.
6 Conclusion: Experiments across visual SSL and vision-language encoders demonstrate strong and consistent detection performance under diverse evaluation settings.The conclusion highlights generative priors as a potential tool for securing self-supervised encoders in practice.

S1 Impact of Detection Preference

The supplementary analysis shows that L2 and DINOv2 similarity favor different image properties. L2 reflects low-level visual complexity, whereas DINOv2 similarity emphasizes clear, semantically coherent foreground objects.

S1 Impact of Detection Preference: L2 and DINOv2 similarity rank substantially different image types because they emphasize low-level complexity and semantic coherence, respectively.The analysis covers 150,000 CC3M images.
S1 Impact of Detection Preference: L2 is harder on images with strong contrast, local fluctuations, bright regions, clutter, text, logos, icons, and fine-grained structures.Smooth scenes with homogeneous backgrounds and weak texture variation are typically easier under L2.
S1 Impact of Detection Preference: DINOv2 similarity favors single salient objects, clear categories, limited object count, clean composition, centered subjects, and sizable backgrounds.It scores crowded, ambiguous, multi-object, fine-grained, or symbolically distracting images lower.

S2 Generalization of ImageNet-Trained CDDM to OOD Domains

The study tests whether an ImageNet-finetuned SDXL detector transfers beyond natural-image data. It reports strong performance on medical and remote sensing datasets.

S2 Generalization of ImageNet-Trained CDDM to OOD Domains: The ImageNet-finetuned SDXL detector is evaluated on medical and remote sensing datasets to test transfer beyond natural-image domains.The experiment examines whether the generator captures transferable semantic structure rather than ImageNet-specific appearance statistics.
S2 Generalization of ImageNet-Trained CDDM to OOD Domains: The detector maintains strong performance on the evaluated medical and remote sensing datasets.The result is reported in Table S2.

S3 Do Objects Matter for Caption-based Similarity?

Object-focused captions improve Caption CLIP’s ability to detect backdoored images, indicating that object semantics drive much of its discriminative power.

AUPRC improves from 0.904 to 0.938 when captions describe the five main objects rather than overall image content.The intervention modifies the Qwen3-VL-235B-A22B prompt to produce object-centric English descriptions.

S4 Masked Autoencoders are not Good CDDMs

Masked autoencoders are unsuitable as conditional diffusion models because their auxiliary visual information can dominate reconstruction and suppress the suspect encoder’s representation.

Masked autoencoders provide rich visual information that facilitates reconstruction but causes the decoder to largely ignore backdoor-encoder information.This creates a trade-off between reconstruction quality and dependence on the representation being evaluated.

S5 Low-level Distances are Susceptible to the Curse of Dimensionality

Low-level distances become unreliable for comparing high-dimensional reconstructions because Euclidean distances converge toward equivalent scales, motivating semantic metrics such as DINOv2.

For high-dimensional images, low-level distances such as L1 and L2 lose sensitivity and are therefore inadvisable for evaluation.The section motivates alternatives that preserve meaningful semantic differences.
As dimensionality increases, Euclidean distances become nearly equivalent up to constant factors.
Figure S2 compares the top-20 and bottom-20 reconstructed images under the L2 and DINOv2 metrics.

S6 Experimental Details

The experiments cover vision-language and visual SSL encoders, multiple domains, reconstruction metrics, triggers, and training configurations. Supplementary tables and figures document these settings and downstream evaluations.

Experimental setup: Stable Diffusion XL experiments use an IP-Adapter- and Diffusers-based implementation with a default 50 × 50 HTBA trigger for CLIP.The poisoned CLIP model is fine-tuned for 10 epochs with learning rate 1 × 10−4 and batch size 256.
Experimental setup: Visual SSL experiments pretrain SimSiam-ResNet18 for 300 epochs and evaluate ASR with a linear-probe protocol.The setup uses an HTBA trigger of size 50×50 and the ImageNet-100 class split from prior work.
Supplementary analyses: Table S1 reports explanatory variables for L2 and DINOv2 similarity on 150,000 CC3M images.The full-feature model reaches R2=0.503 for L2 and R2=0.066 for DINOv2 similarity.
Supplementary analyses: Tables S2–S6 cover cross-domain generalization, finetuning hyperparameters, SimSiam pretraining, training-free versus finetuned variants, and downstream performance.The reported experimental materials include medical and remote-sensing domains, ImageNet-finetuned SDXL, and comparisons involving SSLBKD, BadEncoder, CLIP, Backdoor, and BadCLIP.
Supplementary analyses: Figure S3 evaluates Caption CLIP scores using overall-image and object-centric descriptions, while Figure S4 varies DEDE trigger size.