PaperlayerView exact revision on arXiv

Source-linked AI summary

PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors

Xinwei Liu, Xiaojun Jia, Yuan Xun, Hua Zhang, Xiaochun Cao

arXiv:2502.16167v2cs.CVcs.AI

TL;DR

Perturbation-based defenses for unauthorized T2I personalization assume training images are pre-perturbed and may be weakened by clean images or transformations. PersGuard embeds protective backdoors in pre-trained models, pairing backdoor behavior with prior preservation and a retention loss designed to mirror personalization loss. The paper reports superior protection to existing perturbation-based methods across gray- and black-box settings, multi-object protection, and facial identity protection.

  • Problem

    Perturbation-based defenses assume all training images are pre-perturbed and can lose effectiveness with mixed-source data or transformations, while degraded outputs may still expose protected features.

    Source: §I · p. 1, 3) Perturbation-based… · p. 4

  • Method

    PersGuard embeds protective backdoors in pre-trained T2I models and combines behavior, prior-preservation, and retention losses, with retention designed to mirror personalization loss.

    Source: §I · pp. 1–2

  • Results

    PersGuard reports superior privacy protection to existing perturbation-based methods across gray- and black-box settings, multi-object protection, and facial identity protection.

    Source: §I · p. 2, §V · p. 12

  • Takeaways & Limitations

    The framework is intended to retain protective behavior for protected images while allowing normal outputs for unprotected images after fine-tuning.

    Source: §I · p. 2

    Protection depends on centralized model distribution: it becomes ineffective if attackers obtain clean models elsewhere or train models from scratch.

    Source: §V · p. 12

Abstract

from arXiv · show

Diffusion models (DMs) have advanced text-to-image (T2I) synthesis, yet their personalization capabilities raise serious privacy and copyright concerns. Malicious actors can misuse these models to generate unauthorized portraits or artistic style replicas. Existing proactive defenses primarily rely on applying adversarial perturbations to reference images to disrupt training. However, these approaches face limitations: they assume all training images are pre-perturbed and are prone to failure when datasets contain unperturbed images or undergo minor data transformations. In this paper, we introduce PersGuard, a novel backdoor-based framework designed to prevent unauthorized personalization of pre-trained T2I diffusion models. Unlike perturbation-based methods, we assume protectors can embed protective backdoors into the models before their release. This mechanism ensures that if a downstream user fine-tunes the model on protected images, the model retains the backdoor and generates predefined protective outputs; conversely, for unprotected images, the backdoor is effectively removed during fine-tuning to ensure normal model utility. We formulate the backdoor injection as a unified optimization problem incorporating three objectives: a backdoor behavior loss to activate protection, a prior preservation loss to maintain standard generation capabilities, and a novel backdoor retention loss. The retention loss is specifically designed to mirror personalization loss, ensuring the backdoor remains robust during downstream fine-tuning. Extensive experiments across gray-box and black-box settings, multi-object protection, and facial identity protection demonstrate that PersGuard provides superior privacy protection compared to existing perturbation-based methods.

I. INTRODUCTION

Perturbation-based defenses can falter with mixed-source data and transformations, motivating a model-level approach to unauthorized personalization. PersGuard embeds protective backdoors before release, aiming to retain protection for protected images while preserving normal outputs for unprotected images.

I. INTRODUCTION

  • Existing perturbation defenses assume all training images are pre-perturbed and can weaken with mixed-source images or minor transformations.They may also degrade image quality without fully concealing protected visual features.Source: §I · p. 1
  • PersGuard retains protective backdoors after fine-tuning on protected images, while fine-tuning on unprotected images removes them to preserve normal outputs.The framework is designed for protectors who can embed backdoors into pre-trained models before release.Source: §I · p. 2, Figure 1 · p. 2
  • PersGuard combines backdoor behavior, prior preservation, and retention losses; the retention loss mirrors personalization loss to help the backdoor persist during fine-tuning.Prior preservation supports standard outputs for prompts without the identifier.Source: §I · p. 2
  • The paper introduces a backdoor-based alternative to perturbation-based protection for preventing unauthorized personalization.The authors describe it as the first such approach.Source: §I · p. 2
  • Experiments cover gray- and black-box settings, multi-object protection, and facial identity protection, with reported privacy protection superior to existing perturbation-based methods.The abstracted contribution statement does not provide quantitative comparisons.Source: §I · p. 2

II. RELATED WORK

The related work situates PersGuard among T2I personalization and backdoor research, then reviews text-conditioned diffusion models. Its stated gap is upstream backdoor protection that remains relevant after downstream personalization fine-tuning.

A. Personalization in T2I Diffusion Models

  • DreamBooth fine-tunes a pre-trained Stable Diffusion model on reference images to associate a rare identifier with a new concept; the paper focuses on it for its widespread adoption.Other methods described include textual embedding optimization, singular-value tuning, low-rank adaptation, and ID embeddings.Source: A. Personalization in T2I Diffusion… · p. 3

B. Backdoor Attacks on T2I Diffusion Models

  • Prior T2I backdoor studies target model synthesis, introduce bias, embed backdoors through personalization or model editing, or target the text encoder.The paper identifies no prior study of an upstream T2I backdoor passed to users who then fine-tune with personal data.Source: B. Backdoor Attacks on T2I Diffusion… · p. 3

1) Text-to-Image Diffusion Models:

  • T2I diffusion models condition the reverse diffusion process on text, with a denoiser predicting added noise to enable text-conditioned generation.The cited examples include Stable Diffusion.Source: 1) Text-to-Image Diffusion Models · p. 3

2) Personalization:

Personalization adapts a pre-trained T2I model to user-specific concepts using reference images and prompts. Perturbation-based defenses modify training images to disrupt personalization, but their practical effectiveness is constrained by mixed data and transformations.

2) Personalization:

  • DreamBooth personalizes a pre-trained model using a few reference images and prompts, while prior preservation helps prevent overfitting and maintain diverse class generation.Its objective includes a term weighted by λ for prior preservation.Source: 2) Personalization · p. 3

3) Perturbation-based Anti-personalization:

  • Perturbation-based defenses add imperceptible perturbations to reference images to disrupt the fine-tuned model’s performance.The surveyed methods use different strategies, including alternating surrogate and perturbation learning, adaptive search, meta-learning, and prompt-distribution modeling.Source: 3) Perturbation-based… · p. 3
  • Perturbation defenses depend on control of training data and lose effectiveness with clean images or standard transformations; degraded outputs may still retain identifiable features.Perturbations can also incur high computational costs because of iterative optimization.Source: 3) Perturbation-based… · p. 4

B. Threat Model

The threat model assumes protectors control distribution of the pre-trained model and downstream users fine-tune that protected model. It defines protector capabilities from full knowledge to little or no knowledge of the training images.

B. Threat Model

  • The approach repurposes backdoors from a security threat into a protection mechanism, focusing on DreamBooth because of its strong personalization capabilities.The paper assumes adversaries can trigger behavior while benign inputs retain normal generation performance.Source: B. Threat Model · p. 4

1) Protection Scenarios:

  • Protectors are modeled as providers that embed backdoors before release; protection does not apply if attackers instead use a clean model from another source.The authors consider this less common because public pre-trained models are widely adopted and attackers often lack resources to train diffusion models from scratch.Source: 1) Protection Scenarios · p. 4

2) Protector’s Background Knowledge and Capabilities:

  • The protector intervenes only before release, with downstream personalization unknown and uncontrollable; capability settings range from white-box to black-box.White-box knowledge includes identifiers, class names, prompts, and the protected dataset; gray-box lacks exact text details, while black-box has little or no control over training images.Source: 2) Protector’s Background Knowledge… · p. 4

3) Protector’s goal:

PersGuard aims to make protected-image personalization trigger protective outputs while leaving unprotected images unaffected. Its behavior loss associates identifiers with three possible protective targets: a pattern, erasure, or replacement object.

3) Protector’s goal:

  • PersGuard defines pattern, erasure, and target backdoors that respectively add a predefined pattern, remove the protected object, or replace it with a specified target.For example, an identifier for a protected dog can trigger generation of a rabbit.Source: 3) Protector’s goal · p. 4

C. Our Method: PersGuard

  • LLM-generated prompts build prior, identifier, erasure, and target prompt sets, which a clean Stable Diffusion model uses to synthesize reference images.Identifier prompts prepend tokens to descriptions containing the protected class; erasure prompts use negation, and target prompts substitute another class.Source: C. Our Method: PersGuard · p. 4
  • The training objective optimizes the backdoored model’s denoiser conditioned on prompts, with a frozen clean model used to synthesize reference images.The method defines the encoder and decoder and the latent representation of an image.Source: C. Our Method: PersGuard · p. 4

1) Backdoor Behavior Loss.:

  • The backdoor behavior loss associates identifiers with backdoor targets, using reference images with a preset pattern, erasure outputs, or target-object outputs.The pattern reference images are formed by adding the preset pattern to prior images.Source: 1) Backdoor Behavior Loss · pp. 4–5

2) Prior Preservation Loss.:

PersGuard combines prior preservation for normal generation with a retention loss designed for downstream personalization. The retention loss is intended to preserve protection after fine-tuning on protected images while allowing normal behavior for unprotected images.

2) Prior Preservation Loss.:

  • Prior preservation uses prior images to maintain normal functionality without an identifier, promote output diversity, and reduce backdoor overfitting.The loss is inspired by DreamBooth’s prior preservation loss.Source: 2) Prior Preservation Loss · p. 5

3) Backdoor Retention Loss.:

  • Backdoor retention encourages the model to learn the personalized training loss for protected images, preserving backdoor behavior through downstream fine-tuning.Because the loss is tailored to protected images, personalization on unprotected images can still diminish the backdoor and restore normal outputs.Source: 3) Backdoor Retention Loss · p. 5
  • PersGuard maintains consistently low DINOc similarity under mixed training data or data augmentations, while baselines fail significantly in those settings.Lower DINOc indicates better protection.Source: Figure 4 · p. 6

4) Optimization Problem.:

PersGuard optimizes a weighted combination of its protective objectives and trains by gradient descent. Experiments use DreamBooth object and subject categories, alongside facial identities, with specified training settings and evaluation metrics.

4) Optimization Problem.:

  • Training samples mini-batches from the backdoor behavior, prior preservation, and protected-image datasets, then updates the backdoored model by gradient descent.The loss weights λ1 and λ2 balance the objective terms; training repeats until the maximum epoch count.Source: 4) Optimization Problem · p. 6

A. Experimental Setup

  • DreamBooth evaluation covers 30 categories—21 object classes and 9 living-subject classes—with 20 training and 10 evaluation prompts per category.Facial privacy evaluation uses 307 CelebA-HQ identities, each with at least 15 center-cropped images resized to 512×512.Source: A. Experimental Setup · p. 6
  • In backdoor training, following DreamBooth’s default configuration, both the text encoder and UNet are fine-tuned for 300 steps with batch size 2 and learning rate 5×10−6.During validation, both components are fine-tuned for 50 steps. Most experiments use a white-box assumption, with grey- and black-box results also included.Source: A. Experimental Setup · p. 6
  • Backdoor effectiveness is measured with DINOb and CLIPb, subject fidelity with DINOc and CLIPc, and image quality with FID.Higher backdoor-effectiveness scores indicate successful activation; lower protected-image subject-fidelity scores indicate suppression, while higher unprotected-image scores indicate preserved utility.Source: A. Experimental Setup · p. 7

B. Main Results

PersGuard outperforms perturbation-based defenses in concealing protected features and remains robust when clean images or augmentations challenge those defenses. Unlike baseline backdoors, its protective behavior survives personalization while preserving utility on unprotected images.

1) Comparison with Perturbation-Based Protections:

  • PersGuard consistently outperforms perturbation-based baselines under MLLM evaluation and conceals protected features, while those baselines can degrade image quality without hiding recognizable features.Protection succeeds when four multimodal LLM judges classify personalized outputs and original protected images as different categories.Source: 1) Comparison with Perturbation-Based… · p. 7
  • Perturbation-based baselines lose efficacy when clean external images or data augmentations enter training, whereas PersGuard remains robust across the evaluated settings.The scenarios include training with one clean external image or mostly external images, as well as common transformations and their combinations.Source: 1) Comparison with Perturbation-Based… · p. 7

2) Comparison with Baseline Backdoors.:

  • Existing backdoor baselines are erased during downstream personalization, while PersGuard reliably protects protected instances without disrupting personalization quality for unprotected images.Unprotected images are evaluated from both the same class as the protected target and different classes.Source: 2) Comparison with Baseline Backdoors · p. 7
  • PersGuard variants maintain general-generation capabilities comparable to a clean model on both prior and neutral prompts.The authors report this as evidence of high stealthiness and preserved utility for legitimate users.Source: 2) Comparison with Baseline Backdoors · p. 8

C. Visualization

PersGuard’s three variants produce distinct protective outputs for protected images while leaving unprotected personalization normal. Attention visualizations and loss ablations describe how selective protection is achieved and maintained.

1) Visualization Results:

  • Protected-image outputs show a warning patch for Pattern-Backdoor, object removal for Erasure-Backdoor, and target substitution for Target-Backdoor; unprotected outputs remain like clean-model results.These examples illustrate selective protection across four protected and unprotected image categories.Source: 1) Visualization Results · p. 8, Table IV · p. 9

2) Attention Map:

  • Clean-model attention for “sks” centers on a dog’s head, while Pattern and Erasure redirect it to the upper-left pattern and background; Target retains head focus during rabbitlike transformation.The attention maps are generated with DAAM.Source: 2) Attention Map · pp. 8–9, Figure 5 · p. 8

1) Loss Components Analysis:

  • Backdoor Retention Loss is necessary for robust protection because its absence allows catastrophic forgetting to remove the backdoor during fine-tuning.The ablation evaluates protection with DINOc and DINOb and general-generation alignment with DINOpr.Source: 1) Loss Components Analysis · p. 9, Table IV · p. 9
  • Prior Preservation Loss regularizes against trigger overfitting and preserves the model’s original generative prior.The table caption reports that combining all three losses achieves the optimal balance between protection robustness and model utility.Source: 1) Loss Components Analysis · p. 9, Table IV · p. 9

2) Gray-Box Setting:

Universal training variants address gray-box uncertainty, and PersGuard-UID retains efficacy in black-box evaluation and across model architectures. Protection remains strong for weight-tuning personalization, with attenuation reported for Textual Inversion.

2) Gray-Box Setting:

  • Vanilla PersGuard’s protection degrades when attacker identifiers, classes, or prompts differ from the white-box assumptions; prompt diversification improves gray-box robustness more than token diversification, and combining both generalizes best.The evaluated unseen attacker settings use identifier “xyz,” class “puppy,” and prompt “A picture of xyz puppy”; PersGuard-UI diversifies identifiers, PersGuard-UD prompts, and PersGuard-UID both.Source: 2) Gray-Box Setting · p. 9

3) Black-Box Setting:

  • PersGuard-UID retains significant efficacy in the black-box setting, where the user fine-tunes on unseen images; perturbation defenses are ineffective there because they rely on access to the exact perturbed images.The evaluation reserves two-thirds of target images for protector training and uses the remaining images for downstream fine-tuning.Source: 3) Black-Box Setting · p. 9, Table V · p. 10

4) Model Generalizability:

  • Across Stable Diffusion versions v1.5 to v3.5, PersGuard suppresses protected-image alignment while unprotected-image metrics remain close to the clean baseline.The reported protected metrics are DINOc and CLIPc.Source: 4) Model Generalizability · p. 9, Table VI · p. 10

5) Robustness against Personalization Techniques:

E. Case Study (Face Protection)

The face case study reports protection of five identities during personalization, while multi-backdoor experiments show simultaneous protection across categories with a marginal trade-off at higher capacity.

E. Case Study (Face Protection)

  • The multi-identity face setup requires including all five identities in the backdoor retention-loss training set, even though they share the same trigger identifier and class name.The study treats simultaneous identity protection as a practical face-privacy case.Source: E. Case Study (Face Protection) · p. 10
  • PersGuard’s target backdoor triggers its protective output for all five CelebA-HQ identities during downstream fine-tuning, using shared identifier “sks” and class “person.”The backdoor target class is “Superman”; the reported table shows low DINOc and high DINOb across subjects.Source: E. Case Study (Face Protection) · pp. 10–11, Table VIII · p. 11

1) Impact of Multi-Backdoor Integration:

  • A unified model protects dogs, backpacks, and toys simultaneously, with marginally lower effectiveness than single-backdoor models, likely from parameter contention.Single-backdoor models show category-confined protection without cross-interference; the figure describes stable inter-category capacity and mild protected-performance decline at high intra-category capacity.Source: 1) Impact of Multi-Backdoor Integration · p. 11, Figure 6 · p. 11, Table IX · p. 11

2) Scalability and Protection Effectiveness Curves:

PersGuard scales across protection sets, with stable protected performance for distinct semantic classes and a mild decline for same-category targets. The conclusion reports strong protection across tested settings while identifying distribution assumptions and responsible-deployment concerns.

2) Scalability and Protection Effectiveness Curves:

  • Protected performance remains highly stable as capacity grows across distinct semantic classes, while unprotected personalization quality remains favorable.The paper reports that interference among backdoors is minimal in this inter-category regime.Source: 2) Scalability and Protection… · pp. 11–12
  • Same-category protection is more demanding: protected performance declines mildly with capacity, while unprotected utility varies only modestly.The authors attribute the decline to semantic saturation and increased competition for representation space.Source: 2) Scalability and Protection… · p. 12

V. CONCLUSION

  • PersGuard uses a unified optimization framework to embed Pattern, Erasure, and Target backdoors, with Backdoor Retention Loss keeping triggers active after fine-tuning.The conclusion reports superior protection efficacy and stability to state-of-the-art baselines across gray-box, black-box, and multi-identity facial-protection scenarios.Source: §V · p. 12
  • Protection assumes centralized model distribution; it is ineffective if attackers obtain clean models elsewhere or train from scratch.The authors note that unrestricted weight-level access or alternative personalization strategies may attenuate protection.Source: §V · p. 12
  • Because protective backdoor techniques could be misused, responsible deployment requires governance such as verified protection requests and transparent disclosure policies.Source: §V · p. 12

Related papers

Loading 2502.16167v2…