PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models

Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, Yu-Gang Jiang

arXiv:2407.12383v2cs.CV

TL;DR

Concept-erasure methods for text-to-image models can be incomplete, costly, and damaging to generation ability. RECE derives and erases embeddings through closed-form edits with regularization; experiments report effective removal and strong red-teaming robustness.

  • Problem

    Existing concept-erasure methods can require substantial fine-tuning, degrade generation ability, and leave erased concepts vulnerable to red-teaming.

    Source: §1 · p. 2

  • Method

    RECE derives embeddings that make the edited model regenerate erased concepts, then repeatedly erases them with regularization using closed-form cross-attention edits.

    Source: §1 · p. 2, Finding Target Contents · p. 6, Regularization Term · p. 7

  • Results

    RECE yields the lowest number of nude body parts while preserving normal COCO-30k content, and has best average red-teaming robustness, with Ring-A-Bell ASR at 13.38%.

    Source: §4.1 · p. 9, Results · p. 12

  • Takeaways & Limitations

    The closed-form process takes 3 seconds while preserving unerased-concept generation ability and robustly refraining from generating erased-concept content.

    Source: §1 · p. 3

    RECE and other methods show erasure bias on female information, attributed to Stable Diffusion's nudity concept tending to generate more female-oriented content.

    Source: Nudity Bias · p. 10

Abstract

from arXiv · show

Text-to-image models encounter safety issues, including concerns related to copyright and Not-Safe-For-Work (NSFW) content. Despite several methods have been proposed for erasing inappropriate concepts from diffusion models, they often exhibit incomplete erasure, consume a lot of computing resources, and inadvertently damage generation ability. In this work, we introduce Reliable and Efficient Concept Erasure (RECE), a novel approach that modifies the model in 3 seconds without necessitating additional fine-tuning. Specifically, RECE efficiently leverages a closed-form solution to derive new target embeddings, which are capable of regenerating erased concepts within the unlearned model. To mitigate inappropriate content potentially represented by derived embeddings, RECE further aligns them with harmless concepts in cross-attention layers. The derivation and erasure of new representation embeddings are conducted iteratively to achieve a thorough erasure of inappropriate concepts. Besides, to preserve the model's generation ability, RECE introduces an additional regularization term during the derivation process, resulting in minimizing the impact on unrelated concepts during the erasure process. All the processes above are in closed-form, guaranteeing extremely efficient erasure in only 3 seconds. Benchmarking against previous approaches, our method achieves more efficient and thorough erasure with minor damage to original generation ability and demonstrates enhanced robustness against red-teaming tools. Code is available at \url{https://github.com/CharlesGong12/RECE}.

1 Introduction

Existing concept-erasure methods can be costly, damage generation ability, and leave inappropriate concepts recoverable. RECE iteratively derives and erases new embeddings using closed-form edits, with regularization to preserve unrelated generation.

1 Introduction

  • Earlier safeguards can be costly or circumventable, while concept-erasure methods often require extensive fine-tuning and still leave concepts vulnerable to red-teaming.The introduction also identifies copyright infringement and NSFW generation as safety concerns.Source: §1 · p. 2
  • RECE derives embeddings that make an edited model regenerate erased concepts, then erases those embeddings through repeated closed-form edits.The method builds on UCE and edits cross-attention key and value projection matrices.Source: §1 · p. 2
  • A regularization term limits parameter deviation to preserve generation ability for unerased concepts, and the closed-form process takes 3 seconds.RECE is also described as robustly refraining from generating erased-concept content.Source: §1 · pp. 2–3
  • Experiments assess erasure of unsafe content, artistic styles, and object classes, as well as robustness against three red-teaming tools and fine-tuning duration.Source: §1 · p. 3

2 Related Work

Related work surveys training-data filtering, output-side interventions, and parameter editing as approaches to inappropriate T2I content. It also describes red-teaming tools that probe safety mechanisms, including model-agnostic attacks.

2 Related Work

  • Prior safety strategies filter training data and retrain, censor outputs or steer inference, or fine-tune model parameters to erase inappropriate concepts.The text notes computational costs, performance degradation, circumvention of post-hoc interventions, and red-teaming vulnerability among these approaches.Source: T2I Diffusion Models with Safety… · p. 3

T2I Diffusion Models with Safety Mechanisms.

  • T2I red-teaming tools probe whether deployed safety mechanisms can be bypassed by automatically constructing problematic prompts.Ring-A-Bell extracts a holistic representation of inappropriate concepts before producing prompts.Source: Red-Teaming Tools for T2I Diffusion… · pp. 3–4

Red-Teaming Tools for T2I Diffusion Models.

3 Method

RECE combines closed-form cross-attention editing with iterative derivation and erasure of embeddings that can recover erased concepts. A regularization term is intended to limit effects on unrelated concepts.

3.1 Preliminaries

  • Latent diffusion models use a U-Net to denoise a VAE’s low-dimensional latent space, supporting text-conditioned image synthesis.Text embeddings are integrated into the U-Net through cross-attention.Source: §3.1 · p. 4

Text-to-Image (T2I) Diffusion Models

  • Cross-attention transforms text embeddings into keys and values, which interact with visual queries to produce attention outputs.The projections use Wk and Wv; keys compute an attention map with visual queries, and values are attended over.Source: §3.1 · p. 4, Text-to-Image (T2I) Diffusion Models · p. 4

Concept Erasing with Closed-form Solution

  • UCE edits cross-attention weights in closed form, mapping erased concepts toward destination outputs while preserving designated concepts.The method avoids fine-tuning and is used as RECE’s fast editing step.Source: Concept Erasing with Closed-form… · pp. 4–5

3.2 Reliable and Efficient Concept Erasure (RECE)

Finding Target Contents

  • The derived embedding c′ matches the original concept’s projected representation under the edited model, and can restore nudity generation in UCE.The target is to make W new c′ resemble W old c; the paper uses this recovery to identify incomplete erasure.Source: Finding Target Contents · p. 6

Regularization Term

  • A regularization term favors small derived embeddings to reduce their impact on unrelated concepts, while RECE iteratively erases the refined embeddings.Directly erasing c′ without this safeguard can reduce image quality for unrelated concepts.Source: Regularization Term · p. 7

4 Experiments

Experiments compare RECE with prior methods on unsafe-content and artistic-style removal, red-team robustness, and editing cost. Reported results combine strong removal with preservation of normal or unrelated content and short editing time.

4 Experiments

  • Experiments use SD V1.4, measuring nudity on 4,703 I2P images and normal-image generation on COCO-30k with CLIP-score and FID.I2P includes varied inappropriate prompts, but the reported detector evaluation focuses on nudity.Source: §4 · p. 8, §4.1 · pp. 8–9

4.1 Unsafe Content Removal

  • RECE yields the fewest nude body parts while preserving COCO-30k specificity; its FID is close to UCE and second-best SLD.In Fig. 3’s second-row prompt, only UCE, SLD, and RECE also capture facial features.Source: Table 1 · p. 9, §4.1 · pp. 9–10, Figure 3 · p. 10

4.2 Artistic Style Removal

  • RECE achieves the best LPIPSd, the reported overall measure of artistic-style erasure.LPIPSe measures erased artists and LPIPSu measures unerased artists; lower LPIPSu indicates less impact on unrelated artists.Source: Table 3 · p. 11, Removal Results · p. 11

4.3 Robustness Against Red-teaming Tools

  • 13.38% ASR under Ring-A-Bell was RECE’s lowest black-box result; RECE was best on average and best or near second-best against white-box attacks.CA performed best against the white-box attack.Source: Table 4 · p. 12, Results · p. 12

4.4 Model Editing Duration

  • 3 seconds: RECE achieves the best reported concept-erasure effect; despite similar editing duration, it significantly outperforms UCE in removal effectiveness.RECE and UCE modify the lowest percentage of parameters using closed-form editing.Source: Table 5 · p. 13, §4.4 · p. 13

Effect of Derived Embeddings among Epochs

  • Epochs 0–2 yielded derived embeddings with nudity information, while epochs 3–4 did not; the authors select the epoch-2 checkpoint to avoid harming normal generation.Fig. 5 illustrates the generated images used to assess each embedding.Source: Figure 5 · p. 13, Effect of Derived Embeddings among… · p. 13
  • As the regularization coefficient increases, CLIP-score rises while LPIPS and parameter change fall, but nude-part counts do not strictly increase.The authors interpret the trends as preserving unerased-content generation, while noting that this preservation does not always affect erasure.Source: Figure 6 · p. 14, Effect of Regularization Coefficient · p. 14

5 Conclusion

RECE edits cross-attention K&V matrices in closed form, deriving and erasing new target embeddings across epochs to improve concept removal while limiting effects on unrelated concepts.

5 Conclusion

  • RECE modifies only the U-Net cross-attention K&V matrices, which constitute 2.23% of model parameters.The closed-form editing is designed for rapid model modification.Source: §5 · p. 14
  • RECE derives embeddings that can represent target concepts in the edited model and erases those embeddings across multiple epochs.The iterative derive-erase process is intended to make erasure thorough and robust.Source: §5 · p. 14
  • A regularization term is introduced to reduce the impact of erasure on unrelated concepts.Experiments cover artistic styles, unsafe content, and object classes, and assess robustness against red-teaming tools.Source: §5 · p. 14

A Deriving the New Embedding

The derivation seeks an embedding whose projections in the edited model resemble those of the original concept, allowing the edited model to regenerate the erased content.

A Deriving the New Embedding

  • The objective finds c′ so that the edited model's projections of c′ resemble the original model's projections of the target embedding c.The objective also includes a squared-norm regularization term for c′.Source: A Deriving the New Embedding · p. 18
  • The objective is convex and has a unique global-minimum solution for c′, obtained by setting its gradient to zero.The passage attributes convexity to the squared 2-norm and its preservation under linear transformation.Source: A Deriving the New Embedding · p. 18
  • The derived c′ can guide the edited model to generate nude images even when the word “nudity” cannot.The passage characterizes c′ as the target concept's representation within the edited model.Source: A Deriving the New Embedding · p. 18

B Proof of Regularization Term

The regularization objective limits changes to unrelated concepts by minimizing how much their projections change between successive editing epochs.

B Proof of Regularization Term

  • The regularization objective minimizes the change in an unrelated concept's projection from the previous epoch's matrices to the current epoch's matrices.The unrelated concept is represented by embedding d, and the matrices are W new1 and W new2.Source: B Proof of Regularization Term · pp. 18–19
  • The proof bounds projection change using the submultiplicative matrix-norm property and differences involving the derived embeddings.The proof sets up a bound on the change between successive projection matrices.Source: B Proof of Regularization Term · pp. 19–20
  • When the UCE destination concept is unknown, the proof uses zero as the minimizer and states that c′ closer to zero has less effect on model performance.The passage also identifies the UCE destination embedding as another possible minimizer when known.Source: B Proof of Regularization Term · p. 20

C Object Removal

On Imagenette object removal, RECE is reported to erase target classes more effectively while minimizing interference with non-target classes.

C Object Removal

  • The evaluation compares RECE with ESD and UCE on Imagenette classes, generating 500 images per class and assessing erased and unerased class accuracy.The experiments use one epoch, following iterative erasure validation.Source: C Object Removal · p. 20
  • The regularization coefficient is set to 1e −3 for the harder-to-erase church and garbage truck classes, and 1e −1 for three easier classes.The three easier classes are English Springer, golf ball, and parachute.Source: C Object Removal · pp. 20–21
  • RECE exhibits superior erasure capability while minimizing interference with non-targeted classes in the object-removal comparison.Table C.1 compares classification accuracy for object-removal methods.Source: Table D.1 · p. 21, C Object Removal · p. 21

D Extended Inappropriate Content Removal

The appendix extends the erasure evaluation to larger categories of inappropriate content. On I2P, RECE effectively erases multiple sensitive concepts across listed categories.

D Extended Inappropriate Content Removal

E Qualitative Results

The appendix compares qualitative erasure of Van Gogh’s style and nudity, while also checking generation of unerased content. RECE removes nudity while retaining image quality, minimally affects unerased styles, and is more robust than UCE under attack.

E Qualitative Results

Related papers

Loading 2407.12383v2…