PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Concept Editing in Diffusion Models

Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, David Bau

arXiv:2308.14761v2cs.CVcs.LG

TL;DR

Text-to-image models face bias, copyright, and offensive-content issues that prior methods typically address separately. UCE uses closed-form post-training edits to modify text-to-image projections, and experiments report stronger single-edit and scalable multi-edit performance than prior methods. Its debiasing scope is limited by interdependent and compositional biases across attributes and prompt words.

  • Problem

    Text-to-image models exhibit concurrent safety problems involving bias, copyright, and offensive content, while prior methods have addressed them separately.

  • Method

    UCE is a unified closed-form model-editing method that edits text-to-image projections without training for erasure, moderation, and debiasing.

  • Results

    UCE outperforms prior methods in single edits and scales to many simultaneous edits while minimizing interference with unedited concepts.

  • Takeaways & Limitations

    UCE provides one scalable post-training framework for debiasing, style erasure, and offensive-concept moderation.

  • Takeaways & Limitations

    Joint debiasing must account for interdependent attribute biases and word-level compositional biases in prompts.

Abstract

from arXiv · show

Text-to-image models suffer from various safety issues that may limit their suitability for deployment. Previous methods have separately addressed individual issues of bias, copyright, and offensive content in text-to-image models. However, in the real world, all of these issues appear simultaneously in the same model. We present a method that tackles all issues with a single approach. Our method, Unified Concept Editing (UCE), edits the model without training using a closed-form solution, and scales seamlessly to concurrent edits on text-conditional diffusion models. We demonstrate scalable simultaneous debiasing, style erasure, and content moderation by editing text-to-image projections, and we present extensive experiments demonstrating improved efficacy and scalability over prior work. Our code is available at https://unified.baulab.info

1. Introduction

Text-to-image diffusion models face concurrent problems involving bias, copyright, and offensive content, while data curation alone can be costly and unpredictable. UCE addresses these issues with one closed-form post-training editing approach that supports many edits while preserving unedited concepts.

  • Models can clone artistic styles, amplify stereotypes, and facilitate offensive image creation.
  • Data curation is expensive and can produce counterintuitive effects, including new biases or incomplete removal of unwanted content.
  • UCE provides a single closed-form model-editing formulation for addressing different safety issues after training.
  • UCE supports hundreds of editorial modifications in one text-to-image model while preserving generative quality for unedited concepts.
  • UCE applies the same editing framework to erasure, moderation, and debiasing, with simultaneous edits designed to minimize interference with unedited concepts.

2. Related Work

Prior work addresses copyright, offensive content, and social bias through separate training, inference, or parameter-editing techniques. UCE adopts closed-form post-training editing to remove many concepts at once and extends model editing toward unified, scalable safety interventions.

  • Copyright issues: Copyright mitigation has used corpus preparation, image cloaking, and post-training concept erasure to reduce style imitation or memorization.
  • Copyright issues: UCE differs from prior concept-erasure methods by using a closed-form edit that removes many concepts simultaneously.
  • Offensive content: Offensive-content defenses include training-data filtering, inference-time safety checkers, image editing, and weight-level removal of unwanted concepts.
  • Offensive content: Inference-time filters can be disabled when open-source code and model weights are available, motivating removal of unwanted concepts from model weights.
  • Social biases: Social-bias methods modify parameters or representations, including projection-based removal of biased directions and algebraic manipulation.
  • Model editing: Model editing changes a targeted subset of weights to control model behavior without training, including prior closed-form edits for generative models.

3. Background

Diffusion models use cross-attention to connect text embeddings with visual features through key and value projections. UCE edits these projections in closed form, mapping selected concepts to targets while preserving specified concepts and reducing broad interference.

  • Diffusion background: Diffusion models generate images by iteratively denoising Gaussian noise across T time steps.
  • Cross-attention: Cross-attention injects text conditioning into the U-Net, with image-space queries and text-derived keys and values projected by W_k and W_v.
  • Cross-attention: For embedding c_i, the projections produce k_i = W_kc_i and v_i = W_vc_i, which align text tokens with visual features and determine attended values.
  • Cross-attention: Cross-attention links text and image information by assigning visual meaning to text tokens before the output continues through the diffusion U-Net.
  • Prior editing: TIME edits cross-attention projections using source and destination prompts to change implicit visual assumptions such as object color or gender.
  • UCE formulation: UCE finds edited weights that map concepts in E to target values while preserving concepts in P, using a closed-form solution for efficient simultaneous editing.
  • Preservation: TIME’s general regularization affects all vector representations equally, whereas UCE introduces targeted preservation for maintaining pretrained model capabilities.

4. Method

UCE edits linear cross-attention projections to support erasure, moderation, and debiasing while preserving selected concepts. Its closed-form formulation unifies prior editing methods and supports concurrent concept edits.

  • General editing framework: The method finds edited weights that map edit concepts to target outputs while preserving outputs for selected concepts.This general linear-projection objective has a closed-form solution for the updated weights.
  • Relation to prior methods: The formulation generalizes TIME and MEMIT as special cases: preservation with canonical input directions yields TIME, while solving for ΔW yields MEMIT’s closed-form update.This places the proposed formulation as a unified generalization of prior editing techniques.
  • General editing framework: UCE edits Wk and Wv using concept embeddings, modified target outputs, and preservation concepts for surrounding model behavior.For multi-token concepts, the method aligns the last token of the concept with the last token of its target output.
  • Erasure: Erasure redirects an abstract concept’s output toward a different concept, such as mapping an artistic style toward “art,” to remove the undesired characteristic from generations.The resulting edit is intended to prevent the erased concept from appearing in model generations.
  • Debiasing: Debiasing adjusts a concept’s output along attribute directions so the diffusion model generates desired attribute probabilities, including multiple attributes simultaneously.The attribute coefficients αi determine the desired probability for each attribute; the method is presented as handling more than the dual-attribute scope of TIME and concept ablation.
  • Moderation: Moderation replaces a concept’s output with a generic unconditional output, reducing extreme attributes in the model’s response.For nudity, the target is aligned with an unconditional prompt rather than the original concept output.

5. Experiments

Experiments evaluate UCE for artistic-style and object erasure, preservation of unrelated concepts, profession debiasing, sensitive-content moderation, and simultaneous edits. Across these settings, the method supports scalable edits while retaining image fidelity, text-image alignment, or targeted changes.

  • Artist erasure: UCE consistently erases multiple artistic styles while preserving the remaining artists better than competing methods as erased concepts increase.The method can erase up to 100 artists simultaneously before image fidelity and CLIP scores degrade; preservation is important to this result.
  • Preservation and interference: UCE produces the smallest changes to unrelated concepts and maintains higher COCO text-image alignment after editing.Lower LPIPS indicates less change to unerased concepts, while higher CLIP scores indicate better alignment.
  • Object erasure: UCE removes Imagenette objects while limiting interference on non-target classes; erasing all 10 classes reduces image-generation accuracy to 4.0% while COCO-CLIP remains 31.02 versus 31.32 for original SD.The comparison reports improved targeted erasure and preservation of unrelated classes relative to ESD-u.
  • Racial diversity and moderation: UCE improves racial diversity in generated professions and moderates nudity with lower distortion and better text-image alignment than the compared approaches.For nudity, UCE has comparable erasure to ESD-x while inducing substantially lower LPIPS distortion than ESD-u and ESD-x.
  • Unified editing: A single jointly edited model combines style erasure, profession debiasing, and nudity moderation, achieving a gender-ratio deviation of 0.27 and a 58% nudity reduction.These results are compared with 0.22 for the gender-debiasing model, 0.67 for original Stable Diffusion, 49% for nudity erasure, and 64% for ESD-u.

6. Limitations

UCE’s joint debiasing must account for interacting attributes and compositional prompt effects, while extensive style erasure can damage general image generation.

  • Interdependent biases: 96% male outputs for “a native american person” versus 48% for “a black person” reveal compounding cross-attribute biases.The reported figures use 100 generated images per prompt.
  • Compositional effects: 88% male outputs for “successful person” versus 50% for “person” show that word-level cues can introduce compositional gender bias.These figures are reported for 100 generated images per phrase.
  • Style-erasure limits: Erasing over 500 artists degrades general image generation even when preservation terms are used.The result suggests that maintaining generative capabilities requires preserving a critical mass of artists.

7. Conclusion

The paper concludes that UCE provides a unified, text-driven way to edit diffusion models for debiasing, copyrighted-content erasure, and offensive-content moderation. Its closed-form formulation supports scalable concurrent edits while preserving unedited concepts and model capabilities.

  • 7. Conclusion: UCE edits diffusion models using text descriptions to target debiasing, copyrighted-content erasure, and offensive-concept moderation.The approach is presented as a unified algorithm for post-training model editing.
  • 7. Conclusion: UCE modifies cross-attention key and value weights using concept embeddings, desired target outputs, and preservation concepts.The edited weights are obtained by minimizing an objective and solving its derivative in closed form.
  • 7. Conclusion: The closed-form edit requires a full-rank matrix, so UCE adds d canonical-basis preservation terms when the existing preservation set may be insufficient.Here d denotes the text-embedding dimension.
  • 7. Conclusion: UCE optimizes both cross-attention key and value weights under the same editing principles.This extends the formulation beyond a single projection type.

B. UCE Generalizes to TIME

UCE generalizes TIME and MEMIT by adding explicit preservation of surrounding diffusion concepts and solving for weight updates. This formulation is intended to reduce interference during simultaneous multi-concept editing.

  • B. UCE Generalizes to TIME: UCE generalizes TIME by replacing equal preservation across canonical directions with preservation of important surrounding concepts.TIME regularizes cross-attention weights equally across all directions, whereas UCE adds explicit concept preservation.
  • B. UCE Generalizes to TIME: The preservation strategy is described as enabling multiple edits with less interference.The formulation targets concepts beyond canonical directions while retaining the model’s surrounding knowledge.
  • C. UCE Generalizes to MEMIT: UCE generalizes MEMIT by incorporating preservation terms from the diffusion vocabulary and solving for the weight update ∆W.This differs from directly solving for the edited weight matrix W.

D.1. Erasing Style

UCE can erase multiple artistic styles while preserving selected artists and limiting interference on holdout concepts, but excessive erasure eventually harms general performance.

  • D.1. Erasing Style: After 100 erased artists, holdout quality declines as LPIPS increases; 50 or fewer erasures cause negligible interference on non-erased concepts.These observations come from Figure D.1’s holdout-artist evaluation.
  • D.1. Erasing Style: Preserving at least 500 artists is essential for retaining performance when erasing styles.The preservation-limit experiment erased up to 1500 artists and evaluated 100 non-preserved, non-erased artists using LPIPS divergence.
  • D.1. Erasing Style: The unified model achieves comparable debiasing performance to an individually finetuned model on average.Table D.1 measures percentage deviation from equal gender ratios, where ∆=0 denotes perfect equality.

D.2. Debiasing

The unified model maintains debiasing performance while reducing bias and increasing gender diversity across professions. Its iterative procedure updates cross-attention weights using current attribute ratios and protects sufficiently debiased concepts from later interference.

  • Performance: Unified models show similar average performance to individual debiasing models.The comparison is reported in Table D.1.
  • Debiasing results: UCE consistently reduced bias and increased gender diversity for most of 36 WinoBias professions.Additional examples show improved gender and racial diversity in Stable Diffusion outputs.
  • Method: The debiasing algorithm iteratively edits cross-attention weights using concepts to edit, concepts to preserve, and attribute prompts.It computes current attribute-ratio distributions from validation prompts and CLIP classification.
  • Method: Once a concept is within 5% of its target ratio, it moves from the edit list to the preservation list.This avoids unnecessary validation-image generation and protects the concept from interference by later edits.
  • Related safety editing: For multi-concept moderation, UCE erases better than ESD methods across different NSFW classes.Its individual-class erasure is similar to ESD-x while producing less interference on other concepts.

D.4. Erasing Objects

The method achieves strong object and artistic-style erasure while limiting interference with concepts that are not targeted. Its results contrast with partial object removal and reveal quality boundaries as the number of erasures grows.

  • Style erasure: UCE shows strong multi-concept artistic-style erasure with the least interference on holdout artists.The comparison includes artists that were neither erased nor explicitly preserved.
  • Object erasure: UCE clearly erases the full intended object, unlike ESD-u's partial removal of objects such as churches.ESD-u could remove major attributes while leaving the building intact.
  • Style erasure: Previous methods start showing interference when erasing 10 artists, whereas UCE's reported interference remains lowest in the multi-concept comparison.The figures specifically compare erasing 10 and 50 artists.
  • Style erasure: Changes in holdout-style output quality appear after erasing 300 artists, and artistic nuance begins to decline at 1000 erasures.These are reported limits in the holdout-artist experiments.
  • Debiasing comparison: The debiasing table compares Original-SD, Concept Algebra, Debias-VL, TIME, TIME + Preserve, and Ours across profession results.The listed comparison includes multiple prior debiasing methods and the proposed method.
  • Object erasure: The method demonstrates complete erasure of intended objects with minimal interference to unerased objects.This behavior is shown across Figures D.12–D.14.
Loading 2308.14761v2…