Source-linked AI summary
M-ErasureBench: A Comprehensive Multimodal Evaluation Benchmark for Concept Erasure in Diffusion Models
Ju-Hsuan Weng, Jia-Wei Liao, Cheng-Fu Chou, Jun-Cheng Chen
TL;DR
Concept-erasure research mainly evaluates text prompts, despite diffusion models also being used with learned embeddings and image latents. M-ErasureBench tests these modalities, and IRECE targets vulnerabilities during inference. Existing methods often fail beyond text prompts, while IRECE reduces CRR by up to 40% in the most challenging white-box latent-inversion setting while maintaining visual quality.
Problem
Existing concept-erasure approaches primarily address text prompts, while learned embeddings and inverted latents used in personalization and image editing are less covered by evaluation.
Source: §1 · p. 1Method
M-ErasureBench evaluates text prompts, learned embeddings, and inverted latents, while IRECE localizes target concepts via cross-attention and perturbs associated latents during denoising without retraining.
Source: §1 · pp. 1–2Results
Erasure methods suppress text prompts but largely fail under learned embeddings and inverted latents; IRECE reduces CRR by up to 40% in the most challenging white-box latent-inversion setting.
Source: §1 · p. 2, §6.3 · p. 8, §7 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
Text-to-image diffusion models may generate harmful or copyrighted content, motivating research on concept erasure. However, existing approaches primarily focus on erasing concepts from text prompts, overlooking other input modalities that are increasingly critical in real-world applications such as image editing and personalized generation. These modalities can become attack surfaces, where erased concepts re-emerge despite defenses. To bridge this gap, we introduce M-ErasureBench, a novel multimodal evaluation framework that systematically benchmarks concept erasure methods across three input modalities: text prompts, learned embeddings, and inverted latents. For the latter two, we evaluate both white-box and black-box access, yielding five evaluation scenarios. Our analysis shows that existing methods achieve strong erasure performance against text prompts but largely fail under learned embeddings and inverted latents, with Concept Reproduction Rate (CRR) exceeding 90% in the white-box setting. To address these vulnerabilities, we propose IRECE (Inference-time Robustness Enhancement for Concept Erasure), a plug-and-play module that localizes target concepts via cross-attention and perturbs the associated latents during denoising. Experiments demonstrate that IRECE consistently restores robustness, reducing CRR by up to 40% under the most challenging white-box latent inversion scenario, while preserving visual quality. To the best of our knowledge, M-ErasureBench provides the first comprehensive benchmark of concept erasure beyond text prompts. Together with IRECE, our benchmark offers practical safeguards for building more reliable protective generative models.
1. Introduction
Concept erasure is intended to suppress harmful or copyrighted content, but existing evaluations and defenses largely focus on text prompts. M-ErasureBench evaluates additional modalities, and IRECE offers an inference-time mitigation without retraining.
1. Introduction
- Large-scale uncurated training data can lead diffusion models to reproduce copyrighted or inappropriate content, while retraining on filtered data is costly and can degrade quality.Concept erasure instead suppresses selected concepts by fine-tuning cross-attention layers.Source: §1 · p. 1
- M-ErasureBench evaluates erasure across text prompts, learned embeddings, and inverted latents, including white-box and black-box settings for the latter modalities.The benchmark addresses a gap in text-prompt-centered evaluation.Source: §1 · pp. 1–2
- IRECE localizes target concepts via cross-attention and selectively perturbs associated latents during denoising, without retraining.It is proposed as a plug-and-play inference-time module.Source: §1 · p. 2
- Existing methods suppress concepts under basic prompts but largely fail with learned embeddings and latent inversion, exceeding 90% CRR in white-box unconditional settings without adversarial training.The authors attribute this to disrupting text–image alignment rather than fully removing concepts.Source: §1 · pp. 1–2
2. Related Works
Prior concept-erasure work focuses on text-driven suppression and attacks, leaving robustness to non-textual inputs less explored. This paper positions multimodal evaluation as a way to assess that gap.
2.1. Concept Erasure in Diffusion Models
- Concept-erasure methods fine-tune diffusion-model cross-attention; anchor-based methods substitute concepts, while anchor-free methods steer generation away without replacement.Examples include ESD, UCE, and Receler, respectively using negative guidance, closed-form projection edits, and adversarial training with parameter-efficient fine-tuning.Source: §2.1 · p. 2
- Text-driven methods struggle with learned embeddings and inverted latents, while EraseBench measures post-erasure side effects only within text prompts.The authors motivate evaluation across diverse modalities.Source: §2.1 · p. 2
2.2. Adversarial Attacks on Concept Erasure
- Prior attacks commonly use crafted prompts; examples include P4D, which aligns noise predictions with the original model, and Ring-A-Bell, which constructs prompts from concept vectors.Receler incorporates adversarial prompt optimization to improve resilience.Source: §2.2 · p. 2
- Concept Inversion uses Textual Inversion to learn embeddings from reference images, but most existing attacks remain text-driven and may lose effectiveness against adversarially trained models.The authors identify non-textual pathways as an area needing further robustness evaluation.Source: §2.2 · p. 2
3. Preliminaries
The preliminaries introduce deterministic DDIM denoising and inversion as latent-space operations, then describe Textual Inversion as learning a concept embedding from a reference image.
3.1. Denoising Diffusion Implicit Models (DDIM)
- DDIM deterministically denoises a latent from timestep t to t−1 using the conditioning prompt, denoiser, and cumulative noise schedule.The full denoising trajectory is represented as DDIMProcess from T to 0.Source: §3.1 · p. 2
- DDIM inversion reverses the denoising process to reconstruct a reference image’s noisy latent trajectory for later resampling or manipulation.The full inversion trajectory maps x0 to xT.Source: §3.1 · p. 3
3.2. Textual Inversion
- Textual Inversion learns a dedicated embedding from a reference image, which is inserted into a prompt so the frozen diffusion model can evoke the target concept.The embedding is optimized to minimize reconstruction error in the frozen model.Source: §3.2 · p. 3
4. Multimodal Concept Erasure Evaluation
The evaluation framework tests concept erasure through text prompts, learned embeddings, and inverted latents. The learned-embedding and inversion setups vary access to the erased model, and inversion also varies prompt pairs.
4. Multimodal Concept Erasure Evaluation
- The benchmark covers text prompts, learned embeddings, and latent inversion to assess erasure beyond text-only evaluation.The paper identifies learned embeddings and image-based latents as richer user interaction modalities.Source: §4 · p. 3
4.1. Text Prompt Evaluation
- Text-prompt evaluation includes basic prompts naming the target concept and adversarial prompts generated by Ring-A-Bell.The adversarial-prompt setting is included because prior work reports prompt-based attacks that bypass suppression.Source: §4.1 · p. 4
4.2. Learned Embedding Evaluation
- Learned-embedding evaluation uses Textual Inversion from reference images, with white-box training on the erased model or surrogate-model training followed by transfer.A further black-box variant perturbs the reference image before training to induce semantic shifts in the learned embedding.Source: §4.2 · p. 4
4.3. Latent Inversion Evaluation
- Latent-inversion evaluation initializes generation from a reference-image latent and combines it with a sampling prompt; inversion and resampling use either the erased model or a surrogate for inversion.The evaluation uses unconditional, generic, coarse, and explicit-target prompt pairs.Source: §4.3 · p. 4
5. Inference-time Robustness Enhancement for Concept Erasure (IRECE)
IRECE strengthens erased models at inference time by using cross-attention to locate target-concept regions and perturbing those latent regions during denoising, without retraining.
5. Inference-time Robustness Enhancement for Concept Erasure (IRECE)
- IRECE intervenes at a chosen denoising step, using the target concept embedding and cross-attention maps from the standard model to locate concept-linked regions.The sample prompt continues to guide denoising with the erased model.Source: §5 · p. 4
- Attention maps from different layers are upsampled to a common resolution and aggregated before thresholding creates a binary mask of concept-associated pixels.The threshold is controlled by τ.Source: §5 · pp. 4–5
- IRECE replaces masked latent regions with Gaussian noise, disrupting target-related information while largely preserving the surrounding content.Denoising then resumes from the updated latent.Source: §5 · p. 5, Figure 3 · p. 5
- The plug-and-play procedure operates entirely at inference time and can extend to targeted object removal or replacement.Its stated aim is to prevent erased concepts from re-emerging while maintaining visual coherence elsewhere.Source: §5 · p. 5
6. Experiments
Experiments test three erasure methods across prompt, learned-embedding, and latent-inversion settings. Existing methods often lose suppression outside standard prompts, while IRECE reduces CRR in the most challenging latent-inversion setting.
6.1. Experimental Setup
- The study uses Stable Diffusion v1.4, ESD, UCE, and Receler, with CIFAR-10 classes as targets and four datasets covering normal prompts, adversarial prompts, learned embeddings, and inverted latents.CRR is the percentage of generated samples containing the erased concept, assessed using GroundingDINO; lower CRR indicates stronger suppression.Source: §6.1 · pp. 5–6
6.2.3. Latent Inversion Evaluation
- Under white-box latent inversion, the null prompt drives CRR above 92% for all methods, while the image strategy exceeds 70% across methods.Even the TARGET strategy exceeds 50%.Source: Figure 4 · p. 6, §6.2.3 · pp. 6–7
6.2.1. Text Prompt Evaluation
- On standard text prompts, CRR falls to 26.5% for ESD, 18.5% for UCE, and 15.0% for Receler; adversarial prompts raise ESD and UCE CRR but leave Receler at 14.8%.ESD and UCE rise to 66.7% and 36.9%, respectively, under adversarial prompts.Source: §6.2.1 · p. 6
6.2.2. Learned Embedding Evaluation
- With white-box learned embeddings, CRR rises to 91.1% for ESD, 90.4% for UCE, and 56.0% for Receler, compared with their text-prompt results.The cited passage reports increases of 64.6, 71.9, and 41.0 percentage points, respectively.Source: §6.2.2 · p. 6
- With perturbed reference images in black-box embedding evaluation, CRR increases to 74.0% for ESD and 54.2% for UCE; Receler shows only a marginal increase.The per-pixel perturbation budget is 8; Figure 6 shows successful target regeneration in this setting.Source: §6.2.2 · p. 6, Figure 6 · p. 7
6.3. Inference-time Robustness Enhancement for Concept Erasure (IRECE)
- CRR drops from 92.9% to 35.1% for ESD, 94.4% to 42.2% for Receler, and 94.9% to 54.4% for UCE with IRECE under white-box null-prompt latent inversion.The comparison covers white-box and black-box settings; qualitative examples show target removal or replacement while non-target regions remain largely intact.Source: Figure 5 · p. 7, §6.3 · pp. 7–8, Figure 7 · p. 8
6.2.3. Latent Inversion Evaluation
- Under black-box latent inversion with the null prompt, CRR reaches 57.3% for ESD, 95.2% for UCE, and 79.0% for Receler.The passage reports that even the weakest black-box strategy, TARGET, increases CRR by around 13.0% relative to text-prompt evaluation; regenerated images retain reconstruction quality comparable to references.Source: §6.2.3 · p. 7
7. Conclusion
M-ErasureBench evaluates concept-erasure robustness beyond text prompts, where learned embeddings and inverted latents expose substantial weaknesses. IRECE improves robustness against these attacks while maintaining visual quality.
7. Conclusion
- M-ErasureBench finds that erasure methods that perform well on text prompts degrade with learned embeddings and nearly fail under image latent inversion.The conclusion describes these results as exposing critical vulnerabilities in erased models.Source: §7 · p. 8
- IRECE is a plug-and-play inference-time module that restores robustness without retraining, reducing CRR by up to 40% in the most challenging white-box latent inversion setting.The conclusion also reports that visual quality is maintained.Source: §7 · p. 8
- IRECE also supports broader applications, including targeted object removal and replacement.The conclusion connects these applications to reliable concept erasure in generative models.Source: §7 · p. 8
Concept Erasure in Diffusion Models Supplementary Material
IRECE uses target prompts and cross-attention to identify regions for intervention during diffusion. It perturbs those regions, then continues denoising with the erased model to generate an output.
A. IRECE Algorithm
- IRECE takes a sample prompt, target prompt, initial latent, intervention step, localization threshold, and standard and erased models as inputs.At the intervention step, it extracts target-prompt cross-attention maps from the erased model in white-box access or the standard model in black-box access.Source: A. IRECE Algorithm · p. 12
- IRECE upsamples and aggregates cross-attention maps, then thresholds the aggregate to construct a binary mask of target regions.The mask marks locations whose aggregate attention meets or exceeds threshold τ.Source: A. IRECE Algorithm · p. 12
- At masked locations, IRECE replaces latent values with Gaussian noise before the erased model performs the denoising step.The algorithm repeats denoising and returns the generated output.Source: A. IRECE Algorithm · p. 12
B. Implementation Details
The supplementary evaluation fixes sampling and IRECE settings, then reports class-level results for text prompts and learned embeddings. Learned embeddings weaken erasure, while perturbations can raise CRR even in black-box evaluation.
B. Implementation Details
- Sampling uses DDIMScheduler with guidance scale 7.5 and 50 inference steps; the latent-inversion setup includes four prompt configurations.IRECE uses localization threshold τ = 0.4 and intervention timestep t∗ = 781.Source: B. Implementation Details · p. 12, Table 1 · p. 12
C.1. Text Prompt Evaluation
- Automobile has the highest CRR across methods in the text and adversarial prompt evaluation, but all methods reduce its CRR by at least 39%.The authors suggest automobiles’ large intra-class diversity as a plausible reason for weaker suppression.Source: §C.1 · p. 12, Table 2 · p. 13
C.2. Learned Embedding Evaluation
- Learned embeddings raise white-box CRR above 50% for many categories across methods; black-box CRR stays below 50% for most categories, but perturbations push ESD and UCE above 50% for some classes.The per-class results identify bird, cat, deer, and truck among classes where perturbations raise CRR.Source: §C.2 · pp. 12–13, Table 3 · p. 13
C.3. Latent Inversion Evaluation
IRECE reduces CRR across most classes and prompt strategies in latent inversion evaluation. The results also show that inversion access and surrogate-backbone mismatch affect how effectively erased models suppress target concepts.
C.3. Latent Inversion Evaluation
- In white-box latent inversion, nearly all categories exceed 50% CRR with the null-text and “image” strategies; in black-box evaluation, the null-text strategy exceeds 50% for most categories.The “image” strategy captures overall semantics in most white-box cases, while TARGET behaves differently across access settings.Source: §C.3 · p. 13, Figure 9 · p. 13, Figure 10 · p. 14
C.4. (Surrogate-based) Black-box with Different Backbones
- With different surrogate and erased-model backbones, CRRs are lower than in white-box evaluation but remain higher than in the corresponding same-backbone black-box setting.The authors attribute this pattern to mismatch between the surrogate’s and erased model’s representation spaces.Source: §C.4 · p. 14, Table 5 · p. 15
C.5. Inference-time Robustness Enhancement for Concept Erasure (IRECE)
- IRECE substantially reduces CRR across most classes and prompt strategies in white-box and black-box latent inversion evaluation.Tables 6 and 7 report per-class CRR before and after IRECE for ESD, UCE, and Receler.Source: §C.5 · p. 14, Table 6 · p. 16, Table 7 · p. 17
- IRECE’s intervention timestep trades off image structure against blending artifacts and incomplete suppression: acting too early disrupts structure, while acting too late leaves too few denoising steps.The localization threshold also trades erasure strength against preservation of nearby non-target regions; Figure 11 varies both parameters.Source: §C.5 · p. 14, Figure 11 · p. 18