Source-linked AI summary
No Concept Escapes the Audit: Auditing-Aware Unlearning for Verifiable Concept Erasure in Diffusion Models
Kaiyuan Deng, Yuchen Li, Gen Li, Yang Xiao, Geng Yuan, Xiaoyong Yuan, Bo Hui, Xiaolong Ma
TL;DR
Robust concept-erasure methods can block adversarial prompts while leaving concepts recoverable from diffusion models’ internal representations, a gap that widens with more erased concepts. AVCE uses latent-space audits to guide edits to text-conditioning and visual pathways and refinement for multi-concept erasure. Across three architectures, it reports lower attack success and better auditing scores than the strongest baseline.
Problem
Prompt-based and output-level evaluations do not establish whether an erased concept remains recoverable from a diffusion model’s internal representations.
Source: §1 · p. 2Method
AVCE audits concept-embedding neighborhoods to find a vulnerable anchor, edits cross-attention and self-attention projections there, then refines both pathways with auditing losses and orthogonal gradient projection.
Source: §1 · p. 2Results
Across SD v1.5, SDXL, and Flux 1.0, AVCE reduces attack success rates by 4.92× and improves auditing scores by 3.84× over the strongest baseline.
Source: §1 · p. 2Takeaways & Limitations
Robustness to prompt attacks does not establish concept removal; latent-space auditing tests whether target features remain recoverable inside the denoising network.
Source: §1 · p. 2The audit uses one partial-diffusion protocol, tests robustness against three prompt-level attacks, and assumes white-box access for analytical edits.
Source: D LIMITATIONS · p. 20
Abstract
from arXiv · showhide
Text-to-image diffusion models can generate prohibited content, which motivates concept erasure through machine unlearning. Most erasure methods intervene at the text interface, through prompt modification or localized updates to text-conditioning weights, and they are evaluated by what the model outputs for given prompts. Such evaluation cannot see what the network still encodes. Latent-space auditing, which bypasses text conditioning and probes the denoising network directly, shows that erased concepts remain recoverable from internal representations. We find that this also holds for methods built to be robust against adversarial prompts, and that the problem grows with the number of erased concepts. We propose Auditing-Aware Unlearning for Verifiable Concept Erasure in Diffusion Models (AVCE), a framework that grounds erasure in the model's latent representations. AVCE audits the embedding neighborhood of each concept and condenses the discovered vulnerable directions into an anchor at the weakest geometric point. It edits cross-attention and self-attention projections in closed form at this anchor, then fine-tunes the two pathways with pathway-level auditing losses, using orthogonal gradient projection to consolidate multiple concepts. Experiments on SD v1.5, SDXL, and Flux 1.0 across object, explicit-content, and artistic-style unlearning show that AVCE reduces attack success rates by 5.07x and improves auditing scores by 3.84x over the strongest baseline, while preserving competitive generation quality.
1 INTRODUCTION
Text-interface evaluations can miss concepts that remain recoverable in a diffusion model’s internal representations, including after robust erasure. AVCE targets both text-conditioning and visual pathways, using latent-space audits to guide erasure and multi-concept updates.
1 INTRODUCTION
- AVCE searches each concept’s embedding neighborhood for vulnerable directions and condenses them into a Verifiable Vulnerability Anchor at the neighborhood’s weakest point.At the anchor, closed-form cross-attention edits sever text-concept associations, while self-attention shrinkage suppresses residual visual patterns.Source: §1 · p. 2
- Audit-Guided Refinement alternately fine-tunes both pathways with pathway-level auditing losses and uses orthogonal gradient projection to consolidate multiple concepts.The refinement targets nonlinear leakage left by the linear edits.Source: §1 · p. 2
- Latent-space audits show that robust methods can block adversarial prompts while leaving target-concept visual features recoverable from internal representations.Existing methods’ discrepancy between attack resistance and representation-level erasure grows as the number of erased concepts increases.Source: §1 · p. 2
- 4.92× lower attack success rates and 3.84× better auditing scores than the strongest baseline are reported for AVCE across SD v1.5, SDXL, and Flux 1.0.The reported comparison covers multi-concept erasure.Source: §1 · p. 2
2 RELATED WORK
Prior work erases concepts through training, pruning, guidance, analytical edits, or adapters, while robustness methods focus on resisting prompt attacks. Representation-level auditing bypasses the text interface to test whether concepts remain recoverable inside the network.
2 RELATED WORK
- Concept-erasure methods include weight fine-tuning, neuron pruning, and sampling guidance, while multi-concept approaches also use iterative optimization, closed-form edits, or adapters.Examples include ESD (Gandikota et al., 2023), ConceptPrune (Chavhan et al., 2024), SLD (Schramowski et al., 2023), UCE (Gandikota et al., 2024), and MACE (Lu et al., 2024).Source: §2 · pp. 2–3
- On SD v1.5, robust methods block adversarial text inputs, yet latent-space probes show the erased golf-ball concept’s visual features remain in the model.Figure 1 contrasts the text-interface result at left with latent-space auditing in the middle and right panels.Source: Figure 1 · p. 3
- Robust unlearning methods counter prompt attacks through adversarial training, vulnerable-embedding erasure, or iterative closed-form alignment.Examples include AdvUnlearn (Zhang et al., 2024c), STEREO (Srivatsan et al., 2025), and RECE (Gong et al., 2024).Source: §2 · p. 3
- Output-level and attack-level evaluations observe the model through its text interface, whereas partial-diffusion auditing injects a concept-informative latent mid-denoising and measures reconstruction with CRS and CCS.The paper adopts this representation-level protocol as its primary evaluation and uses its signal during erasure.Source: §2 · p. 3
3 METHOD
AVCE grounds concept erasure in audited latent representations, then edits and refines both attention pathways. Its stages target vulnerable embedding directions, residual visual patterns, and leakage that remains after analytical edits.
3.2 ANALYTICAL EDITING FOR A SAFER MODEL SPACE
- Closed-form edits at the anchor rewrite cross-attention projections and shrink self-attention projections, providing initialization for subsequent fine-tuning.The edits act on disjoint parameter sets: one removes text–concept association, while the other suppresses residual visual patterns.Source: §3 · p. 3, §3.2 · p. 5
3 METHOD
- AVCE’s three-stage framework audits embedding neighborhoods, edits both attention pathways at the resulting anchor, and refines the model with pathway-level losses.The framework also consolidates multiple concepts through orthogonal gradient projection.Source: §3 · p. 3
3.1 VERIFIABLE VULNERABILITY ANCHOR
- AVCE audits each concept’s embedding neighborhood, searches for directions that revive it, and condenses successful probes into an anchor at the weakest point.The search holds cross-attention maps fixed in one evaluation to distinguish visual-content revival from prompt-alignment effects.Source: §3.1 · p. 4
3.2.1 (I) CROSS-ATTENTION ANALYTICAL ERASURE
- Cross-attention erasure uses a sparse channel modulator and audited vulnerable directions while preserving neutral-prompt responses to limit collateral damage.The modulator combines channel stability and target-versus-neutral discriminability; preservation terms are needed because text embeddings share common components.Source: §3.2.1 · p. 5
3.2.2 (II) SELF-ATTENTION ANALYTICAL SHRINKAGE
- Self-attention shrinkage localizes concept-specific features using cross-attention outputs, then compresses representations along those visual axes while leaving orthogonal directions unchanged.The visual anchor pools conditional self-attention inputs with a mask derived from the difference between conditioned and unconditional cross-attention outputs.Source: §3.2.2 · pp. 5–6
3.3 AUDIT-GUIDED REFINEMENT
- Audit-guided refinement alternates pathway-specific updates with auditing losses, checking pathway contributions rather than relying on output-level erasure alone.The cross-attention audit discourages self-attention from compensating for its contribution; the self-attention update also audits at the vulnerability anchor.Source: §3.3 · p. 6
- For multiple concepts, refinement samples harder concepts more often and projects conflicting gradients to remove components that oppose progress on another concept.The sampling rule retains a uniform exploration floor, and the projection is applied to conflicting gradient pairs.Source: §3.3 · p. 7
4 EXPERIMENT
The experiments evaluate AVCE on object, explicit-content, and artistic-style unlearning, primarily on SDXL, with additional backbones also reported. Across tasks, results cover erasure, robustness, latent-space auditing, and generation quality.
4 EXPERIMENT
- The study evaluates multi-concept object, explicit-content, and artistic-style unlearning under a shared experimental protocol.Appendix B also reports another backbone, large-scale unlearning, and runtime results.Source: §4 · p. 7
4.1 EXPERIMENTAL SETTINGS
- Experiments primarily use SDXL, with additional evaluations on SD v1.5 and Flux 1.0, across Imagenette objects, I2P explicit content, and five artistic styles.Evaluation includes attack success, CRS and CCS auditing, and COCO-30K FID and CLIP scores.Source: §4.1 · p. 7
4.2 OBJECT UNLEARNING
- Over 50 percentage points on average, AVCE reduces object-unlearning ASR versus STEREO and achieves higher CRS and lower CCS while remaining usable with comparable CLIP and FID.UCE and RECE reach zero Unlearn Acc only after model collapse, making them unusable in this setting.Source: §4.2 · pp. 7–8, Table 1 · p. 8
4.3 EXPLICIT-CONTENT UNLEARNING
- Over two-thirds, AVCE reduces total NudeNet detections versus the next-best baseline and lowers explicit-content ASR by over 50 percentage points versus STEREO.It also achieves higher CRS and lower CCS than the baselines.Source: §4.3 · p. 8, Table 2 · p. 9, Table 3 · p. 9
4.4 ARTISTIC-STYLE UNLEARNING
- AVCE attains the highest CLIPd by a large margin, suppressing target styles while preserving unrelated-concept alignment, with FID comparable to most baselines.The audit also reports higher CRS and lower CCS for AVCE than for the baselines.Source: §4.4 · pp. 8–9, Table 4 · p. 10
5 CONCLUSION
The conclusion argues that prompt-level robustness can leave concepts recoverable inside diffusion models. AVCE targets internal representations, and its reported results support assessing erasure through both model encoding and outputs, subject to the scope of the audit used.
5 CONCLUSION
- Robust methods can block adversarial prompts while leaving erased concepts recoverable in diffusion-model representations.The conclusion bounds its claims to one auditing protocol and three prompt-level attacks.Source: §5 · p. 10
- AVCE audits concept embedding neighborhoods, edits cross-attention and self-attention projections at a weakest-point anchor, then refines both pathways with auditing losses and orthogonal gradient projection.Source: §5 · p. 10
- Across three diffusion architectures and three unlearning tasks, AVCE lowers attack success rates and improves auditing scores over existing baselines while keeping generation quality competitive.Source: §5 · p. 10
ETHICS STATEMENT
The ethics statement describes the study's focus on prohibited concepts and safety-related erasure evaluation. The appendix lists implementation details for attacks, auditing, datasets, and AVCE parameters.
ETHICS STATEMENT
- The study evaluates removal of prohibited concepts, including explicit content, using public benchmarks and methods, with no human subjects or newly collected data.I2P and NudeNet are used solely to measure unsafe content remaining after erasure.Source: ETHICS STATEMENT · p. 10
A DETAILED SETTINGS
- The appendix documents attack configurations, the latent-space auditing protocol, task datasets, and AVCE parameter settings.Source: A DETAILED SETTINGS · p. 15
A.1 ADVERSARIAL ATTACK SETTINGS
The appendix specifies three adversarial attacks and their configurations, then describes a partial-diffusion audit that bypasses text conditioning. CRS and CCS assess whether concept-related patterns remain recoverable.
A.1 ADVERSARIAL ATTACK SETTINGS
- The robustness evaluation uses three representative attacks spanning black-box and white-box settings, with concept-specific classifiers determining attack success.ResNet50 detects objects; NudeNet detects specified unsafe body-part categories.Source: §A.1 · p. 15
- Ring-A-Bell optimizes discrete prompt tokens using a genetic algorithm; UnlearnDiffAtk optimizes token perturbations, while P4D searches continuous embeddings before projecting to tokens.The configurations specify attack-specific token counts, timesteps, and iterations.Source: §A.1 · p. 15
A.2 LATENT-SPACE AUDITING SETTINGS
- Partial diffusion injects noise into a reference-image latent, then denoises it with the unlearned model from an intermediate timestep using an empty prompt.This bypasses text conditioning and probes whether internal parameters retain concept-related visual patterns.Source: §A.2 · p. 15
- CRS measures CLIP-feature similarity to the reference image and should be high after genuine erasure, while CCS measures domain-classifier confidence and should be low.The protocol averages both metrics across a guidance-scale sweep and 50 reference images per concept.Source: §A.2 · p. 16
A.3 DATASET CONSTRUCTION
The evaluation covers object, explicit-content, and artistic-style erasure, plus a larger benchmark for testing scalability. AVCE’s detailed settings specify its vulnerability search, analytical edits, and audit-guided refinement.
A.3 DATASET CONSTRUCTION
- Object erasure uses Imagenette’s ten classes, varied prompts, and 500 generated images per class, with Unlearn Acc measured by a pretrained ResNet50.Unlearn Acc is the classifier’s top-1 accuracy on images generated for erased classes.Source: §A.3 · p. 16
- Explicit-content evaluation targets nudity on I2P, generating 4,703 images and applying NudeNet at a 0.6 threshold.I2P also includes prompts associated with violence, self-harm, and shocking imagery.Source: §A.3 · p. 16
- Style erasure covers five artists, using 20 prompts per artist and CLIPa, CLIPc, and CLIPd to assess style alignment, generation quality, and selectivity.CLIPd is defined as CLIPc−CLIPa, averaged over the five styles.Source: §A.3 · p. 16
- The extended benchmark tests 5, 10, 15, and 20 forgotten concepts with standardized prompts, evaluation splits, and concept-specific classifiers.It is used for scalability evaluation across Unlearn Acc, ASR, CCS, and CRS.Source: §A.3 · p. 16
A.4 DETAILED SETTINGS OF AVCE
- Anchor search uses ten random probes per concept, optimized for 200 iterations, then selects successful perturbations to extract a principal direction by truncated SVD.If no probe passes the success criterion, the best probe is kept.Source: §A.4 · p. 17
- Analytical editing uses 50 calibration samples for cross-attention and shrinks self-attention key and value projections across UNet layers.The preservation set contains neutral-prompt embeddings; self-attention localization uses a noisy latent at timestep 500.Source: §A.4 · p. 17
- Refinement alternates cross- and self-attention updates for 500 iterations, while multi-concept consolidation samples four concepts per mini-batch and resolves gradient conflicts by orthogonal projection.Updates use Adam at a learning rate of 1 × 10^-5.Source: §A.4 · p. 17
B.1 SCALABILITY OF AVCE
AVCE’s erasure degrades more slowly than the baselines as the number of forgotten objects grows, while concurrent consolidation keeps the added computation modest.
B.1 SCALABILITY OF AVCE
- AVCE maintains low Unlearn Acc, ASR, and CCS and high CRS even when erasing 20 objects simultaneously.As the number of erased concepts grows, the baselines’ Unlearn Acc, ASR, and CCS rise while CRS falls.Source: §B.1 · p. 17
- Sequential methods’ training time grows as O(N), whereas AVCE updates concepts concurrently through consolidation.The reported comparison measures total unlearning time across concept counts on SD v1.5 and SDXL.Source: §B.1 · p. 17
- 20 concepts take 6.74 minutes on SD v1.5 and 16.86 minutes on SDXL, roughly 12% longer than erasing one concept.The single-concept times are 6.02 minutes on SD v1.5 and 15.05 minutes on SDXL, measured on one NVIDIA A100 GPU.Source: §B.1 · p. 17
B.2 GENERALIZATION ACROSS ARCHITECTURES AND TASKS
On Flux 1.0, AVCE outperforms the reported baselines for object and explicit-content erasure; the section also reports scalability behavior and lists code repositories for comparison methods.
B.2 GENERALIZATION ACROSS ARCHITECTURES AND TASKS
- Several baselines leave Unlearn Acc around 75%–85% on Flux 1.0, and the paper attributes transfer difficulty to more entangled internal representations in larger architectures.The passage characterizes the difficulty for existing closed-form and fine-tuning strategies.Source: §B.2 · pp. 17–18
- On Flux 1.0, AVCE lowers Unlearn Acc to 16.32% and keeps ASR below 19%; explicit-content ASR is 13.4%–16.5%, while every baseline exceeds 80%.The comparison covers object unlearning and explicit-content unlearning under three attacks.Source: §B.2 · p. 18, Table 6 · p. 19, Table 7 · p. 19
- As the number of forgotten concepts grows, AVCE remains stable across the reported metrics, whereas the baselines degrade.Figure 6 reports Unlearn Acc, ASR, CCS, and CRS for 5, 10, 15, and 20 concepts.Source: Figure 6 · p. 18
C CODE REFERENCE
- The code reference lists official baseline repositories and separates unlearning methods from attack and auditing tools, including CCS/CRS and UnlearnDiffAtk.The paper states that baseline results use official open-source implementations with recommended settings unless otherwise noted.Source: C CODE REFERENCE · p. 20
D LIMITATIONS
The paper’s evaluation and analytical edits have explicit scope boundaries.
D LIMITATIONS
- The audit relies on one protocol, partial diffusion with CRS and CCS; other internal-representation probes may expose traces it misses.Robustness testing covers three prompt-level attacks: Ring-A-Bell, UnlearnDiffAtk, and P4D.Source: D LIMITATIONS · p. 20
- Anchor-adaptive attacks and recovery through fine-tuning the edited model are not evaluated.Source: D LIMITATIONS · p. 20
- The analytical edits act on attention projections and assume white-box access to model weights.Source: D LIMITATIONS · p. 20