PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept Acupuncturist: An Efficient and Precise Concept Erasure Method for Diffusion Models via Sparse Autoencoders

Yiran Xiahou, Siliang Qin, Kai Chen, Shengzhi Zhang, Renyang Liu

arXiv:2610.32059v1cs.CVcs.AIcs.CRcs.LG

TL;DR

Output-level evaluation can provide incomplete evidence of concept forgetting, while internal concept representations remain underexplored. ConceptACP uses feature-guided supervision for localized, inference-free unlearning and improves forgetting success rate by 36.34 percentage points over the best-performing baseline in the standard one-step setting.

  • Problem

    Output-level metrics provide incomplete evidence of genuine concept forgetting, while analysis of internal representations remains largely unexplored.

    Source: D. The Limitations of Output-based… · p. 5

  • Method

    ConceptACP uses SAE-derived feature corrections as supervision to fine-tune a single cross-attention block without full diffusion trajectories.

    Source: §I · p. 2

  • Results

    36.34 percentage points higher forgetting success rate on average than the best-performing baseline under the standard one-step setting.

    Source: §I · p. 2

  • Takeaways & Limitations

    ConceptACP achieves precise concept forgetting while maintaining image quality and avoiding repeated diffusion trajectories or inference-time overhead.

    Source: §VIII · p. 14

    Output-level metrics alone provide incomplete evidence of genuine concept forgetting.

    Source: D. The Limitations of Output-based… · p. 5

Abstract

from arXiv · show

Diffusion-based image generation models produce high-quality images but may generate harmful or policy-violating content, motivating machine unlearning (MU) to selectively remove target concepts while preserving general capabilities. However, existing MU approaches often struggle with the distributed and entangled nature of concept representations, leading to inefficient optimization and imprecise edits when applied through large-scale fine-tuning over full diffusion trajectories. To bridge this gap, we propose Concept Acupuncturist (ConceptACP), a novel MU framework that integrates interpretability-guided analysis with inference-free fine-tuning for efficient and precise concept removal. By establishing mappings between semantic concepts and cross-attention blocks in the U-Net denoiser and localizing the most concept-relevant block for targeted optimization, ConceptACP avoids full diffusion trajectory optimization, thereby improving fine-tuning efficiency. To mitigate concept entanglement, we propose a feature-level correction mechanism based on Sparse Autoencoders (SAEs), where SAE-derived sparse, semantically meaningful feature activations are manipulated to construct targeted correction signals, which are then projected back to the output space of the corresponding block and used as supervision for precise concept removal. Consequently, ConceptACP eliminates the need for external inference-time modules while achieving efficient and fine-grained unlearning. Instantiated on SDXL-Turbo, ConceptACP improves the forgetting success rate by 36.34 percentage points in the standard one-step generation setting on average over the best-performing baseline, while preserving image quality and prompt fidelity, and generalizes effectively across other Stable Diffusion backbones. Our code is available at https://github.com/Eran-Xia/ConceptACP.

I. INTRODUCTION

Diffusion models can generate harmful or policy-violating content, while existing fine-tuning methods struggle to remove concepts precisely and efficiently. CONCEPTACP uses SAE-derived feature corrections and localized block-level supervision; on SDXL-Turbo it improves one-step forgetting success while preserving unrelated concepts and reducing training cost.

I. INTRODUCTION

  • Fine-tuning-based unlearning can interfere with unrelated concepts and incur high cost by updating many parameters and requiring denoising trajectories for supervision.The introduction frames its two challenges as limited edit precision and full-trajectory optimization overhead.Source: §I · p. 1
  • CONCEPTACP uses SAE-derived feature corrections as supervision and localizes parameter optimization to a concept-relevant cross-attention block.Corrected SAE outputs are paired with original block inputs to build supervision, while cached activations across timesteps avoid repeated denoising during fine-tuning.Source: §I · p. 2
  • 36.34 percentage points: CONCEPTACP improves average forgetting success rate over the best-performing baseline in the standard one-step setting.The method also reduces computational cost by 46.4% versus a full diffusion-based method without block-level design, and gains transfer to SDXL and SD v1.4.Source: §I · p. 2

A. Text-to-Image Models

  • Diffusion models combine a text encoder, a U-Net denoiser, and a decoder, while SDXL-Turbo generates images in typically 1–4 steps.Traditional Stable Diffusion 1.x and 2.x models typically use tens of denoising steps.Source: A. Text-to-Image Models · p. 2

B. Machine Unlearning

  • Machine unlearning targets removal of undesired learned concepts while preserving performance on unrelated content, using fine-tuning, model editing, or inference-time intervention.Prior examples span semantic, stylistic, and object concepts, including nudity, Van Gogh, and church.Source: B. Machine Unlearning · p. 3

C. Sparse Autoencoder in Stable Diffusion

The section motivates feature-level localization by describing SAE representations of U-Net cross-attention residuals and showing why neuron-level edits can affect unrelated concepts. Its empirical analysis identifies concept-related features in specific blocks.

C. Sparse Autoencoder in Stable Diffusion

  • SAEs analyze cross-attention residual updates and encode activations into sparse spatial feature maps that expose feature distributions across an image.The described training strategy applies to few-step models such as SDXL-Turbo and multi-step models such as SD v1.4.Source: C. Sparse Autoencoder in Stable… · p. 3
  • The analyzed cross-attention blocks have distinct roles: down.2.1 controls composition, mid.0 captures low-level semantics, up.0.0 adds structural detail, and up.0.1 refines color and style.Source: C. Sparse Autoencoder in Stable… · p. 3

D. Threat Model

  • The threat model covers controlled online access with prompt-based black-box attacks and public checkpoint release that allows local model control.The model provider is assumed trusted in the public-release scenario.Source: D. Threat Model · pp. 3–4

III. EMPIRICAL STUDY

  • The empirical study examines localization, editing, optimization scope, and evaluation choices because many existing design decisions remain largely empirical.Source: §III · p. 4

A. The Limitations of Neuron-level Localization

  • Neuron-level localization can affect unrelated representations because neurons are polysemantic, and coarse edits may degrade image quality.The section argues that choosing a low-interference update space does not by itself identify where the target concept is represented.Source: A. The Limitations of Neuron-level… · p. 4
  • For the naked concept, SAE features show strong activation in the down.2.1 cross-attention block, supporting feature-level localization rather than uniform network-wide editing.Figure 1 visualizes the top six activated SAE features across four representative cross-attention blocks.Source: A. The Limitations of Neuron-level… · p. 4, Figure 1 · p. 4

B. The Fragility of Inference-time Steering

The section identifies limitations in inference-time steering, large-scale fine-tuning, output-based evaluation, and cross-architecture transfer. Its evidence highlights recurring deployment overhead and gaps between automated assessment and broader concept-unlearning settings.

B. The Fragility of Inference-time Steering

  • Inference-time steering methods add 14.4%–125.8% latency on SDXL-Turbo, while external modules can be bypassed and may introduce visual artifacts.The cited methods are TraSCE, SAFREE, and SAEUron.Source: B. The Fragility of Inference-time… · pp. 4–5

C. The Inefficiency of Large-scale Fine-tuning

  • 53.6%: single-block fine-tuning without the complete denoising trajectory reduces per-iteration training time to this share of the original approach.The comparison indicates that full-trajectory dependence accounts for a substantial part of optimization cost.Source: C. The Inefficiency of Large-scale… · p. 5

D. The Limitations of Output-based Evaluation

  • Classifier-based evaluation can miss human-perceived differences in concept presence, so output-level metrics alone provide incomplete evidence of forgetting.Figure 2 illustrates images with similar classifier predictions but different apparent concept presence to humans.Source: D. The Limitations of Output-based… · p. 5, Figure 2 · p. 5

E. The Generalization Gap Across Diffusion Architectures

  • Existing unlearning methods developed for SD 1.x/1.5 can degrade on SDXL-Turbo, which differs architecturally and uses a fast 1–4-step sampling regime.The section hypothesizes that fast sampling partly explains degradation for methods such as ESD.Source: E. The Generalization Gap Across… · p. 5

IV. CONCEPT ACUPUNCTURIST

Concept Acupuncturist localizes concept-related SAE features and a target cross-attention block, then embeds feature corrections through block-level fine-tuning. Its training pairs use corrected block outputs as supervision, avoiding repeated diffusion inference during optimization.

A. Overview

  • ConceptACP fine-tunes one concept-relevant cross-attention block using SAE-corrected outputs as supervision, internalizing feature corrections without inference-time components.A joint objective balances concept suppression with preservation of generation quality.Source: A. Overview · p. 6

B. Feature Localization

  • The automated localization algorithm compares feature activations on concept-containing and concept-excluding data, then selects the highest-scoring block and its high-scoring features.It searches a broader SAE pool and uses concept-specific thresholds; a single concept may involve multiple features.Source: B. Feature Localization · p. 7

C. Fine-tuning Dataset Generation

  • For target prompts, selected SAE features are attenuated and decoded; combining the resulting residual with the block input produces concept-erased outputs for fine-tuning pairs.The most relevant feature is scaled by a, while other above-threshold features are scaled proportionally to their relevance scores.Source: C. Fine-tuning Dataset Generation · pp. 7–8
  • Block input–output pairs enable inference-free fine-tuning without rerunning preceding pipeline stages or storing the full denoising trajectory.Unmodified outputs for prompts excluding the target concept are also included to mitigate collateral effects; the suppression factor is chosen empirically and may vary by concept.Source: C. Fine-tuning Dataset Generation · pp. 7–8

D. Loss Function

The block-level loss combines concept erasure with preservation of outputs for prompts excluding the target concept. An empirically chosen weight balances the two terms.

D. Loss Function

  • The block-level objective combines a concept-erasure loss with a preservation loss, weighted by λ chosen empirically for the concept type.Each term compares the block’s output against its paired target output.Source: D. Loss Function · p. 8

V. NEW METRICS

The paper introduces SAE-derived internal metrics to complement output-level evaluation and interpret representation changes during unlearning. Feature Overlap tracks changes in dominant feature identities, while Feature Activation Correlation tracks activation-distribution shifts.

V. NEW METRICS

  • Output-level metrics may not reveal whether target concepts have been erased from internal representations, motivating complementary internal diagnostics.The proposed metrics are post-hoc analysis tools, not a standalone benchmark.Source: §V · p. 8
  • The metrics are diagnostic rather than replacements for output-level evaluation, and their SAE-derived feature space may inherently favor SAE-based methods.The paper frames successful unlearning as disrupting target-feature activations while preserving unrelated-feature activations.Source: §V · p. 9
  • Feature Overlap (FO) measures changes in the most activated SAE feature identities before and after unlearning.It compares top-K feature sets for prompts containing and excluding the target concept.Source: §V · p. 9
  • Feature Activation Correlation (FAC) measures shifts in feature activation distributions, complementing FO’s focus on whether the same features remain dominant.The proposed interpretation is lower similarity for target-related activations and preserved similarity for unrelated concepts.Source: §V · p. 9

VI. EVALUATION

The evaluation tests CONCEPTACP on four concepts, comparing forgetting, preservation, precision, robustness, and efficiency. It reports stronger forgetting across fast and multi-step generation, with some concept-specific caveats.

A. Setup

  • The evaluation uses SDXL-Turbo at one and four inference steps, with additional tests on SDXL and SD v1.4; it benchmarks four concepts and compares four unlearning baselines.EvalIGMU supplies the benchmark tasks; evaluation uses FSR, CLIP Acc., FID, ASR, and internal FO and FAC scores.Source: A. Setup · pp. 9–10

B. Unlearning Effectiveness

  • The T4 advantage extends to SDXL and SD v1.4, and forgetting persists across generation schedules of up to 30 steps.The authors attribute ESD’s weaker SDXL-Turbo result partly to its reliance on interventions across many denoising timesteps.Source: B. Unlearning Effectiveness · p. 10, Table III · p. 11
  • Parachute forgetting is less consistent: residual fragments remain in some outputs, while ESD’s T4 FSR of 82.80 accompanies less realistic, less coherent images.The authors caution that a high FSR does not necessarily indicate better visual quality or more faithful removal.Source: B. Unlearning Effectiveness · p. 10
  • For one-step VanGogh removal, the main subject can be affected; the authors recommend avoiding an excessively large suppression coefficient for global styles.For other concepts and under T4, they report strong forgetting with comparable image quality and no noticeable degradation.Source: B. Unlearning Effectiveness · p. 10
  • At T1, CONCEPTACP improves average FSR by 36.34 and reduces CLIP Acc. by 32.10 versus the best baseline; at T4, the gains are 24.10 and 12.48, respectively.Table II covers four concepts under both settings.Source: B. Unlearning Effectiveness · p. 10, Table II · p. 11

C. Unrelated Concept Preservation

CONCEPTACP balances forgetting with preservation, reports stronger internal feature-level precision, and performs robustly under adversarial prompts. Results also identify limits in some attack settings and outputs.

C. Unrelated Concept Preservation

  • Compared with ConceptPrune, CONCEPTACP achieves substantially higher FSR with T1 FID only 2.50 higher on average and T4 FID 6.89 lower on average.Against SAEUron, FSR is higher with FID increases of 1.99 in T1 and 2.73 in T4.Source: C. Unrelated Concept Preservation · p. 11

D. Feature-Level Unlearning Precision

  • On average, CONCEPTACP improves FO and FAC over the second-best method by 0.14 and 0.23, respectively.It has the highest scores on Nudity, VanGogh, and Church, and performs comparably to ESD on Parachute.Source: D. Feature-Level Unlearning Precision · p. 12
  • ConceptPrune has low feature-level precision despite reasonable forgetting, while CONCEPTACP changes target-concept features more and preserves unrelated-concept features.The analysis links ConceptPrune’s low top-K feature overlap on unrelated concepts to disturbance of irrelevant features.Source: D. Feature-Level Unlearning Precision · p. 12

E. Robustness

  • Across the I2P, MMA, and UnlearnDiffAtk tests, CONCEPTACP’s ASRs are consistently lower than ESD’s and UCE’s, though ConceptPrune is more robust.Across three additional forgetting tasks, CONCEPTACP ranks second in average ASR, behind ConceptPrune.Source: E. Robustness · p. 12
  • Under adversarial prompts, CONCEPTACP sometimes removes the whole human subject, whereas clean prompts retain it; the authors report no broader human-body erasure.The observed behavior differs between benchmark prompts and adversarial prompts, where clothed appearances occur more often.Source: E. Robustness · p. 12

F. Multi-Concept Unlearning

The paper examines joint concept removal, parameter and efficiency trade-offs, and limitations of feature localization. It concludes that localized, interpretability-guided fine-tuning can support efficient concept forgetting, while identifying areas for further work.

FSR (%)

  • For concepts in different blocks, combining individually unlearned checkpoints preserves comparable FSR without extra training; same-block sequential removal suppresses both concepts, with FID reaching 66.61.The same-block FID is measured on five object categories excluding the unlearned concepts.Source: FSR (%) · p. 13, Table VII · p. 13

G. Ablation Study

  • Moderate suppression balances forgetting and image quality: increasing the magnitude of a generally improves FSR but may increase FID, while excessive suppression can over-edit images.The reported settings are a = −8, −7, −9, and −5 for Nudity, VanGogh, Church, and Parachute; milder a = −6 or −5 is also recommended for VanGogh.Source: G. Ablation Study · p. 13, Figure 8 · p. 13, Figure 9 · p. 13
  • AMP further improves training efficiency, with reported training times of 1.34 s/epoch on SDXL and 0.63 s/epoch on SD v1.4.Source: G. Ablation Study · p. 14

VII. DISCUSSION

  • General-purpose SAEs support localization, but Church and Parachute features are more numerous and dispersed; Parachute’s weaker forgetting suggests less clean sparse-feature decomposition.The authors identify concept-specific SAE training as a possible way to concentrate relevant features.Source: §VII · p. 14

VIII. CONCLUSION AND FUTURE WORK

  • CONCEPTACP internalizes feature interventions in model parameters and localizes updates to one cross-attention block, avoiding repeated diffusion trajectories and inference-time overhead.The conclusion reports superior forgetting over state-of-the-art baselines while maintaining image quality and improving fine-tuning efficiency.Source: §VIII · p. 14
  • The authors identify targeted SAE training, adaptive feature selection across layers and concepts, and stronger adversarial robustness as future work.Source: §VIII · p. 14

ETHICAL CONSIDERATIONS

  • The paper frames concept erasure as dual-use and encourages governance when deploying edited models.Its experiments use established benchmarks and minimize manual inspection of sensitive outputs.Source: ETHICAL CONSIDERATIONS · p. 14

APPENDIX OVERVIEW

The appendices collect supplementary methods, evaluation details, and results, including a feature-localization procedure that selects a concept-relevant block and features for targeted unlearning.

APPENDIX OVERVIEW

  • Appendix B details the evaluation datasets, baselines, and metrics used to assess unlearning effectiveness, image quality, robustness, and precision.Source: APPENDIX OVERVIEW · pp. 15–16
  • Appendix C covers additional evaluations of forgetting, cross-backbone generalization, adversarial robustness, multi-concept unlearning, ablations, and computational efficiency.Source: APPENDIX OVERVIEW · p. 16

APPENDIX A FEATURE LOCALIZATION ALGORITHM

  • Appendix A formalizes feature localization: score features by concept relevance, select the block with the strongest feature, then retain features above a threshold.The selected block and feature set support downstream feature-level interventions.Source: Appendix A · p. 16

A. Datasets

  • The evaluation datasets include I2P and MMA adversarial prompts for nudity robustness, plus UnlearnDiffAtk prompts transferred from SD v1.5 to test SDXL-Turbo.EvalIGMU supplies 250 prompts each for Nudity and VanGogh and 300 each for Church and Parachute; five images are generated per prompt.Source: A. Datasets · p. 16

B. Evaluation Baselines

The evaluation compares CONCEPTACP with representative unlearning methods spanning fine-tuning, model editing, neuron pruning, and inference-time intervention.

B. Evaluation Baselines

  • ESD uses frozen-model predictions to form a negative-guidance update, with parameters and evaluation steps varying by concept and setting.The study reproduces it on SDXL-Turbo with one- and four-step fine-tuning and matching generation settings.Source: B. Evaluation Baselines · p. 17
  • UCE edits cross-attention parameters through a closed-form update and requires no additional training.Source: B. Evaluation Baselines · p. 17
  • ConceptPrune removes relevant neurons using weight-importance scores computed while generating a target concept.It is reproduced with four-step generation and evaluated in both one- and four-step settings.Source: B. Evaluation Baselines · p. 17
  • SAEUron mounts Sparse Autoencoders on SD v1.4 U-Net cross-attention blocks to suppress concepts during inference.Its focus is objects and artistic styles rather than safety-related concepts.Source: B. Evaluation Baselines · p. 17

C. Metrics

The appendix defines output and feature-level measures and reports forgetting across generation schedules and backbones, alongside robustness results under adversarial evaluation.

C. Metrics

  • FSR measures the share of generated images lacking the target concept, while CLIP Accuracy classifies images using positive- versus negative-anchor similarity.Higher FSR indicates stronger unlearning; CLIP Accuracy is the fraction of images correctly classified.Source: C. Metrics · p. 17
  • FID measures generated-to-reference image distribution distance, ASR measures adversarial reactivation, and FO and FAC capture changes in SAE features.Lower FID indicates better image quality and fidelity; lower ASR indicates stronger robustness.Source: C. Metrics · p. 17

A. Forgetting Effectiveness

  • CONCEPTACP shows strong forgetting across four concepts in single- and four-step generation; parachute fabric remnants remain in some examples.Figure 14 presents sampled visual outcomes for each concept and generation setting.Source: A. Forgetting Effectiveness · p. 17
  • Fine-tuning on a small number of early timesteps achieves effective forgetting even in generation schedules extending to 30 steps.Figure 10 evaluates the one-step fine-tuned model under up to 30-step generation.Source: A. Forgetting Effectiveness · p. 17, Figure 10 · p. 18
  • CONCEPTACP forgets target concepts across SDXL and SD v1.4 backbones, supporting generalization beyond SDXL-Turbo.The authors report robustness across architectures and sampling configurations without architecture-specific modifications.Source: A. Forgetting Effectiveness · p. 18

B. Robustness

  • Against UnlearnDiffAtk, CONCEPTACP ranks second overall, with one best and one second-best result across four concept tasks.Together with I2P and MMA results, the authors report strong overall robustness under intrinsic and adversarial evaluations.Source: B. Robustness · p. 18
  • UnlearnDiffAtk results may not transfer consistently across models, motivating SDXL-Turbo-specific adversarial forgetting benchmarks for fuller robustness assessment.Source: B. Robustness · p. 18

C. Multi-concept Unlearning

The appendix presents qualitative sequential concept erasure, analyzes parameter effects and efficiency, and examines how selected features differ across concepts and backbones.

C. Multi-concept Unlearning

D. Ablation Study

  • Increasing the magnitude of scaling factor a strengthens concept suppression but can reduce image quality; values below -10 are not recommended.For nudity, stronger suppression may make the person disappear; the authors hypothesize that overly strong feature activation or suppression causes imbalance.Source: D. Ablation Study · p. 18, Figure 12 · p. 19
  • With automatic mixed-precision training, average per-epoch fine-tuning time drops from 6.7 seconds to 1.34 seconds on SDXL-Turbo and 0.63 seconds on SD v1.4, demonstrating additional efficiency gains.Latent I/O fine-tuning bypasses diffusion inference and avoids storing intermediate activations, reducing GPU memory use and enabling a larger batch size under the same hardware budget.Source: D. Ablation Study · p. 19
  • CONCEPTACP adds no inference cost, while SAEUron incurs a 73.3% increase in inference overhead on SDXL-Turbo.Concept erasure with CONCEPTACP takes less than three minutes per concept, though adaptation takes slightly longer upfront.Source: D. Ablation Study · p. 19

APPENDIX D RELEVANT FEATURES

  • Highly activated features for Nudity, Church, and Parachute are largely consistent between SDXL-Turbo and SDXL, while VanGogh shows noticeable differences.The comparison uses shared SAEs and the same feature-identification dataset.Source: Appendix D · pp. 19–20, Table X · p. 20
  • Church and Parachute involve more associated features than Nudity and VanGogh; limited diversity in SAE training samples may also prevent feature concentration.The authors connect multi-component object visuals with harder forgetting that avoids affecting unrelated concepts.Source: Appendix D · p. 20

Related papers

Loading 2610.32059v1…