Source-linked AI summary
One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications
Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, Guiguang Ding
TL;DR
Existing diffusion-model concept erasure methods can damage non-target concepts, scale poorly across models and deployment scenarios, and leave concrete-object preservation challenging. This paper introduces transferable one-dimensional SPM adapters with Latent Anchoring and prompt-dependent Facilitated Transport to erase multiple concepts while preserving other generations. Across diverse concepts, models, and applications, the method reports precise erasure, reduced alteration and erosion, and substantial efficiency gains.
Problem
Existing erasure methods can cause generation alterations and concept erosion, require model-specific redesign with costs that scale across deployments, and still face challenges preserving coherence when removing concrete objects.
Method
The framework injects one-dimensional SPM adapters into diffusion models, uses Latent Anchoring during fine-tuning, and combines transferable SPMs with prompt-dependent Facilitated Transport during generation.
Results
SPM achieves precise multi-concept erasure while mitigating generation alteration and erosion across diverse models and applications, with reported speed improvements exceeding 160× over state-of-the-art methods.
Takeaways & Limitations
SPMs support customizable, training-free transfer across diffusion models while preserving non-target generations and reducing time, computational, and storage costs.
Takeaways & Limitations
Existing approaches still face difficulty eliminating concrete objects while maintaining coherence and quality.
Abstract
from arXiv · showhide
The prevalent use of commercial and open-source diffusion models (DMs) for text-to-image generation prompts risk mitigation to prevent undesired behaviors. Existing concept erasing methods in academia are all based on full parameter or specification-based fine-tuning, from which we observe the following issues: 1) Generation alternation towards erosion: Parameter drift during target elimination causes alternations and potential deformations across all generations, even eroding other concepts at varying degrees, which is more evident with multi-concept erased; 2) Transfer inability & deployment inefficiency: Previous model-specific erasure impedes the flexible combination of concepts and the training-free transfer towards other models, resulting in linear cost growth as the deployment scenarios increase. To achieve non-invasive, precise, customizable, and transferable elimination, we ground our erasing framework on one-dimensional adapters to erase multiple concepts from most DMs at once across versatile erasing applications. The concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to learn targeted erasing, and meantime the alteration and erosion phenomenon is effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once obtained, SPMs can be flexibly combined and plug-and-play for other DMs without specific re-tuning, enabling timely and efficient adaptation to diverse scenarios. During generation, our Facilitated Transport mechanism dynamically regulates the permeability of each SPM to respond to different input prompts, further minimizing the impact on other concepts. Quantitative and qualitative results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated the superior erasing of SPM. Our code and pre-tuned SPMs are available on the project page https://lyumengyao.github.io/projects/spm.
1. Introduction
Diffusion-model safety efforts face shortcomings in moderation and targeted erasure, including concept erosion, unpredictable alterations, and costly model-specific adaptation. The proposed framework uses lightweight SPM adapters with latent anchoring to erase multiple concepts while preserving other generations and enabling transfer across models.
- Diffusion models’ synthesis capabilities introduce risks including copyright infringement, privacy breaches, and mature-content dissemination.
- Dataset cleansing, blacklisting, and post-hoc safety checkers are limited by labor costs, introduced stereotypes, annotated-data dependence, and circumvention.
- Full-parameter or specification-based erasure can distort non-target generations and cause concept erosion, especially when multiple concepts are erased simultaneously.
- Existing erasure methods require repeated redesign for each security specification and model, causing time and computational costs to grow linearly with deployment changes.
- SPM is a one-dimensional adapter that learns concept erasure while preserving the pretrained model, and Latent Anchoring retains other concepts by anchoring their generations to corresponding origins.SPM adds a minimum size increase of 0.0005× and requires no auxiliary real or synthetic training data.
- Across more than 100 tasks, SPM erased objects, styles, sexual content, and memorized images while suppressing alterations and erosion; its model-independent transfer achieved over 160× speed improvement versus state-of-the-art methods.
2. Related Work
Prior mitigation methods intervene during data preparation, model training, generation, or post-generation screening, but each has practical or quality limitations. The paper presents SPM as a non-invasive, precise, customizable, and transferable alternative that can also complement layered defenses.
- Existing mitigations intervene through training-data filtering, pretrained-model fine-tuning, in-generation guidance, or post-generation content screening.
- Training-data filtering can require extensive curation and may introduce detector-derived bias despite improving commercial safety.
- Parameter-based concept erasure can remove targets but may collapse generation quality because unconstrained training produces arbitrary images.
- Figure 2 depicts SPM training with Latent Anchoring, plug-and-play transfer of combined SPMs, and prompt-dependent permeability that targets risky prompts while preserving safe generations.
- Prompt blacklisting and multilayered moderation can depend on closed-source resources, while post-generation detectors require iterative data curation and remain vulnerable to circumvention.
- Eliminating concrete objects while maintaining coherence and quality remains a challenge for existing approaches.
- Unlike prior interventions, SPM is described as non-invasive, precise, customizable, and transferable, and can integrate with interventions at different stages as a layered defense.
3. Method
SPM is a lightweight, model-independent adapter for targeted concept erasure that preserves the pretrained diffusion model and supports combining and transferring erasure modules. Latent Anchoring preserves non-target generations during training, while Facilitated Transport dynamically limits interference during inference.
- SPM as a 1-dim Lightweight Adapter: SPM is a one-dimensional adapter injected into diffusion models to learn concept-specific semi-permeability without modifying pretrained parameters.It introduces a thin membrane that learns targeted erasure while preserving the original model.
- SPM as a 1-dim Lightweight Adapter: The adapter suppresses undesired content through an erasing signal whose strength is controlled by a trainable regulator.The original forward operation y = W x is intervened by the SPM.
- Customization and Transferability: SPMs can be combined to erase multiple concepts and transferred across diffusion models without repeating the full erasing pipeline.A corpus of concept-specific SPMs supports plug-and-play overlay on other models.
- Latent Anchoring: Latent Anchoring aligns the target concept with a surrogate while anchoring other concepts to their original generations.Sampling in conceptual space emphasizes consistency for distant concepts and suitable influence on target synonyms.
- Facilitated Transport: Facilitated Transport dynamically increases permeability for concepts relevant to a prompt and deactivates transport when relevance is low.It combines global semantic similarity with token-level similarity to reduce effects on safe concepts.
- Facilitated Transport: SPM exhibits sufficient Snoopy erasure while keeping impacts on non-target concepts negligible.The figure samples illustrate the intended targeted-versus-non-target behavior.
4. Experiments
Experiments evaluate SPM across concept-removal settings, applications, models, and efficiency conditions. SPM preserves non-target generation while suppressing targets, transfers without retraining, and scales more efficiently than prior methods.
- Experimental scope: SPM evaluation covers varied concepts, personalized models, and practical erasing applications, including object removal, artistic-style removal, and nudity mitigation.The experiments assess single- and multi-concept erasing and report more than 100 tasks.
- Single-concept erasure: SPM preserves non-target generation more effectively than prior methods while maintaining sufficient target erasure.Previous methods show generation alteration, concept erosion, or degraded general FID, whereas SPM keeps general FID intact.
- Multi-concept erasure: SPM suppresses multi-concept alteration and erosion, with general FIDg increasing by ≤0.02 versus 10× to 200× variances for prior methods.The comparison is conducted in the more challenging setting where several concepts are erased simultaneously.
- Efficiency analysis: For c = 20, n = 5, and p = 60, SPM reduces time consumption by 65.1% versus ESD and 39.1% versus ConAbl, while storing only 0.0005× the parameters.The comparison also reports a 167.4× speed margin over SA in the stated setting.
- Training-free transfer: SPMs transfer from SD v1.4 to SD v1.5 and community checkpoints without model-specific fine-tuning while preserving generation consistency.The evaluated community models include Chilloutmix1, RealisticVision2, and Dreamshaper-83.
- Nudity mitigation: SPM achieves an 85.3% reduction in generated nudity and eliminates explicit contents across different nude categories.The evaluation uses the I2P benchmark and compares SPM with dataset filtering and prior concept-erasing methods.
5. Conclusion
The paper concludes that lightweight one-dimensional SPMs enable precise multi-concept erasure while preserving general generation quality. Latent Anchoring and Facilitated Transport reduce alteration and erosion, and one-dimensional capacity is sufficient for effective, efficient operation.
- Conclusion: SPMs erase multiple concepts with a minimum model-size increase of 0.0005× while supporting practical erasing across diverse applications.The framework combines lightweight adapters with training-time and inference-time safeguards.
- Conclusion: One-dimensional SPMs retain erasing, preservation, and general-generation performance despite their reduced intermediate dimensionality.Dimension analysis reports minimal performance fluctuation as dimensionality increases from one upward.
- Conclusion: Latent Anchoring and Facilitated Transport jointly mitigate alteration and erosion of non-target concepts while maintaining target erasure.Without these components, non-target FID can rise sharply; LA and FT improve non-target and general-caption metrics.
- Conclusion: Increasing η improves target removal but can increase alteration in inspected non-targets, while general generative consistency remains strong.The sensitivity analysis identifies a trade-off between target erasure and non-target alteration.
A.3. Sensitivity Analysis of η
The sensitivity and generality analyses show that SPMs remain effective across η settings, diffusion-model architectures, prompts, concepts, and applications. They preserve non-target content while supporting compositional, transferable, and reconsolidation-based erasure.
- A.3. Sensitivity Analysis of η: Increasing η improves removal of the targeted concept but can also increase alteration in inspected non-targets.Despite this trade-off, FIDg indicates strong generative consistency on general concepts.
- Model generality: One-dimensional SPMs generalize to SD v2.1 and SDXL v1.0 without intricate parameter search, preserving precise erasure across different generative structures.The results extend conclusions obtained on SD v1.4 to newer models.
- Prompt preservation: SPMs preserve non-target descriptions such as outdoor, sailing, cyberpunk, and wheat within rich prompts.The SDXL v1.0 examples specifically test preservation of non-target prompt elements during erasure.
- Cross-application erasure: Cross-application SPM composition erases comic style and Snoopy independently or jointly, leaving unrelated prompt content such as sailing out of the sea.The examples demonstrate interaction between artistic-style and instance erasure.
- Memorized-image erasure: SPMs erase memorized artworks while preserving related artists and other paintings, with targeted SSCD scores reduced below 0.1.The evaluation covers works by Hokusai and Vincent van Gogh and compares generated images before and after erasure.
- Concept reconsolidation: SPMs support reconsolidation by rewriting target concepts toward specified surrogates such as Gal Gadot, Darth Vader, Heath Ledger, or Batman.The method is presented as flexible for application requirements involving concept replacement.
B.6. Numerical Results of Artistic Style Erasure
Numerical artistic-style experiments show that SPMs outperform comparative approaches in target-style erasure, preservation of other styles, and general generation capability. Multi-concept overlays also preserve prompt-aligned content without erosion or excessive alteration.
- Numerical comparison: SPMs significantly surpass comparative approaches in targeted style erasure, preservation of other styles, and general generation capability.Table 7 evaluates target erasure, non-target preservation, and general generation capacity using CS, Clip Score, FID, and FIDg indicators.
- Multi-concept preservation: SPMs preserve prompt-aligned objects, backgrounds, and actions during multiple concept erasure without erosion or substantial alteration.COCO-30k Caption generations retain general content while cartoon characters are erased.
B.7. Samples of Nudity Removal
SPMs suppress nudity-related generation through a single-word erasure and transfer training-free to a content-oriented model. Their latent-space behavior also preserves distinct concepts that share words while extending erasure to related patterns.
- Nudity removal: Erasing the single word “nudity” suppresses exposure of different body parts, including after training-free transfer to ChillOutMix.The transfer result is tested on a model optimized toward generating such content.
- Shared-word concepts: SPMs erase synonyms such as Mickey Mouse and Mickey while largely preserving distinct shared-term concepts such as Mouse and Batman versus {*}man.The result supports latent-distance similarity as a more accurate preservation signal than token overlap.
- Erasure granularity: Erasure can extend beyond the object level to high-level patterns associated with the target, raising granularity questions for interconnected concepts.The paper cites Minnie and Mickey as an example where universally agreed erasure standards may be lacking.
C.1. Implementation Details
Experiments fix the pretrained model parameters while training only SPM modules under standardized settings, using official implementations for comparative methods. Results indicate that SPM preserves other concepts better than competing erasure methods while retaining effective removal.
- Implementation: SPMs are injected into U-Net linear and convolution layers while pretrained parameters remain fixed, adding approximately 0.05% parameter overhead.
- Implementation: Training uses 3,000 iterations with batch size 1 and four latent-anchoring samples unless otherwise specified.
- Implementation: SPM optimization uses AdamW8bit with learning rate 1e-4, cosine restarts, and a 500-iteration warmup period.
- Implementation: Experiments generally use the empty prompt as surrogate concept, with λ=10^3, sampling factor α=1.0, and erasing guidance η=1.0.
- Comparisons: Comparative methods use their official public codebases, while qualitative comparisons report that SPM preserves other concepts more effectively than ESD, ConAbl, and SA.
D.2. Additional Samples of 20 Concepts Erasure
Additional experiments examine progressively erasing 20 concepts and test style and cross-model object erasure. SPM remains robust under multiple overlays and preserves non-target styles while transferring across community models.
- Multi-concept erasure: Twenty Disney characters are progressively erased to test robustness under increasing numbers of target concepts.
- Multi-concept erasure: ESD can converge toward empty-background images, whereas latent anchoring and facilitated transport keep object-centric generations robust with 20 SPMs overlaid.
- Style erasure: SPM erases Van Gogh, Picasso, and Rembrandt styles while preserving image content and consistency of non-targeted styles.
- Transfer: A cat-erasing SPM trained on official Stable Diffusion with one class name also suppresses cat patterns in SD-derived community models.
D.5. Full Numerical Results of Object Erasure
The extended evaluation distinguishes target erasure from non-target alteration and documents both SPM’s preservation strengths and its failure boundaries. Quantitative indicators require joint interpretation because CS and CER can miss alterations that FID reveals.
- Metrics: CS and CER measure target-content correlation, whereas FID captures generation alteration but cannot alone establish accurate target removal.
- Metrics: CS and CER may overlook generation alterations until they become substantial concept erosion, while increased FID indicates significant alteration after erasure.
- Interpretation: ConAbl and SA can retain similar CS and CER to the original model while showing significant FID increases, indicating generation alteration.
- Failure cases: When prompts trigger SPM despite the original model not generating the target, the resulting activation can still disrupt generation.
- Failure cases: Minnie generations can erode because of semantic similarity to the erased Mickey target, while the desired granularity of erasure remains unresolved.
- Failure cases: On community diffusion models, nudity erasure may fail to clothe characters at γ=1.0 and require manually increasing strength to 2.0.
- Implications: The authors conclude that safe deployment requires clearer erasure definitions, more precise methods, and multi-stage mitigation.
E. Comparison with Concept-based Manipulation Methods
SPM differs from concept personalization and image editing by targeting risk detection and erasure while preserving unrelated generations. Its surrogate-based design also supports concept rewriting, but the paper notes governance risks from neutral target selection.
- Concept personalization: Concept personalization introduces new objects or styles, whereas SPM modifies or erases existing concepts while preserving other relevant concepts.
- Image editing: Image editing and inpainting alter specified content in an input image, while SPM detects risks from user prompts and generates safe content without requiring an input image.
- Concept editing: SPM can rewrite one concept with another through surrogate concepts, extending its use beyond erasure toward concept editing.
- Societal impact: Because target selection is neutral, SPM could be misused to conceal or reconsolidate specific events, motivating stronger oversight and review.
- Analysis: SPM’s latent-space effects are examined through reconsolidation, shared-word concepts, nearby semantic representations, and single-concept preservation tests.
- Style erasure: Style-erasure examples compare SPM with prior methods across Van Gogh, Picasso, and Rembrandt prompts, emphasizing preservation of non-target styles.
- Transferability: Training-free transfer examples compare original and SPM-applied outputs across RealisticVision, Dreamshaper, and ChillOutMix community models.
- Limitations: The paper also reports suboptimal and failure cases for Mickey and nudity erasure, delimiting the scope of the demonstrated manipulation capabilities.