Source-linked AI summary
Mass Concept Erasure in Diffusion Models with Concept Hierarchy
Jiahang Tu, Ye Li, Yiming Wu, Hanbin Zhao, Chao Zhang, Hui Qian
TL;DR
Mass concept erasure in diffusion models is limited by growing parameter costs and degradation of general or supertype generation. The paper uses a supertype-subtype hierarchy for shared group-wise suppression and introduces SuPLoRA, with experiments reporting a stronger trade-off between erasure and preserved generative quality. Its scope depends on having shared supertype structures.
Problem
Mass concept erasure can become inefficient as erased concepts multiply, while repeated suppression can degrade general and shared-supertype generation.
Method
The paper groups related erased concepts under shared supertypes for joint suppression and uses SuPLoRA with a frozen down-projection and updated up-projection to preserve supertype generation.
Results
The method achieves a more favorable trade-off between erasing undesired concepts and preserving generative quality across a benchmark spanning celebrities, objects, and pornographic content.
Takeaways & Limitations
Shared supertype structures support parameter-efficient mass erasure while SuPLoRA mitigates degradation of general and supertype-level generation.
Takeaways & Limitations
The approach may be less effective when erased concepts have limited overlap in shared supertype structures.
Abstract
from arXiv · showhide
The success of diffusion models has raised concerns about the generation of unsafe or harmful content, prompting concept erasure approaches that fine-tune modules to suppress specific concepts while preserving general generative capabilities. However, as the number of erased concepts grows, these methods often become inefficient and ineffective, since each concept requires a separate set of fine-tuned parameters and may degrade the overall generation quality. In this work, we propose a supertype-subtype concept hierarchy that organizes erased concepts into a parent-child structure. Each erased concept is treated as a child node, and semantically related concepts (e.g., macaw, and bald eagle) are grouped under a shared parent node, referred to as a supertype concept (e.g., bird). Rather than erasing concepts individually, we introduce an effective and efficient group-wise suppression method, where semantically similar concepts are grouped and erased jointly by sharing a single set of learnable parameters. During the erasure phase, standard diffusion regularization is applied to preserve denoising process in unmasked regions. To mitigate the degradation of supertype generation caused by excessive erasure of semantically related subtypes, we propose a novel method called Supertype-Preserving Low-Rank Adaptation (SuPLoRA), which encodes the supertype concept information in the frozen down-projection matrix and updates only the up-projection matrix during erasure. Theoretical analysis demonstrates the effectiveness of SuPLoRA in mitigating generation performance degradation. We construct a more challenging benchmark that requires simultaneous erasure of concepts across diverse domains, including celebrities, objects, and pornographic content.
1 Introduction
Diffusion models can learn unsafe or sensitive concepts, motivating concept erasure, but erasing many concepts increases parameter costs and can degrade general and supertype generation. The paper organizes related concepts hierarchically for group-wise erasure and introduces SuPLoRA to preserve supertype capability while evaluating erasure across diverse domains.
- Diffusion models can generate unsafe, copyrighted, offensive, or privacy-sensitive content learned from unfiltered or filtered datasets.
- Concept-wise erasure becomes inefficient as parameter sets grow linearly with the number of erased concepts, while repeated suppression degrades general generation.
- Erasing more celebrity concepts noticeably degrades Stable Diffusion’s ability to generate the supertype person.
- The proposed hierarchy groups semantically related erased concepts as child nodes under shared supertype parents, enabling joint group-wise suppression.
- SuPLoRA preserves supertype generation by encoding supertype information in a frozen down-projection while updating the up-projection during erasure.
- The benchmark simultaneously erases concepts across celebrities, objects, and pornographic content, with results favoring a balance between erasure and generative quality.
2 Related Work
Related work develops inference-time interventions and fine-tuning methods for erasing targeted concepts, including single-concept and mass-erasure approaches. Preserving unerased concepts remains underexplored in mass erasure because shared supertype generation can degrade as more concepts are removed.
- Inference-time methods modify sampling, often through classifier-free guidance, to steer latent representations away from undesired concepts without changing model parameters.
- Fine-tuning approaches include single-concept erasure methods that selectively update parameters associated with one target concept.
- Mass concept erasure methods target multiple concepts simultaneously, including approaches that prune unions of concept-responsive expert neurons.
- Regularization, auxiliary losses, and anchoring losses have been used to retain unerased concepts during erasure.
- A persistent limitation is that generating a shared supertype can degrade as more related concepts are erased, motivating SuPLoRA.
3 Methodology
The method organizes erased concepts into supertype–subtype groups, jointly suppresses related concepts, and uses SuPLoRA to preserve supertype generation while retaining denoising capability. It implements this through shared group-wise parameters, diffusion regularization, and constrained low-rank updates.
- Methodology: The objective suppresses targeted concepts while preserving generation quality for general and supertype concepts.The modified model changes mappings for targeted concepts without degrading general-concept generation, with additional attention to supertype preservation as erasure scales.
- Concept Hierarchy Construction: A concept hierarchy maps each erased child concept to a semantically related supertype, enabling shared parameters for joint erasure.Examples include grouping celebrities under “person”; the hierarchy is constructed using LLM semantic understanding.
- Group-wise Suppression: Group-wise suppression jointly minimizes attention to concept-relevant regions for related concepts rather than allocating independent suppression parameters.The method uses MACE-style attention modulation and segmentation priors to identify relevant image areas.
- Group-wise Suppression: Diffusion regularization trains denoising on regions unrelated to erased concepts, reducing disruption to the diffusion process.The final objective combines attention suppression and diffusion losses, with the latter applied outside concept-associated regions.
- Supertype-Preserving Low-Rank Adaptation: SuPLoRA uses low-rank updates in which the down-projection defines an orthogonal subspace and only the up-projection is fine-tuned for erasure.The linear-layer formulation is o_j = W h_j + A_j B_j h_j; SuPLoRA constructs B_j from the orthogonal complement of the supertype gradient subspace.
- Supertype-Preserving Low-Rank Adaptation: Because the erasure update is orthogonal to the supertype gradient subspace, freezing B_j and tuning A_j is intended to preserve supertype generation.The analysis connects this orthogonality to reduced interference with supertype generation, and the resulting capability is described as preserved.
4 Experiments
Experiments evaluate mass concept erasure across celebrities, objects, and pornography, measuring both removal effectiveness and preservation of desirable generation. The method achieves a favorable balance through strong erasure, preserved domain-specific and supertype generation, and reduced storage and training time.
- Experimental Setup: The benchmark simultaneously erases concepts across celebrities, objects, and pornography while retaining additional concepts in the object and celebrity domains.The evaluation compares erasure effectiveness with preservation of domain-specific, MS-COCO, and supertype concepts.
- Quantitative Analysis: Our method achieves the best trade-off among mass-concept methods, combining strong erasure effectiveness with high domain-specific generation and competitive MS-COCO results.The comparison includes UCE, SRS-ME, and MACE.
- Quantitative Analysis: Our method excels in supertype generation while reducing storage and training time compared with MACE.
- Qualitative Comparison: Qualitatively, our method removes target concepts effectively while maintaining high-quality domain-specific and supertype generation.Several baselines either produce noisy outputs, fail to suppress some targets, or lose coherent generation when erasing 30 concepts under person or bird.
- Ablation Studies: Removing the concept hierarchy raises parameter cost from 7.11MB to 28.5MB, while removing SuPLoRA or diffusion loss reduces generation quality and preservation.The full method best balances erasure with general and supertype-level generation.
5 Conclusion and Limitations
The paper concludes that shared supertype structures enable parameter-efficient group-wise erasure, while SuPLoRA mitigates degradation of supertype generation. Its effectiveness is constrained when erased concepts have limited overlap under shared supertypes.
- Conclusion: The method organizes related erased concepts under shared supertypes and jointly suppresses them, improving parameter efficiency.
- Conclusion: SuPLoRA freezes the down-projection and updates only the up-projection matrix to mitigate supertype generation degradation.
- Limitations: When erased concepts share limited supertype structure, suppression may be less effective.The paper identifies adaptive, structure-independent erasure as future work for better scalability.
Supplementary Materials: Mass Concept Erasure in Diffusion Models with
The supplementary materials document implementation, evaluation-prompt construction, and scope decisions for mass concept erasure. They exclude the style domain because SD v1.4 produces unreliable style generations across random seeds.
- Supplementary Materials: The supplementary materials provide concept lists, implementation details, additional experiments, qualitative comparisons, diffusion preliminaries, and societal-impact discussion.
- Multiple Concepts: The mass concept erasure task covers celebrity, object, and pornography domains.The object-domain procedure uses Stable Diffusion v1.4 and a ViT-L/16 classifier to select highly recognized object categories.
- Multiple Concepts: General-concept evaluation uses 10,000 MS-COCO prompts selected for low relevance to erased concepts using token-embedding similarity.
- Multiple Concepts: The style domain is excluded because SD v1.4 generation quality depends heavily on the random seed and most seeds yield unsatisfactory results.
B Implementation Details
The hierarchy is constructed by clustering semantically related erased concepts and assigning each cluster a supertype with GPT-4o. Experiments compare the proposed method against established baselines using their original settings where applicable.
- Hierarchy Construction: CLIP pairwise similarities and clustering group semantically related erased concepts before GPT-4o assigns each cluster an appropriate supertype.For example, jay, macaw, and bald eagle are assigned the shared supertype bird.
- Implementation Details: The experiments compare the proposed method with ESD-u, UES-x, FMN, CE-SDWV, UCE, SRS-ME, SPM, and MACE.For single-domain experiments, baselines use the original hyperparameter settings supplied by their authors when compatible with the task.
B.3 Experimental Setup
The experiments evaluate mass concept erasure across domain-specific concepts, using standardized prompts and image-generation settings.
- Evaluation prompts use the template “a photo of the {erased concept}”.
- Table 4 organizes the mass erasure task by domain-specific concepts, erased concepts, and supertype concepts.
- Each concept is evaluated with eight generated samples using DDIM with 50 sampling steps and classifier-free guidance scale 7.5.
C.1 Scaling the Number of Concepts and Domains
As erasure expands across concepts and domains, preserving useful generation becomes harder; the proposed method achieves a better balance than MACE while maintaining strong erasure.
- As erased concepts and domains increase, balancing target suppression with preservation of general and supertype generation becomes increasingly difficult.
- Under the 20/20 setting, celebrity and object classification drops by 5.000% and 1.875%, compared with MACE’s 5.480% and 5.263%.
- The method improves domain-specific generation preservation by 14.76% for objects and 5.380% for celebrities.
- It achieves CLIP scores of 31.10 on MS-COCO concepts and 26.33 on supertype concepts, exceeding MACE’s 30.53 and 25.91.
- The results indicate improved mitigation of useful generative-capacity degradation during multi-domain erasure.
C.2 Erasure effect on SuPLoRA Ablation
SuPLoRA variants remove targeted celebrity and object concepts comparably, but SuPLoRA provides the strongest preservation of generation quality.
- Celebrity accuracy ranges from 4.160% to 5.420%, while object accuracy ranges from 2.080% to 2.410% across update strategies.
- SuPLoRA achieves the best domain-specific accuracy, with 83.38% on celebrities and 61.50% on objects.
- SuPLoRA also obtains FID 17.94 and the highest CLIP scores for MS-COCO concepts at 30.66 and supertype concepts at 26.21.
- These results identify SuPLoRA as the update strategy with the strongest balance between erasure and generation preservation.
C.3 Effect on the Rank of Bj Ablation.
The ablations examine the rank of the frozen down-projection and the hierarchy design, showing that moderate rank preserves supertype information efficiently and complex semantics remain effective.
- Rank ablation: At r = 1, the projection captures 89.79% of orthogonal supertype information and yields Supertype CLIP Score 25.43.
- Rank ablation: At r = 3, the information ratio reaches 98.01% and Supertype CLIP Score rises to 26.10.
- Rank ablation: At r = 5, the ratio is 98.53% and Supertype CLIP Score peaks at 26.21; larger ranks provide minimal gains while increasing parameter cost.
- Complex hierarchy design: The main hierarchy uses two levels, while broader semantic relations can include nested and intersecting categories such as actor, director, and person.
- Complex hierarchy design: A complex hierarchy combines gender, occupation, and skin color into composite supertypes and applies SuPLoRA to suppress leaf-level celebrities.
- Complex hierarchy design: Supertype CLIP Score increases as additional semantic dimensions are introduced, indicating preserved supertype integrity under more granular hierarchies.
D Additional Qualitative Results
The paper presents qualitative visual results for erasing targeted concepts and preserving unerased and supertype concepts across celebrity and object domains. The figures organize these comparisons by targeted erasure, domain-specific preservation, and supertype preservation.
- Qualitative assessment: The qualitative-results section presents visual outputs for assessing concept erasure and preservation, with figure indices listed in Table 9.The surrounding material describes qualitative assessment through an array of visual results.
- Targeted concept erasure: Figures 4 and 5 compare erasure of targeted concepts in the celebrity and object domains, respectively.The listed examples include celebrity names and object concepts such as angora, dalmatian, fireboat, and ice bear.
- Unerased-concept preservation: Figures 6 and 7 compare preservation of unerased concepts in the celebrity and object domains, respectively.These figures evaluate whether concepts outside the erased targets remain represented in the two domains.
- Supertype preservation: Figure 8 provides a qualitative comparison of preserving supertype concepts.The figure focuses specifically on supertype preservation rather than targeted concept removal or domain-specific preservation.