Source-linked AI summary
Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts
Leyang Li, Shilin Lu, Yan Ren, Adams Wai-Kin Kong
TL;DR
Concept erasure must prevent harmful or inappropriate imagery while preserving unrelated content and natural-image generation. ANT uses trajectory-aware finetuning to modify mid-to-late denoising while preserving early-stage score fields, and it reports superior performance across single- and multi-concept erasure.
Problem
Existing finetuning-based concept-erasure methods can disrupt early denoising trajectories or depend on heuristic anchor selection, threatening preservation and output quality.
Method
ANT reorients denoising trajectories during mid-to-late stages, preserves early-stage score fields, avoids anchor concepts, and uses augmentation-enhanced saliency for single-concept erasure.
Results
ANT demonstrates superior performance and state-of-the-art results in both single- and multi-concept erasure scenarios.
Takeaways & Limitations
ANT balances removal of unwanted concepts with preservation of unrelated elements and structural integrity in generated images.
Takeaways & Limitations
Concept-erasure methods must preserve early-stage score functions because altering them can prevent convergence to the natural-image manifold and degrade output quality.
Abstract
from arXiv · showhide
Ensuring the ethical deployment of text-to-image models requires effective techniques to prevent the generation of harmful or inappropriate content. While concept erasure methods offer a promising solution, existing finetuning-based approaches suffer from notable limitations. Anchor-free methods risk disrupting sampling trajectories, leading to visual artifacts, while anchor-based methods rely on the heuristic selection of anchor concepts. To overcome these shortcomings, we introduce a finetuning framework, dubbed ANT, which Automatically guides deNoising Trajectories to avoid unwanted concepts. ANT is built on a key insight: reversing the condition direction of classifier-free guidance during mid-to-late denoising stages enables precise content modification without sacrificing early-stage structural integrity. This inspires a trajectory-aware objective that preserves the integrity of the early-stage score function field, which steers samples toward the natural image manifold, without relying on heuristic anchor concept selection. For single-concept erasure, we propose an augmentation-enhanced weight saliency map to precisely identify the critical parameters that most significantly contribute to the unwanted concept, enabling more thorough and efficient erasure. For multi-concept erasure, our objective function offers a versatile plug-and-play solution that significantly boosts performance. Extensive experiments demonstrate that ANT achieves state-of-the-art results in both single and multi-concept erasure, delivering high-quality, safe outputs without compromising the generative fidelity. Code is available at https://github.com/lileyang1210/ANT
1. Introduction
Concept erasure protects against harmful or inappropriate imagery, but finetuning methods can damage natural-image trajectories or depend on heuristic anchors. ANT instead reorients denoising only later, preserving early structure while avoiding unwanted concepts.
- Concept erasure targets harmful or inappropriate visual content in text-to-image models.
- Finetuning-based methods modify score-function fields, while anchor-free and anchor-based approaches differ in how they alter conditional scores.
- Anchor-free methods can disrupt early movement toward natural-image manifolds, causing artifacts or unintended content.They emphasize moving away from unwanted manifolds throughout denoising, although early stages establish plausible layouts.
- Anchor-based methods align unwanted-concept scores with benign anchors, but their effectiveness depends on heuristic anchor selection.Poor anchors can reduce image quality, especially in complex scenarios.
- ANT reverses classifier-free guidance direction during mid-to-late denoising, preserving early-stage score fields without requiring anchor concepts.Its trajectory-aware objective keeps early trajectories largely unchanged while reorienting later trajectories away from unwanted concepts.
3. We introduce an augmentation-enhanced weight saliency
ANT combines trajectory-aware concept erasure with an augmentation-enhanced saliency map and supports strong performance across single- and multi-concept settings.
- ANT preserves overall structure and contextual integrity while removing the target concept in cat-conditioned generations.
- The augmentation-enhanced saliency map identifies parameters most responsible for generating the undesired concept, enabling more effective and efficient erasure.
- ANT’s objective substantially enhances existing multi-concept erasure frameworks and achieves state-of-the-art results in single- and multi-concept settings.
2. Related Work
Related work spans inference-time guidance, parameter editing, attention manipulation, and anchor alignment. Across these approaches, balancing target removal with preservation remains central, while early-stage score preservation is under-addressed.
- SLD suppresses undesired concepts at inference time without modifying model parameters, whereas ESD performs permanent concept removal through direct parameter editing.
- FMN manipulates attention mechanisms for computational efficiency, while AC aligns unwanted-concept scores with anchor-concept scores.
- Concept-erasure research emphasizes eliminating target concepts while preserving unrelated concepts during finetuning.
- Insufficient attention to early-stage score functions can prevent convergence to the natural-image manifold and degrade output quality and reliability.
3. Method
ANT steers denoising away from unwanted concepts by reversing classifier-free guidance in mid-to-late stages while preserving early-stage movement toward the natural image manifold. Its trajectory-aware objective, saliency-guided parameter selection, and multi-concept integration support targeted and efficient concept erasure.
- Trajectory-Aware Loss Function: ANT reverses the classifier-free guidance condition direction during mid-to-late denoising to modify detailed content while preserving fundamental image structure.The condition direction is δ(c) = ϵθ(zt, t, c) − ϵθ(zt, t), and t′ determines when reversal begins.
- Trajectory-Aware Loss Function: Selecting t′ too early damages structural integrity, whereas selecting it too late limits changes to fine details.An appropriate reversal point removes targeted attributes while preserving image naturalness.
- Trajectory-Aware Loss Function: The trajectory-aware objective preserves the early-stage score function field and applies erasure guidance only at later timesteps.Early preservation keeps samples moving toward the natural data manifold, while later erasure steers them away from undesirable modes.
- Trajectory-Aware Loss Function: ANT preserves unconditional score functions because modifying them can impair preservation of unrelated concepts during multi-concept erasure.The objective includes unconditional early- and late-stage preservation terms to limit these effects.
- The Heavy Hitters Among the Parameters: For single-concept erasure, prompt- and seed-augmented saliency maps are intersected to identify stable, concept-specific parameters for finetuning.The resulting map narrows optimization to parameters consistently influential for erasing the target concept, reducing unnecessary perturbations.
- Boosting the Performance of Multi-Concept Erasure Frameworks: For multi-concept erasure, ANT’s trajectory-aware loss integrates with MACE and substantially enhances performance through a plug-and-play solution.The framework trains separate LoRA modules and fuses them into cross-attention layers using a closed-form objective.
4. Experiments
ANT is evaluated on single- and multi-concept erasure tasks, including NSFW content, celebrities, and artistic styles. It reduces unwanted content while maintaining preservation and image-quality performance.
- Evaluation setup: Experiments benchmark ANT on NSFW removal, 100-celebrity erasure, 100-artistic-style erasure, and component ablations.The evaluation uses Stable Diffusion models and compares ANT with state-of-the-art baselines.
- NSFW removal: ANT generates significantly less NSFW content on I2P prompts, including in challenging breast-related cases, while retaining competitive FID and CLIP scores.NudeNet detections on I2P measure residual nudity, while MS-COCO FID and CLIP assess benign-content preservation.
- Celebrity erasure: ANT achieves the highest Hc among celebrity-erasure baselines, balancing low accuracy for erased identities with high accuracy for preserved identities.Hc is the harmonic mean of erased-celebrity accuracy and preserved-celebrity accuracy.
- Art-style erasure: ANT achieves the highest Ha for artistic-style erasure, substantially surpassing baselines in balancing removal of targeted styles with preservation of unrelated styles.Ha is defined as CLIPp − CLIPe, where higher values indicate better balance.
- Ablation study: Applying the erasure loss only at mid-to-late stages slightly improves Hc, while unconditional-loss and preservation terms further improve overall and preservation performance.The full method applies the unconditional loss across all stages and achieves superior overall performance.
- Ablation study: Intersecting saliency maps from multiple prompts and seeds outperforms single-pass parameter selection for NSFW removal by identifying more precise concept-associated parameters.The single-pass saliency map can be inaccurate, whereas the multi-map intersection yields substantially improved performance.
5. Conclusion
ANT uses a geometric view of diffusion trajectories to erase unwanted concepts while preserving unrelated content. Its central insight is that reversing classifier-free guidance during mid-to-late denoising modifies details without compromising overall image structure.
- Conclusion: Reversing classifier-free guidance during mid-to-late denoising modifies detailed content without compromising generated-image structure.This insight motivates ANT’s trajectory-aware approach to concept erasure.
- Conclusion: ANT balances unwanted-concept removal with preservation of unrelated elements and outperforms current state-of-the-art methods on single- and multi-concept erasure.The conclusion reports superior performance across both evaluation settings.
Appendix
The appendix situates concept erasure within broader efforts on safety, balanced preservation, efficiency, scalability, and robustness. Prior work increasingly targets practical editing across many concepts while limiting disruption to unrelated concepts.
- Related work: Concept erasure research addresses harmful or inappropriate visual content in generative models and increasingly emphasizes minimizing effects on unrelated concepts.The related-work discussion frames erasure and preservation as a central trade-off.
- Related work: Recent methods improve finetuning efficiency through low-rank adapters, null-space constraints, closed-form updates, and structure-aware operations.Some reported approaches erase hundreds of concepts within seconds or reduce erasure time by orders of magnitude.
A.3. Scalability.
Scalable concept-erasure research targets simultaneous editing of many concepts with lightweight or structured updates. These methods aim to support broader, reliable deployment while preserving specificity and visual fidelity.
- Scalability: Recent frameworks support simultaneous modification of multiple concepts through lightweight model updates and modular LoRA-based editing.The related work describes methods that eliminate more than 100 concepts with limited interference.
- Scalability: Embedding-space operations and adversarially robust objectives further improve scalability, specificity, and visual fidelity for high-volume editing.These techniques collectively target broader practical deployment scenarios.
A.4. Robustness.
The paper notes that erased models can remain vulnerable to adversarial prompts, motivating methods that improve reliability against prompt-based attacks.
- Erased models remain vulnerable to adversarial prompts that can reactivate undesirable content.
C. Limitations and Future Work
The framework has primarily been evaluated on UNet-based diffusion models, leaving compatibility with newer architectures and robustness to several attacks as future work.
- The approach has primarily been tested on UNet-based diffusion models.Evaluating compatibility with MMDiT architectures is identified as a key next step.
- Future work includes testing robustness against adversarial prompts and methods for learning personalized concepts.
D. Additional Qualitative Results
Additional qualitative results show that ANT removes targeted artistic styles while preserving plausible image quality and unrelated artists’ visual characteristics.
- Figure 8 compares art-style erasure and preservation across different baselines using matched random seeds within each row.The erasure group includes Chris Van Allsburg and Claude Monet; the retention group includes Adriaen Van Outrecht and Adrian Ghenie.
- ANT eliminates Chris Van Allsburg and Claude Monet styles while retaining high-quality, plausible generations.
- ANT preserves the visual characteristics of Adriaen Van Utrecht and Adrian Ghenie with minimal unintended impact on non-target styles.