PaperlayerView exact revision on arXiv

Source-linked AI summary

GRACE: Adaptive Concept Erasure with Geometry-Guided Retention in Diffusion Models

Qinghui Gong, Yihuai Liang, Yuanlun Xie, Deepak Kumar Jain, Vitomir Štruc, Zhengchun Zhou

arXiv:2609.12731v1cs.CV

TL;DR

T2I models internalize sensitive concepts, while erasure methods can modify broad feature directions and struggle to preserve non-target semantics. GRACE localizes target-related feature directions, constrains lightweight adapters, derives a prompt-specific safe anchor, and dynamically gates intervention during denoising. Compared with SOTA concept erasure methods under identical evaluation settings, GRACE improves fine-grained NSFW reduction rate by 17.86%, while reducing target CLIP Score by 4.75% and preservation-oriented FID by 50.58% on average.

  • Problem

    T2I models internalize sensitive concepts, while erasure methods can modify broad feature directions and struggle to preserve non-target semantics.

    Source: §I · p. 1

  • Method

    GRACE localizes target-related feature directions, constrains lightweight adapters, derives a prompt-specific safe anchor, and dynamically gates intervention during denoising.

    Source: §I · p. 2

  • Results

    Compared with SOTA concept erasure methods under identical evaluation settings, GRACE improves fine-grained NSFW reduction rate by 17.86%, while reducing target CLIP Score by 4.75% and preservation-oriented FID by 50.58% on average.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported results indicate stronger target suppression alongside improved preservation of the original model’s generative utility.

    Source: §I · p. 2

Abstract

from arXiv · show

Text-to-image (T2I) diffusion models inevitably internalize sensitive or non-compliant concepts from large-scale pretraining data, necessitating post-hoc concept erasure. However, existing erasure methods often lack explicit constraints on parameter updates, leading to over-intervention and unintended semantic drift. In addition, many methods rely on manually crafted counterfactual supervision, such as surrogate prompts, which incurs substantial data construction costs that limit scalability to new concepts. To address these limitations, we propose GRACE, a structured concept erasure framework designed to enable localized and selective intervention. Specifically, we introduce a semantically weighted sensitive subspace estimation to precisely lock intervention directions, and employ lightweight subspace-constrained adapters to prevent global semantic disturbance. To eliminate the dependency on manual prompt engineering, we design an automatically decoupled safe-anchor mechanism. To mitigate semantic drift induced by excessive intervention, we introduce an energy-driven dynamic gating mechanism that adaptively controls the timing and strength of intervention at inference. Extensive experiments demonstrate that our method achieves a superior balance between erasure effectiveness and generation fidelity. Compared with the average performance of five state-of-the-art (SOTA) concept erasure methods, our method improves the fine-grained NSFW reduction rate by $17.86\%$, while reducing the macro-averaged target CLIP Score and preservation-oriented Fr\'echet Inception Distance (FID) by $4.75\%$ and $50.58\%$, respectively, indicating stronger concept suppression with substantially improved preservation of the original model's generative utility.

I. INTRODUCTION

GRACE targets selective concept erasure by constraining where and when model interventions occur while retaining prompt-consistent content. Its reported comparisons show stronger target suppression alongside improved preservation of generative utility.

I. INTRODUCTION

  • T2I models internalize sensitive concepts, and existing erasure methods struggle to suppress targets without disrupting non-target semantics.The paper identifies unauthorized or offensive content as a persistent compliance risk and describes suppression-preservation as a central challenge.Source: §I · p. 1
  • GRACE estimates target-related feature directions and trains subspace-constrained adapters to limit changes to unrelated semantics.The sensitive subspace localizes intervention, while the adapter constraint limits modifications outside it.Source: §I · pp. 1–2
  • Energy-driven gating adjusts adapter activation across denoising timesteps to reduce unnecessary trajectory deviation and semantic drift.The gate measures target-related response in the sensitive subspace and dynamically adjusts intervention timing and strength.Source: §I · p. 2
  • Under identical evaluation settings, GRACE improves fine-grained NSFW reduction rate by 17.86% versus SOTA concept erasure methods, while reducing target CLIP Score by 4.75% and preservation-oriented FID by 50.58% on average.Source: §I · p. 2
  • An automatically derived safe anchor uses the prompt’s non-target CLIP semantics for preservation supervision, reducing reliance on manually constructed surrogate prompts and benign datasets.The method removes the target-aligned component from the prompt representation and uses the remaining semantics as the preservation target.Source: §I · p. 2

II. RELATED WORK

  • Prior erasure methods use parameter updates, direct weight edits, or inference-time interventions; related subspace methods typically use static constraints.The paper characterizes unconstrained over-intervention as a source of unintended disruption to benign semantics and manifold deviation.Source: §II · p. 2

A. Semantic-Aware Sensitive Subspace Localization

Semantic-Aware Sensitive Subspace Localization identifies compact, layer-wise directions associated with a target concept. SA-PCA weights feature variations by prompt relevance and retains the dominant directions for later use.

A. Semantic-Aware Sensitive Subspace Localization

  • Target-related variation is represented by a compact, layer-wise low-dimensional subspace that provides a shared reference for training constraints and inference-time response measurement.The paper points to empirical layer-wise rank analysis as support for this compact structure.Source: A. Semantic-Aware Sensitive Subspace… · p. 3
  • SA-PCA assigns prompt weights from CLIP similarity to the target concept, giving more influence to prompts with stronger semantic relevance.Target-related prompts are encoded by the frozen diffusion model to collect intermediate UNet features at selected layers.Source: A. Semantic-Aware Sensitive Subspace… · pp. 3–4
  • The weighted covariance’s top K eigenvectors form an orthonormal basis for dominant feature variations semantically associated with the target.The retained rank K determines the number of directions in the estimated sensitive subspace.Source: A. Semantic-Aware Sensitive Subspace… · p. 4
  • The resulting layer-wise subspace bank is reused to constrain adapter optimization and determine which non-target semantics should be preserved.This geometric localization is retained for the subsequent training stage.Source: A. Semantic-Aware Sensitive Subspace… · p. 4

B. Subspace-Constrained Adapter Optimization

GRACE trains lightweight adapters with the pretrained text encoder, UNet, and VAE decoder frozen, balancing safe-anchor semantic retention with changes constrained to target-related directions. Its automatically decoupled safe anchor derives preservation supervision from each training prompt rather than manually constructed prompts or external benign datasets.

B. Subspace-Constrained Adapter Optimization

1) Automatically Decoupled Safe Anchor for Joint Erasure and Retention:

2) Subspace-Constrained Residual Regularization:

GRACE decomposes each adapter-induced feature change into target-related and off-subspace components, then penalizes only the latter to limit unnecessary modification while retaining erasure capacity.

2) Subspace-Constrained Residual Regularization:

3) Timestep-Aware Optimization:

GRACE trains adapters on an intermediate-noise range and uses projection energy at inference to gate intervention according to current target-sensitive responses. The gate activates weakly below a threshold and increases adapter contribution above it.

3) Timestep-Aware Optimization:

  • On the original 1,000-step schedule, training samples timesteps 400–800 to avoid states with weak semantic information or limited room for semantic modification.Source: 3) Timestep-Aware Optimization · p. 5

C. Energy-Driven Dynamic Gating

1) Projection-Energy Layer-wise Gating:

  • The inference gate uses each selected layer’s spatially averaged feature vector to measure response within the sensitive subspace; larger energy indicates a stronger target-sensitive response.Source: 1) Projection-Energy Layer-wise Gating · p. 5
  • The soft gate controls adapter activation using a threshold and transition sharpness, with weak activation below the threshold and increased contribution above it.Source: 1) Projection-Energy Layer-wise Gating · p. 5
  • The threshold uses the expected projection magnitude of concept-agnostic features onto a random subspace plus a margin, providing a data-free reference for incidental projections.Source: 1) Projection-Energy Layer-wise Gating · p. 5

2) Temporal Intervention Scaling.:

GRACE scales adapter intervention by denoising stage, using a concept-specific transition point alongside a layer-wise gate. Across the supplied evaluations, it reports target erasure with preservation across concepts and model settings.

2) Temporal Intervention Scaling.:

  • A temporal scaling factor adjusts adapter intervention by denoising stage, with a concept-specific transition point and a layer-wise gate controlling activation.Earlier interventions improve target erasure but disrupt neighboring concepts and overall quality; the figure marks a Pareto-optimal timestep for each concept.Source: 2) Temporal Intervention Scaling · p. 5, Figure 8 · p. 8

A. Experimental Setup

  • Experiments use SD v1.5 as the primary model and assess sensitive content, styles, IP characters, identities, and general-generation preservation with category-specific metrics.The setup also evaluates architectural generalizability on SD v2.1, SDXL, and FLUX.1 schnell.Source: A. Experimental Setup · p. 5

B. Main Results

  • GRACE reports strong fine-grained NSFW reductions, low target CLIP Scores with preserved MS-COCO fidelity in IP erasure, and stable preservation in artistic-style and identity erasure.The supplied evaluations include Fig. 3, Table I, and Fig. 4.Source: B. Main Results · p. 6, Figure 3 · p. 6, Table I · p. 6

C. Robustness, Scalability, and Generalization

  • Under Ring-A-Bell attacks, GRACE achieves strongest or competitive target suppression across categories, while Fig. 6 shows nudity suppression with coherent, natural-quality images.The caption contrasts GRACE with baselines that may fail to suppress unsafe content or exhibit semantic drift.Source: C. Robustness, Scalability, and… · p. 7, Table II · p. 7, Figure 6 · p. 7
  • Across different erasure scales, GRACE maintains competitive target erasure and strong retained-identity preservation, achieving the best overall trade-off as measured by the harmonic mean of erased-target and retained-identity top-1 GCD accuracy.Source: C. Robustness, Scalability, and… · p. 7
  • Across SD v2.1, SDXL, and FLUX.1, GRACE removes target concepts while preserving generation fidelity.The passage attributes this transfer across heterogeneous models to subspace-constrained intervention.Source: C. Robustness, Scalability, and… · p. 7, Figure 7 · p. 7
  • For erased IP concepts, GRACE preserves parent categories with substantially lower Parent FID than baselines, including natural dog images after Snoopy erasure.The table caption identifies lower Parent FID as better superclass preservation.Source: C. Robustness, Scalability, and… · p. 7, Table III · p. 8
  • GRACE requires substantially less optimization time than conventional fine-tuning baselines while maintaining strong erasure and preservation performance.The timing comparison uses 20 concepts, 5 models, and 60 prompts.Source: C. Robustness, Scalability, and… · p. 8, Table IV · p. 8

D. Ablation Studies

The ablations examine how GRACE’s components, intervention timing, regularization weight, and adapter placement affect erasure and preservation. The conclusion presents GRACE as combining localized subspace updates with adaptive intervention to balance those goals.

D. Ablation Studies

  • The full GRACE model achieves the best preservation while maintaining a 0.0% Nudity Rate.The ablation attributes preservation differences to semantic weighting, the off-subspace constraint, and Energy Gating.Source: D. Ablation Studies · p. 8, Table VI · p. 9
  • Earlier intervention (larger t) strengthens erasure but disrupts neighboring concepts and image structure more, whereas later intervention may under-suppress the target.The identified concept-specific transition point tstart is used to dynamically adjust intervention strength during denoising.Source: D. Ablation Studies · p. 8
  • The optimal subspace regularization weight varies by concept: 0.2 for Van Gogh, 0.5 for Snoopy, 0.6 for Nudity, and 0.7 for Ariana.The passage describes these values as reflecting concept-dependent sensitivity to the erasure-preservation trade-off.Source: D. Ablation Studies · p. 8
  • Mid + Up adapter placement reaches a 0.0% nude rate and improves COCO preservation over All Layers, while saving 15.6 MB of parameters.Its CLIP Score is 25.38 vs. 24.87 and its FID is 19.27 vs. 20.46; Table VII evaluates adapter placement across U-Net blocks.Source: D. Ablation Studies · p. 8, Table VII · p. 9

V. CONCLUSION

  • GRACE combines SA-PCA, off-subspace regularization Lsub, and Energy Gating to focus modifications on relevant semantic directions and regulate activation across denoising timesteps.The conclusion reports a balance between erasure effectiveness, image quality, and semantic preservation.Source: §V · p. 9

APPENDIX A. Pseudocode of the GRACE Framework

The appendix summarizes GRACE as a three-stage procedure: sensitive-subspace localization, subspace-constrained adapter optimization, and energy-driven gated inference. Its threshold analysis uses an isotropic benign-feature assumption to characterize random projection energy and motivate a data-free activation threshold.

APPENDIX A. Pseudocode of the GRACE Framework

  • The GRACE procedure is organized into semantic-aware sensitive-subspace localization, subspace-constrained adapter optimization, and energy-driven gated inference.The appendix assigns these stages to Algorithms 1, 2, and 3, respectively.Source: Appendix A · p. 11

B. Theoretical Analysis of the Activation Threshold τl

  • The threshold analysis models benign features as isotropic Gaussian vectors and examines their normalized projection energy in the sensitive subspace.Under this assumption, the squared normalized projection energy follows a Beta distribution with parameters K/2 and (C_l−K)/2.Source: B. Theoretical Analysis of the… · p. 11
  • For high-dimensional features, benign normalized projection energy concentrates around K/C_l, motivating τ_l = K/C_l + δ as a data-free activation threshold.The passage introduces δ > 0 to account for slight non-Gaussianity in real-world neural network features.Source: B. Theoretical Analysis of the… · p. 12

C. Evolution of Projection Energy

Projection energy decreases across the down-sampling, middle, and up-sampling blocks during GRACE optimization, indicating that target-aligned feature responses are progressively weakened.

C. Evolution of Projection Energy

  • Projection energy decreases across down-sampling, middle, and up-sampling blocks during optimization, indicating progressively weakened target-aligned feature responses.The passage defines projection energy as the normalized response of intermediate features within the sensitive subspace.Source: C. Evolution of Projection Energy · p. 12

D. Empirical Evidence of Low-Rank Semantic Subspaces

For Ariana Grande, SA-PCA selects compact layer-wise subspaces using a 99% cumulative-variance criterion. The reported ranks remain below 2% of each examined layer’s channel dimension.

C. Empirical Evidence of Low-Rank Semantic Subspaces

  • SA-PCA selects the smallest rank K retaining more than 99% of semantically weighted variance, avoiding a manually specified rank for each layer.The passage describes this as an adaptive rank-selection rule based on PCA’s low-rank approximation property.Source: D. Empirical Evidence of Low-Rank… · p. 12
  • For Ariana Grande, the retained subspace rank ranges from 6 to 12, averages 8.86, and occupies less than 2% of each examined layer’s channel dimension.The report covers selected cross-attention layers.Source: D. Empirical Evidence of Low-Rank… · p. 12
  • The reported compact subspaces represent dominant semantically weighted variations and provide a restricted space for localized concept intervention and incremental updates.Source: D. Empirical Evidence of Low-Rank… · p. 12

E. Ablation Study of Regularization Weight λ

The regularization weight λ trades target erasure against near-neighbor and general-content preservation, with concept-specific settings selected from that trade-off.

E. Ablation Study of Regularization Weight λ

F. Sensitivity Analysis of the Spatial-Gating Sharpness α

The analyses report a preferred spatial-gating sharpness, a smooth temporally selective activation setting, and qualitative erasure results across concepts and diffusion architectures.

F. Sensitivity Analysis of the Spatial-Gating Sharpness α

  • α = 40 achieves the best trade-off between smooth activation and gating selectivity; smaller values may under-distinguish responses, while excessively large values may cause abrupt adapter activation.The energy-gating margin is δ = 0.03 for all layers and target concepts.Source: F. Sensitivity Analysis of the… · p. 12

G. Analysis of Temporal Scaling Sharpness

  • With β = 1.0, temporal activation spans approximately six denoising steps, or 12% of the 50-step sampling trajectory.This setting provides smooth activation while retaining temporal selectivity.Source: G. Analysis of Temporal Scaling… · p. 12
  • Projection energy consistently decays across down-sampling, middle, and up-sampling U-Net blocks during training.The figure describes this trend as reflecting progressive adaptation and shifting of internal feature representations.Source: Figure 9 · p. 13
  • For Ariana Grande, 6–12 principal directions retain more than 99% of semantically weighted variance, each occupying less than 2% of its channel dimension.Source: Table VIII · p. 13
  • Fig. 10 compares Target CS and Target FID for erasure, and FID for near-neighbor and COCO preservation across λ settings.Star markers indicate the selected concept-specific λ.Source: Figure 10 · p. 13

H. Extended Qualitative Results

  • For artistic styles, GRACE achieves a better balance between target-style removal and preservation of neighboring styles and general generation quality.Table IX evaluates Van Gogh and Picasso erasure using CLIP Score and FID for erased concepts, neighboring concepts, and general generation.Source: H. Extended Qualitative Results · p. 13, Table IX · p. 14
  • GRACE removes celebrity-specific features as the erased set grows from 10 to 50 identities while preserving non-target identities and visual quality.The comparison illustrates suppression of Amanda Seyfried and preservation of Adele.Source: H. Extended Qualitative Results · p. 13, Figure 11 · p. 14
  • On SDXL, GRACE qualitatively suppresses nudity, Van Gogh style, and Snoopy features while maintaining natural generation quality.The figure presents individual erasure for these three concepts.Source: H. Extended Qualitative Results · p. 13, Figure 12 · p. 14
Loading 2609.12731v1…