PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaussian Core LoRA: Distribution-Aware Dynamic Adaptation for Broad Concept Erasure

Qinghui Gong, Xunlei Chen, Yu-Xuan Zhang, Hua Meng, Zhengchun Zhou

arXiv:2609.01433v1cs.CV

TL;DR

Broad concept erasure is difficult because static adapter updates cannot represent heterogeneous latent modes without risking incomplete suppression or interference with benign semantics. Gaussian Core LoRA fits a prompt-feature GMM and uses responsibilities to generate bounded, prompt-specific reconfigurations of one shared LoRA space. It reports improved erasure, preservation, adversarial robustness, scalability, and cross-architecture compatibility.

  • Problem

    Static LoRA assigns a fixed erasure direction to each broad concept, although its prompts may occupy multiple latent semantic modes requiring different local directions.

  • Method

    Gaussian Core LoRA models target prompts with a feature-space GMM and conditions a norm-bounded dynamic core on posterior responsibilities to reconfigure a shared LoRA rank space.

  • Results

    Gaussian Core LoRA improves erasure effectiveness, semantic preservation, and adaptability across unsafe prompt groups, adversarial prompts, multi-identity and multi-style erasure, SDXL, and FLUX.

  • Takeaways & Limitations

    A single lightweight adapter can provide prototype-adaptive erasure for broad concepts while retaining shared adaptation structures.

Abstract

from arXiv · show

Concept erasure aims to suppress unsafe, privacy-sensitive, or undesirable generations in text-to-image diffusion models while preserving benign semantics, visual quality, and deployment efficiency. Existing adapter-based methods, such as Low-Rank Adaptation (LoRA), typically freeze the diffusion backbone and learn lightweight parameter updates to steer generation away from target semantics. However, these methods usually assign a static semantic erasure direction to each target concept. This assumption is overly coarse for broad and complex target concepts, since a concept often contains multiple latent semantic prototypes involving different objects, scenes, or relations, and requires different local erasure directions. A single LoRA update averages these heterogeneous erasure demands, leading to under-erasure on difficult prototypes and over-editing of nearby benign semantics. To address this limitation, we propose Gaussian Core LoRA, a distribution-aware low-rank adaptation framework. It fits a Gaussian mixture model in the prompt feature space to estimate latent semantic prototypes within the target concept. During inference, each input prompt is projected into this feature space to compute its Gaussian posterior responsibilities, which condition the core generator to produce a prompt-specific, norm-bounded residual reconfiguration of the shared LoRA rank space. This enables prototype-adaptive erasure with a single lightweight adapter. Compared with the strongest baseline on each metric, Gaussian Core LoRA reduces average Attack Success Rate (ASR) by 7.95%, lowers COCO Fr'echet Inception Distance (FID) by 14.72%, and improves CLIP Score by 4.98%. Further experiments show robustness to adversarial prompts, scalability to multi-identity and multi-style erasure, and compatibility with SDXL and FLUX.

1 Introduction

Concept erasure must suppress undesirable generations while preserving benign semantics, quality, and efficiency. Gaussian Core LoRA addresses the heterogeneity of broad concepts by adapting erasure updates to latent-mode positions.

  • Concept erasure seeks to suppress undesirable concepts without retraining the entire diffusion model or sacrificing benign semantics, visual quality, and deployment efficiency.
  • Broad concepts contain multiple latent modes that can require different local erasure directions, making a single fixed LoRA update prone to incomplete coverage and semantic interference.
  • Gaussian Core LoRA models target prompts with a feature-space distribution and dynamically reconfigures a shared LoRA space according to each prompt’s position.
  • A Gaussian support gate screens activation, GMM responsibilities route prompts across latent modes, and a dynamic core produces prompt-conditioned rank-space updates.
  • 7.95% lower average ASR is reported relative to the strongest average baseline, alongside improved COCO FID and CLIP Score.
  • Gaussian Core LoRA uses one shared adapter to cover heterogeneous modes without separate mode-specific training, while supporting adversarial prompts and cross-architecture evaluation.

2 Related Work

Concept-erasure methods trade off deployment efficiency, direct behavioral modification, and scalability. Adapter-based approaches remain limited by predefined concept units and fixed updates for heterogeneous concepts.

  • Inference-time interventions are efficient but leave the underlying model unchanged and can be bypassed by prompt rewriting or adversarial attacks.
  • Parameter-efficient adaptation freezes the diffusion backbone while learning lightweight modules, offering a practical trade-off for concept erasure.
  • Existing adapter methods improve scalability through merging, sharing, or dynamic composition, but commonly remain organized around predefined concept units.
  • For broad concepts with heterogeneous latent modes, fixed adapter updates may average different editing demands, motivating distribution-aware prompt-conditioned adaptation.

3 Method

Gaussian Core LoRA treats broad concept erasure as distribution-aware parameter-efficient adaptation. It combines feature-space mode modeling, support gating, soft routing, and bounded dynamic reconfiguration with paired erasure and retention training.

  • Problem formulation: The method learns a lightweight adapter that suppresses target prompts while preserving non-target and semantically neighboring benign prompts.
  • Gaussian Core LoRA: At each injected attention layer, shared LoRA bases are combined with a prompt-conditioned core that reconfigures the shared rank space according to mode responsibilities.
  • Task-Feature Space and Mode Modeling: Prompt representations are projected into a compact feature space, where a diagonal-covariance GMM models latent modes using mixture weights, means, and covariances.
  • Task-Feature Space and Mode Modeling: Posterior responsibilities provide a soft assignment of each prompt among the target latent modes.
  • Gaussian Support Gate: Because responsibilities alone can confidently assign benign open-set prompts, a Mahalanobis-distance support gate screens whether prompts lie within the target distribution.
  • Gaussian Support Gate: The gate strengthens adapter updates inside target support and suppresses them outside all target supports, with Tg = 0.2 controlling boundary softness.
  • Gaussian Core LoRA: A shared generator uses mode responsibilities and layer embeddings to produce layer-aware low-rank core factors.
  • Gaussian Core LoRA: Core normalization and a strength parameter bound the residual reconfiguration, while paired safe prompts impose retention constraints during training.

4 Experiment

Experiments evaluate Gaussian Core LoRA on broad unsafe-concept erasure, adversarial robustness, scalability, storage, routing, and cross-architecture compatibility. Results consistently indicate strong erasure-preservation trade-offs with prompt-conditioned adaptation.

  • Evaluation setup: Gaussian Core LoRA is evaluated against established erasure methods on broad unsafe concepts, adversarial attacks, scalability, and diffusion architectures.Experiments include ESD, RECE, MACE, SAFREE, AdaVD, Prototype-Guided, and SuPLoRA in relevant settings.
  • Evaluation setup: The evaluation uses I2P unsafe prompts, independently optimized adversarial prompts, benign MS-COCO prompts, and multi-identity or multi-style targets.Adversarial attacks include Ring-A-Bell, P4D, and UnDiff; preservation uses FID and CLIP Score.
  • Broad unsafe-concept erasure: Gaussian Core LoRA achieves the lowest average ASR across seven I2P prompt groups and the best COCO FID and CLIP Score among erasure methods.It performs best on illegal activity, self-harm, sexual, and violence categories.
  • Adversarial robustness: Under adversarial attacks, Gaussian Core LoRA performs best on Ring-A-Bell and P4D and remains competitive on UnDiff.The prompt-conditioned core responds to each prompt’s latent-mode position rather than relying on a fixed erasure direction.
  • Routing and mechanism: Different target prompts activate distinct core patterns in the shared rank space, while editing gradients show higher within-mode than across-mode similarity.These findings support prompt-specific reconfiguration and editing-relevant Gaussian modes.
  • Ablation: GMM-conditioned dynamic cores outperform standard LoRA, soft routing outperforms hard routing, and non-diagonal cores outperform diagonal scaling.The ablation attributes these gains to mixed-mode information preservation and cross-channel rank-space reconfiguration.
  • Scalability and storage: Gaussian Core LoRA uses a single shared adapter under 50-target erasure, avoiding storage growth from concept-specific adapter components.Figure 8 compares its storage with MACE and SuPLoRA.

5 Conclusion

The conclusion frames concept erasure as a distributional problem and presents Gaussian Core LoRA as a shared-adapter solution for prototype-adaptive control. It reports gains across effectiveness, preservation, adaptability, and architectures while identifying richer prototype modeling as future work.

  • 5 Conclusion: Broad target concepts can contain multiple semantic prototypes requiring different erasure directions, making separate adapters costly for scalable erasure.The paper therefore models target concepts through their prompt-feature distributions.
  • 5 Conclusion: Gaussian Core LoRA fits a Gaussian mixture and uses posterior responsibilities to condition a dynamic core that reconfigures shared LoRA rank space.This provides prototype-adaptive erasure within a single lightweight adapter.
  • 5 Conclusion: Experiments show consistent gains in erasure effectiveness, semantic preservation, and adaptability across diffusion architectures.The conclusion characterizes distribution-aware LoRA as promising for scalable concept control.
  • 5 Conclusion: Gaussian mixtures provide a compact approximation, but open-ended prompts may contain finer-grained or entangled semantic patterns.Future work targets richer prototype discovery and boundary characterization for open-set erasure.

A Theoretical Analysis

Gaussian Core LoRA is analyzed as a prompt-conditioned reconfiguration of a shared LoRA rank space. The analysis explains how dynamic cores address mode averaging when different latent modes require different erasure directions.

  • Fixed LoRA Coupling: Standard LoRA uses a fixed identity coupling between input and output rank channels for every prompt.
  • Prompt-Conditioned Dynamic Core: Gaussian Core LoRA expresses its update as a gated base LoRA term plus a gated prompt-conditioned residual correction.The residual form is ΔW(z) = g(z)BA + g(z)BδC(z)A.
  • Prompt-Conditioned Dynamic Core: The dynamic residual is an exact first-order reconfiguration of the standard LoRA update within the shared rank space.Because F(C) = BCA is linear in C, DF|Ir[δC] = BδCA.
  • Mode Averaging: A single static core incurs mode-averaging error when different latent modes require different average cores.Dynamic routing is useful when between-mode variation is non-negligible.

Gate and Routing Error Decomposition

The error decomposition separates whether the adapter activates from how it reconfigures the shared rank space. This motivates combining Gaussian support screening with soft responsibility-based routing for smoother prompt-conditioned adaptation.

  • Gate and Routing Error Decomposition: The decomposition separates support-screening error from conditional routing error.The first controls adapter activation, while the second controls rank-space reconfiguration.
  • Gate and Routing Error Decomposition: Normalized GMM responsibilities alone cannot suppress false activation on benign prompts.They affect only the routing term, not support screening.
  • Soft Routing: Soft routing uses a responsibility-weighted router as an idealized analysis of posterior calibration.The implemented core generator generalizes this linear form with a bounded nonlinear mapping.
  • Shared Subspace: The residual core has bounded rank, while all prompt-conditioned updates remain inside the shared LoRA input and output subspaces.Only rank-space coordinates are reconfigured across prompts.
  • Benign Perturbation: Benign-prompt preservation depends on gate activation, dynamic core strength, and the operator norms of the shared LoRA bases.
  • Local Stability: Soft routing avoids discontinuous changes near GMM boundaries but does not guarantee robustness to large adversarial prompt transformations.

Implementation Details

The implementation freezes the diffusion backbone and trains a lightweight adapter with Gaussian modeling, shared LoRA bases, and a dynamic core. Component count, residual rank, and support percentile are selected according to concept complexity and preservation–erasure trade-offs.

  • Training Setup: The backbone and text encoder are frozen while shared LoRA bases, the dynamic core generator, and layer embeddings are trained.The adapter is inserted into UNet attention projection layers.
  • Prompt Construction: Target prompts are constructed from concept-specific subjects, modifiers, and templates, with paired safe prompts preserving scene context.Sensitive-1K contains 1,000 sampled training prompts after removing invalid and duplicate combinations.
  • Prompt Construction: Template diversity supports evaluation of style erasure while preserving non-target semantic content.
  • Distribution Modeling: The final Gaussian component count is selected with BIC rather than fixed across concepts.This allows concentrated concepts to use fewer modes and diverse concepts to use more.
  • Residual Core Rank: Residual core rank is matched to the selected component count and normalized GMM mode entropy, giving heterogeneous tasks more adaptive capacity.
  • Scalability: The shared adapter scales to joint erasure of up to 50 identities or styles while retaining a compact shared rank space.
  • Hyperparameter Sensitivity: Table 8 reports that BIC-selected K = 5 achieves the lowest ASR, while K = 7 degrades preservation without further improving erasure.Increasing residual rank improves Violence erasure with a mild preservation trade-off.
  • Hyperparameter Sensitivity: β = 0.95 gives the best erasure-preservation trade-off, whereas larger support percentiles provide limited ASR gains but degrade COCO preservation.Compact concepts use β = 0.90; medium- and high-dispersion concepts generally use β = 0.95.

C Additional Distribution-Aware Analysis and Generalization Results

Feature visualizations show that large-scale identity and style targets share semantic structures while retaining multiple local modes. This supports a shared low-rank space with prompt-specific reconfiguration rather than independent adapters or a fixed update.

  • Distribution-Aware Analysis: Fifty target concepts overlap in CLIP prompt-embedding space and form several high-density regions.
  • Generalization Results: The overlap supports a shared low-rank erasure space instead of independent adapters or isolated rank blocks for each target.
  • Distribution-Aware Analysis: Multiple local modes remain, so fixed LoRA updates may average erasure directions and disturb retained concepts.Gaussian Core LoRA uses GMM responsibilities to reconfigure the shared rank space for each prompt.

Additional Results on 50-Style Erasure

The 50-style setting tests whether Gaussian Core LoRA scales beyond manually defined concept boundaries while preserving retained styles and general COCO semantics. Its distribution-aware routing addresses distinct style modes without relying on a static, mode-averaged update.

  • Evaluation Setup: The evaluation erases up to 50 artistic styles with one adapter and measures suppression on target styles alongside preservation on retained styles and COCO prompts.The setup uses 50 artistic styles for erasure and another 50 for retention.
  • Motivation: Distinct artists and style families form different visual modes, making a static LoRA update prone to mode averaging.The style modes involve appearance patterns such as color distribution, brushstroke texture, composition preference, and local rendering statistics.
  • Quantitative and Qualitative Results: As erased styles increase from 1 to 50, Gaussian Core LoRA maintains stronger target-style suppression while better preserving retained-style and COCO quality than static LoRA baselines.Target-style suppression is assessed with lower Target Style CLIP Score and higher Target Style FID; preservation uses higher CLIP Score and lower FID.
  • Mechanism: The dynamic core adaptively reconfigures the shared LoRA rank space according to each prompt’s latent style mode rather than simply increasing overall erasure strength.The broader framework uses a Gaussian support gate and GMM responsibilities for soft latent-mode routing.
  • Architecture Adaptation: For FLUX, the same gate–router–core formulation is retained while adapter injection locations and the backbone-native prediction parameterization are changed.The adapters can be inserted into both visual-stream and context-stream projections in multimodal Transformer blocks.

Cross-Architecture Qualitative Results on Sexual Erasure

Gaussian Core LoRA is evaluated for Sexual erasure on SDXL and FLUX, two architectures that differ from Stable Diffusion v1.5. Across both, it suppresses target cues while preserving prompt-relevant non-target content and scene structure.

  • Evaluation Scope: SDXL and FLUX provide cross-architecture tests because they differ from Stable Diffusion v1.5 in scale, denoising architecture, and generation behavior.The evaluation examines whether the method generalizes beyond its original backbone.
  • Qualitative Results: On both SDXL and FLUX, Gaussian Core LoRA more consistently suppresses Sexual-related visual cues while preserving scene layout, object context, and overall image structure than baseline methods.The qualitative results are reported in Figures 12 and 13.
  • Implication: The results suggest distribution-aware dynamic adaptation is not tied to one diffusion backbone and maintains a favorable balance between erasure effectiveness and semantic preservation.The reported intervention is localized in the shared LoRA rank space rather than broadly over-suppressing generation.
Loading 2609.01433v1…