PaperlayerView exact revision on arXiv

Source-linked AI summary

RASteer: Retain-Aware Activation Steering for Concept Erasure in Diffusion Models

Yongliang Wu, Haori Lu, Yulun Wu, Jinqi Luo, Xingyu Zhu, Yaoyao Liu

arXiv:2610.01969v1cs.CV

TL;DR

Activation steering can suppress retained concepts when target and retain representations overlap, complicating concept erasure that must preserve other content. RASteer refines the erasure direction using a retain subspace and overlap-adaptive removal, and achieves a better erasure–preservation trade-off across evaluated tasks and backbones.

  • Problem

    Activation-steering erasure directions can contain shared components that retained concepts rely on, risking collateral suppression when erasing a target.

    Source: §1 · p. 2

  • Method

    RASteer builds a retain subspace, removes aligned erasure-direction components with ROS, and uses overlap-adaptive calibration to set removal strength.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across multiple backbones and benchmarks, RASteer achieves the best erasure–preservation trade-off among compared methods for unsafe-content, instance, and artistic-style erasure.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    RASteer leaves model weights unchanged while balancing target suppression with preservation of retained concepts and general generation.

    Source: §1 · p. 2, §5 · p. 10

    Experiments focus on Stable Diffusion; RASteer relies on a predefined retain set, and high target–retain overlap limits the direction available for erasure.

    Source: §5 · p. 10

Abstract

from arXiv · show

Concept erasure aims to remove a target concept, such as a copyrighted style, a recognizable character, or unsafe content, from a pretrained text-to-image diffusion model while preserving its ability to generate other content. Existing activation steering methods build an erasure direction mainly from the target concept and adjust model activations along it at inference time. However, target and retained concepts often overlap in the model's representation space, so this direction also contains shared components that retained concepts rely on. Steering directly along this direction can therefore suppress retained concepts and harm the generation of non-target content. To address this issue, we propose Retain-aware Activation Steering (RASteer), a training-free method. RASteer first builds a retain subspace from the concepts to preserve. Retain-Orthogonal Steering (ROS) then removes components aligned with this subspace from the erasure direction, making steering more specific to the target. Since fully removing the shared components can weaken erasure, we further introduce Overlap-Adaptive Calibration (OAC). At each layer and denoising step, OAC uses the overlap between the erasure direction and the retain subspace to control how much of each shared component is removed, balancing target erasure and concept preservation. Experiments on unsafe-content, instance, and artistic-style erasure across multiple backbones and benchmarks show that RASteer matches or outperforms the activation steering and weight editing baselines we evaluate, achieving a better balance between erasure and preservation.

1 INTRODUCTION

RASteer addresses shared target–retain representations that make activation steering damage retained concepts or weaken erasure. Across unsafe-content, instance, and artistic-style tasks, it achieves the best erasure–preservation trade-off among the compared methods.

1 INTRODUCTION

  • Concept erasure edits a released model to prevent target generation while preserving other content, avoiding costly retraining and bypassable prompt- or output-level filters.Targets include copyrighted styles, recognizable characters, and unsafe images.Source: §1 · p. 1
  • Weight editing changes model parameters, whereas activation steering adjusts intermediate activations at inference while leaving weights fixed.Activation steering edits can be enabled, disabled, or combined.Source: §1 · pp. 1–2
  • Shared components can make original-direction steering damage retained concepts, while full projection can leave the target recognizable.For the default gate threshold, it triggers on 65% of retained activations and misses 21% of target activations.Source: §1 · p. 2, Figure 1 · p. 2
  • RASteer uses ROS to reduce erasure-direction components aligned with a retain subspace and OAC to adapt removal strength to overlap.OAC adjusts removal at each layer and denoising step to balance target erasure and concept preservation.Source: §1 · p. 2
  • Across unsafe-content, instance, and artistic-style erasure, RASteer achieves the best erasure–preservation trade-off among compared methods.Evaluations span multiple backbones and benchmarks.Source: §1 · p. 2

2 RELATED WORK

Related work covers parameter-editing approaches, inference-time steering, and continual or sequential unlearning. RASteer retains inference-time editing while incorporating the concepts to preserve into its steering direction.

Model Editing for Concept Erasure

Inference-Time Steering

  • Existing inference-time steering methods leave weights untouched but do not use retained concepts as input, so shared target directions can suppress retained concepts.RASteer builds a retain subspace and sets layerwise steering strength by overlap.Source: §2 · p. 3

Sequential and Continual Unlearning

  • Sequential unlearning must accommodate new forgetting requests while retaining other knowledge; repeated parameter editing can accumulate interference.RASteer instead edits at inference time, allowing erasure constraints to be introduced, removed, or combined without parameter drift.Source: §2 · p. 3

3 METHOD

RASteer extracts erasure and retain directions from paired prompts, then adjusts cross-attention steering to reduce shared components while preserving target erasure. Its ROS and OAC modules use retain-subspace geometry and layer- and step-specific overlap to calibrate the steering direction.

3.1 PRELIMINARIES

  • At each cross-attention layer and denoising step, concept directions come from normalized differences between mean activations for paired prompts with and without the concept.RASteer reduces retain-shared components of the target direction before computing the gate and applying the update.Source: §3.1 · p. 3

3 METHOD

  • RASteer uses paired prompts to extract erasure and retain directions, then calibrates a direction injected into cross-attention during denoising.ROS decomposes the erasure direction into retain-aligned and retain-orthogonal components; OAC adapts steering strength according to their overlap.Source: §3 · p. 3, Figure 2 · p. 4

3.2 RETAIN-AWARE ACTIVATION STEERING

  • ROS builds an orthonormal retain subspace and separates the erasure direction into shared and target-specific components; the latter is orthogonal to retained activations in that subspace.Removing the shared component fully can also weaken erasure when the target has a component along a retain direction, so RASteer allows controllable removal.Source: §3.2 · p. 4
  • The Figure 3 ablation caption reports that removing ROS damages retained concepts, removing OAC collapses erasure, and the full method maintains both.The caption notes that measured overlap varies widely across layers and denoising steps.Source: Figure 3 · p. 5
  • OAC sets the removal fraction from overlap o, the erasure direction’s energy fraction in the retain subspace, separately at each layer and denoising step.Its default rule is λi = 1 − o: it removes most of a small shared component at low overlap and less at high overlap.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate RASteer on unsafe-content, instance, and artistic-style erasure alongside retained- and general-content preservation. Results indicate a favorable erasure–preservation balance, with ablations examining the effects of components, layer scope, and steering strength.

4.1 SETUP

  • Evaluation covers SD-v1.4, SD-v1.5, SD-v2.1, and SDXL, using unsafe, instance/style, and COCO tasks with joint erasure and preservation metrics.RASteer updates no weights: steering vectors are precomputed offline and applied at every cross-attention layer using OAC without per-concept tuning.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

  • Across unsafe-content, instance, and style tasks, RASteer offers a favorable erasure–preservation balance, reducing unsafe detections while keeping retained and open-domain generations close to the unedited model.The comparison spans unsafe-content, instance, and artistic-style erasure on SD-v1.4; Table 1 reports unsafe detections.Source: Table 1 · p. 6, §4.2 · p. 6
  • RASteer leaves COCO image–caption alignment essentially unchanged and has lower paired perceptual drift than several methods that meaningfully erase nudity.For instances and styles, it also lowers erased-target accuracy than CASteer while keeping retained concepts close to the unedited reference in recognition and FID.Source: Table 2 · p. 6, §4.2 · pp. 6–8, Table 3 · p. 7

4.3 ABLATION STUDY

  • Low+Mid matches All in target accuracy, while retained FID is lower for Low+Mid; the authors retain All as the default across backbones.Figure 4 compares five scopes on target CLIP accuracy, retained-concept accuracy, and retained FID.Source: Figure 4 · p. 8, §4.3 · p. 9
  • Stronger β lowers instance and style target accuracy but increases retained and COCO drift, while unsafe detections do not improve consistently.The default β = 2 on SD-v1.4 keeps retained-concept accuracy above 98% while limiting drift relative to larger strengths.Source: Figure 5 · p. 8, §4.3 · p. 9
  • Removing ROS raises retained-concept FID, while omitting OAC applies full retain projection and leaves erased outputs closer to target semantics and distribution.Van Gogh accuracy is an exception to the latter trend, so the ablation interprets erasure using alignment and distributional change together.Source: Table 4 · p. 7, §4.3 · p. 8

4.4 QUALITATIVE RESULTS

  • On SD-v2.1, RASteer removes Snoopy in one scene but leaves face-like remnants and texture artifacts in other erased rows, while better preserving Mickey than CASteer.Figure 6 compares erased Snoopy and retained Mickey across the unedited model, baselines, and RASteer.Source: Figure 6 · p. 9, §4.4 · p. 9

5 CONCLUSION

RASteer uses retain-aware activation steering to balance target suppression with preservation while leaving model weights unchanged. Its scope is limited by the Stable Diffusion experiments, predefined retain sets, and overlap between target and retain representations.

5 CONCLUSION

  • RASteer leaves model weights unchanged, using ROS to remove retain-aligned components and OAC to adjust removal strength according to target–retain overlap.The method is training-free.Source: §5 · p. 10
  • Across multiple concept-erasure benchmarks and four Stable Diffusion backbones, RASteer balances target suppression with retained-concept preservation and general generation.Source: §5 · p. 10
  • Experiments focus on Stable Diffusion; RASteer also relies on predefined retain sets, and substantial target–retain overlap limits the direction available for erasure.The authors identify diffusion transformers and FLUX, adaptive retain-set construction, and iterative steering as future work.Source: §5 · p. 10

A.1 ADDITIONAL IMPLEMENTATION DETAILS

The implementation evaluates four Stable Diffusion backbones with released pipeline defaults and a fixed steering strength per backbone. It applies clipping at every denoising step.

A.1 ADDITIONAL IMPLEMENTATION DETAILS

  • SD-v1.4, SD-v1.5, and SD-v2.1 use 50 denoising steps, while SDXL uses 30.Source: §A.1 · p. 14
  • Steering strength is fixed per backbone rather than tuned per concept: β = 2 for SD-v1.4 and SD-v1.5, and β = 3 for SD-v2.1 and SDXL.Source: §A.1 · p. 14
  • All models use half precision and released pipeline defaults, with the clipping rule in Eq. 1 applied at every denoising step.Figure 5 reports sensitivity to β on SD-v1.4.Source: §A.1 · p. 14

A.2 CROSS-MODEL VALIDATION

Across SD-v1.5, SD-v2.1, and SDXL, the erase–preserve pattern from the main results transfers, with gains narrower on SD-v2.1. The evaluation re-estimates steering vectors and method-specific weights for each backbone.

A.2 CROSS-MODEL VALIDATION

  • The full six-setting instance and style protocol is repeated on SD-v1.5, SD-v2.1, and SDXL, with steering vectors and method-specific weights re-estimated on each backbone.Source: §A.2 · p. 14
  • Relative to CASteer, RASteer generally lowers target accuracy while maintaining higher retained accuracy and lower retained FID on instances and styles across the tested backbones.The gains are largest on SD-1.x and narrower on SD-v2.1, where targets are harder to suppress.Source: §A.2 · pp. 14–15
  • On SDXL, retain-aware construction transfers to dual text encoders and a higher-resolution denoiser without changing the projection rule.Methods whose released implementations do not support SDXL are omitted rather than replaced by custom variants.Source: §A.2 · p. 15

A.3 ADDITIONAL QUALITATIVE COMPARISONS

The additional qualitative comparisons assess both target removal and retained-content stability across backbones. They distinguish semantic removal with coherent replacements from visual degradation and collateral changes.

A.3 ADDITIONAL QUALITATIVE COMPARISONS

  • Each comparison uses the same prompt and seed across columns, which show the unedited backbone, supported baselines, and RASteer.The grids include three erased instances, three retained instances, three erased styles, and three retained styles.Source: §A.3 · p. 15
  • The grids distinguish semantic removal from visual degradation: recognizable target cues indicate under-erasure, while repeated textures, collapsed structure, or empty scenes indicate over-erasure.Retained rows are compared with the unedited output for identity, composition, color, and texture.Source: §A.3 · pp. 15–16
  • Across SD-v1.4, SD-v1.5, SD-v2.1, and SDXL, RASteer generally removes targets while producing structured alternatives; residual cues are more common on SD-v2.1, while retained concepts remain comparatively stable.On SDXL, target removal is strong and retained Pikachu and artistic-style rows show the preservation advantage, though erased-style rows can retain texture changes.Source: §A.3 · p. 16
  • Figures 7–10 provide additional instance and style comparisons across Stable Diffusion v1.4, v1.5, v2.1, and SDXL.Each group of three rows shows representative prompts from the indicated erased or retained set.Source: Figure 7 · p. 17, Figure 8 · p. 18, Figure 9 · p. 19

Related papers

Loading 2610.01969v1…