PaperlayerView exact revision on arXiv

Source-linked AI summary

GAPS: Dimension-Level Gates for Conditional Activation Steering

Moghis Fereidouni, Muhammad Umair Haider, Hassan Sajjad, A. B. Siddique

arXiv:2609.01878v1cs.CL

TL;DR

Existing conditional activation-steering methods decide when to intervene but still apply dense interventions across all hidden dimensions, even when neurons lack concept information or are already in the desired regime. GAPS adds training-free neuron-level conditioning through separability and posterior gates. Across toxicity mitigation and concept removal, it matches or improves token-level Pareto fronts, including a reduction in Gemma-3 toxicity from 6.52% to 0.48% at matched capability cost.

  • Problem

    Conditional steering improves when interventions occur, but active methods still perturb every hidden dimension regardless of concept information or current activation regime.

  • Method

    GAPS combines a static AUROC-based separability gate with a Gaussian posterior gate that steers neurons only when their activations favor the undesired concept.

  • Results

    Across two models, two token-level methods, and two tasks, GAPS matches or improves Pareto fronts; Gemma-3 toxicity falls from 6.52% to 0.48% versus 3.52% for DSAS alone.

  • Takeaways & Limitations

    Dimension-level conditioning complements token-level conditioning and can improve behavior-capability trade-offs with O(D) per-token overhead, with most gains attributed to the posterior gate.

  • Takeaways & Limitations

    The posterior gate assumes equal source and target priors, and systematic study of prior selection is left for future work.

Abstract

from arXiv · show

Activation steering suppresses undesired behaviors in language models by adding a steering vector to the hidden state during generation. Recent conditional methods such as CAST and DSAS improve the behavior-capability trade-off by deciding when to intervene, but once active, they apply the full dense vector to all hidden dimensions, regardless of whether a neuron carries concept information or already lies in the desired regime. We introduce dimension-level conditioning as a complementary axis of selectivity that also decides which neurons to intervene on. Our method, GAPS (Gated Activation steering via Posterior and Separability), combines two training-free gates: a static separability gate that restricts steering to neurons with statistically reliable concept information (via AUROC), and a dynamic posterior gate that steers a neuron only when its current activation is better explained by the undesired concept under a Gaussian model. The gates add O(D) overhead per token, and they plug into existing conditional methods. On toxicity mitigation (RealToxicityPrompts) and concept removal (OneSeC) with Gemma-3 (4B) and Qwen-3 (1.7B), GAPS consistently matches or improves the Pareto front of its token-level counterparts; under a fixed capability budget, DSAS+GAPS reduces Gemma-3's toxicity rate from 6.52% to 0.48%, versus 3.52% for DSAS alone. Ablations attribute most of the gain to the posterior gate.

1 Introduction

Activation steering reduces undesired behavior without retraining, but token-level methods still perturb every hidden dimension when active. GAPS adds neuron-level selectivity through static separability and dynamic posterior gates, improving the behavior-capability trade-off across evaluated tasks and models.

  • Motivation: Activation steering adds a behavior-direction vector during generation, avoiding gradient updates but potentially degrading fluency and downstream accuracy.Its effectiveness is evaluated through the trade-off between behavior change and capability loss.
  • Motivation: Token-level methods such as CAST and DSAS decide when to intervene, but active interventions still apply the full dense vector to all D neurons.This leaves neuron-level selectivity unresolved.
  • GAPS: GAPS adds dimension-level conditioning by deciding which neurons to steer in addition to deciding when to intervene.The approach targets only the hidden-state changes required by the steering objective.
  • GAPS: The static separability gate retains neurons with reliable concept information, while the dynamic posterior gate intervenes when activation is more likely under the undesired concept.Separability uses AUROC, and the posterior gate models concept-conditional activations as Gaussians.
  • GAPS: GAPS uses O(D) elementwise operations per token, adds only constant-factor overhead, and strictly generalizes existing conditional methods.The required statistics come from the same contrastive activations used to construct the steering vector.
  • Results: Across two models, two token-level methods, and two tasks, GAPS matches or improves the Pareto front of its token-level counterpart.Ablations attribute most of the gain to the posterior gate.
  • Results: Under a fixed capability budget, DSAS+GAPS lowers Gemma-3 toxicity from 6.52% to 0.48%, versus 3.52% for DSAS alone.The evaluation covers toxicity mitigation and concept removal with Gemma-3 and Qwen-3.

2 Preliminaries

Activation steering operates on hidden-state neurons and can be conditioned at the token level. CAST uses a hard gate, whereas DSAS uses a probe-derived continuous gate; GAPS further organizes neuron-level gating.

  • Definitions: A neuron is one coordinate of a transformer hidden state, with h ∈ R^D and h_t,d denoting neuron d at generation step t.The hidden state is represented as h, and each neuron corresponds to one coordinate.
  • Activation steering: Standard activation steering adds αv to the hidden state, where α controls intervention strength and v is computed from target and source activations.The standard direction is the mean difference between target and source concept activations.
  • Token-level conditioning: Token-level conditional steering modulates intervention strength with a scalar gate g(h_t) ∈ [0, 1] instead of applying it at every generation step.This gate determines when or how strongly the intervention is applied.
  • Token-level methods: CAST uses a thresholded hard binary gate, while DSAS uses a probe output as a continuous gate that scales intervention strength.Both methods condition steering on detected undesired behavior.
  • Dimension-level conditioning: GAPS combines a static separability gate and a dynamic posterior gate whose product forms the final per-neuron steering mask.The static gate is computed offline, while the posterior gate responds to current activation.

3 Neuron-Level Conditional Steering

GAPS adds neuron-level conditioning to token-level steering by selecting concept-informative neurons statically and intervening dynamically only when their activations indicate the undesired concept.

  • Motivation: Token-level methods still apply the full steering vector to all D neurons whenever their gate is active.This can perturb neurons lacking concept information or neurons already in the desired regime.
  • Static separability gate: The separability gate retains neurons whose activations reliably distinguish the two concepts using AUROC-based statistical testing.Its threshold adapts to contrastive sample size and tests deviations on either side of chance.
  • Dynamic posterior gate: The posterior gate steers a retained neuron only when its current activation is more likely under the source than target concept.Concept-conditional activations are modeled with Gaussian distributions estimated from contrastive samples.
  • Combined gating: Combining the token-level gate with both dimension-level gates yields GAPS, which strictly generalizes CAST, DSAS, and unconditional steering.Setting both dimension-level gates to one recovers token-level conditional steering; setting the token-level gate to one as well recovers unconditional steering.
  • Combined gating: The combined dimension-level gates are complementary: separability identifies neurons that can encode the behavior, while posterior gating identifies when they currently express it.The gates are evaluated statically or per generation step according to their roles.

4 Experiments and Results

Experiments evaluate GAPS across toxicity mitigation and concept removal on two language models, finding that dimension-level gates generally improve or preserve behavior-capability trade-offs and that posterior gating drives most gains.

  • Experimental setup: The evaluation covers toxicity mitigation and seven-concept OneSeC removal on Gemma-3 4B and Qwen-3 1.7B, using CAST and DSAS baselines.Trade-offs are traced across intervention strength using Wikipedia perplexity, toxicity or concept scores, and MMLU capability measures.
  • Toxicity mitigation: GAPS consistently shifts toxicity-mitigation Pareto fronts toward lower toxicity and lower Wikipedia perplexity for both models and token-level baselines.Figure 2 compares unconditional steering, CAST or DSAS, and their +GAPS variants over steering strength α.
  • Concept removal: GAPS matches or improves the concept-removal Pareto front on both models, with the largest gains on Gemma-3 (4B).The trade-off averages normalized concept scores across seven OneSeC concepts.
  • Fixed-budget ablations: 6.52% to 0.48%: DSAS+GAPS lowers Gemma-3 toxicity under a fixed capability budget, compared with 3.52% for DSAS alone.The budget constrains perplexity to at most 5% above and MMLU correct-option probability to at most 3% below the unsteered model.
  • Dimension-level gate analysis: The posterior gate outperforms the separability gate in 7 of 8 settings and contributes to the best-performing variant in every setting.In 7 of 8 settings, posterior-gated variants occupy both top-two positions; the exception is Gemma-3 with CAST on toxicity.
  • Dimension-level gate analysis: Posterior gating also outperforms the range gate across all 8 settings, while GAPS outperforms the separability-plus-range combination in every case.The range gate uses only the undesired-concept distribution, whereas posterior gating compares undesired and desired explanations.

5 Related Works

Related activation-steering work distinguishes unconditional dense interventions from token-level methods that decide when to intervene; GAPS adds neuron-level selectivity to this progression.

  • Activation steering: Activation steering typically adds a concept direction during inference without gradient updates, using contrastive examples or probes, but conventional methods intervene unconditionally and densely.The intervention is applied at every generation step and across all hidden dimensions.
  • When to steer: CAST and DSAS introduce token-level selectivity by deciding when to intervene based on hidden-state alignment or a trained probe.Related methods likewise use probes or controllers to trigger, scale, or calibrate interventions.

6 Conclusion

GAPS adds dimension-level selectivity to activation steering by deciding which neurons to steer at each generation step, complementing token-level decisions about when to intervene. Across two models, two conditional methods, and two tasks, it matches or improves token-level Pareto fronts, with most gains attributed to the posterior gate.

  • GAPS introduces dimension-level conditioning that decides which hidden-state neurons to steer beyond deciding when to intervene.
  • The static separability gate targets neurons carrying reliable concept information, while the dynamic posterior gate intervenes when activations better match the undesired concept.
  • GAPS is training-free, adds negligible overhead, and strictly generalizes existing conditional methods.
  • Across two models, two token-level methods, and two tasks, GAPS matches or improves the Pareto front of its token-level counterparts.
  • Most of the gain is attributed to the posterior gate, indicating that effective steering requires identifying when concept-related neurons are active.

Limitations

The posterior gate assumes equal source and target priors, although deployment users may adjust them based on expected context; systematic prior selection remains future work. The concept-classification evaluation uses an LLM-based binary classifier with WordNet definitions for seven OneSeC concepts.

  • The posterior gate assumes equal priors over the source and target concepts.
  • Practitioners expecting mostly benign contexts could lower the source-concept prior to make gating more conservative, while adversarial settings may warrant the opposite.
  • The paper leaves systematic study of prior selection to future work.
  • The posterior gate is evaluated in log space for numerical stability using Gaussian concept-conditional densities.
  • The posterior gate has O(D) per-token cost, using a fixed number of elementwise operations per neuron and no additional forward passes.
  • Concept-reference evaluation uses an LLM-based yes-or-no classifier supplied with WordNet definitions for seven OneSeC concepts.

C MMLU Results - Toxicity

On Gemma-3, dimension-gated methods retain more MMLU capability than token-level counterparts at matched toxicity, while Qwen-3 shows comparable performance across most of the toxicity range. Figure 4 compares capability–toxicity trade-offs as steering strength varies.

  • Figure 4 plots MMLU correct-option probability against toxicity score while sweeping steering strength α.
  • On Gemma-3 (4B), +GAPS variants retain higher MMLU than token-level counterparts at matched toxicity levels.
  • On Gemma-3 (4B), GAPS variants remain closer to the unsteered baseline across most α values and degrade more gracefully at strong interventions.
  • On Qwen-3 (1.7B), all methods perform comparably across most of the toxicity range.
  • At large α, baseline performance drops steeply, whereas GAPS variants degrade more slowly at high intervention strengths.

D MMLU Results - OneSeC

For concept removal, GAPS adds little capability benefit on Gemma-3 because token-level methods already preserve MMLU, but improves matched-removal MMLU on Qwen-3 at low to moderate intervention strengths. The layer-18 toxicity experiment indicates that gains are robust to intervention-layer choice.

  • C MMLU Results - OneSeC: Figure 5 compares MMLU correct-option probability with average normalized concept score across seven OneSeC concepts as α varies.
  • C MMLU Results - OneSeC: On Gemma-3 (4B), +GAPS variants perform nearly identically to their token-level counterparts for concept removal.
  • C MMLU Results - OneSeC: At the strongest interventions, CAST+GAPS also declines slightly and cannot fully recover unconditional-steering performance because it inherits degradation from the underlying CAST gate.
  • D MMLU Results - OneSeC: With interventions at layer 18, dimension-level gates shift the toxicity-mitigation Pareto frontier toward the bottom-left for both models and steering families.

E Robustness to the Intervention Layer

Repeating toxicity mitigation at layer 18 reproduces the layer-15 ordering: conditional steering improves over unconditional steering, and GAPS shifts the trade-off further toward lower perplexity at matched toxicity.

  • Robustness to the Intervention Layer: The robustness experiment applies all interventions at layer 18 instead of layer 15 while keeping the protocol components recomputed for that layer.The evaluation uses both models and both token-level methods.
  • Robustness to the Intervention Layer: At layer 18, CAST and DSAS again improve substantially over unconditional steering, while adding GAPS shifts the Pareto frontier further toward the bottom-left.The layer-18 setup recomputes contrastive data, steering vectors, gate statistics, and evaluation components at the new layer.
  • Robustness to the Intervention Layer: At matched toxicity levels, GAPS achieves equal or lower perplexity than token-level counterparts across both models and both token-level methods.This pattern holds for Gemma-3 (4B) and Qwen-3 (1.7B), with CAST and DSAS.
  • Robustness to the Intervention Layer: The ordering unconditional < token-level < token-level+GAPS also holds at a different intervention layer, indicating gains are not specific to one site.The gates adapt to the selected layer through layer-specific contrastive activations, separability statistics, and concept-conditional Gaussians.

G Separability Threshold

The separability threshold τz controls how stringently GAPS retains statistically informative neurons. Sensitivity results identify a band around τz ∈[7, 8] as best for toxicity mitigation, with degradation below and above it.

  • Threshold Sensitivity: The statistical calibration floor is approximately τz ≈5.0 for Qwen-3 and 5.1 for Gemma-3 under Bonferroni-corrected family-wise level α0 = 10^-3.The admissible range is bounded below by calibration across all hidden dimensions.
  • Threshold Sensitivity: τz ∈[7, 8] minimizes toxicity in the Gemma-3 (4B) DSAS+GAPS sensitivity sweep under both fixed-budget and budget-averaged protocols.The sweep evaluates toxicity, Wikipedia perplexity, and MMLU across intervention strengths and 15 capability budgets.
  • Threshold Sensitivity: At τz = 6, fixed-budget toxicity rises to 0.56% and budget-averaged toxicity to 0.77 ± 0.30%, because weaker dimensions are admitted.This threshold lies below the empirically optimal band.
  • Threshold Sensitivity: At τz = 9, toxicity rises to 0.62% under the fixed budget and 0.69 ± 0.27% when averaged across budgets, as useful dimensions are discarded.The higher threshold lies above the empirically optimal band.
  • Threshold Sensitivity: The paper fixes τz = 7 for all experiments after the sensitivity sweep, noting that its toxicity is essentially on par with τz = 8.The lower threshold retains more dimensions.
Loading 2609.01878v1…