PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does Activation Steering Control? Attribution Across Answer Encodings and Output-Sensitive Subspaces

Zhiwei Gao, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

arXiv:2608.22985v1cs.CL

TL;DR

Activation-steering gains may reflect compatibility with answer identifiers rather than the intended semantic judgment, making attribution under a shared answer encoding ambiguous. The paper freezes interventions while re-encoding answers and finds extraction-index following in several settings, with results varying across models, tasks, and evaluation formats. It concludes that steering claims require evidence directly testing the claimed form of control.

  • Problem

    Activation-steering evaluations can conflate semantic-label control with dependence on answer identifiers used to construct the intervention.

  • Method

    The paper freezes an intervention and re-encodes answer scaffolds for the same held-out items, using remapping, factorial, localization, and open-ended audits.

  • Results

    Extraction-index following exceeds semantic-label following on NormBank, while MNLI favors extraction-index following in aggregate and SC101 favors semantic-label following.

  • Takeaways & Limitations

    A steering gain under one answer encoding does not by itself identify what the intervention controls.

  • Takeaways & Limitations

    The audit’s localization analyses use NormBank, and the gradient-defined subspace localizes effects relative to identifier readout without identifying a complete circuit.

Abstract

from arXiv · show

Activation steering is often evaluated under the answer encoding used to construct the direction. A reported gain may reflect the intended judgment or compatibility with answer identifiers seen during construction. We introduce Cross-Encoding Steering Evaluation, which freezes an intervention while re-encoding answers to the same held-out items. On NormBank, after A/B/C identifiers are reassigned, contrastive activation addition (CAA) induces larger target-versus-source score changes for the extraction indices than for the semantic labels under the new mapping. We call this extraction-index following. Varying identifier vocabulary (A/B/C, X/Y/Z, or 1/2/3) and row order shows that the effect tracks extraction index rather than row position. After matching direction norms across layers, extraction-index following emerges mainly at later depths. A low-rank output-sensitive component containing 15.4% of the direction's squared norm retains 96.3% of this effect. An Inference-Time Intervention (ITI)-style method also favors extraction-index over semantic-label following on NormBank in three models. In aggregate, MNLI favors extraction-index following, whereas Social Chemistry 101 (SC101) favors semantic-label following. Multiple-choice and open-ended evaluations can yield different behavioral conclusions. Thus, a steering gain under one answer encoding does not by itself identify what the intervention controls.

1 Introduction

The paper audits what activation-steering interventions control by freezing directions while changing answer encodings, then validates attribution across evaluation formats. It finds extraction-index following in key settings, with attribution varying by depth, subspace, model, task, and evaluation granularity.

  • Motivation: Activation steering changes LLM behavior without updating weights, but reported score gains may not reveal what the intervention controls.CAA constructs a direction from hidden-state differences between positive and negative examples and injects it at inference time.
  • Method: Cross-Encoding Steering Evaluation freezes the intervention while counterfactually re-rendering answer scaffolds for the same held-out items.The method holds the item, model, direction, layer, position, and dose fixed while varying answer encoding.
  • Findings: On NormBank, extraction-index following exceeds semantic-label following under all five remappings, emerges at later tested depths, and concentrates in output-sensitive components.A low-rank component containing 15.4% of the direction’s squared norm retains 96.3% of the effect.
  • Findings: Attribution varies across evaluation settings: MNLI favors extraction-index following in aggregate, SC101 favors semantic-label following, and open-ended behavior can diverge from multiple-choice scores.The paper therefore treats the evidence required as dependent on the claimed form of control.
  • Method: The audit distinguishes semantic-label following, extraction-index following, and extraction-row following under remapped answer encodings.Identifier vocabularies align A/X/1, B/Y/2, and C/Z/3 for cross-vocabulary comparisons.

2 Related Work

Related work covers activation-steering methods and reliable evaluation practices, alongside analyses connecting hidden representations to model outputs. The paper adapts projection–intervention analysis to identifier-readout attribution.

  • Steering evaluation: CAA and representation-engineering methods inject contrastive activation directions, while ITI edits probe-informative attention heads.Prior work applies these interventions to moral profiles, values, and cultural adaptation.
  • Steering evaluation: Prior evaluation research emphasizes context-matched tasks, likelihood-aware metrics, standardized comparisons, and explicit baselines.It also documents input sensitivity, spurious dependence, and out-of-distribution brittleness.
  • Output-sensitive subspaces: The paper adapts projection–intervention logic from output-related representation analyses to an identifier-readout-specific attribution target.Its output-sensitive subspace analysis connects hidden-state directions to option-identifier logit differences.

3 Method

The method freezes contrastive steering interventions while varying answer encodings, then measures whether behavior follows semantic labels, extraction indices, or displayed rows. Depth, position, output-sensitive subspace, and cross-method analyses localize the attribution.

  • Contrastive Activation Steering: The canonical CAA direction averages contrastive hidden-state differences and is injected at inference time with fixed extraction and injection settings.The primary audit uses the pre-answer position and fixes the direction, block, position, and strength before testing answer encodings.
  • Cross-Encoding Evaluation: Cross-encoding evaluation holds the intervention fixed while changing label mappings, identifier vocabularies, row orders, and candidate-answer renderings.Comparisons use within-encoding mean-token log-likelihood margins because each encoding has its own output sequences and margin scale.
  • Attribution Definitions: Extraction-index effects follow identifiers at the indices assigned during direction extraction, whereas semantic-label effects follow labels under the current test mapping.Extraction-index alignment is preserved across A/B/C, X/Y/Z, and 1/2/3 vocabularies, separating it from current labels and displayed rows.
  • Localization: Depth and position sweeps test when attribution emerges and whether it depends on the pre-answer scaffold, with per-model direction norms matched to a common reference.The sweep evaluates 50%, 62.5%, 75%, and 87.5% depths, while matched-position injection tests scenario-end versus pre-answer placement.
  • Output-Sensitive Subspaces: The output-sensitive subspace is built from normalized option-logit gradients, whose leading singular vectors capture dominant identifier-logit sensitivities.The smallest rank among 2, 4, 8, and 16 capturing at least 90% of validation-gradient energy defines the subspace; held-out prompts test projection and residual interventions.
  • Output-Sensitive Subspaces: Effect concentration is assessed by comparing each projected component’s extraction-index effect with its share of the full CAA direction’s squared norm.Effect retention divides the component effect by the full canonical effect, using norm-matched variants and covariance-matched random controls.
  • Cross-Method Validation: An ITI-style replication tests whether attention-head interventions reproduce the extraction-index-over-semantic-label ordering under validation-selected settings.The method ranks heads by validation probe accuracy and freezes selected layers, head count, and strength before mapping evaluation.

4 Experimental Setup

The experiments evaluate controlled and cross-task answer-attribution behavior across NormBank, MNLI, and SC101 using multiple instruction-tuned models and fixed intervention protocols. Factorial NormBank analyses vary mappings, vocabularies, and row orders while preserving extraction-index alignment.

  • Datasets: NormBank provides matched taboo, normal, and expected contrasts, while MNLI and SC101 extend evaluation to non-norm and action-level three-label tasks.The public CAA behaviors additionally extend evaluation to generation.
  • Splits: NormBank splits complete setting-and-behavior groups across train, validation, and test without shared pairs, groups, or endpoints.The factorial audit evaluates all 108 conditions, and MNLI splits analogously by normalized premise.
  • Models: CAA experiments use Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-2-9B-IT, while ITI retains three models after validation screening.The public protocol uses Llama-2-7B/13B-Chat.
  • Interventions: Primary CAA interventions use fixed near-75%-depth blocks and α = .8, with norm-matched depth audits and validation-selected ITI settings.SC101 uses previously selected middle layers and α = 1; test answer encodings do not alter extracted directions or intervention hyperparameters.
  • Scoring: Cross-encoding and factorial comparisons use mean-token log-likelihood margins, whereas localization uses normalized option-choice probability margins on a separate scale.Intervals resample complete setting–behavior groups, and effect retention compares equal-stratum component effects with canonical CAA.
  • Factorial Audit: Figure 2 reports extraction-index advantage, extraction-index-minus-row effects, and extraction-index effects across identifier vocabularies on the NormBank margin scale.All reported intervals exclude zero, and the corresponding sign-flip tests remain significant after Holm correction.

5 Results

Across remapped NormBank evaluations, steering effects preferentially follow extraction indices, emerge at later layers, and concentrate in output-sensitive components. Attribution also varies across methods, models, tasks, and behavioral evaluation formats.

  • Cross-encoding attribution: 4.63 is the random-adjusted extraction-encoding effect; after remapping, extraction-index effects range from 3.00 to 4.13 and exceed semantic-label effects by 1.95–4.04.This ordering holds across all five alternative mappings on the strict split.
  • Cross-encoding attribution: 0.32–1.86 is the extraction-index effect’s advantage over extraction-row effects across every model, identifying identifier index rather than row position as the stronger association.The pooled effect remains 1.455 [1.443, 1.467] with advantage .426 [.408, .443] on 309 pairs solved correctly under all mappings.
  • Layer and position localization: 0.065, 0.130, 0.215, and 0.160 are the extraction-index effects across increasing tested depths, with larger effects and advantages at 75% and 87.5% depth after norm matching.All four models show positive extraction-index effect and advantage at 75% and 87.5% depth.
  • Layer and position localization: 98.3% is the reduction in extraction-index probability-margin effect when direction extraction moves from pre-answer to scenario end, remaining near zero after norm rescaling and matched-position injection.The pre-answer site includes the question, displayed identifiers, label descriptions, and output instruction; the scenario-end site precedes this scaffold.
  • Output-sensitive subspaces: 15.4% of the direction’s squared L2 energy retains 96.3% [95.8, 96.7] of its extraction-index effect, while the residual retains 1.0% [0.6, 1.3].A mean local-gradient controller also produces the same ordering, with extraction-index effect 0.417 and advantage 0.526 versus 0.215 and 0.236 for canonical CAA.
  • Cross-model and behavioral validation: ITI favors extraction-index following across all five alternative mappings in three retained models, while MNLI favors it in aggregate and SC101 favors semantic-label following.Qwen reverses the MNLI pattern, and multiple-choice and open-ended evaluations produce different verdicts for some behaviors.

6 Discussion

The audit separates score changes under a shared answer encoding from semantic-label following. It localizes the observed dependence to identifier-sensitive late states without identifying a complete circuit.

  • On NormBank, extraction-index following is weak before the answer scaffold and concentrates at late pre-answer states sensitive to identifier logits.
  • A selected output-sensitive subspace carries nearly all of the effect, while removing it does not increase semantic-label following.
  • MNLI extends the pooled pattern, whereas Qwen and SC101 show model- and evaluation-dependent profiles.
  • Extraction-index following is an evaluation-level attribution, not evidence that the model represents an abstract index variable.

7 Limitations

The study’s scope and localization claims are bounded by task, model, and intervention choices. Several analyses use NormBank, and the localization procedures do not identify a complete circuit.

  • The six-mapping audit covers three three-label tasks and four instruction-tuned model families, while full factorial, localization, and ITI analyses use NormBank.
  • The gradient-defined subspace localizes the effect relative to identifier readouts at tested layers and positions without identifying a complete circuit.
  • Layer-wise norm matching equates direction magnitude but not layer-specific model sensitivity, and position localization uses a single-site intervention.
  • The public CAA comparison follows protocol-specific doses.

8 Conclusion

Cross-encoding evaluation freezes interventions while re-encoding answers to distinguish semantic-label from extraction-index following. The results favor identifier-linked control in some settings but vary across tasks and models.

  • CAA preferentially follows extraction indices on NormBank, with the effect emerging at later tested depths and concentrating in a low-rank output-sensitive component.
  • ITI replicates the extraction-index-over-semantic-label ordering in three models.
  • MNLI reproduces the pooled extraction-index ordering despite a Qwen reversal, whereas SC101 favors semantic-label following.
  • These contrasts show that steering gains under one answer encoding do not by themselves identify what the intervention controls.

A Reproducibility and Data Isolation

The evaluation fixes interventions and systematically varies answer encodings, splits, prompts, scoring scales, and datasets. Reproducibility controls isolate mapping effects while answer-key checks constrain interpretable open-ended analyses.

  • Scoring and evaluation: NormBank and MNLI cross-encoding tests use mean-token log-likelihood margins, while localization analyses use normalized option-choice probability margins.
  • Pair construction: NormBank preprocessing retains taboo, normal, and expected labels, groups rows by exact setting and behavior, and deterministically matches contrast endpoints without reuse.
  • Data isolation: The strict NormBank split assigns complete setting–behavior groups to partitions and removes overlap in pair IDs, context groups, and endpoints.
  • Factorial audit: The factorial audit reuses strict CAA directions across six mappings, three identifier vocabularies, and six row orders, producing 108 conditions per frozen direction.
  • Cross-task replication: MNLI pairs hypotheses within normalized premise groups and assigns complete premise groups to one split, while SC101 evaluates three label contrasts under all six mappings.
  • Prespecification: The audits use fixed layers, strengths, templates, and pre-specified settings, with SC101 settings differing from NormBank and MNLI and therefore not supporting cross-dataset magnitude comparison.
  • Prompt rendering: Candidate answers are scored as alternative continuations of an unchanged prompt prefix, so changing encoding changes label expression while retaining sequence-likelihood scoring.
  • Answer-key checks: Opaque open-ended results require temporary-key decoding; published CAA models fail a stricter counterbalanced key test and are excluded from open-ended behavioral evidence.

B.1 Factorial Attribution of Semantic Label, Extraction Index, and Extraction Row

The factorial audit separates semantic-label, extraction-index, and extraction-row following by varying answer mappings while freezing the intervention. Extraction-index effects generally dominate, emerge at later depths, and concentrate in an output-sensitive subspace.

  • Factorial audit: The audit independently varies semantic mapping, identifier vocabulary, and row permutation while scoring semantic-label, extraction-index, and extraction-row margins.Identifier indices are aligned across A/B/C, X/Y/Z, and 1/2/3 vocabularies.
  • Model-level attribution: Extraction-index effects exceed extraction-row effects in all four models, and exceed semantic-label effects in Llama, Mistral, and Gemma but not Qwen.All identifier-minus-label and identifier-minus-row comparisons remain nonzero after Holm correction at p < .001.
  • Vocabulary sensitivity: Changing identifier vocabulary attenuates the extraction-index effect for every model, measuring compatibility of the frozen intervention with each identifier set.The comparison uses steered-minus-base mean log-likelihood-margin changes for single-token identifiers.
  • Layer localization: The extraction-index advantage emerges mainly at later depths rather than from direction-norm growth alone.All four models show positive extraction-index effect and advantage at 75% and 87.5% depth after norm matching.
  • Position localization: Scenario-end extraction reduces the extraction-index effect by 98.3%, while norm matching does not restore it.Pre-answer and scenario-end directions also show low cosine alignment across model–contrast cells.
  • Output-sensitive subspace: A low-rank output-sensitive projection containing 15.4% of squared L2 energy retains 96.3% of the extraction-index effect.Across models, the projection retains 93.3–103.5% of the full effect while the residual remains between −.0056 and +.0085.

C.1 ITI-Style Attention-Head Replication

An ITI-style attention-head intervention replicates extraction-index preference on NormBank, while broader evaluations show model- and task-dependent attribution and limits for matched-context interpretation.

  • ITI replication: The ITI-style replication trains head-level probes, selects heads and intervention strength on held-out validation data, and tests only interventions meeting a positive-gain criterion.Heads are selected from candidate layer fractions using validation probe accuracy and target-margin gain.
  • NormBank results: Extraction-index effects range from 6.26 to 7.34 across alternative mappings, while semantic-label effects range from −3.05 to +1.71.Extraction-index advantage is positive under all five mappings, and every reported interval excludes zero.
  • Cross-method comparison: Both CAA and ITI show positive extraction-index effects under all five alternative mappings, whereas semantic-label effects vary with the mapping.The comparison concerns ordering rather than effect magnitudes because the methods use different strengths.
  • MNLI: On MNLI, extraction-index effects and advantages are positive across all five alternative mappings, but model-level advantages range from 3.090 to −1.966.Gemma, Llama, and Mistral are positive across mappings, while Qwen is negative across mappings.
  • SC101: SC101 reverses the aggregate NormBank pattern: all five mappings have positive semantic-label effects and negative extraction-index advantage.SC101 values use a probability scale and are not numerically compared with NormBank.
  • Matched-context diagnostic: The matched-context diagnostic applies one additive direction uniformly to every item, so it measures context-dependent response rather than an explicit conditional gate.This design leaves selective activation as a scope boundary of the diagnostic.

D.2 Additional Answer-Encoding and Label-Cue Controls

Additional controls test answer-encoding, lexical-label, open-ended, and robustness effects beyond the core factorial attribution. These analyses distinguish direct-label behavior from selective contextual control and show that evaluation granularities can diverge.

  • Label-cue control: The label-cue direction is a scenario-free lexical control formed from target and source prompts containing explicit classification labels.It is norm-matched to canonical CAA and excludes scenarios and options.
  • Label-cue results: The label-cue direction produces a direct-label gain of 2.618, while other gains are .114–.277 and context effects range from −.240 to .075.Ranking changes range from −.02 to .27 points, showing no robust selective-context control.
  • Open-ended controls: For hallucination and refusal, swapped A/B effects retain 0.94–1.10 times the original fixed-direction effect within the shared A/B token space.Direct-completion effects remain on a separate log-likelihood-margin scale, so cross-encoding retention ratios are not formed.
  • Judge robustness: Judge agreement is high, but numerical judge scales are not interchangeable.Pairwise agreement spans Pearson .886–.943, Spearman .801–.829, and 80.3–86.3% within one point.
  • Scoring validation: Human and automated scores correlate at .887 across 12 nonzero-dose cells and agree in sign for 10, supporting the scoring procedure.The 180-response sample supports the procedure, while Table 36 estimates use all 918 generations.
  • Robustness checks: Group-disjoint and pair-ID splits preserve the main extraction-index conclusions in 17 of 18 encoding–metric cells.Both splits retain positive extraction-index effects and advantages under all five alternative mappings.

E.2 Mapping-Balanced Full Factorial Control

A mapping-balanced direction averages directions built under all six semantic mappings before repeating the factorial audit. This reduces extraction-index advantage, but the resulting attribution remains model-dependent.

  • Direction construction: The mapping-balanced direction averages directions from all six semantic mappings, rescales the average to canonical CAA norm, and freezes it for the complete audit.The control uses the same six-mapping family as direction construction and evaluates the same strict test pairs.
  • Attribution shift: The mapping-balanced direction reduces extraction-index advantage in all four models, with a pooled reduction of 1.536 [1.525, 1.547].Qwen remains semantic-label dominant; Llama and Gemma switch dominance, while Mistral retains a small positive extraction-index advantage.
  • Encoding-specific gains: Canonical CAA gains 4.624 under extraction encoding and 2.845 under direct-label completion, compared with 1.255 and 3.910 for the mapping-balanced direction.Each pair is compared only within its own encoding scale.
  • Implication: The mapping-balanced control shows that direction construction can reduce answer-encoding dependence, while outcomes remain model-dependent.This is a construction-level control rather than a universal elimination of extraction-index following.
Loading 2608.22985v1…