Minimally Invasive Steering of Language Models

1 Introduction

• Test-time alignment adapts generation to user- or context-dependent rewards, changing rewards, or black-box evaluators while keeping model parameters fixed.Unlike post-training alignment, this approach targets deployment-time rewards.Source: §1 · p. 1
• Unregularized pre-logit steering can push residual states far from the reachable manifold, degrading coherence or diversity; imperfect rewards can amplify oversteering.Pre-logit interventions avoid backpropagation through the transformer body, but still require sufficient regularization.Source: §1 · p. 1
• MISVO penalizes steering with a Fisher-information quadratic derived from token-level KL geometry, with an analytic gradient and first-order agreement to the full KL gradient near zero steering for fixed horizon.The sequence-level KL gradient separates into an analytic Fisher term and a suffix score-function term, which is second order for fixed horizon.Source: §1 · pp. 1–2

2 Problem Setup

• For a prompt and fixed horizon, the frozen model supplies hidden states immediately before its language-model head, with the reference policy defined over generated responses.The head is parameterized by W and b.Source: §2 · p. 2
• The alignment objective trades terminal reward from an evaluator that need not be differentiable against KL deviation from the reference policy.The trade-off is controlled by λ, and its ideal optimizer is a reward-tilted distribution.Source: §2 · pp. 2–3
• Position-specific steering vectors change the pre-head logits by W u_t while model parameters remain fixed, so gradients need not backpropagate through the transformer body.Steering can still change the distribution of sampled prefixes.Source: §2 · pp. 2–3

3 Proposed Method: Minimally Invasive Steering

• The ideal reward-tilted distribution may not be representable by position-specific interventions, so the KL-regularized objective is restricted to the steering policy class.By the autoregressive chain rule, sequence-level KL decomposes into expected token-level divergences.Source: §3 · p. 3
• A quadratic surrogate with fixed positive semidefinite cost matrices replaces the KL penalty and gives the analytic gradient Σ_t u_t.The sequence-level prefix distribution depends on steering, making direct KL-gradient estimation include a potentially high-variance score-function term.Source: §3 · p. 3

3.1 Algorithm

• MISVO uses stochastic gradient ascent with K independently sampled responses and their rewards, returning the final steered policy and the highest-reward sampled response.All positions are updated using rollouts and token probabilities from the same policy iterate.Source: §3.1 · pp. 3–4

4.1 From token-level KL to a quadratic penalty

• MISVO averages prefix-dependent Fisher matrices over reference-policy prefixes to obtain fixed cost matrices, replacing token KLs with quadratic expansions and steered prefixes with reference prefixes.The Fisher matrix is derived from the local geometry of token-level KL.Source: §4.1 · p. 4, §4 · p. 4
• The token-level KL’s leading quadratic term weights interventions by their effects on relative logits; a constant shift of all logits has zero cost.The approximation is local: neither KL nor its Fisher quadratic globally bounds the other.Source: §4.1 · pp. 4–5

4.2 Sequence-level KL gradient

• The exact sequence-level KL gradient separates an analytic Fisher term from a suffix score-function term capturing how steering changes future prefixes.The score-function term is the source of potentially high-variance Monte Carlo estimation.Source: §4.2 · p. 5

4.3 Accuracy of the Fisher surrogates

• Three Fisher surrogates differ in how they evaluate the token Fisher and prefix distribution, but their gradient errors are O(∥u∥2) near zero steering.The three choices retain the exact analytic term, evaluate token Fisher at zero steering with steered prefixes, or additionally use the reference prefix law.Source: §4.3 · p. 5
• MISVO’s frozen-Fisher regularizer gradient agrees with the full sequence-level KL gradient to first order near zero steering, for fixed generation horizon T.This is a local guarantee; it does not ensure accuracy for large interventions or bound accumulated error along an optimization trajectory.Source: §4.3 · p. 6

4.4 Computational overhead

• Matrix-free Fisher products cost O(V d) each; retaining S reference hidden states per position gives total cost O(NTSV d) and storage O(TSd).MISVO uses S = K initial rollouts; its N iterations with K rollouts use NK generations and reward evaluations, matching Best-of-N’s candidate budget.Source: §4.4 · p. 6

5.1 Setup

• Four frozen models (~1B–14B) cover SHP preference generation (500 prompts) and MBPP+ executable-test reward (120 problems on three smaller models), with matched rollout budgets and best-found reward reporting.MISVO and AISP generate candidates iteratively, while Best-of-N samples from the reference policy.Source: §5.1 · pp. 6–7

5.2 Main results

• MISVO has the highest reported mean reward in six of seven model–task settings across SHP and MBPP+.It beats both baselines on three of four SHP models, with AISP leading only on Llama-3-8B, and leads on all three MBPP+ models, with the largest gain on the smallest.Source: Table 1 · p. 7, Table 2 · p. 8, §5.2 · p. 8

5.1 Setup

• Diversity measures repeated n-grams and coherence measures prompt–response semantic relatedness; neither establishes distributional proximity, factual accuracy, or absence of reward exploitation.The paper evaluates token-distribution deviation separately.Source: §5.1 · p. 7, §5.2 · p. 8

5.2 Main results

• For LFM2.5-1.2B on SHP, MISVO’s within-step mean reward increases, while AISP’s later decline is consistent with—but does not identify—the cause of oversteering.Figure 2 also reports within-step maximum and cumulative maximum reward against cumulative sampled completions.Source: §5.2 · p. 8, Figure 2 · p. 8

5.3 Direct measurement of policy deviation

• On SHP with LFM2.5-1.2B, MISVO has higher reward and lower reference-prefix KL than AISP and unregularized steering.This diagnostic uses reference-policy prefixes and does not capture all steering-induced changes in prefix visitation.Source: Table 3 · p. 9, §5.3 · p. 9

6 Conclusion

• In six of seven model–task settings, MISVO reports the highest mean reward, with diversity and coherence scores close to BoN.The conclusion reports this comparison across the seven settings.Source: §6 · p. 9
• MISVO optimizes pre-logit interventions with a Fisher-quadratic penalty derived from local KL geometry.The conclusion states that the exact sequence-level gradient decomposition and local error bounds establish first-order agreement between the frozen-Fisher gradient and the full KL gradient.Source: §6 · p. 9

A Related Work

• MISVO differs from training-time alignment, prompt-based control, and output-level methods by optimizing hidden-space interventions against a runtime reward.The related-work discussion also distinguishes its Fisher-geometry regularizer from existing representation-level methods.Source: A Related Work · p. 14

B.1 Preliminary lemmas

• The preliminary lemmas identify the softmax gradient as the probability vector and its Hessian as the softmax covariance matrix, with that covariance positive semidefinite and annihilating the all-ones vector.The covariance matrix also has spectral norm at most 1.Source: §B.1 · p. 15

B.2 Proof of the autoregressive chain rule

• The autoregressive chain rule expresses sequence-level KL through successive terms that depend on prefixes, then sums those terms.The proof obtains each term by marginalizing later tokens and using the autoregressive factorization.Source: §B.2 · p. 16

B.3 Proof of Proposition 1

• For nearby steering vectors, the KL between induced token distributions has a Fisher quadratic as its leading term, with a cubic-order remainder.At the reference steering vector, the Fisher matrix is WᵀCπrefW.Source: §B.3 · p. 17

B.4 Proof of the non-asymptotic KL bound

• The sequence-level KL is bounded above by one-half the squared norm of the logit perturbation.The proof uses the bound that the softmax covariance matrix is no larger than the identity.Source: §B.4 · p. 17
• The Fisher quadratic is only a local second-order proxy for KL and does not dominate it globally.The passage contrasts it with a simpler global upper bound.Source: §B.4 · p. 17

B.5 Proof of Theorem 1

• At fixed context, differentiating the per-position KL gives the exact analytic term F_t(u_t)u_t.The additional trajectory contribution arises because the prefix distribution under the steered policy depends on the steering vectors.Source: §B.5 · pp. 17–18
• The remaining trajectory contribution is the suffix score-function term, which weights the position-t score by the conditional sequence KL of the suffix.The proof identifies this term by factoring the score over the prefix through position t out of the conditional suffix expectation.Source: §B.5 · pp. 18–19

B.6 Proof of Theorem 2

• The Fisher-matrix change is bounded using the logit perturbation at each position, uniformly over prefixes.The proof then takes expectation under the steered policy without enlarging this uniform bound.Source: §B.6 · pp. 20–21
• The proof of the surrogate-gradient bounds controls both pointwise changes in the Fisher matrices and drift in the prefix trajectory law.It applies a sequential coupling argument to the steered and reference trajectory laws.Source: §B.6 · pp. 19–21

B.7 Proof of Proposition 2

• At zero steering, the sequence-level KL gradient vanishes and its Hessian is the reference-policy score covariance.The proof uses the fact that KL is minimized at the reference policy.Source: §B.7 · p. 21
• The reference-policy Hessian has frozen Fisher matrices on its diagonal blocks and zero off-diagonal blocks.Conditioning on each prefix yields the per-position Fisher matrices.Source: §B.7 · p. 22
• For each position, the complete KL gradient, including the suffix term, equals the frozen Fisher times u_t plus O(∥u∥²).The proof obtains the expansion by Taylor expansion around the origin.Source: §B.7 · p. 22

• Figure 3 tracks Gemma 3 4B/SHP mean reward per step, maximum reward per step, and cumulative maximum reward against cumulative sampled completions.Source: Figure 3 · p. 23
• Figure 4 tracks Llama 3 8B Instruct/SHP mean reward per step, maximum reward per step, and cumulative maximum reward against cumulative sampled completions.Source: Figure 4 · p. 23
• Figure 5 tracks Phi 4/SHP mean reward per step, maximum reward per step, and cumulative maximum reward against cumulative sampled completions.Source: Figure 5 · p. 23

C.1 Ablation: Fisher surrogates

• On LFM2.5-1.2B/SHP, Fisher variants differ by at most 0.34 in reward and 0.013 in diversity/coherence, with the frozen-reference surrogate highest in reward.The ablation holds main-experiment hyperparameters fixed; the KL-regularized variant has lower reported reward.Source: §C.1 · pp. 23–24, Table 4 · p. 24

C.2 Runtime and memory

• BoN is fastest on both profiled models, while MISVO's retained reference hidden states and vocabulary-sized Fisher–vector intermediates contribute to higher peak memory.Profiling uses single-prompt batches, so cross-prompt batching may change throughput.Source: §C.2 · p. 24, Table 5 · p. 24

D Relation to per-prompt adapter fitting

• The study matches rollout counts but not wall-clock or memory budgets, and leaves comparison with per-prompt LoRA under both budgets to future work.The paper makes no claim that steering is preferable to adapter fitting.Source: D Relation to per-prompt adapter fitting · p. 24