PaperlayerView exact revision on arXiv

Source-linked AI summary

First-Order Steering: Translating Weight Adaptation into Activation Steering

Sri Pranav Kunda, Alexander Kurz, Tomas Dominik, Uri Maoz

arXiv:2610.04283v1cs.LGcs.AIcs.CL

TL;DR

Composing multiple target behaviors remains difficult for activation steering, despite weight-space merging’s ability to combine adaptations. The paper translates those adaptations into activation directions using first-order approximations and develops HeRD-Merging to reduce approximation error. Its methods improve steering for individual and composed behaviors while retaining performance comparable to conventional model-merging baselines.

  • Problem

    Activation steering methods face challenges composing multiple target behaviors, while model merging can combine behaviors represented as weight adaptations.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    First-order steering derives composable activation directions from hidden-layer Jacobians with respect to steering strengths, and HeRD-Merging reduces Hessian-based approximation-error terms.

    Source: §1 · p. 2

  • Results

    First-order steering controls individual and composed behaviors more accurately than existing activation steering methods, while HeRD-Merging maintains performance comparable to conventional model-merging baselines.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Weight adaptations from model merging can yield composable activation-steering vectors with improved steering accuracy.

    Source: §1 · p. 2

    The formulation assumes a frozen pretrained transformer and defines its distribution-matching setup for a fixed autoregressive context.

    Source: §3 · p. 3, §4.2 · p. 5

Abstract

from arXiv · show

Activation steering exploits interpretable directions in the residual stream to enable inference-time manipulation of model behavior. Composing steering vectors to apply multiple target behaviors simultaneously is important in various fields-including AI alignment and safety-but remains a challenge for existing activation steering methods. In contrast, prior work in model merging shows that target behaviors represented by learned weight adaptations can be combined with high accuracy. A method that translates weight adaptations into activation steering vectors could therefore extend prior work in model merging to generate composable steering vectors that better enable simultaneous inference-time behavioral control. For this, we introduce First-Order Steering, a formulation of activation steering as a first-order approximation of weight update matrices parameterized by a vector of steering strengths, and establish theoretical bounds on the approximation error of first-order steering. We then develop a novel model merging procedure, HeRD-Merging, which minimizes the first-order approximation error terms to enable higher first-order steering accuracy. Together, our method produces steering vectors that control both individual and composed behaviors more accurately than existing activation steering methods. Furthermore, HeRD-Merging matches the performance of conventional model-merging baselines, while producing weight adaptations that admit more accurate first-order steering vectors.

1 INTRODUCTION

Activation steering can induce individual behaviors, but composing multiple behaviors remains challenging; model merging shows that weight adaptations can be composed. The paper translates differentiable weight adaptations into composable activation directions and introduces HeRD-Merging to improve their accuracy.

1 INTRODUCTION

  • Composing multiple target behaviors remains a challenge for activation steering, although model-merging methods can combine behaviors represented as weight adaptations.Source: §1 · p. 1
  • First-order steering uses each hidden layer’s Jacobian with respect to steering strengths to produce composable activation-space directions, with Hessian norms bounding approximation error.Source: §1 · p. 2
  • HeRD-Merging reduces Hessian-based error terms so its weight adaptations yield more accurate first-order steering vectors.Source: §1 · p. 2
  • First-order steering outperforms existing activation steering methods on individual and composed behavioral control, while HeRD-Merging improves steering accuracy and performs comparably to conventional model-merging baselines.Source: §1 · p. 2

2 PRELIMINARIES

Existing activation steering methods can be inconsistent when controlling multiple behaviors, while model merging offers a way to combine behaviors represented by weight adaptations. The paper motivates first-order steering as a bridge between these approaches.

2 PRELIMINARIES

  • Activation steering adds directions to intermediate activations, while model merging combines independently finetuned LoRA adapters through weight-space updates.Source: §2 · pp. 2–3
  • Activation steering methods have shown inconsistent out-of-distribution results, especially when steering multiple behaviors simultaneously.Source: §2 · p. 3
  • Task arithmetic adds or subtracts task vectors to compose or suppress behaviors, and TIES, DARE, and KnOTS modify or decompose those updates before merging.Source: §2 · p. 3
  • Composing weight-space adaptations while retaining strong per-task performance provides a foundation for constructing composable activation steering vectors.Source: §2 · p. 3

3 FIRST-ORDER STEERING

First-order steering approximates how transformer hidden layers change as task-arithmetic steering strengths vary. Its Jacobian supplies behavior-specific directions, while Hessian-based bounds characterize approximation error.

3 FIRST-ORDER STEERING

  • The method treats steering strengths as parameters of task-arithmetic updates to LoRA adapters applied at transformer projection heads.Source: §3 · p. 3
  • For each hidden layer, first-order steering approximates activations by their value at zero steering strengths plus the Jacobian-weighted steering-strength changes.Source: §3 · p. 4
  • Steering vectors are fitted over activation samples to reduce dependence on the residual input, and individual behavior directions are columns of the fitted matrix.Source: §3 · p. 4
  • Hessian norms bound the first-order remainder and also bound a mixed-Jacobian quantity linked to least-squares error, making Hessian regularization a signal of steering accuracy.Source: §3 · p. 4

4 HESSIAN REGULARIZED DIFFERENTIABLE MERGING (HERD-MERGING)

HeRD-Merging is designed to reduce error terms that govern first-order steering accuracy while retaining model-merging performance. It combines learnable low-rank corrections with distribution matching and Hessian regularization.

4 HESSIAN REGULARIZED DIFFERENTIABLE MERGING (HERD-MERGING)

  • HeRD-Merging controls the first-order steering error bound and aims to minimize a mixed-Jacobian quantity while retaining merging performance.Source: §4 · p. 4

4.1 THE HERD ADAPTER FUNCTION

  • The HeRD adapter adds learnable, scaled low-rank correction matrices for interactions among behaviors; setting their scaling coefficients to zero recovers task arithmetic.Source: §4.1 · p. 5

4.2 DISTRIBUTION LOSS BY GRADIENT DESCENT

  • The distribution-matching stage interpolates next-token distributions from the base model and individual behavior models, then optimizes their match over a prompt dataset and steering-strength grid.Source: §4.2 · p. 5

4.3 HESSIAN REGULARIZATION BY THE LEVENBERG-MARQUARDT ALGORITHM

  • The Levenberg–Marquardt stage retains distribution alignment while penalizing Hessian-related mixed-Jacobian terms and correction magnitude; it optimizes the correction scales after fitting the correction matrices.Source: §4.3 · pp. 5–6

5 EXPERIMENTS

The experiments evaluate steering three semantically independent behaviors across three model families, comparing first-order steering with activation-steering baselines and HeRD-Merging with model-merging baselines. First-order steering leads the activation-steering comparison, while HeRD-Merging improves on task arithmetic for steering and remains competitive in model merging.

5.1 SETUP AND TRAINING DETAILS

  • The study evaluates analogy usage, bulletpoint usage, and sophisticated language across Qwen2.5-VL-3B, Gemma-4-E2B, and Mistral-7B-Instruct-v0.3, training separate LoRA adapters for each behavior on each model.Training datasets were synthetically generated and contained approximately 10,000 samples each.Source: §5.1 · pp. 6–7

5.2 DEBERTA-V3 CLASSIFIERS

  • The DeBERTa-v3-base classifier is finetuned on 80% of the labeled data, with 10% each reserved for validation and testing.The test set contains 1,994 samples, and validation accuracy is used as a heuristic for binary-classification thresholds.Source: §5.2 · p. 7

5.3 ACTIVATION STEERING EXPERIMENTS

  • Steering methods are compared at one common layer, selected by maximizing normalized mean activation difference; first-order steering uses the mean Jacobian over UltraChat training prompts.The evaluation tests the 2^n corners of the steering-strength space with 100 outputs each and uses DeBERTa-v3 to measure behavioral adherence.Source: §5.3 · pp. 7–8
  • 46.4% average behavioral adherence for first-order steering with HeRD-Merging, versus 30.3% for the strongest non-first-order baseline, Linear-AcT.Across the three models, HeRD-Merging reaches 64.7%, 42.5%, and 32.0%, respectively; Table 1 reports the aggregate activation-steering comparison.Source: §5.3 · p. 8, Table 1 · p. 8, Table 2 · p. 9

5.4 HERD-MERGING COMPARISONS

  • 79.9% overall behavioral adherence for HeRD-Merging, versus 78.5% for the strongest model-merging baseline, DARE task arithmetic.HeRD-Merging also achieves the highest aggregate adherence for analogy use, bulletpoint use, and sophisticated language; Table 3 reports overall adherence averaged across the three models.Source: §5.4 · p. 9, Table 3 · p. 9

6 CONCLUSION

The conclusion presents first-order steering as a weight-adaptation approximation and HeRD-Merging as a way to control its error. Experiments show improved steering performance alongside model-merging performance similar to established methods; finetuning compute and data remain practical constraints.

6 CONCLUSION

  • First-order steering reformulates activation steering as a first-order approximation of model weight adaptations, with Hessian norms bounding approximation error.HeRD-Merging controls the Hessian norm of each transformer layer and improves the resulting first-order steering performance.Source: §6 · p. 10
  • Experiments show improved overall accuracy and consistency across model families compared with baselines on behavioral adaptation tasks.HeRD-Merging performs similarly to established model-merging methods while improving first-order steering performance.Source: §6 · p. 10
  • First-order steering requires finetuning compute and data, and HeRD-Merging’s Hessian-regularization effectiveness depends on the supplied weight-update matrices.The conclusion identifies low-data finetuning and incorporating Hessian regularization into model training as directions for future work.Source: §6 · p. 10

ETHICS STATEMENT

The authors report that the experiments use publicly available data and models, and do not involve human subjects or user-sensitive data.

ETHICS STATEMENT

  • The experiments use publicly available data and models, without human subjects or user-sensitive data.The authors describe the aim as aiding aligned and controllable language models and acknowledge dual-use potential.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The authors identify public code, datasets, and finetuned checkpoints as reproducibility resources, and report the hardware used.

REPRODUCIBILITY STATEMENT

  • Code, synthetic datasets, and finetuned model checkpoints are available through the authors’ GitHub and Hugging Face repositories.The experiments used four NVIDIA A100 GPUs with 80 GB VRAM.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE DISCLOSURE STATEMENT

This material pairs an AI-use disclosure with proof details for a theoretical result. The proof invokes the mean value inequality and extends it to an open rectangle.

AI USE DISCLOSURE STATEMENT

  • The authors report using LLMs for synthetic datasets, methodological feedback, implementation, literature search, and literature analysis, while manually reviewing AI-assisted work.They state that generative tools were not used to develop theoretical models, formulate mathematical claims, design methodology, or interpret results.Source: AI USE DISCLOSURE STATEMENT · p. 10

A.1 PROOFS OF THEOREMS

  • The proof of the stated theoretical result uses the mean value inequality, which the appendix restates before extending its application.The appendix says the inequality can be extended to an open rectangle and functions f: R^(d+n) → V.Source: §A.1 · p. 15
  • The appendix invokes a Banach-space mean value inequality for x and y in I, then uses it to obtain the desired bound.The supplied proof passages do not state the bound itself.Source: §A.1 · p. 15

A.2 FIRST-ORDER STEERING GENERALIZES TASK ARITHMETIC

The appendix explains how task arithmetic appears in first-order steering and describes the synthetic data generation used in the experiments. It connects differentiable merging procedures to additive steering directions.

A.2 FIRST-ORDER STEERING GENERALIZES TASK ARITHMETIC

  • Any differentiable merging procedure satisfying the stated condition has task arithmetic as its first-order term, including when behavioral updates are considered at later computational-graph blocks.The appendix frames this as support for task arithmetic as a rule for composing behaviors.Source: §A.2 · p. 16
  • Under the generalization, task arithmetic yields the same first-order steering rule, and HeRD-Merging preserves composition by adding steering vectors for target behaviors and composition factors.The passage also notes that this generalization allows incorporation of the matrices RS.Source: §A.2 · p. 16

B.1 SYNTHETIC DATASET GENERATION

  • Synthetic datasets for three target behaviors use the same UltraChat prompts, transformed by GPT-5.5 without access to the original model responses.Sampling uses the default temperature τ = 1.0 and top-p p = 1.0 settings.Source: §B.1 · p. 16

B.2 CLASSIFIER TRAINING

The appendix describes behavior-classifier training, generation settings, and bootstrap confidence intervals. It also identifies supplementary results for first-order steering and HeRD-Merging.

B.2 CLASSIFIER TRAINING

  • DeBERTa-v3-base classifiers are finetuned into three behavior-specific variants, and a manually evaluated subset is checked for agreement with classifier labels.Training uses approximately 10,000 samples over two epochs; Table 4 reports classifier performance with thresholds selected on validation accuracy and fixed for evaluations.Source: §B.2 · p. 17, Table 4 · p. 17

B.3 LLM GENERATION SETTINGS

  • Evaluations use greedy decoding without a system prompt across Qwen2.5-VL-3B, Gemma-4-E2B, and Mistral-7B-Instruct-v0.3.The selected models span 3 billion, 5 billion, and 7 billion parameters, respectively.Source: §B.3 · p. 17
  • The reported 95% confidence intervals resample prompts, retaining every associated generation, across 2,000 bootstrap replicates.The empirical 2.5th and 97.5th percentiles define each interval.Source: §B.3 · p. 17

C ADDITIONAL RESULTS

  • The appendix reports supplementary results for first-order steering and HeRD-Merging.The supplied passage does not specify the supplementary findings.Source: C ADDITIONAL RESULTS · p. 17

C.1 HERD-MERGING PER-MODEL RESULTS

This section presents per-model results for HeRD-Merging and first-order steering, covering adherence changes, steering-vector energy distributions, and example outputs for website redirect-loop and missing-cookie prompts.

C.1 HERD-MERGING PER-MODEL RESULTS

  • HeRD-Merging achieves 92.4% mean adherence on Qwen and 78.9% on Mistral, but on Gemma matches task arithmetic at 68.5% while DARE-TIES leads at 71.9%.It exceeds the strongest baseline by 2.2 percentage points on Qwen and task arithmetic's 76.7% on Mistral, but is not universally superior.Source: §C.1 · pp. 17–18, Table 5 · p. 18

C.2 FIRST-ORDER STEERING AT ALL LAYERS

  • The all-layer evaluation uses a single layer for baseline comparison, while first-order steering also supports computing vectors for all layers simultaneously without output degradation.The all-layer procedure follows Section 5.3 but uses LoRA adapters trained at all layers for Section 5.4.Source: §C.2 · p. 18
  • All-layer HeRD-Merging steering raises mean adherence from 64.7% to 81.2% on Qwen and from 32.0% to 34.3% on Gemma, but lowers it from 42.5% to 35.4% on Mistral.HeRD-Merging retains higher mean adherence than task arithmetic at all layers, but adding layers does not uniformly improve behavioral control.Source: §C.2 · p. 18, Table 6 · p. 19
  • Effective layer counts are lowest for Qwen (5.5–8.6, mean 7.4), intermediate for Mistral (9.9–12.9, mean 11.1), and highest for Gemma (19.9–20.8, mean 20.3).Within Qwen, sophisticated-language steering is most concentrated; Gemma distributes energy similarly across all three behaviors.Source: Table 7 · p. 19, §C.2 · p. 19

C.4 STEERING OUTPUT EXAMPLES

  • For a redirect-loop prompt, CAA, RE-Control, RepE, and ODESteer explain the error or cite lack of real-time access, while Linear-AcT begins an analogy.Source: §C.4 · p. 25
  • For a missing-cookie prompt, First-Order (Task Arithmetic) begins a key-and-door analogy, while other methods produce repeated, garbled, or redirect-focused responses.Source: §C.4 · p. 25
  • On a redirect-error prompt, First-Order (Task Arithmetic) and several baselines give browser-error explanations, while Linear-AcT, CAA, and ODESteer produce repetitive or garbled text.Source: §C.4 · p. 25

Related papers

Loading 2610.04283v1…