PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Instruction-Following in Language Models through Activation Steering

Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, Besmira Nushi

arXiv:2410.12877v2cs.CLcs.AIcs.LG

TL;DR

Language models need reliable control over diverse output constraints, whose many possible forms make exhaustive post-training coverage difficult. The paper derives instruction-specific activation differences and applies them during inference to steer models toward those constraints. Across several models and constraint types, steering improves adherence without explicit instructions, reinforces adherence when instructions are present, supports multiple constraints, and can transfer from instruction-tuned to base models.

  • Problem

    Diverse, modular output constraints are important for language-model applications, but their many possible variations make exhaustive post-training coverage impractical.

  • Method

    The paper computes instruction-specific steering vectors as activation differences between inputs with and without instructions and adds them during inference.

  • Results

    Activation steering improves adherence to format, length, and word-inclusion or exclusion constraints, including when instructions are absent and when multiple instructions are applied simultaneously.

  • Takeaways & Limitations

    Activation steering provides inference-time control over instruction following and can extend instruction-tuned behavior to base models.

  • Takeaways & Limitations

    Effectiveness can vary for input-dependent instructions whose averaged steering vectors do not represent a single consistent target.

Abstract

from arXiv · show

The ability to follow instructions is crucial for numerous real-world applications of language models. In pursuit of deeper insights and more powerful capabilities, we derive instruction-specific vector representations from language models and use them to steer models accordingly. These vectors are computed as the difference in activations between inputs with and without instructions, enabling a modular approach to activation steering. We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion, providing inference-time control over instruction following. Our experiments across four models demonstrate how we can use the activation vectors to guide models to follow constraints even without explicit instructions and to enhance performance when instructions are present. Additionally, we explore the compositionality of activation steering, successfully applying multiple instructions simultaneously. Finally, we demonstrate that steering vectors computed on instruction-tuned models can transfer to improve base models. Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation. Our code and data are available at https://github.com/microsoft/llm-steer-instruct.

1 INTRODUCTION

The paper asks whether instruction-specific activation vectors can efficiently control diverse, verifiable constraints in language generation. It develops contrastive activation steering and evaluates whether these vectors improve adherence with and without explicit instructions.

  • 1 INTRODUCTION: Instruction-following constraints such as format, length, and word inclusion are important for controlling language-model outputs, but their many variations make exhaustive post-training data impractical.The paper frames efficient control as a need for real-world, reliable language-model applications.
  • 1 INTRODUCTION: The method computes activation differences between inputs with and without an instruction, then applies them during inference to guide instruction adherence.The approach targets lower-level, specific constraints that complement a base request’s semantics.
  • 1 INTRODUCTION: Experiments across Phi-3, Gemma 2, and Mistral evaluate format, length, and word-specific instructions on the IFEval dataset.The study tests instruction-tuned models and investigates transfer to base Gemma 2 models.
  • 1 INTRODUCTION: Activation steering improves adherence both when instructions are absent from the input and when they are explicitly present.The paper also reports simultaneous steering of multiple instructions and transfer from instruction-tuned models to base models.
  • 1 INTRODUCTION: The work presents activation steering as a step toward practical use of mechanistic-interpretability techniques for controllable language generation.The stated contribution concerns real-world utility rather than only internal analysis.

2 STEERING FOR INSTRUCTION ADHERENCE

The paper defines modular output constraints, derives instruction-specific steering directions from residual-stream differences, and evaluates them across datasets, models, and response-quality checks. Its setup also examines representation similarity and token projections to characterize how instructions are encoded.

  • 2.1 TYPES OF INSTRUCTIONS: Instruction constraints are modular requirements on particular output aspects, divided here into format, length, and word-specific categories.This definition separates instruction adherence from factual knowledge and domain-specific skills.
  • 2.2 STEERING PROCEDURE: Activation engineering adds vectors to a frozen language model’s forward pass, while difference-in-means isolates instruction-related directions in the residual stream.The residual stream is the per-token hidden state formed by summing previous component outputs.
  • 2.2 STEERING PROCEDURE: The method averages residual-stream differences across paired base queries and instructed queries to obtain a direction associated with the instruction rather than a particular query.Representations are taken at the last input token because they capture behavior relevant to the generation that follows.
  • 2.2 STEERING PROCEDURE: The direction is normalized and rescaled before being added at one residual-stream layer across token positions, with layer selection based on held-out performance and a perplexity quality check.Format weights are mapped toward mean instructed activations; length and word-specific weights use coefficient experiments or held-out tuning.
  • 2.3 EXPERIMENTAL SETUP: The evaluation uses an augmented IFEval set containing 25 instructions, multiple base queries and phrasings, and 541 total prompts, with format assessed on 163 examples.Metrics measure format accuracy, output length, and keyword inclusion or exclusion.
  • 2.3 EXPERIMENTAL SETUP: Response quality is evaluated separately with GPT-4o-generated yes/no questions about each base query, excluding the explicit instructions from that quality judgment.The score averages positive judgments across questions and repeated evaluation runs.
  • 2.3 EXPERIMENTAL SETUP: Experiments use instruction-tuned Phi-3, Gemma 2 2B and 9B, and Mistral 7B, while also testing transfer from instruction-tuned to base Gemma models.All models are evaluated zero-shot; language constraints are grouped with format because they alter surface presentation.
  • 2.3 EXPERIMENTAL SETUP: Residual-stream analysis compares same-instruction inputs across different queries with matched and unmatched instruction conditions, while vocabulary projections examine instruction-related token preferences.Higher similarity across different queries sharing an instruction supports a shared instruction feature.

3 FORMAT INSTRUCTIONS

The paper tests activation steering for format instructions using internal representations and finds improved adherence both with and without explicit text instructions.

  • Representational analysis finds instruction-specific similarity patterns in residual-stream activations, including clear separation for quotation and stronger same-instruction similarity for Urdu language.
  • Accuracy rises from around 10% without steering to approximately 30% with steering when format instructions are absent.The non-zero baseline reflects accidental constraint satisfaction, while the increase indicates that steering vectors encode instruction-relevant information.
  • Steering improves instruction-following accuracy across all models without explicit instructions and boosts accuracy in two of four models when instructions are provided.
  • Phi-3 benefits most on instructions such as Lowercase and Highlight Text, while End Checker remains difficult because its required sentence varies by input.Averaging examples into one vector likely misses the specific sentence needed for each End Checker case.
  • Steering can slightly reduce response quality, with decreases comparable to adding instructions as text and occasional nonsensical or repetitive outputs.

4 LENGTH INSTRUCTIONS

Length steering represents general conciseness and verbosity rather than every possible length constraint, enabling adjustable output length and improved maximum-length adherence.

  • Quality-score changes from steering are evaluated against the effect of adding instructions without steering, with steering decreasing quality similarly to instruction text.
  • Steering improves adherence to explicitly specified maximum-length constraints, including outputs limited to at most five sentences.
  • Table 2 shows that increasing steering weights produces progressively shorter responses in both sentence count and word count.
  • Increasing the steering weight c effectively shortens responses, enabling continuous control over conciseness and verbosity.The method uses vectors for general requests to be brief or provide a long answer, rather than separate vectors for each numerical length constraint.

5 WORD-SPECIFIC INSTRUCTIONS

The paper constructs word-specific activation vectors to control keyword inclusion and exclusion, adapting the steering procedure to each target word.

  • Word-inclusion vectors are computed by appending differently phrased requests to include a specific word w to base queries.A separate vector is required for each keyword, and vectors can be generated on the fly at inference time.
  • Naively computed exclusion vectors can increase the probability of generating the very keywords they are intended to suppress.Their vocabulary projections produced high logits for excluded-word tokens, motivating a correction to the exclusion procedure.

6 MULTI-INSTRUCTION STEERING

The paper composes activation steering across multiple instructions by injecting separate vectors at their selected layers, improving adherence to both constraints simultaneously.

  • Cross-model steering vectors from instruction-tuned models transfer effectively to base models, with cross-model steering outperforming same-model steering in Gemma 2 2B.For length instructions, increasing the cross-model steering weight also shortens output.
  • Multiple vectors are injected simultaneously at different locations instead of being combined into one vector.This design follows prior evidence that combining multiple steering vectors into a single vector is largely unsuccessful.
  • Steering two instructions at once improves performance on both constraints for format plus length and lowercase plus keyword exclusion.
  • The approach may require refinement for conflicting instructions because competing constraints can be difficult to balance.

7 CROSS-MODEL STEERING

Cross-model steering transfers instruction representations from instruction-tuned models to base models, improving instruction adherence and, in some cases, outperforming same-model steering.

  • Cross-model steering with vectors from instruction-tuned counterparts yields performance gains for base models with and without explicit instructions.
  • In Gemma 2 2B, cross-model steering outperforms same-model steering in both query-only and explicitly instructed settings.
  • Increasing the steering weight shortens outputs when a conciseness vector from instruction-tuned Gemma 9B is applied to base Gemma 9B.
  • The results suggest that instruction-tuned models provide transferable representations for steering base models and potentially composable transfer across specialized domains.

8 RELATED WORK

Related work establishes instruction following as an important modeling objective, connects the approach to learned vector representations, and situates it within activation-editing methods for controlling generation.

  • Instruction-following research develops training methods and datasets to improve models’ adherence to user instructions and support safe deployment.
  • Prior studies show that language models can encode trained tasks, in-context tasks, and features such as truthfulness in vector representations.
  • Activation editing has been used to manipulate generation and steer models toward properties including honesty, sycophancy, and moral alignment.

9 CONCLUSION

The paper concludes that activation steering can improve instruction adherence across several constraint types, support simultaneous constraints, and transfer instruction representations from tuned to base models.

  • Activation differences between instructed and uninstructed inputs produce steering vectors that improve adherence to format, length, and word inclusion or exclusion constraints.
  • The vectors can guide models to satisfy constraints without explicit instructions and enhance performance when instructions are present.
  • Activation steering supports multiple constraints simultaneously, providing a flexible framework for controlled language generation.
  • Vectors computed on instruction-tuned models can improve base-model behavior and sometimes surpass same-model steering.

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies where the paper’s methodology, datasets, evaluation procedures, metrics, and implementation details are documented.

  • Methodology, instruction definitions, and experimental setup are described in §2, including data, metrics, and evaluation procedures.
  • Appendices C and D provide additional dataset and augmentation details, while Appendix F documents the quality score metric implementation.
  • The paper states that its code and data were submitted as supplementary materials and would be open-sourced upon publication.

A LIMITATIONS AND FUTURE WORK

The paper identifies limitations in intervention selection, representation quality, evaluation coverage, and cross-model transfer. Future work targets better optimization, quality assessment, broader instruction coverage, and transfer across architectures or sizes.

  • Method limitations: The intervention space remains only partially explored because layer, steering weight, and intervention location are selected systematically but not exhaustively optimized.Representation quality can also suffer when vectors are computed from examples where the model fails to follow instructions.
  • Future optimization: A fixed steering weight is applied throughout generation, leaving dynamic modulation as an avenue for improving control.The authors also propose more exhaustive searches over layers and steering weights.
  • Data efficiency: The effect of the number of examples used to compute steering vectors was not extensively investigated, despite vectors from as few as 20 examples proving effective.Future work could optimize the balance between computational efficiency and steering performance.
  • Evaluation limitations: Quality evaluation may miss factual inaccuracies and logical inconsistencies because GPT-2 perplexity primarily measures fluency and coherence.The authors suggest more nuanced quality metrics or task-specific evaluations.
  • Transferability: Cross-model transfer was tested only between instruction-tuned and base models with the same family and parameter size.Future research could examine transfer across architectures or sizes, potentially using latent-space transformations.

F.3 ADDITIONAL RESULTS

Additional analyses examine how activation steering affects response quality and instruction-following accuracy. Steering generally causes limited quality changes for length and word-specific constraints, while steering weight provides a controllable accuracy–quality trade-off for word exclusion.

  • Evaluation setup: Quality score analyses compare steering without instructions, steering with instructions, and explicit-instruction effects without steering across length and word-specific tasks.These conditions are summarized in Figure 9, while Figure 10 visualizes word-inclusion score-delta distributions.
  • Quality-score examples: Quality-score examples include repetitive uppercase outputs, factual inaccuracies, and a limerick rhyme failure producing drops from 1 to 0.4, 0.8 to 0, and 0.8 to 0.2.These cases illustrate distinct ways steering can degrade judged response quality.
  • Word-specific instructions: Word-inclusion steering produces minimal quality loss across scenarios, with the largest drop around -0.03 without explicit instructions.The score-delta distributions are centered near zero, with a few outliers exceeding 0.4 in magnitude.
  • Format instructions: Format analyses break down quality-score deltas by individual instructions across Phi-3, Gemma 2 2B IT, Mistral 7B IT, and Gemma 2 9B IT.Figure 11 compares steering without explicit instructions, steering with explicit instructions, and no-steering instruction effects.
  • Word exclusion: Larger absolute steering weights increase word-exclusion accuracy while gradually decreasing quality in both instruction settings.Smooth transitions in both metrics indicate that tuning c provides fine-grained control over this trade-off.

H ADDITIONAL RESULTS: REPRESENTATIONS

Additional analyses show that instruction vectors have structured geometry and can steer format, length, and word-specific behavior, while layer and weight choices affect both adherence and output quality. The results also reveal useful design trade-offs, including continuous length control and more reliable word exclusion by subtracting inclusion vectors.

  • Representations: Instruction vectors exhibit structured geometry: format vectors form distinct clusters for some constraints, while length vectors follow a concise-to-verbose trend.The clearest format clusters include “No Comma,” “Lowercase,” and “JSON Format”; length vectors place longer-output instructions opposite concise-output instructions.
  • Validation: Layer selection balances instruction-following accuracy against output quality, preventing degraded perplexity in some cases and avoiding over-steering in later layers.For Gemma 2 2B IT with explicit “JSON Format” instructions, later-layer steering degrades both accuracy and quality.
  • Length Instructions: Length steering continuously controls output verbosity: increasing the scalar steering weight produces longer responses for verbosity vectors.The effect is demonstrated on 50 base prompts without explicit length instructions.
  • Length Instructions: Steering significantly improves adherence to exact sentence-count constraints in 4 of 5 cases and shifts the n = 4 distribution toward the target.The steered distribution has a sharper peak at four sentences while retaining some natural variability.
  • Word-Specific Instructions: Subtracting word-inclusion vectors outperforms adding word-exclusion vectors in accuracy and the accuracy-to-low-perplexity trade-off.Word-exclusion vectors can promote logits for the very words they should exclude, potentially counteracting the intended steering effect.
Loading 2410.12877v2…