PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Supervised Fine-Tuning Reduce Instruction Sensitivity?

Jaekeol Choi

arXiv:2608.26661v1cs.IR

TL;DR

The paper asks whether conventional task-specific SFT reduces sensitivity to alternative formulations of the same instruction. It evaluates fixed checkpoints across paraphrased instructions and finds that robustness effects vary with scale, training instruction, and model, while prediction and scoring protocols can change the measured conclusion.

  • Problem

    It remains unclear whether conventional task-specific SFT consistently reduces instruction sensitivity and whether its effect depends on the instruction used during adaptation.

  • Method

    The study performs controlled pre–post SFT evaluations across paraphrased instructions, Qwen3 scales, selected model families, and alternative prediction and scoring protocols.

  • Results

    SFT reduces sensitivity at Qwen3-1.7B and 4B, shows training-instruction contrasts at 8B and Gemma-2-9B but not Mistral-7B, and yields protocol-dependent robustness conclusions on ESCI-English.

  • Takeaways & Limitations

    SFT should not be assumed to uniformly reduce instruction sensitivity; robustness should be evaluated across paraphrases and controlled prediction protocols rather than a single instruction or average performance.

  • Takeaways & Limitations

    The systematic scale analysis covers only three Qwen3 sizes, while cross-family checks are targeted rather than a full model-family-by-scale analysis.

Abstract

from arXiv · show

Large language models can exhibit substantial performance variation across alternative formulations of the same task instruction, yet it remains unclear how conventional task-specific supervised fine-tuning (SFT) changes this instruction sensitivity. We study this question by evaluating fixed model checkpoints under multiple paraphrased instructions and defining instruction sensitivity as the standard deviation of task performance across them. We conduct a controlled scale analysis with Qwen3 models at 1.7B, 4B, and 8B on MS MARCO, together with targeted cross-family checks using Mistral-7B and Gemma-2-9B. Before SFT, instruction sensitivity decreases sharply with Qwen3 model scale. At 1.7B and 4B, SFT consistently reduces sensitivity across training instructions, with reductions of approximately 54--71%. At 8B, individual sensitivity changes are not statistically distinguishable from zero, but paired contrasts between training instructions are statistically reliable under query-level bootstrap analysis and have consistent directions across all three random seeds. Gemma-2-9B shows the same directional training-instruction contrast as Qwen3-8B, whereas Mistral-7B does not, suggesting that the strength of this effect also varies across models. Experiments on ESCI-English further show that free-generation and likelihood-based forced-choice evaluation can yield qualitatively different robustness conclusions even when valid-label generation is nearly perfect and average task performance is similar. Overall, SFT does not uniformly reduce instruction sensitivity: its robustness effect depends on the adaptation setting, while measured sensitivity can additionally depend on the prediction and scoring protocol.

1 Introduction

This paper examines whether conventional task-specific SFT changes sensitivity to paraphrased instructions and whether the effect depends on training instruction, model scale, and evaluation protocol. Controlled experiments find that SFT can reduce sensitivity at smaller Qwen3 scales, but its robustness effect is not uniform and measured conclusions can vary with prediction and scoring.

  • Motivation: Instruction sensitivity remains unclear under conventional task-specific SFT, which typically uses one fixed instruction rather than explicit robustness training.Existing robustness approaches often introduce instruction diversity, paraphrases, contrastive objectives, or prompt augmentation.
  • Approach: The study evaluates fixed checkpoints across paraphrased instructions and defines instruction sensitivity as the standard deviation of task performance across them.The primary scale analysis uses Qwen3 models at 1.7B, 4B, and 8B, with Mistral-7B and Gemma-2-9B as cross-family checks.
  • Results: At 1.7B and 4B, SFT reliably reduces sensitivity to unseen instruction paraphrases under both tested training instructions.The MS MARCO analysis reports a clear scale-dependent pattern within Qwen3.
  • Results: At 8B, individual sensitivity changes are not statistically distinguishable from zero, but paired contrasts between training instructions are reliable and consistent across all three random seeds.Gemma-2-9B shows the same directional contrast, whereas Mistral-7B does not.
  • Evaluation: Free-generation and likelihood-based forced-choice evaluation yield qualitatively different robustness conclusions despite similar average performance and valid-label generation above 99.8%.This discrepancy cannot be explained simply by failures to produce the required output format.

2 Related Work

Prior work studies prompt sensitivity, instruction-diverse robustness methods, and interactions between prompt choice and parameter adaptation. This paper instead isolates within-model changes from conventional fixed-instruction SFT across paraphrased instructions, scales, and model families.

  • Prompt Sensitivity: Prior studies show that prompt wording, formatting, specificity, and alternative task descriptions can change performance even when the intended task is unchanged.Such variation motivates evaluating multiple instruction formulations rather than relying on a single prompt.
  • Open Gap: Larger models and supervised or instruction-tuned models have been associated with lower prompt sensitivity, but these comparisons do not reveal within-model changes caused by task-specific adaptation.The unresolved question is whether SFT consistently reduces sensitivity and whether its effect depends on the adaptation instruction.
  • Robustness Methods: Instruction-robustness methods commonly use diverse instructions, contrastive objectives, prompt augmentation, or specialized training data.Examples include broad instruction tuning, CoIN, SELF-GUIDE, and PAFT.
  • Prompt–Adaptation Interaction: Research on prompt choice and parameter adaptation reports that prompt formulation can interact with fine-tuning and that adaptation to one prompt can impair generalization to alternatives.Prompt optimization and prompt-diverse fine-tuning address related but different objectives.
  • Positioning: This work uses one fixed training instruction per SFT run, evaluates every resulting model on the same non-overlapping paraphrase set, and studies robustness rather than maximum task performance.The design compares the same model before and after SFT across scale and training-instruction choices.

3 Method

The method measures instruction sensitivity for fixed checkpoints across semantically equivalent instruction variants, then quantifies pre–post SFT changes and contrasts the effects of different training instructions. The measures separate robustness changes from average task-performance changes and preserve checkpoint-level seed structure.

  • 3.1 Instruction Sensitivity: Instruction Sensitivity S(θ) is the sample standard deviation of task performance across K instruction variants for a fixed model checkpoint.Each variant expresses the same task with a different natural-language formulation, while the same task instances are evaluated.
  • 3.1 Instruction Sensitivity: Lower S(θ) indicates more stable performance across instruction formulations, whereas higher S(θ) indicates greater dependence on instruction wording.The controlled variants preserve task semantics and output space.
  • 3.1 Instruction Sensitivity: The sensitivity measure varies only the evaluation instruction for each fixed checkpoint and aggregates across seeds only after computing checkpoint-level sensitivity.It is distinct from variation across random seeds or training runs.
  • 3.2 SFT-Induced Change: ΔS_T = S_T post − S_pre measures the change in instruction sensitivity after SFT with training instruction T.Negative values indicate more consistent performance across variants; positive values indicate increased dependence on instruction formulation.
  • 3.2 SFT-Induced Change: ΔS_T captures robustness changes rather than task-metric improvement, because SFT can raise average performance while either decreasing or increasing sensitivity.A value near zero indicates little change in instruction sensitivity.
  • 3.3 Training-Instruction Dependence: D(T_a,T_b) = ΔS_Tb − ΔS_Ta compares how two training instructions change post-SFT sensitivity relative to the common pretrained baseline.A positive value means T_b produces greater sensitivity than T_a, while a negative value means the opposite.
  • 3.3 Training-Instruction Dependence: The paired contrast can be reliable even when the individual SFT-induced changes are not statistically distinguishable from zero.T_A and T_B form the primary comparison, with T_C additionally evaluated for Qwen3-8B.

4 Experimental Setup

The experiments compare instruction sensitivity across retrieval tasks, model scales, training instructions, and evaluation protocols using controlled SFT settings. MS MARCO provides the main robustness analysis, while ESCI-English tests how prediction protocols affect measured robustness.

  • Models and tasks: MS MARCO measures passage-ranking robustness, while ESCI-English provides a complementary binary product-relevance setting for evaluating output protocols.MS MARCO is the primary benchmark for scale, training-instruction, and family comparisons; ESCI-English focuses on evaluation-protocol effects.
  • Training and evaluation instructions: MS MARCO compares three fixed training instructions whose length and relevance-criterion explicitness vary jointly, so effects are attributed to conditions rather than one linguistic property.T_A and T_B form the primary comparison across models, while T_C is additionally tested for Qwen3-8B.
  • Models and tasks: Qwen3, Mistral-7B, and Gemma-2-9B are evaluated, with Qwen3 providing the primary scale-controlled comparison at 1.7B, 4B, and 8B parameters.Using Qwen3 as the main family minimizes architectural differences when examining scale; the other models provide targeted cross-family checks.
  • Robustness measurement: Instruction sensitivity is the sample standard deviation of task performance across ten task-equivalent evaluation instructions, computed separately by seed after SFT and averaged across three seeds.The MS MARCO evaluation instructions are distinct from all three training instructions, preventing exact instruction overlap.
  • Evaluation protocols: MS MARCO ranks passages with the likelihood difference between Relevant and Irrelevant labels and evaluates effectiveness using nDCG@10.The same likelihood-based scoring procedure is applied across model families and training-instruction conditions.
  • Evaluation protocols: ESCI-English contrasts deterministic free generation with forced-choice selection by full conditional label likelihood and reports valid-label rate for the generation protocol.Free generation maps case-sensitive output prefixes to labels, whereas forced choice directly selects the more likely label.
  • Statistical analysis: All MS MARCO comparisons use 10,000-replicate paired query-level bootstrap resampling that preserves shared query indices across conditions and seeds.Sensitivity is recomputed within each replicate before forming changes and contrasts.

5 Experimental Results

Across Qwen3, SFT’s effect on instruction sensitivity depends on model scale and training instruction, while evaluation protocol can change the apparent robustness pattern. Smaller models show reliable sensitivity reductions, but Qwen3-8B exhibits training-instruction dependence and ESCI-English yields different conclusions under free generation versus forced-choice scoring.

  • Before SFT, Qwen3 instruction sensitivity decreases from 0.0905 at 1.7B to 0.0332 at 4B and 0.0126 at 8B.
  • 5.2 SFT-Induced Changes in Instruction Sensitivity: 70.7% and 67.0% reductions at 1.7B, and 57.1% and 54.3% reductions at 4B, show that SFT compresses sensitivity under both training instructions.
  • 5.3 Training-Instruction Dependence and Cross-Model Comparison: At 8B, SFT changes are individually uncertain, but the paired contrast is 0.0103 with a 95% confidence interval of [0.0021, 0.0173] and P(D > 0) = 0.996.
  • 5.3 Training-Instruction Dependence and Cross-Model Comparison: The Qwen3-8B training-instruction contrast remains positive across all three seeds and all six leave-one-seed-out analyses, while Gemma-2-9B shows the direction without excluding zero and Mistral-7B shows no corresponding contrast.
  • 5.4 Evaluation Protocol and Apparent Instruction Sensitivity: The protocol discrepancy persists despite similar average accuracy and a mean valid-label rate of 0.9983, indicating dependence on prediction and scoring procedure rather than output formatting alone.
  • 5.4 Evaluation Protocol and Apparent Instruction Sensitivity: On ESCI-English, free generation increases sensitivity from 0.0121 to 0.0141 and 0.0171, whereas forced choice changes it from 0.0136 to 0.0119 and 0.0138.

6 Discussion

The discussion shows that task effectiveness and instruction robustness can diverge, and that robustness conclusions depend on model, training instruction, and evaluation protocol. It also identifies scope and inference limitations in the current experiments.

  • Task performance and robustness: At 1.7B and 4B, SFT increases mean nDCG@10 while substantially reducing instruction sensitivity, but this alignment does not consistently hold at 8B.Under T_A, mean nDCG@10 rises from 0.3745 to 0.5244 at 1.7B and from 0.6383 to 0.7240 at 4B.
  • Task performance and robustness: At 8B, individual sensitivity changes are not statistically distinguishable from zero, although paired contrasts between training instructions are statistically reliable.Under T_B, sensitivity increases from 0.0126 to 0.0189 while mean nDCG@10 improves from 0.7096 to 0.7293; under T_A, sensitivity decreases to 0.0086 as performance improves to 0.7405.
  • Cross-model variation: Gemma-2-9B matches Qwen3-8B’s directional training-instruction contrast, whereas Mistral-7B shows no corresponding contrast.The Gemma confidence interval includes zero, and the results indicate that robustness effects depend on adapted-model properties beyond parameter scale.
  • Evaluation protocol: Free-generation evaluation suggests increased post-SFT sensitivity on ESCI-English, whereas likelihood-based forced choice does not, despite similar average performance and over 99.8% valid-label generation.The discrepancy is therefore not simply attributable to output-format failures.
  • Evaluation protocol: Robustness estimates from different prediction or scoring procedures should not be assumed interchangeable, although all MS MARCO comparisons use a consistent likelihood-based protocol.This consistency supports interpreting the MS MARCO scale, training-instruction, and cross-model differences under one scoring procedure.
  • Limitations: The systematic scale analysis covers only three Qwen3 sizes, while cross-family checks do not establish a full model-family-by-scale pattern.Accordingly, Qwen3-8B should not be interpreted as a universal transition point.
  • Limitations: The study uses limited training instructions and ten evaluation paraphrases, leaving the properties driving different robustness outcomes unresolved.The three Qwen3-8B conditions also do not establish whether T_A is exceptional or part of a broader pattern.
  • Limitations: Three seeds are insufficient for strong inference about variability across training runs, despite consistent Qwen3-8B contrast directions and reliable leave-one-seed-out query bootstraps.The statistical analysis primarily quantifies query-level uncertainty while treating trained seeds as observed runs.

7 Conclusion

The study finds that conventional SFT does not uniformly improve robustness to paraphrased instructions: effects vary across scale, training instruction, and model. Evaluation protocol can also change the apparent sensitivity pattern without materially changing average task performance.

  • Conclusion: Within Qwen3, instruction sensitivity is lower at larger pretrained scales, while SFT reliably reduces sensitivity at 1.7B and 4B but yields training-instruction dependence at 8B.Cross-family checks further show the Qwen3-8B directional contrast in Gemma-2-9B but not Mistral-7B.
  • Conclusion: Average task performance and instruction robustness need not improve together after task-specific SFT.The conclusion treats these as distinct properties of fine-tuned models.
  • Conclusion: Free-generation and forced-choice evaluation produce qualitatively different sensitivity patterns despite similar average performance and nearly perfect valid-label generation.The evaluation procedure therefore changes the robustness conclusion without materially changing task-effectiveness conclusions.
  • Conclusion: Assessing fine-tuned models across multiple paraphrases and controlled prediction protocols is more complete than relying on one instruction or average performance alone.The supported scope includes variation across model scale, training instruction, model, and evaluation protocol.

Ethical Considerations

The study analyzes pretrained language models on public benchmarks without human participants, personally identifiable information, or decisions about individuals. It also limits computational demands through relatively small and medium-sized models and parameter-efficient fine-tuning.

  • Scope and data: The study uses publicly available benchmark datasets and pretrained open-weight models, without human participants or personally identifiable information.It does not introduce new user-facing systems or make decisions about individuals.
  • Research purpose: The analysis is intended to improve understanding and evaluation of instruction sensitivity rather than exploit it.The paper reports robustness across multiple instruction formulations and examines evaluation artifacts that could mislead conclusions.
  • Resources and data: Computational cost is reduced by restricting experiments to relatively small and medium-sized models and using parameter-efficient fine-tuning.No proprietary or private user data are used.
Loading 2608.26661v1…