PaperlayerView exact revision on arXiv

Source-linked AI summary

The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives

Henry Lim, Kwan Hui Lim

arXiv:2510.17388v1cs.CL

TL;DR

Atomic instruction-following by instruction-tuned LLMs remains underexplored, despite its foundational role in complex behavior. The paper tests it across modified MMLU and MMLU-Pro prompts with semantically equivalent label formats and controlled changes to instructions, option content, and examples. Models show persistent label-format fragility, and three-shot examples do not significantly improve robustness or fidelity.

  • Problem

    How reliably instruction-tuned LLMs follow simple, self-contained directives remains underexplored, despite atomic adherence underpinning reliable deterministic applications and compositional reasoning.

    Source: §1 · p. 1

  • Method

    The study evaluates 20 instruction-tuned LLMs on modified MMLU and MMLU-Pro benchmarks, varying equivalent option-label formats and prompt components across four experiments.

    Source: §1 · p. 1

  • Results

    Across the benchmarks, semantically equivalent label formats produce persistent accuracy gaps and three-shot examples yield no statistically significant gains in robustness or fidelity; numeric prompts average 63.06% on MMLU.

    Source: §4.1 · p. 4, §7 · p. 8

  • Takeaways & Limitations

    Atomic-instruction failures persist across model scales and are not resolved by iterative instruction-tuning or few-shot prompting.

    Source: §1 · p. 2

    The study covers only multiple-choice tasks, uses 4-bit quantization, samples 5% of the data for 70B models, and excludes proprietary systems.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Instruction-tuned large language models (IT-LLMs) exhibit strong zero-shot reasoning, yet their ability to execute simple, self-contained instructions remains underexplored, despite this being foundational to complex instruction-following. We evaluate 20 IT-LLMs on modified MMLU and MMLU-Pro benchmarks, by systematically varying the format of option labels (alphabetic, numeric, Roman) while keeping their meaning identical under four paradigms, namely: (1) With explicit instructions, label changes cause large performance shifts (e.g., -30.45\% for Roman vs. numeric), revealing instruction-format bias. (2) Without instructions, performance drops further (up to -10.84\%) and label sensitivity intensifies, underscoring the role of explicit guidance. (3) When option contents are removed, models fail random-choice baselines except with numeric labels, suggesting weak adherence to atomic directives. (4) Three-shot exemplars yield no significant gains in robustness or fidelity, and generation analyses show persistent label errors, especially for non-numeric formats. Across model sizes, larger LLMs achieve higher accuracy but remain inconsistent in instruction adherence. These results expose the insufficiencies of current instruction-tuning paradigms and highlight the need for evaluation methods and training strategies that explicitly target atomic instruction-following.

1 Introduction

The ability of instruction-tuned LLMs to follow simple, unambiguous directives remains underexplored, despite its role in reliable deterministic applications and compositional reasoning. The paper evaluates this ability by varying semantically equivalent multiple-choice label formats and prompt components.

1 Introduction

  • Atomic instruction-following remains underexplored, although it underpins reliable performance in deterministic applications and compositional reasoning.Prior work has focused on multi-step or uncommon instructions rather than simple, unambiguous directives.Source: §1 · p. 1
  • 30.44% drop for Roman versus numeric labels on MMLU exposes performance sensitivity to semantically equivalent option formats.The experiments also remove instructions and option content, add three-shot examples, and analyze whether generated answers use the instructed label set.Source: §1 · p. 1, Figure 1 · p. 2
  • With option content removed, models approach chance only with numeric labels; alphabetic and Roman labels score 13.90% and 12.97% versus MMLU’s 25% baseline.The paper also reports that three-shot examples do not resolve failures and that models often generate labels outside the instructed set.Source: §1 · pp. 1–2
  • The paper reports statistically significant drops from label-style changes and persistent fragility despite model scaling, iterative instruction-tuning, and few-shot examples.It also traces symbolic bias to base models and reports below-chance performance when task content is removed.Source: §1 · pp. 1–2

2 Related Work

Prior instruction-following benchmarks emphasize complex or multi-step constraints, while prompt-sensitivity work studies phrasing and option-format effects. This paper isolates sensitivity to semantically neutral instruction changes.

Instruction-Following Evaluation

  • Existing instruction-following benchmarks primarily test compositional or multi-step instructions, leaving atomic instruction handling less understood despite its foundational role.The paper contrasts its focus with benchmarks such as Super-NaturalInstructions, AlpacaEval, and IFEval, which assess diverse tasks or fine-grained constraints.Source: §2 · p. 2

Prompt Sensitivity and Robustness

  • Prior multiple-choice sensitivity work focuses on task-driven changes such as misleading distractors, whereas this study isolates instruction-driven changes to labels or phrasing.Earlier work also documents sensitivity to option positions or formats (Wang et al., 2023).Source: §2 · pp. 2–3
  • The study finds that even trivial variations (i.e., label shifts) disrupt performance, and iterative improvements (e.g., Qwen’s versions) do not guarantee robustness to atomic instruction variations.Evaluation uses around 14,000 MMLU and 12,000 MMLU-Pro samples, versus approximately 500 manually constructed prompts in Zhou et al. (2023a).Source: §2 · pp. 2–3

3 Methodology

The methodology evaluates instruction-following through controlled multiple-choice prompts that vary label formats and other prompt components. It uses exact-match accuracy and a separate measure of whether generated answers conform to the instructed label set.

3.1 Model Selection and Decoding

  • 25 models, including 20 instruction-tuned models, span 0.5B–72B parameters and use 4-bit quantization with 0-shot greedy decoding.Experiments on 70B-class models use 5% of each dataset’s samples.Source: §3.1 · p. 3

3.2 Datasets

  • MMLU covers 57 subjects, while MMLU-Pro adds more challenging questions and increases the number of answer options from four to ten.The study modifies both benchmarks into instruction–question–options prompts.Source: §3.2 · p. 3

3.3 Prompt Setups

  • The prompt experiments compare alphabetic, numeric, and Roman labels, then remove instructions, remove option content, or add three-shot format examples.Across label formats, the semantic content remains constant; the three-shot examples precede the target question.Source: §3.3 · p. 3

3.4 Evaluation Metrics

  • Accuracy requires an exact match to the correct option symbol, and benchmark accuracy averages subsection accuracies.Table 1 reports accuracy, format spread, and failures below the random baselines of 25% for MMLU and 10% for MMLU-Pro; Table 2 defines instruction-removal accuracy and range changes.Source: §3.4 · p. 4, Table 1 · p. 4, Table 2 · p. 4

3.5 Analysis of Generated Answers

  • Label fidelity measures the proportion of generated responses within the acceptable label set, regardless of factual correctness.The analysis covers both 0-shot and 3-shot settings.Source: §3.5 · p. 4

4 Results and Analysis

Across MMLU and MMLU-Pro, semantically equivalent label formats produce large accuracy differences, and removing instructions or option content exposes further fragility. Three-shot examples do not reliably resolve the inconsistency.

4.1 Main Results: Atomic Instruction Failure in Instruction-Tuned LLMs

  • Numeric prompts achieved 63.06% mean accuracy on MMLU and 35.50% on MMLU-Pro; alphabetic and Roman formats were significantly lower (p < 0.05).Five to ten models fell below random baseline on the alternative formats.Source: §4.1 · p. 4, Table 3 · p. 5
  • Qwen-72B reached 84.06% peak accuracy yet varied by up to 7.81 percentage points across formats, while Mistral improved without scaling but still lagged same-size Qwen models.Phi-3 models remained near 0% on alternative formats regardless of size.Source: §4.1 · pp. 4–5
  • LLaMA-70B showed a 72.20pp format range, and 10+ models scored below random chance on Roman labels despite explicit directives.The authors attribute the divergence to implicit numeric-format priors developed through instruction tuning.Source: §4.1 · p. 5

4.2 Instruction-Tuning Benefits and Limitations

  • Removing instructions significantly harmed five of six label-format conditions; Roman-label losses reached -10.84% on MMLU and -8.51% on MMLU-Pro.Without instructions, average min-max ranges expanded by 8.97% on MMLU and 5.96% on MMLU-Pro.Source: §4.2 · p. 5
  • Even with explicit instructions, label-format gaps persist, reflecting symbolic bias that instruction tuning does not overcome.The authors conclude that prompt design remains important for instruction-dependent tasks.Source: §4.2 · pp. 5–6

4.3 A Fundamental Failure: Instruction Following Without Content

  • With option contents removed, numeric labels reached chance-level MMLU accuracy (23.34% against a 25% baseline), while alphabetic and Roman formats fell below chance.On MMLU-Pro, numeric labels averaged 10.23% against a 10% baseline; alphabetic and Roman labels averaged 4.94% and 5.21%.Source: §4.3 · p. 6

4.4 The Ineffectiveness of Few-Shot Learning

  • Three-shot examples yielded gains in 15 of 20 models, but improvements were inconsistent and only 12 models reduced format variability.The table summarizes accuracy changes and variability across label formats on MMLU-Pro.Source: Table 4 · p. 5, §4.4 · p. 6
  • Across label formats, three-shot examples produced no statistically significant accuracy improvements in paired tests, leaving atomic instruction-following unresolved.All Bonferroni-corrected p-values exceeded 0.05.Source: §4.4 · p. 6

5 Analysis of Generated Answers: Label Fidelity and Bias

Generated-label analyses find that acceptable output remains sensitive to label style, with numeral responses favored and few-shot demonstrations providing only modest, inconsistent gains.

5.1 Label Fidelity

  • Three-shot prompting slightly increased acceptable-label generation, but the change was not significant and substantial variation across models and formats remained.On MMLU-Pro, the Min–Max range decreased but remained 45.01%.Source: §5 · p. 6, Table 5 · p. 7, §5.1 · p. 7
  • The acceptable-output Min–Max range narrowed from 54.69% to 45.01%, yet some models still differed by over 80 percentage points across label styles.Numeric labels were handled robustly by most models, unlike letter-based and Roman labels.Source: §5.1 · p. 7

5.2 Error Analysis: Preference for Numeral Responses

  • Allowing numeral responses alongside intended alphabetic or Roman labels raised the number of models above 80% acceptable outputs from 7 to 13 in 0-shot and 6 to 15 in 3-shot.The authors interpret this as a strong bias toward numeral-style responses despite contrary instructions.Source: §5.2 · p. 7

5.3 Effects of Model Family and Scale

  • Qwen label adherence rose from 7.07% for its 3B model to 84.73% for its 7B variant, while scaling did not ensure uniform fidelity in LLaMA models.LLaMA 3.1 70B achieved 10.61% on Roman labels; Mistral, Phi-3, and OLMo-2 also varied by label type.Source: §5.3 · p. 7

5.4 Ability to Learn from Examples

  • Of 20 models, 12 increased acceptable outputs and 10 reduced variability with demonstrations, but benefit did not rise monotonically with model size.Within Qwen, only the 1.5B, 3B, 32B, and 72B variants benefited from demonstrations.Source: §5.4 · p. 8

6 Ablation: Effect of Instruction Tuning vs. Base Models

Instruction tuning generally improves mean accuracy but does not consistently reduce sensitivity to option-label formats.

6 Ablation: Effect of Instruction Tuning vs. Base Models

  • Instruction tuning improved mean accuracy in 9 of 10 instances, but reduced label-format sensitivity in only 5 of 10.The comparison covers instruction-tuned and corresponding base models across label formats.Source: §6 · p. 8

7 Conclusion

IT-LLMs often rely on surface patterns rather than treating atomic instructions as abstract operations. The conclusion reports persistent label infidelity despite explicit instructions and three-shot examples, and argues that robust adherence requires targeted optimization.

7 Conclusion

  • Numeric labels consistently outperform semantically equivalent Roman or alphabetic labels, while explicit instructions do not close the format gap.The conclusion also states that stronger task performance does not entail better instruction-following.Source: §7 · p. 8
  • Three-shot exemplars yield no statistically significant gains in adherence or robustness, with only marginal mitigation of label errors.Source: §7 · p. 8
  • Label infidelity persists across model scales and families, indicating that scaling alone does not ensure robust atomic instruction-following.Source: §7 · p. 8
  • The authors argue that robust instruction adherence requires explicit optimization or dedicated tuning, rather than emerging from scaling.Source: §7 · p. 8

Limitations

The study is limited to multiple-choice tasks and publicly available open-weight models, with 4-bit quantization and reduced sampling for 70B-class experiments.

Limitations

  • Experiments cover only multiple-choice tasks, a controlled setting the authors consider sufficient for this preliminary study.They identify open-ended and multi-turn tasks as future extensions.Source: Limitations · p. 9
  • All models use 4-bit quantization, and 70B-class experiments use only 5% of the dataset because of hardware constraints.Source: Limitations · p. 9
  • Cost limits the study to publicly available open-weight models, excluding proprietary systems that have demonstrated stronger instruction-following performance.Source: Limitations · p. 9

B Generated Output Numeral

The supplied table captions identify model-version coverage and generated-label proportions across two prompting settings.

B Generated Output Numeral

  • Table 7 lists model versions and marks models whose non-instruction-tuned versions were also used.Source: Table 7 · p. 10
  • Table 8 reports the proportion of generated labels that are numerals or the supposed alphabetical or Roman label styles.Source: Table 8 · p. 11
  • Table 8 separates results into 0-shot and 3-shot settings.Source: Table 8 · p. 11
Loading 2510.17388v1…