PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

Melanie Sclar, Yejin Choi, Yulia Tsvetkov, Alane Suhr

arXiv:2310.11324v2cs.CLcs.AIcs.LG

TL;DR

The paper asks whether meaning-preserving prompt-format choices materially affect LLM evaluation. It defines equivalent formats, measures their performance spread, and introduces FORMATSPREAD for efficient estimation; it finds large, persistent sensitivity and recommends spread-aware evaluation, while noting single-format use remains valid for some systems.

  • Problem

    Prompt-formatting choices are rarely characterized or reported, despite potentially influencing performance and confounding comparisons attributed to models or training.

  • Method

    FORMATSPREAD samples plausible, semantically equivalent formats and uses budgeted bandit search to estimate performance spread without requiring model weights.

  • Results

    Format sensitivity remains large across models, model sizes, few-shot counts, and instruction tuning, reaching up to 76 accuracy points for LLaMA-2-13B.

  • Takeaways & Limitations

    Prompt-based evaluations should report performance ranges across plausible formats, especially for model comparisons because format effects correlate only weakly between models.

  • Takeaways & Limitations

    Single-format evaluation can remain sufficient for researchers or practitioners building larger systems, where a sufficiently effective format is a valid methodological choice.

Abstract

from arXiv · show

As large language models (LLMs) are adopted as a fundamental component of language technologies, it is crucial to accurately characterize their performance. Because choices in prompt design can strongly influence model behavior, this design process is critical in effectively using any modern pre-trained generative language model. In this work, we focus on LLM sensitivity to a quintessential class of meaning-preserving design choices: prompt formatting. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B. Sensitivity remains even when increasing model size, the number of few-shot examples, or performing instruction tuning. Our analysis suggests that work evaluating LLMs with prompting-based methods would benefit from reporting a range of performance across plausible prompt formats, instead of the currently-standard practice of reporting performance on a single format. We also show that format performance only weakly correlates between models, which puts into question the methodological validity of comparing models with an arbitrarily chosen, fixed prompt format. To facilitate systematic analysis we propose FormatSpread, an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights. Furthermore, we present a suite of analyses that characterize the nature of this sensitivity, including exploring the influence of particular atomic perturbations and the internal representation of particular formats.

1 INTRODUCTION

Prompt formatting choices can substantially change LLM performance despite preserving prompt meaning. The paper measures this sensitivity and proposes reporting performance spreads across plausible formats rather than relying on one arbitrary format.

  • Up to 76 accuracy points separate equivalent formats for LLaMA-2-13B, with approximately 10 accuracy points of variation on average across 50+ tasks and several models.
  • Prompt-template choices are often unreported because their performance variance is assumed insignificant, despite evidence that formatting can influence model behavior.
  • Ignoring format variance can distort conclusions attributed to training data, parameter count, or architecture and may cause users to select poorly performing formats.
  • FORMATSPREAD evaluates a broad set of semantically equivalent formats within a user-specified computational budget.
  • Format effects only weakly correlate between models, making a fixed format a potential confound when comparing model performance.

2 OVERVIEW

The paper defines plausible, meaning-equivalent prompt formats and analyzes sensitivity across them. It also examines which formatting features and internal representations relate to observed performance variation.

  • Plausible formats are defined by a grammar representing choices a non-adversarial user might make when designing a task prompt.
  • Sensitivity is measured as the performance range across formats equivalent to a task’s original format.
  • FORMATSPREAD searches this format space as a bandit problem, while additional analyses quantify atomic-feature contributions and representation-based format identifiability.

3 MEASURING SENSITIVITY WITH FORMATSPREAD

FORMATSPREAD formalizes semantically equivalent prompt formats, measures accuracy spread across them, and uses budgeted bandit search to estimate high- and low-performing formats efficiently.

  • 3.1 GRAMMAR OF PLAUSIBLE PROMPT FORMATS: The manually constructed grammar defines plausible formats and semantic equivalence, and is tested against formatting from 100+ Super-NaturalInstructions tasks.
  • 3.1 GRAMMAR OF PLAUSIBLE PROMPT FORMATS: Basic fields combine descriptors, separators, casing functions, and text placeholders to build prompt formats such as ‘Passage: <text> || Answer: <text>’.
  • 3.1 GRAMMAR OF PLAUSIBLE PROMPT FORMATS: Fields can be joined and enumerated using separators, item labels, number formats, and wrappers such as ‘(A)’ or ‘[A]’.
  • 3.1 GRAMMAR OF PLAUSIBLE PROMPT FORMATS: The grammar permits casing changes while preserving meaning and imposes contextual restrictions to exclude unnatural combinations of spaces and separators.
  • 3.1 GRAMMAR OF PLAUSIBLE PROMPT FORMATS: Two formats are equivalent when they apply the same rule with the same descriptors and equivalent sub-elements; sensitivity is evaluated only within the original format’s equivalence class.
  • 3.2 MEASURING SENSITIVITY: Performance spread is max_i m(p_i,D) − min_i m(p_i,D), and higher spread indicates greater sensitivity among semantically equivalent formats.
  • 3.2 MEASURING SENSITIVITY: Full evaluation scales linearly with the number of formats, so the method models formats as bandit arms and estimates rewards from minibatches.
  • 3.2 MEASURING SENSITIVITY: FORMATSPREAD splits its evaluation budget between searching for the highest- and lowest-performing formats, using Thompson sampling with Bernoulli accuracy rewards.

4 CHARACTERIZING PROMPT FORMAT VARIANCE WITH FORMATSPREAD

Across 53 tasks, semantically equivalent prompt formats produce substantial and often model-specific performance variation. FORMATSPREAD measures this variance efficiently, while analyses show that individual formatting changes can have large, non-monotonic effects and remain identifiable in model representations.

  • 4.1 EXPERIMENTAL SETUP: The evaluation covers LLaMA-2-{7B,13B,70B}, Falcon-7B, Falcon-7B-Instruct, and GPT-3.5-Turbo using exact prefix matching and probability ranking, with ranking accuracy reported by default.The tasks comprise 19 multiple-choice and 34 classification tasks from Super-NaturalInstructions.
  • 4.2 PROMPT FORMATS HAVE A LARGE PERFORMANCE SPREAD, NOT ELIMINATED BY INCREASING FEW-SHOT EXAMPLES OR MODEL SIZE, NOR WITH INSTRUCTION TUNING: 7.5 accuracy points is the median spread across sampled formats, while 20% of tasks show at least 15 points for every LLaMA-2 setting and at least 9 points for every Falcon setting.Some tasks exceed 70 accuracy points, and evaluating only 10 formats makes these estimates lower bounds on the true spreads.
  • 4.1 EXPERIMENTAL SETUP: 0.141 and 0.140 are the probabilities that model-comparison trends reverse by at least d = 0.02 for the LLaMA-2-13B versus -70B and LLaMA-2-7B versus Falcon-7B comparisons, respectively.Both evaluations were often statistically significant: 76% and 47% of the two comparisons, respectively.
  • 4.1 EXPERIMENTAL SETUP: Formats with high performance for one model may not perform well for another, indicating that formats are not inherently good or bad.This model-specificity complicates fixed-format comparisons across models.
  • 4.3 HOW DO INDIVIDUAL FEATURES CONTRIBUTE TO PERFORMANCE?: S1 separators and Fitem2 enumeration-number formats have the most individual impact, whereas S2, Fitem1, and Fcasing do not independently predict performance differences.The analysis used 500 random formats with 250 samples each on 31 tasks for 1-shot LLaMA-2-7B.
  • 4.3 HOW DO INDIVIDUAL FEATURES CONTRIBUTE TO PERFORMANCE?: 24% of atomic changes shift exact-prefix accuracy by at least 5 points, compared with 11% under probability ranking.Thus, changing a single formatting constant can produce large accuracy differences under either scoring criterion.
  • 4.3 HOW DO INDIVIDUAL FEATURES CONTRIBUTE TO PERFORMANCE?: 32.4% and 33.6% of sampled atomic-change triples were monotonic for multiple-choice and non-multiple-choice tasks, respectively, near the 33.3% random-order baseline.This supports the conclusion that the prompt-format accuracy space is highly non-monotonic and locally smooth search may be ineffective.
  • 4.4 PROMPT FORMATS ARE IDENTIFIABLE TRANSFORMATIONS OF PROMPT EMBEDDINGS: Using only the top 100 principal components of 4,096-dimensional prompt embeddings, a classifier achieved at least 0.98 format-identification accuracy on all 31 analyzed tasks.Embeddings came from LLaMA-2-7B prompts across 10 formats and both 1- and 5-shot settings.

5 RELATED WORK

Prior prompt-optimization research largely targets finding high-performing prompts, while this paper studies sensitivity to meaning-preserving formatting choices without requiring model internals.

  • Gradient-based methods can find effective prompts but often produce disfluent or unnatural text.Langevin-dynamics methods have been proposed to mitigate this issue.
  • Continuous prompt methods learn and insert prompt or task representations but require access to model parameters.
  • FORMATSPREAD differs by searching prompt-format spaces without assuming access to model internals, enabling use with API-gated models.
  • Related work examines prompt language, few-shot example ordering and patterns, textual interpretations of continuous prompts, and masked-language-model variance.

6 DISCUSSION

The paper defines semantically equivalent prompt formats, analyzes their performance spread, and uses FORMATSPREAD to search this space efficiently under a computational budget. Its analyses motivate grammar-based format construction and show why local search is not advantageous.

  • 6 DISCUSSION: FORMATSPREAD estimates performance spread across prompt-format choices and evaluates widely used open-source LLMs on few-shot classification tasks.The reported spread remains large across model size, few-shot count, and instruction-tuning conditions.
  • 6 DISCUSSION: Using only 5% of a 320-format search space, FORMATSPREAD estimates the true spread within 2 accuracy points for tasks with 2,500 test examples.
  • 6 DISCUSSION: Format spaces are largely non-monotonic, while few atomic features alone predict performance; format-embedding separability correlates highly with observed spread.These findings inform the search procedure, making local search methods disadvantageous.
  • 6 DISCUSSION: The paper recommends reporting performance ranges across plausible formats when comparing models, while allowing single-format evaluation for systems built around one sufficiently effective format.FORMATSPREAD can also identify lower-bound performance and the range a non-adversarial user might encounter.
  • 6 DISCUSSION: Prompt formats are generated from a grammar whose equivalence relation is defined over structured components, though string-identical formats may not be equivalent under that relation.
  • 6 DISCUSSION: The grammar represents a full prompt as an instruction, n few-shots, and a data point, with formatting variables constrained to preserve naturalness.Restrictions govern separators, newlines, descriptor spacing, and empty-string cases; users may customize additional rules.
  • 6 DISCUSSION: FORMATSPREAD initializes its first exploration with a Beta prior whose expected value equals the original format accuracy, fixing β = 5 to avoid overconfidence.The second exploration uses information from the first exploration’s counters.

B.1 TASK SELECTION

The evaluation uses a filtered subset of Super-NaturalInstructions tasks selected for scale, tractable inference, classification outputs, and dataset diversity.

  • B.1 TASK SELECTION: Tasks needed at least 1,000 samples, instructions under 3,000 characters, inputs under 2,000 characters, and no more than 20 valid output strings.
  • B.1 TASK SELECTION: Tasks with 0% a priori LLaMA-2-7B 1-shot accuracy were removed, and no more than four tasks were retained from the same original dataset.
  • B.1 TASK SELECTION: Socially impactful tasks were specifically sought, and only those tasks received a format when the dataset did not provide one.
  • B.1 TASK SELECTION: The selected task list is enumerated explicitly in the appendix.

B.2 ADDITIONAL RESULTS FOR SECTION 4.2

Additional experiments show that prompt-format sensitivity persists across metrics, models, shot counts, and task types, while format choices can also affect valid-response frequency and output distributions.

  • Spread remains high across model sizes, architectures, and few-shot counts under exact prefix matching.Performance-trend reversals between models also remain common when the metric changes to exact prefix matching.
  • Prompt formats can affect whether models produce valid answers, not only whether those answers are correct.Accuracy and centered mass are moderately or highly correlated, with an almost perfect correlation for Falcon models.
  • Spread remains high across models and shot counts on Instruction Induction generation tasks.These tasks require generating unique valid strings rather than selecting among classification options.
  • 25% of LLaMA-2-7B 5-shot sentence-generation tasks have ROUGE-L spread of 0.098 or higher and BERTScore spread of 0.09 or higher.The pattern persists across both metrics and across the numbers of few-shot examples considered.
  • Text-generation spread can have a smaller median than accuracy-task spread, but metric noise makes some decreases ambiguous as performance losses.The authors leave differences in induced style or content for future work.
  • Accuracy distributions contain variation beyond their range: across 30 tasks, the median standard deviation is σ ≈0.04.Higher standard deviation indicates a greater likelihood of dissimilar outcomes after selecting different formats.
  • Prompt length does not reliably predict performance spread across models and shot counts.Joint correlations are negligible for both exact prefix matching and option ranking, although LLaMA-2-7B shows r = 0.559 for average prompt length with exact prefix matching.

B.3 PCA EXAMPLES

The paper finds that prompt formats have separable internal representations: the top two principal components of prompt embeddings identify the formats for two tasks.

  • The top two principal components of the last decoder-layer prompt representation easily distinguish all 10 formats for each of two tasks.The representation is treated as encoding the output probability distribution.

B.4 NOTABLE FEATURES

Atomic formatting features can alter accuracy ranges, but individual constant choices do not consistently predict performance across tasks and metrics.

  • Constant choices sometimes produce strongly different accuracy ranges across 53 Super Natural-Instructions tasks.The analysis covers both accuracy metrics used throughout the work.
  • Individual constant values do not consistently predict performance in isolation.Figures 22–24 show strongly dissimilar choices across the evaluated tasks and metrics.

B.5 THOMPSON SAMPLING RESULTS

Thompson Sampling was used to search prompt formats efficiently, with reported best–worst accuracy differences across several tasks and models.

  • 320 formats were searched with Thompson Sampling under a budget of 40000 evaluations.LLaMA-2-70B used option ranking, while GPT3.5 used prefix matching because all logits were unavailable.
  • For task280 with Llama-70B, the reported best and worst accuracies are 0.84 and 0.04.The compared formats differ substantially in capitalization, separators, and line structure around passage and answer fields.
  • For task317 with Llama-70B, the reported best and worst accuracies are 0.83 and 0.07.The formats differ in capitalization, punctuation, spacing, and line breaks around passage and answer fields.
  • For task1502 with GPT3.5, the reported best and worst accuracies are 0.54 and 0.48.The table lists the corresponding best and worst formats as input/output variants.
  • For task155 with GPT3.5, the reported best and worst accuracies are 0.48 and 0.40.The compared formats differ in capitalization, punctuation, spacing, and line breaks around sentence and answer fields.
  • For task158 with GPT3.5, the reported best and worst accuracies are 0.63 and 0.60.The compared formats use different capitalization, separators, spacing, and punctuation.

C LIMITATIONS

The format grammar admits ambiguous parses and can generate unusual but semantically equivalent character combinations. The evaluation also focuses on reasonably short instructions and inputs, leaving input-length effects for future work.

  • Equivalent formats can have multiple valid parses, depending on whether punctuation belongs to the passage or the format.The paper uses one parsing consistently to ensure full sentences.
  • The chosen parsing can produce less usual character combinations while preserving semantic equivalence.The paper gives punctuation-adjacent separator combinations as an example.
  • The paper observes high performance spread for these unusual formatting cases and beyond them.Contextual relations may restrict such cases when desired by the end user.
  • The evaluation focuses on tasks with reasonably short input instructions and input field lengths.Future work may investigate how input length affects final performance.
  • A manual inspection found the parsing-related unusual combinations in fewer than 20% of cases across 10 formats and 20 tasks.
Loading 2310.11324v2…