PaperlayerView exact revision on arXiv

Source-linked AI summary

What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering

Federico Errica, Giuseppe Siracusano, Davide Sanvito, Roberto Bifulco

arXiv:2406.12334v4cs.LGcs.SE

TL;DR

Developers need a way to quantify prompt sensitivity beyond accuracy-based evaluation, which depends on ground-truth labels. The paper introduces sensitivity and consistency metrics for classification and finds that they provide distinct information and help identify problematic samples for prompt improvement.

  • Problem

    Developers lack a practical way to quantify prompt sensitivity at individual LLM steps, while accuracy-only estimates require enough ground-truth labels.

    Source: §1 · p. 2

  • Method

    The paper proposes sensitivity to prediction changes under prompt rephrasings and consistency of predictions across examples of the same class as diagnostics complementary to accuracy.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Sensitivity and consistency convey different information, and help identify different sets of problematic samples for prompt improvement.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Sensitivity and consistency add criteria about prediction behavior alongside accuracy for guiding prompt-engineering decisions.

    Source: §6 · p. 8

    The metrics apply only to classification, and their estimates trade off quality against sampler bias, rephrasing count, and query cost.

    Source: §7 · pp. 8–9

Abstract

from arXiv · show

Large Language Models (LLMs) changed the way we design and interact with software systems. Their ability to process and extract information from text has drastically improved productivity in a number of routine tasks. Developers that want to include these models in their software stack, however, face a dreadful challenge: debugging LLMs' inconsistent behavior across minor variations of the prompt. We therefore introduce two metrics for classification tasks, namely sensitivity and consistency, which are complementary to task performance. First, sensitivity measures changes of predictions across rephrasings of the prompt, and does not require access to ground truth labels. Instead, consistency measures how predictions vary across rephrasings for elements of the same class. We perform an empirical comparison of these metrics on text classification tasks, using them as guideline for understanding failure modes of the LLM. Our hope is that sensitivity and consistency will be helpful to guide prompt engineering and obtain LLMs that balance robustness with performance.

1 Introduction

Minor prompt variations can silently disrupt LLM predictions, making their behavior difficult to debug. The paper proposes complementary diagnostic metrics and tests how they can inform prompt analysis and improvement.

1 Introduction

  • Minor prompt edits, such as adding an article to a label definition, can drastically change predictions while leaving developers unaware of the cause.The programmer might expect changes to label descriptions or ordering to leave results unchanged.Source: §1 · p. 1, Figure 1 · p. 2
  • Accuracy alone offers limited help for developers seeking to quantify prompt sensitivity, especially when ground-truth labels are insufficient or individual LLM steps need analysis.The authors call for a computationally feasible way to assess steps that may occur within multi-step systems.Source: §1 · p. 2
  • The paper proposes sensitivity, which does not require ground-truth labels, and consistency across examples of the same class as diagnostics complementary to accuracy.The metrics target prediction changes under prompt variation and differences across same-class examples.Source: §1 · p. 2
  • Empirical tests find that the metrics convey different information, and their use helps identify problematic samples for prompt improvement.The authors also qualitatively analyze datasets in a manner intended to mimic an LLM engineer debugging a system.Source: §1 · pp. 2–3

2 Related Work

Related work addresses prompt-driven performance changes, output uncertainty, and prompt optimization. The paper distinguishes its focus on prompt sensitivity and prediction consistency from these directions.

Spurious Features

  • Studies of spurious prompt features report performance changes with prompt likelihood, few-shot example order, and prompt format, while other work predicts performance under prompt variations.Related work also connects prompt features to security vulnerabilities and evaluates semantic-preserving transformations against accuracy changes.Source: §2 · p. 3

Uncertainty Quantification

  • Uncertainty research examines differing outputs under the same prompt, confidence estimates, voting, calibration, user trust, and information-theoretic distinctions.The paper excludes output uncertainty from its scope and notes that pseudo-deterministic behavior limits some probability-based uncertainty metrics.Source: §2 · p. 3

Prompt Optimization

  • Prompt optimization includes calibration and LLM-based prompt search, but automatic optimizers can struggle to identify error causes.The authors hope their work will inspire prompting strategies beyond evaluating task accuracy alone.Source: §2 · p. 3

3 Methodology

The method averages class predictions over semantically equivalent prompts, then uses sensitivity and consistency to characterize prompt-related variation. The metrics capture different aspects of behavior and have distinct interpretive limits.

3 Methodology

  • The method treats semantically equivalent prompts as a task-dependent, nonuniform distribution and defines class predictions averaged over that distribution.Because the prompt distribution is unknown, the average is approximated using a sampler over Q equivalent prompts; this work uses an LLM to generate rephrasings.Source: §3 · p. 4
  • Sensitivity is normalized entropy of the prompt-averaged class distribution, with expected sensitivity averaging this measure across samples.It requires no ground-truth labels and need not correlate with task performance.Source: §3 · p. 4
  • The experiments use semantically equivalent prompt rephrasings, whose scope can range from minor wording changes to different prompting strategies.The interpretation of sensitivity depends on the use case.Source: §3 · p. 4
  • Consistency measures the Total Variation Distance between prompt-averaged prediction distributions for two samples of the same class, then averages across such pairs.A consistent LLM produces similar distributions across samples of that class.Source: §3 · p. 4
  • When sensitivity is zero, consistency reflects inherent task difficulty and is expected to be most useful when sensitivity is sufficiently high, such as 0.1.High consistency with nonzero sensitivity can mean prompt rephrasings produce similar mistakes across same-class samples; inconsistency can instead point to the classifier rather than the prompt.Source: §3 · pp. 4–5
  • The authors suggest selecting LLMs with low sensitivity and high consistency to reduce production unpredictability.In their notation, this corresponds to Sτ →0 and Cy →1 for each class y.Source: §3 · p. 5

4 Experiments

The experiments are designed to demonstrate how sensitivity and consistency can help analyze English-language text classification, rather than to benchmark many datasets and models. They compare four LLMs across five datasets and three prompting strategies.

4 Experiments

  • The study targets English-based text classification and explicitly does not attempt an extensive benchmark across many datasets and LLMs.Its stated goal is to show developers how to use sensitivity and consistency to analyze prompt variations and strategies.Source: §4 · p. 5
  • The comparison includes four LLMs—two open-source and two closed-source—and five multiclass text-classification datasets.The datasets are TREC, CB, RTE, DBPedia, and Web of Science; invalid class outputs receive an extra N/A label.Source: §4 · p. 5
  • The experiments compare simple, detailed-class-description, and one-shot prompts, using an LLM to generate rephrasings of task descriptions.They fix Q = 30 after observing no significant result variation from increasing it; the chosen Q balances computational resources and statistical stability.Source: §4 · p. 5

5 Results

Across the tested tasks, sensitivity and consistency reveal distinct behavior that accuracy alone does not capture, while sample- and class-level analyses support targeted prompt debugging.

Quantitative Results

  • -0.07 Pearson correlation between sensitivity and consistency (excluding sensitivity below 0.05) supports their distinct signals; GPT-4o with 1-shot showed excellent performance across all tasks.Across open-source LLMs and prompting strategies, the metrics showed no consistent agreement when sensitivity was sufficiently high. Table 1 demonstrates practical utility rather than an extensive benchmark across datasets and models.Source: §5 · p. 6, Table 1 · p. 6
  • A prompt that works with one LLM may cause instability and worse performance with another; on CB, Detail may be preferable with Mixtral.Source: §5 · p. 6

Sensitivity Analysis

  • Highly sensitive samples can be manually inspected or sent to labelers without ground-truth labels; prompt adjustments targeting them reduced sensitivity and improved classification accuracy.Source: §5 · p. 6
  • Sensitivity was highest for Description, Entity, and Number on TREC, neutral statements on CB, and Artists Building, Animal, and Written Work on DBPedia.For CB, the paper suggests adding few-shot examples or clarifying the neutral class in the prompt.Source: §5 · pp. 6–7

Consistency Analysis

  • Entity had more inconsistent than consistent pairs, with color/other samples differing from mostly-animal samples; Description consistency was mostly below 0.75.The authors suggest that refining Entity consistency may also resolve Description inconsistencies because the classes are often confused.Source: §5 · p. 7, Figure 4 · p. 8
  • On TREC, 1-shot reduced medium consistency but yielded more zero-consistency pairs than omitting Simple or Detail; on CB, Detail increased sensitivity for some classes.The authors caution that prompting-strategy effects can be counterintuitive and that qualitative results may be more informative than mean values.Source: §5 · pp. 7–8

6 Conclusions

The paper presents sensitivity and consistency as diagnostics that complement accuracy and can inform prompt-engineering decisions. It also identifies production use and extension beyond classification as areas requiring further consideration.

6 Conclusions

  • Sensitivity and consistency complement accuracy as diagnostic metrics for evaluating LLM predictions under prompt variation.The authors describe these metrics as additions to developers’ toolboxes for guiding prompt-engineering decisions.Source: §6 · p. 8
  • Prompt changes across multiple intermediate production steps may make a highly sensitive, accurate LLM a poor choice.The authors note that such steps can introduce minor prompt alterations.Source: §6 · p. 8
  • Extending sensitivity to tasks beyond classification and integrating the metrics into automatic prompt engineering remain open questions.The authors name code generation as one possible non-classification task.Source: §6 · p. 8

7 Limitations

The proposed metrics are limited to classification, and their approximation involves trade-offs tied to sampling, prompt-rephrasing count, and query cost.

7 Limitations

  • The proposed metrics currently apply only to classification problems.The authors identify extending at least sensitivity to more general problems as future work.Source: §7 · p. 8
  • Figure 5 presents Llama3 consistency across same-class samples, grouped by prompting technique and dataset.Source: Figure 5 · p. 9
  • Approximation quality trades off against possible sampler bias, the number of prompt rephrasings Q, and the cost of additional queries.The authors also suggest investigating whether higher moments, such as variance, add information without class labels.Source: §7 · p. 9

8 Ethical Considerations

The authors describe potential uses of the metrics for identifying LLM failure modes, while noting that they do not indicate how to perform jailbreaks.

8 Ethical Considerations

  • The metrics could reveal whether an LLM is more sensitive to a minority class than another, pointing to a failure mode to address.Source: §8 · p. 9
  • Attackers could use the metrics to compare LLM susceptibility to jailbreaks, but the metrics provide no indication of how to jailbreak a model.Source: §8 · p. 9
  • The authors suggest that making LLMs robust and optimized for these metrics may increase their trustworthiness.Source: §8 · p. 9

A Comparison with Noisy and Random Predictors

On TREC, the authors compare sensitivity and average consistency for Llama3 with noisy and random predictors. With Q = 30, the joint metric distributions differ, although consistency alone does not distinguish the random predictor in this setting.

A Comparison with Noisy and Random Predictors

  • Q = 30 produces a joint sensitivity–consistency distribution that differs from the noisy and random predictor distributions on TREC.The noisy predictor swaps LLM class predictions randomly 50% of the time.Source: A Comparison with Noisy and Random… · p. 12
  • Q = 30 is not enough to distinguish random consistency values from Llama3’s on TREC, where sampling is not perfectly uniform.Source: A Comparison with Noisy and Random… · p. 12
  • Figure 6 plots each TREC sample’s sensitivity against its average consistency with other samples of the same class.Source: Figure 6 · p. 12

B Using Sensitivity to Improve Prompts

The sensitivity ranking surfaces date questions as problematic TREC samples, and adding an explicit date-classification instruction improves their predictions across prompt rephrasings.

B Using Sensitivity to Improve Prompts

C Using Consistency to Improve Prompts

Consistency analysis identifies problematic samples that sensitivity alone does not; prompt refinements targeting these patterns improve classification on examples involving colors and descriptions.

C Using Consistency to Improve Prompts

  • Consistency inspection identifies color-related Entity questions as difficult for Llama3, except samples in the “other” subclass.Source: C Using Consistency to Improve Prompts · p. 12
  • A prompt correction significantly improved classification of the three examples about color, raising correct predictions over 30 rephrasings from [0, 4, 3, 15, 0] to [30, 4, 30, 15, 30].Source: C Using Consistency to Improve Prompts · p. 13
  • For high-sensitivity samples, inconsistency can indicate classification hardness; quasi-zero sensitivity with inconsistency and misclassification instead suggests intrinsic sample diversity.Standard misclassification at Q = 1 does not reveal the same examples as consistency analysis.Source: C Using Consistency to Improve Prompts · p. 13

D Analysis of Standard Deviation Values

The supplement cautions against interpreting standard deviations for the non-Gaussian results and provides examples of prompt rephrasings for the evaluated tasks and datasets.

D Analysis of Standard Deviation Values

E Examples of Prompt Rephrasings

  • The listed prompt variants change task-description wording while leaving the rest of the prompt unchanged; Llama3-generated examples are shown, with similar variants for other models.Source: E Examples of Prompt Rephrasings · pp. 13–14

DBPedia

  • DBPedia prompt examples phrase classification over categories including companies, institutions, people, places, living things, and creative works.Source: DBPedia · p. 15

WoS

  • WoS prompt examples ask classification among seven fields, including Computer Science, Psychology, and Biochemistry.Source: WoS · p. 16
Loading 2406.12334v4…