Source-linked AI summary
ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, Kai Chen
TL;DR
Prompt sensitivity complicates reliable evaluation because prompt variants can change LLM responses, while existing work underexamines instance-level and subjective effects. ProSA introduces instance-level evaluation with PromptSensiScore and decoding-confidence analysis, finding variation across models and datasets, improved robustness with few-shot examples, and a correlation between higher confidence and robustness.
Problem
Prompt variants can substantially change LLM performance, while prior research mainly overlooks instance-level variation, subjective evaluations, and the reasons for sensitivity.
Method
ProSA combines instance-level objective and subjective evaluations with PromptSensiScore and decoding-confidence analysis across multiple task capabilities.
Results
Prompt sensitivity varies across models, datasets, and task types; few-shot examples alleviate it, larger models gain greater robustness, and higher confidence correlates with robustness.
Takeaways & Limitations
PromptSensiScore and confidence analysis provide a framework for studying LLM prompt robustness and stability.
Takeaways & Limitations
Limited computational resources prevented testing larger numbers of few-shot examples, and API costs restricted subjective evaluations to three prompt variants.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but their performance is highly sensitive to the prompts utilized. This variability poses challenges for accurate assessment and user satisfaction. Current research frequently overlooks instance-level prompt variations and their implications on subjective evaluations. To address these shortcomings, we introduce ProSA, a framework designed to evaluate and comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity metric, PromptSensiScore, and leverages decoding confidence to elucidate underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers that prompt sensitivity fluctuates across datasets and models, with larger models exhibiting enhanced robustness. We observe that few-shot examples can alleviate this sensitivity issue, and subjective evaluations are also susceptible to prompt sensitivities, particularly in complex, reasoning-oriented tasks. Furthermore, our findings indicate that higher model confidence correlates with increased prompt robustness. We believe this work will serve as a helpful tool in studying prompt sensitivity of LLMs. The project is released at: https://github.com/open-compass/ProSA .
1 Introduction
ProSA addresses prompt sensitivity by analyzing how prompt variants affect individual instances, objective and subjective evaluations, and model confidence. Across tasks and models, sensitivity varies, few-shot examples can improve robustness, and higher confidence correlates with greater robustness.
- Different expressions of the same requirement can substantially change LLM performance, challenging precise capability evaluation and requiring users to iterate prompts for better outputs.
- Existing research largely studies dataset-level template differences while overlooking instance-level variation, subjective evaluations, and explanations for prompt sensitivity.
- ProSA evaluates prompt sensitivity at the instance level across objective and subjective tasks spanning understanding, reasoning, coding, and alignment.
- PromptSensiScore measures the average response discrepancy across semantic variants of the same instruction.
- Across 8 LLMs, 4 datasets, and 12 prompt variants per model, sensitivity varies by model and dataset, while Llama3-70B-Instruct shows the highest robustness.
- Subjective evaluations show greater robustness for straightforward or established-knowledge queries but higher sensitivity in complex, coding, and creative tasks.
- Higher decoding confidence correlates with greater robustness to semantic prompt variations.
2 Instance Level Prompt Sensitivity
Instance-level prompt sensitivity measures how differently an LLM responds to prompt variants within the same instance, addressing nuances that dataset-level aggregates can conceal. ProSA operationalizes this sensitivity with PromptSensiScore across objective and subjective evaluations.
- 2.1 Definition: Different expressions of the same requirement are treated as prompt variants of one instance, and they can produce different LLM responses.
- 2.1 Definition: Dataset-level template comparisons can overlook performance differences between prompts for individual instances.
- 2.1 Definition: Prompt sensitivity can vary substantially across instances because they differ in complexity, context, and information type.
- 2.1 Definition: LLMs may be robust to prompt templates for some instances but sensitive to them for others.
- 2.1 Definition: Aggregate metrics can mask individual cases where slight prompt changes produce drastic response changes.
- 2.2 Evaluation Metric: PromptSensiScore is the average discrepancy in LLM responses across different semantic variants of the same instruction.
- 2.2 Evaluation Metric: PSS averages pairwise absolute performance differences within each instance, using correctness for grounded tasks and quality scores in [0, 1] otherwise.
3 Prompt Sensitivity on the Objective Evaluation
The study evaluates prompt sensitivity across objective tasks, datasets, model sizes, and few-shot settings. Results show that sensitivity depends on task and dataset, while few-shot examples generally improve prompt robustness.
- 3 Prompt Sensitivity on the Objective Evaluation: Objective evaluation measures response variation across prompts for instances with specific ground-truth answers.
- 3.1 Experimental Setup: The benchmark covers CommonsenseQA, ARC-Challenge, MATH, and HumanEval, spanning commonsense knowledge, science reasoning, mathematics, and programming.
- 3.1 Experimental Setup: Experiments use 0-shot prompts for three datasets, 4-shot prompts for MATH, and Llama3-70B-Instruct for MATH response extraction.
- 3.1 Experimental Setup: Each prompt aspect has three manually constructed prompts, with examples covering formats such as emotional support and output requirements.
- 3.2 Main Results and Analysis: On easier datasets, average performance and PSS are approximately linearly related, whereas MATH combines poor performance with prompt sensitivity because it requires extensive reasoning.
- 3.2 Main Results and Analysis: Prompt sensitivity varies by task within a model: Qwen1.5-14B-Chat is robust on CommonsenseQA but most sensitive on MATH.
- 3.3 Prompt Sensitivity and Model Size: Figure 4 compares model parameter size with average PSS across four datasets; Llama3-70B-Instruct is exceptionally robust, while Qwen models show similar sensitivity across sizes.
- 3.4 Few-shot Enhances Prompt Robustness: Few-shot examples enhance robustness across models, with the largest ARC-Challenge reduction occurring from 0-shot to 1-shot; increasing shots especially benefits larger models.
4 Prompt Sensitivity on the Subjective Evaluation
The subjective evaluation study examines prompt sensitivity across two benchmarks, using rewritten prompts and PSS comparisons for five LLMs. Results show greater robustness on relatively simple questions and higher sensitivity on more challenging, creative, or content-intensive tasks.
- 4.1.1 Dataset Selection: The study uses LC AlpacaEval 2.0 and Arena Hard Auto to evaluate prompt sensitivity in subjective evaluations.Both benchmarks compare tested-model responses with reference-model responses.
- 4.1.1 Dataset Selection: LC AlpacaEval 2.0 contains 805 instances, while Arena Hard Auto contains 500 questions across 250 categories.Arena Hard Auto scores each response pair twice after swapping positions, producing 1000 scores.
- 4.1.2 LLMs Selection: Five relatively strong models were selected: InternLM2-20B-Chat, Llama3-8B-Instruct, Llama3-70B-Instruct, Qwen1.5-14B-Chat, and Qwen1.5-72B-Chat.The selection reflects the benchmarks’ comparison against reference-model responses.
- 4.1.3 Prompt Rewriting: Prompts were rewritten with GPT-4o and GPT-4-0409, then manually verified and refined to produce varied question styles.BERTScore and human annotation were used to verify semantic similarity between original and rewritten prompts.
- 4.2 Main Results and Analysis: PSS was calculated across one original and two generated prompt versions, using the average response quality of two Llama3 models as references.The reference models were Llama3-8B-Instruct and Llama3-70B-Instruct.
- 4.2 Main Results and Analysis: All evaluated LLMs had lower PSS than the reference on LC AlpacaEval 2.0, indicating prompt robustness on that benchmark.The corresponding PSS values on Arena Hard Auto were higher, indicating greater sensitivity there.
- 4.2 Main Results and Analysis: Arena Hard Auto showed that models were more robust on relatively simple questions but more prompt-sensitive on challenging questions.This comparison motivates analyzing sensitivity by task category.
- 4.3 Prompt Sensitivity and Categories: Knowledge-focused categories such as business solutions and IT problems were more robust, whereas data visualization and scripting tasks were more sensitive.The latter tasks require substantial content generation and some creativity.
5 Why LLMs are Sensitive to Prompts
The paper studies why prompt sensitivity varies by examining task categories and decoding confidence. It reports that confidence is highest for robust instances and decreases as sensitivity increases.
- 5 Why LLMs are Sensitive to Prompts: Existing work largely selects suitable prompts but gives less attention to interpreting why LLMs are sensitive to prompts.The paper therefore investigates prompt sensitivity through model confidence.
- 5 Why LLMs are Sensitive to Prompts: Figure 6 compares the five most and five least sensitive Arena Hard Auto categories using category-level PSSc.PSSc is the average PSS of five LLMs within a category.
- 5 Why LLMs are Sensitive to Prompts: On CommonsenseQA, the study found that prompt sensitivity is related to model confidence across Mistral-7B-Instruct, InternLM2-7B-Chat, and InternLM2-20B-Chat.For an instance, greater confidence was associated with greater prompt robustness.
- 5.1 Decoding Confidence: Decoding confidence is computed from token probabilities, using only the option token predicted by the model for each CommonsenseQA instance.This adapts confidence measurement to the dataset’s multiple-choice format.
- 5.1 Decoding Confidence: For a prompt set P, instance confidence averages the maximum next-token probability across prompts.The paper then averages instance-level confidence across instances for each LLM.
- 5.2 Experiments and Analysis: The confidence analysis uses the same 12 prompt templates for the three LLMs and reports the results in Figure 7.The evaluated models are Mistral-7B-Instruct, InternLM2-7B-Chat, and InternLM2-20B-Chat.
- 5.2 Experiments and Analysis: Robust instances with low PSS have the highest decoding confidence, while confidence decreases for prompt-sensitive instances.The paper interprets this correlation as an external manifestation of decoding logic.
6 Related Work
Related work evaluates LLM capabilities across objective tasks and increasingly uses subjective evaluation to assess alignment with human needs. Prior studies also establish prompt sensitivity but leave instance-level and explanatory questions open.
- 6 Related Work: Prior LLM evaluation spans reading comprehension, mathematical problem solving, and code generation.These tasks provide multiple perspectives on model capabilities.
- 6 Related Work: Subjective evaluation has gained attention as objective capabilities improve, because it assesses alignment between generated responses and human needs.LC AlpacaEval 2.0 is cited as an example.
- 6 Related Work: Earlier studies show that prompt perturbations can significantly change model performance, including through option-order changes in multiple-choice questions.Related work also connects model robustness with cherry-picking of performance.
7 Conclusion
The paper concludes by combining an instance-level prompt sensitivity metric with objective and subjective evaluation and confidence analysis. It positions ProSA as guidance for further sensitivity analysis and robust LLM development.
- 7 Conclusion: The paper proposes PSS and analyzes prompt sensitivity through both objective and subjective evaluation.It also examines the relationship between prompt sensitivity and model confidence.
- 7 Conclusion: The authors present ProSA as a tool that can provide guidance for further sensitivity analysis and building robust LLMs.
8 Limitations
The study evaluates prompt sensitivity in objective and subjective settings but identifies two resource-related limitations. It does not examine increasing numbers of few-shot examples and uses only three prompt variants for the subjective benchmarks.
- Computational resources prevented exploring prompt sensitivity as the number of few-shot examples increased.
- The subjective evaluation used only three prompt variants on LC AlpacaEval 2.0 and Arena Hard Auto because OpenAI API access was expensive.
9 Ethical Considerations
The paper uses publicly available datasets and LLMs for analytical experiments and reports how responses and PSS are evaluated. It also describes LLM-assisted answer extraction for MATH and acknowledges possible cherry-picked evaluation reports.
- The analytical experiments use publicly available datasets and LLMs.
- The authors acknowledge that their findings could potentially be used to create cherry-picked evaluation reports.
- Response lists record correctness across 12 prompt templates, with 1 denoting correct and 0 denoting incorrect.
- Because MATH answers can use equivalent formats, Llama3-70B-Instruct performs post-processing answer extraction to improve response assessment.
C.2 Subjective Evaluation
The subjective evaluation uses representative instances from LC AlpacaEval 2.0 and Arena Hard Auto, rewriting each original prompt with GPT-4 to create two additional prompts for separate evaluation.
- Four representative instances from each subjective benchmark are presented as evaluation examples.
- The subjective datasets use prompts generated from specified requirements, so the original prompts are rewritten with GPT-4.
- Each original subjective prompt produces two new prompts that are evaluated separately.
E Results and Analysis about the Proprietary Models
The proprietary-model analysis compares Claude-3.5-sonnet and GPT-4o on HumanEval with 12 prompts, alongside advanced open-source models. Despite strong average performance, the proprietary models are not necessarily the most prompt-robust, and model behavior differs on challenging instances.
- The HumanEval comparison evaluates the two proprietary models with 12 prompts against several advanced open-source LLMs.
- Claude-3.5-sonnet and GPT-4o achieve optimal average HumanEval performance but are less prompt-robust than some lower-scoring models.
- When instances are challenging, Llama-3-70B-Instruct tends to remain incorrect across prompt templates, whereas GPT-4o can answer correctly under some prompts after earlier errors.
- The differing responses of GPT-4o and Llama-3-70B-Instruct on difficult instances contribute to their observed disparity in prompt sensitivity.
- The evaluation materials include objective-task prompt templates, few-shot examples for ARC-Challenge and CommonsenseQA, and rewritten prompts for subjective benchmarks.