Source-linked AI summary
Does Prompt Formatting Have Any Impact on LLM Performance?
Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, Sadid Hasan
TL;DR
Prompt-format sensitivity remains insufficiently understood despite its potential to affect LLM evaluation. This paper compares equivalent contexts formatted as plain text, Markdown, YAML, and JSON across GPT models and multiple tasks, finding substantial variation and greater robustness in GPT-4. It concludes that no single format is universally best and that evaluations should use diverse formats.
Problem
Prompt-format effects are underexamined, although limited evidence suggests that template choices can produce substantial performance variation and fixed templates may mislead evaluations.
Method
The study evaluates four GPT models across six benchmarks using identical prompt content rendered in plain text, Markdown, YAML, and JSON, and measures sensitivity, consistency, and transferability.
Results
Prompt formatting significantly affects GPT-model performance, with no universally optimal format; GPT-4-turbo is less susceptible to structural variation than earlier models.
Takeaways & Limitations
LLM evaluations should use diverse prompt formats because fixed structures can misjudge model abilities, while larger models may process prompt variations more consistently.
Takeaways & Limitations
The study examines GPT-based models only and holds other prompt-design elements constant, leaving other model families and interactions with prompt techniques for future work.
Abstract
from arXiv · showhide
In the realm of Large Language Models (LLMs), prompt optimization is crucial for model performance. Although previous research has explored aspects like rephrasing prompt contexts, using various prompting techniques (like in-context learning and chain-of-thought), and ordering few-shot examples, our understanding of LLM sensitivity to prompt templates remains limited. Therefore, this paper examines the impact of different prompt templates on LLM performance. We formatted the same contexts into various human-readable templates, including plain text, Markdown, JSON, and YAML, and evaluated their impact across tasks like natural language reasoning, code generation, and translation using OpenAI's GPT models. Experiments show that GPT-3.5-turbo's performance varies by up to 40\% in a code translation task depending on the prompt template, while larger models like GPT-4 are more robust to these variations. Our analysis highlights the need to reconsider the use of fixed prompt templates, as different formats can significantly affect model performance.
1 Introduction
This study examines whether broader prompt-format changes affect GPT-model performance, addressing limited evidence and evaluation practices that often ignore prompt structure. It compares plain text, Markdown, YAML, and JSON across multiple tasks and finds substantial sensitivity, with GPT-4-turbo more resilient than earlier models.
- Research motivation: Prompt format can significantly change model performance, yet existing research and evaluation practices have often overlooked this factor.Prior evidence focused mainly on minor template changes, while fixed evaluation templates may produce misleading conclusions.
- Study scope: The study compares plain text, Markdown, YAML, and JSON formats across multiple-choice questions, code generation, and translation tasks.It evaluates four OpenAI GPT models across six benchmarks.
- Research questions: The paper asks whether prompt formats alter performance, whether responses remain consistent across structures, and whether any format transfers optimally across models.It also examines relationships with task-specific competencies and model size.
- Main findings: There is no universally optimal prompt format, even within the same GPT generation.Performance discrepancies occur across models and benchmarks.
- Main findings: GPT-4-turbo is less susceptible to prompt-structure variations than earlier GPT models.The evaluation compares behavioral patterns across GPT iterations, including GPT-4-turbo.
2 Experimental Setup
The experiments evaluate GPT models on six datasets spanning natural-language understanding, natural-language-to-code, and code-to-code tasks. Prompts preserve identical content while varying only structural format and syntax across plain text, Markdown, YAML, and JSON.
- Tasks and datasets: Experiments cover six datasets grouped into NL2NL, NL2Code, and Code2Code task categories.NL2NL includes MMLU and NER Finance; NL2Code includes HumanEval and FIND; Code2Code includes CODEXGLUE and HumanEval-X.
- Evaluation: Model performance is first assessed with task-specific scalar scores, followed by metrics addressing the study’s research questions.Dataset descriptions and metric details are provided in Appendix B.
- Prompt construction: Prompts use plain text, Markdown, YAML, and JSON while keeping persona, instructions, examples, output requirements, and user asks constant.Only prompt structure and syntax differ across formats, preventing context and meaning from becoming confounding variables.
- Models: The experiments use GPT-3.5 and GPT-4 model configurations accessed through Azure OpenAI.GPT-3.5 configurations compare 4k and 16k context sizes, while GPT-4 configurations include a newer variant with a 128k context window.
3 Sensitivity
The sensitivity analysis tests whether prompt-template choice changes task performance by comparing the best and worst formats and assessing statistical significance. Results show widespread, sometimes very large, performance variation across formats.
- Metric and test: Sensitivity is measured by comparing the maximum and minimum template scores for each task and testing their difference with a matched-pairs t-test.A significant test indicates that prompt format matters; otherwise performance is considered relatively insensitive to the prompt.
- Statistical sensitivity: Prompt-format differences are statistically significant for most model–benchmark comparisons.The one-sided matched-pairs t-test produces p-values mostly below 0.01; one reported exception is GPT-4-1104-preview on HumanEval.
- Performance variation: 200% improvement occurs on FIND for both GPT-35-turbo configurations when prompts switch from Markdown to plain text.The comparison is reported for GPT-35-turbo-0613 and GPT-35-turbo-16k-0613.
- Performance variation: Over 300% improvement occurs for GPT-4-32k-0613 on HumanEval when the prompt format changes from JSON to plain text.This result illustrates that even a GPT-4 configuration can show substantial format sensitivity on a particular benchmark.
4 Consistency
The consistency analysis measures how often identical queries produce identical answers under different prompt templates. GPT-4 is more consistent than GPT-3.5 across the evaluated MMLU and FIND settings, although reliable format-invariant behavior remains incomplete.
- Metric definition: Consistency is the proportion of test samples producing identical responses for two prompt templates.For templates Pa and Pb, the metric compares model answers across the test set; higher scores indicate greater consistency.
- MMLU results: Only 16% of MMLU responses are identical between Markdown and JSON for the GPT-3.5-turbo series.MMLU evaluations use temperature zero to remove response variability.
- FIND results: GPT-4 also outperforms the GPT-3.5-turbo series in consistency on the FIND dataset.Across the evaluated datasets, larger models provide more uniform outputs across different prompts.
- MMLU results: GPT-3.5 models score below 0.5 across MMLU format pairs, whereas GPT-4 consistently exceeds 0.5.Figure 2 presents this cross-format consistency comparison.
5 Transferability
Prompt-template transferability varies substantially across GPT models: models can prefer different formats, and top-template overlap is low across series but high within closely related sub-series.
- IoU measures overlap between models’ sets of statistically indistinguishable top-performing templates.The top-template sets are identified using a matched pairs t-test.
- GPT-3.5-turbo prefers JSON, whereas GPT-4 favors Markdown.
- Higher IoU occurs within same-version model pairs, whereas cross-version pairs exhibit lower IoU.
- IoU is often below 0.2 between different model series, indicating low compatibility of their top prompt templates.
- GPT-35-turbo-16k-0613 and GPT-35-turbo-0613 show IoU above 0.7, indicating high overlap within the same sub-series.
- Prompt formats are not transferable reliably across GPT models, so optimal performance requires model-specific prompt engineering.
6 Conclusion
Prompt formatting significantly affects GPT-based model performance, with no universally best format. Larger models such as GPT-4 are less influenced by prompt changes than GPT-3.5.
- Prompt formatting significantly impacts GPT-based model performance, and no single format excels universally.
- GPT-4’s performance is less influenced by prompt changes than GPT-3.5’s, suggesting that larger models process prompts more consistently.
7 Limitations
The study’s scope and design leave several directions for future work, including broader model coverage, additional prompt formats, and interactions with other prompt-engineering choices.
- The study focuses on GPT-based models rather than other LLM families such as LLaMA, Gemini, PaLM, or Phi.
- The evaluated template formats exclude HTML and XML, limiting the breadth of format exploration.
- The experimental design holds other prompt elements constant, so future work should examine interactions with few-shot examples and instruction precision.
- Related Work: Prior prompt-format studies mainly examined subtle local changes and classification tasks, whereas this work investigates global format modifications across broader task settings.
B Datasets
The study evaluates GPT models on six benchmarks spanning natural-language understanding, entity extraction, code generation, and code translation. It keeps prompt content constant while comparing plain text, Markdown, YAML, and JSON structures.
- Datasets: Six benchmarks are grouped by task type, covering natural-language understanding, entity extraction, code generation, and code translation.The benchmark set includes MMLU, NER Finance, HumanEval, FIND, CODEXGLUE, and HumanEval-X.
- Natural Language Tasks: MMLU contains 57 subjects across humanities, social sciences, STEM, and other domains, measuring question-answering accuracy.The test set contains 14,079 questions, while five development questions per subject are used as few-shot examples.
- Code Generation: HumanEval tests Python code generation with pass@1 over 164 programming problems and unit tests.The metric checks whether generated code passes the provided tests in one attempt.
- Code Generation: FIND evaluates reverse engineering of unknown Python functions from five input-output examples using 500 string-category functions.Generated functions are evaluated by comparing test-case outputs with those of ground-truth functions.
- Prompt Templates: The four prompt templates preserve persona, instructions, examples, output-format instructions, and user ask while changing only structure and syntax.The formats are plain text, Markdown, YAML, and JSON.
D.2 Is there a correlation between the model size and the robustness of the LLM for different prompt templates?
The paper measures robustness to prompt-template changes using CMD across models and benchmarks. GPT-4 models generally show lower dispersion than GPT-3.5 models, with GPT-4-1106-preview the most robust in the reported comparison.
- Robustness Measure: CMD measures performance dispersion across prompt templates, with lower values indicating greater robustness to format variation.The calculation uses each template’s score, the average score across formats, and the number of formats.
- Model-Size Comparison: GPT-3.5 models exhibit larger CMD scores than GPT-4 models across benchmarks, indicating greater sensitivity to prompt-format choice.The figure summarizes CMD across models and datasets.
- Model-Size Comparison: GPT-4-1106-preview maintains CMD below 0.036 across all benchmarks, indicating the strongest reported robustness to format changes.GPT-4-32k-0613 has CMDs not exceeding 0.043, while GPT-3.5 models range from 0.035 to 0.176.
- Conclusion: The results support greater robustness to template variation in larger models, while GPT-4 model variants still differ in robustness.The paper notes that GPT-4-1106-preview is more robust than GPT-4-32k-0613.
E.3 Dotplot on all benchmark datasets
Figure 9 presents dotplots of model performance across prompt formats for all benchmark datasets. The plots are intended for comparing performance distributions across formats and models.
- Dotplot: Figure 9 uses dotplots to display model performance across prompt formats on all benchmark datasets.The supplied materials do not specify additional axis or encoding details.
- Comparison: Use the plots to compare how performance changes among prompt formats across the different benchmark datasets.The captions identify the figure as a cross-benchmark, cross-format performance overview.