PromptResponse: Optimizing Prompts for LLM Coding Tasks

Erik Thureck; Robert Kühnen; Tim Jacobowitz

Source-linked AI summary

PromptResponse: Optimizing Prompts for LLM Coding Tasks

Erik Thureck, Robert Kühnen, Tim Jacobowitz

arXiv:2608.21074v1Aug 21, 2026cs.CLcs.AIcs.HCcs.SE

TL;DR

LLM outputs are sensitive to prompt variation, creating concerns for reliable and reproducible coding workflows. PromptResponse tests five semantically identical HumanEval prompt variants across 8200 GPT-4o executions and finds that formatting, especially JSON, improves efficiency and stability, whereas LLM tuning degrades task performance. The study therefore supports low-effort prompt reformatting while identifying model choice as an important scope boundary.

Problem
Prompt optimization for code generation remains underinvestigated despite concerns about LLM output sensitivity and reproducibility in research.
Method
The study compares five semantically identical HumanEval prompt variants using controlled GPT-4o executions and measures performance, efficiency, and stability.
Results
Consistent formatting, especially JSON, improves generation efficiency and syntactic prompt stability with slight task-performance gains, while LLM-tuned prompts degrade task performance.
Takeaways & Limitations
Reformatting prompts can provide measurable benefits for LLM coding tasks, while prompt tuning should account for model alignment.
Takeaways & Limitations
The findings focus on GPT-4o, so performance may vary with the employed LLM.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used in research workflows and software development pipelines, yet their output remains sensitive to input prompt variations. This paper presents $\unicode{x00AB}$PromptResponse$\unicode{x00BB}$, a controlled study examining how formatting and LLM-based tuning of coding task prompts affect the resulting code's performance, efficiency, and stability. Using five semantically identical yet syntactically distinct variants of the HumanEval dataset$\unicode{x2014}$baseline, JSON, Markdown, YAML, and an LLM-tuned version$\unicode{x2014}$we had GPT-4o solve its coding problems over 8200$\unicode{x00A0}$executions. Our results show that consistent formatting$\unicode{x2014}$especially JSON$\unicode{x2014}$improves generation efficiency and syntactic stability, with minor gains in task performance. Conversely, the LLM-tuned prompts resulted in significantly degraded task performance without significant improvements in any other dimension. These findings suggest that low-effort reformatting alone can yield measurable improvements, while tuning must account for model alignment. We conclude our work with providing a set of practical recommendations informed by our results as well as releasing our dataset variants and evaluation pipeline for future work.

1 Introduction

LLM use in research and software development raises concerns about output sensitivity, stability, and reproducibility. This paper addresses underexplored prompt optimization for code generation through semantically equivalent dataset variants and controlled GPT-4o experiments.

Informal prompt formulation can jeopardize result stability and undermine reproducibility in research workflows.
Semantically identical prompts can produce varying outputs, especially under low-control or real-world conditions.
LLMs are increasingly used to generate executable code and support software engineering workflows.
Prompt optimization for code generation remains underinvestigated compared with natural-language tasks, including autonomous LLM-based prompt tuning.
The paper creates four syntactically different but semantically equivalent HumanEval derivatives and conducts 8200 GPT-4o executions to study performance, efficiency, and stability.
The authors provide deployment implications and guidelines for using LLMs in coding tasks based on the controlled experiment.

2 Related Work

Prior work documents prompt sensitivity, nondeterminism, and reproducibility risks across LLM applications, while evidence on formatting and prompt optimization remains mixed. Existing studies motivate controlled evaluation of coding prompts across formats and models.

Research reviews identify validity, reproducibility, ethical, and bias concerns in academic LLM use and recommend documenting prompts and outputs.
Prompt-stability work shows that semantically similar prompts can yield inconsistent outputs, motivating metrics and benchmarks for sensitivity analysis.
E-Bench found that larger models handle synonymous changes better but struggle with typographical noise, supporting human-centered evaluation under low-control conditions.
Studies of prompt engineering report format preferences and strong effects from seemingly minor changes on consistency or evaluation rankings.
Formatting studies found significant effects on GPT performance but no universally superior format, with weaker effects for GPT-4 than GPT-3.5.
Prior nondeterminism research observed accuracy fluctuations of up to 15% across runs and up to 70% between best- and worst-case outcomes.
The paper frames prompt stability as a reproducibility problem and seeks a foundation for efficient and reliable LLM code generation.

3 Methodology

The study varies five semantically identical HumanEval prompt formats and evaluates GPT-4o outputs across task performance, efficiency, and stability measures. Repeated executions enable comparisons of formatting and LLM-based tuning under default settings.

The experiment queries GPT-4o on Python HumanEval problems presented in five semantically identical variations.
The study avoids custom settings because low-temperature configurations do not reliably produce deterministic or stable output.
The five conditions are vanilla HumanEval, JSON, Markdown, YAML, and an LLM-tuned version with Mistral AI-adjusted docstrings.
The wording of each coding task remained consistent across dataset variations to avoid introducing random error.
Each of the 164 prompts was queried 10 times in fresh ChatGPT windows, producing 8200 total executions.
Task performance is measured by PASSRATE, the proportion of ten responses passing all associated HumanEval tests.
Efficiency measures include generation duration, evaluation duration, pass-only duration, and response length.
Prompt stability is quantified using average pairwise ROUGE-L across ten executions, ranging from completely distinct to perfectly matching responses.

3.2 Datasets

The study constructs five HumanEval prompt variants from the unaltered vanilla dataset: JSON, Markdown, YAML, and an LLM-tuned version. The variants preserve prompt content while regularizing or modifying its structure to enable controlled comparison.

3.2.1 vanilla: The vanilla HumanEval dataset contains 164 Python coding prompts with function signatures, docstrings, and optional sections such as examples, variables, constraints, notes, imports, and helper functions.Examples appear in 97.6% of prompts, while section presence, ordering, labels, and formatting vary across prompts.
3.2.1 vanilla: Vanilla prompts exhibit high structural inter-prompt variability, including inconsistent section labels, example formats, spelling, function signatures, and occasional repair notes.The dataset does not contain empty sections, making vanilla the lightest dataset level.
3.2.2 json: The JSON variant parses prompt sections into a common structure, retains original textual contents and grammar mistakes, and fills absent sections with empty values.Its standardized structure produces a 233 KB dataset, the largest of the five levels.
3.2.3 markdown: The Markdown variant retains all section headers, marks unavailable fields with explicit placeholders, and uses headings, tables, underscores, backticks, italics, and code boxes for formatting.At 218 KB, it is the second-largest dataset level.
3.2.4 yaml: The YAML variant uses the same contents as JSON while leaving empty sections blank and occupies 209 KB, making it the second-smallest reformatted level.Its size increase is described as the most conservative among the three reformatted variants.
3.2.5 tuned: The tuned variant rewrites the unaltered vanilla prompts with Mistral AI’s Mistral-7B-Instruct-v0.2 using instructions specified in Table 1.The passage identifies LLM-based rewriting as the defining change for this dataset level.

3.3 Procedure

The procedure evaluates GPT-4o on five HumanEval prompt variants using independent executions and records generation, execution, and outcome measures. The resulting dataset contains 8200 executions processed under a fixed evaluation pipeline.

3.3.1 Execution: Each of the five dataset variants was evaluated on 164 prompts with 10 independent GPT-4o API requests per prompt, totaling 8200 executions.Each request used a newly created chat completion window without previous context or interactions.
3.3.1 Execution: GPT-4o responses were generated without streaming, and when multiple answers appeared, the first answer was selected and logged with its generation duration.Responses were recorded immediately to prevent data loss.
3.3.3 Controls: The experiment ran sequentially without multiprocessing on July 10, 2025, and took about 205 minutes.The reported setup was intended to avoid external confounding variables beyond the prompt condition.
3.3.2 Evaluation: A Python script cleaned generated responses, appended applicable helper functions, executed solutions against HumanEval tests, and logged success, response length, and evaluation duration.The pipeline also calculated PassRate, PassDuration, and ROUGE-L over each prompt’s 10 executions.

3.4 Analysis

The analysis uses non-parametric tests because the dependent variables failed normality checks. Dataset variants are compared with Kruskal–Wallis tests followed by corrected pairwise tests when omnibus differences are significant.

3.4.1 Statistical tests: Shapiro–Wilk tests found highly significant departures from normality for all dependent variables, motivating non-parametric analyses.The authors therefore used the tests described in the subsequent analysis procedure.
3.4.1 Statistical tests: Kruskal–Wallis tests assessed main effects across the five HumanEval variants, followed by Bonferroni-corrected Mann–Whitney U tests for significant differences.The data were aggregated across 10 executions for each variant’s 164 prompts.
3.4.2 Effect sizes: The analysis reports epsilon-squared ε2 for Kruskal–Wallis effect sizes and rank-biserial correlation r for post hoc comparisons.Both effect sizes are classified as small, medium, or large using thresholds attributed to Cohen’s suggestions.
3.4.3 Visualization: Figures summarize PassRate across the five dataset levels using dots representing aggregate results from each prompt’s 10 executions.The figure-based analysis compares prompt variants at the prompt-aggregate level.

4 Results

Across the five prompt variants, JSON and Markdown generally show the strongest PassRate results, while YAML is fastest to generate and evaluation duration does not differ significantly. The tuned variant has substantially lower task performance and higher generation time than several alternatives.

4.1 Pass Rate: 79.9% of JSON prompts and 78.7% of Markdown prompts achieved perfect PassRate, compared with 75% for vanilla, 73.2% for YAML, and 63.4% for tuned.Average PassRates were 0.901 for JSON, 0.890 for Markdown, 0.886 for vanilla, 0.873 for YAML, and 0.748 for tuned.
4.1 Pass Rate: The PassRate effect of dataset format was significant but small: χ2(4) = 19.18, p < 0.001, ε2 = 0.019.The omnibus result indicates differences among dataset variants without establishing a causal mechanism.
4.2 Generation Duration: YAML generated fastest on average at 1.37 s, followed by Markdown at 1.44 s, JSON at 1.49 s, vanilla at 1.61 s, and tuned at 1.61 s.The dataset effect on generation duration was significant but small, with χ2(4) = 35.05, p < 0.001, ε2 = 0.038.
4.2 Generation Duration: Tuned prompts took significantly longer to generate than JSON, Markdown, and YAML, with differences of 0.21 s, 0.24 s, and 0.23 s respectively.Each reported pairwise comparison had a small effect size.
4.3 Evaluation Duration: Evaluation duration did not differ significantly across dataset variants: χ2(4) = 5.04, p = 0.283, ε2 = 0.001.The reported effect size was negligible despite variation in individual evaluation times.

10 ), ‹json›

Across five HumanEval dataset formats, formatting had little effect on response length or pass duration but significantly affected intra-prompt response similarity. JSON produced the highest average ROUGE-L, while vanilla produced the lowest.

PassDuration showed no significant differences across dataset formats (χ2(4) = 3.95, p = 0.413, ε2 = 0.000).
Response lengths ranged from 41 characters for vanilla/53 to 1321 characters for tuned/129, which did not pass.
ResponseLen also showed no significant differences across dataset formats (χ2(4) = 1.37, p = 0.849, ε2 = −0.003).
JSON achieved the highest average ROUGE-L (x̄ = 0.842, sintra = 0.136), followed by Markdown (x̄ = 0.837, sintra = 0.138) and YAML (x̄ = 0.821, sintra = 0.148).
Vanilla had the lowest average ROUGE-L (x̄ = 0.787, sintra = 0.152), while tuned was slightly higher (x̄ = 0.798, sintra = 0.162).
Dataset format significantly affected ROUGE-L (χ2(4) = 14.95, p = 0.005, ε2 = 0.013), with vanilla significantly below JSON and YAML.

5 Discussion

The discussion finds that consistent reformatting, especially JSON, can improve task performance modestly, generation efficiency, and prompt stability, whereas LLM tuning degraded task performance. The authors therefore recommend reformatting while checking tuning for model alignment.

Task performance: JSON and Markdown had higher PASSRATEs than vanilla, but these gains were not statistically significant.
Task performance: Tuned performed significantly worse than JSON, Markdown, and vanilla on pass@1, leading to only partial support for the task-performance hypothesis.
Practical implications: LLM tuning should not be applied indiscriminately because the Mistral-generated tuned dataset may have been de-optimized for ChatGPT.
Generation efficiency: Reformatted prompts reduced generation duration relative to vanilla and tuned despite their increased prompt lengths, with YAML reaching the shortest average GENDURATION.
Code efficiency: No significant differences were found in EVALDURATION, PASSDURATION, or RESPONSELEN across dataset formats.
Prompt stability: JSON and YAML produced significantly higher ROUGE-L scores than vanilla, while Markdown and tuned were higher but not significantly so.
Practical implications: The authors recommend consistent formatting, particularly JSON, to reduce response variability and improve prompt stability.

6 Limitations & Future Work

The study is limited by its reliance on 164 synthetic HumanEval problems, GPT-4o, sample-verified derivatives, and sequential API requests. Future work should test broader datasets, model pairings, stability measures, and adaptable pipelines.

Dataset scope: The experiment used only 164 relatively short, isolated HumanEval problems, limiting evidence for complex datasets and real-life scenarios.
Model alignment: Using Mistral AI to derive the tuned dataset may have reduced quality or adversely retargeted it for ChatGPT evaluation.
Dataset construction: The four HumanEval derivatives were verified only on a sample basis and may contain overlooked prompt-specific inconsistencies, especially the tuned version.
Future work: Future work could investigate model-internal tuning combinations, specialized stability measures such as PSS, and automatic integration into software-development pipelines.
Generalizability: Because performance may vary across models, the guidelines should be verified with other LLMs, particularly specialized coding models.
Measurement constraints: Sequential, non-counterbalanced API requests leave generation-duration effects partly confounded with temporal API-response variation and server traffic.
Future work: The evaluation pipeline currently supports HumanEval and datasets with the same format, requiring adaptation for other datasets.

7 Conclusion

PromptResponse examined five semantically identical HumanEval dataset versions to assess how prompt structure affects coding-task performance, efficiency, and stability. Reformatting generally improved efficiency and stability, with JSON performing most consistently, while the LLM-tuned version worsened results.

Conclusion: Five semantically identical HumanEval versions covered the original dataset, JSON, Markdown, YAML, and an LLM-tuned format.The study compared these variants in a controlled experiment.
Conclusion: Reformatted prompts produced slight task-performance gains and more substantial improvements in generation efficiency and syntactic prompt stability.The reported improvements were strongest for efficiency and stability rather than task performance.
Conclusion: JSON was the only format that stood out across all measured dimensions, while Markdown and YAML improved selected subcategories.The conclusion distinguishes JSON's broad consistency from the more limited effects of Markdown and YAML.
Conclusion: LLM tuning worsened the results, motivating further research into practical agreement between different LLMs.The authors identify cross-LLM agreement as an unresolved issue for prompt tuning.

Supplementary Information

The supplementary materials provide the HumanEval derivatives and the code-generation and evaluation pipeline used in the work, with an explicit dataset-schema assumption.

Supplementary Information: The four HumanEval derivative versions and the code-generation and evaluation pipeline are available from the authors’ GitHub repository.The repository contains both the dataset variants and the pipeline developed for the study.
Supplementary Information: The implementation assumes that the HumanEval dataset contains the columns task_id, prompt, test, and entry_point.These columns are treated as required inputs by the implementation.