PaperlayerView exact revision on arXiv

Source-linked AI summary

How Prompt Style Affects AI Accuracy and Speed in Competitive Programming

Saad B. Alarifi, Mohammed A. Alam

arXiv:2610.30223v1cs.LGcs.SE

TL;DR

Prompt-style comparisons in competitive programming had addressed accuracy but not runtime speed, and had not situated results against human performance. This exploratory study tests nine prompt styles across accuracy and speed tasks and compares AI accuracy descriptively with human IOI contestants. No style consistently wins: the strongest choice depends on task and summary metric, while outlier runs and differing human-comparison conditions limit interpretation.

  • Problem

    Prior competitive-programming prompt comparisons measured accuracy but not runtime speed, and had not situated prompt-style results against human contestants.

    Source: §I · p. 1

  • Method

    The study evaluates nine prompt styles on three IOI accuracy tasks and two NP-hard speed tasks, with repeated runs and descriptive comparison of accuracy against IOI contestants.

    Source: §I · p. 1, 4) Speed evaluation · p. 3

  • Results

    No prompt style consistently outperformed the others; the strongest depended on task type and whether scores were summarized by mean or median, while many mean differences came from outlier runs.

    Source: §I · p. 1, §VIII · p. 8

  • Takeaways & Limitations

    Within this study’s scope, prompt style should be matched to the objective rather than treated as universal, and a single summary metric can obscure variation in effectiveness.

    Source: §VIII · p. 8

    Each style was represented by one generated prompt per task, so results compare those particular prompts and may partly reflect differences in prompt length or content.

    Source: §VI · p. 7

Abstract

from arXiv · show

Large language models are increasingly used in competitive programming, and their outputs are sensitive to how a prompt is phrased. Prior work has compared prompt strategies for accuracy on competitive programming, but not for runtime speed, and has not situated the results against human performance. In an exploratory study of one prompt in each of nine styles on GPT-5.5 across three accuracy tasks from the International Olympiad in Informatics (IOI) and two NP-hard speed tasks, we found that no single prompt was best: the strongest prompt depended on the task type and on whether results were summarized by the mean or the median, suggesting that, at least for these tasks, prompt style should be matched to the objective rather than treated as universal. Many differences in mean scores came from a few outlier runs, showing that evaluating prompting by a single metric can obscure how much its effectiveness varies. Compared descriptively with human IOI contestants, who worked under different conditions, the AI was neither strictly better nor strictly worse, exceeding the average contestant on one task but falling short on the other two. By extending prompt comparison from accuracy to runtime speed and grounding it against human performance, this study gives a broader picture of how prompt style affects competitive programming performance.

I. INTRODUCTION

Because prompt phrasing can substantially shift language-model performance, this study compares nine prompt styles across competitive-programming accuracy and speed tasks and contextualizes accuracy against human IOI contestants. No style is best across tasks or summary metrics; outlier runs and task-dependent human comparisons further qualify the findings.

I. INTRODUCTION

  • The study evaluates nine prompt styles on three IOI accuracy tasks and two NP-hard speed tasks, with each style run nine times per task.The prompts, generated solutions, grading scripts, and speed test cases are publicly released for re-analysis or extension.Source: §I · p. 1
  • No prompt style was best across all tasks: the strongest style depended on task type and whether performance was summarized by mean or median.Speed was measured by the largest input a solution could handle within a time limit.Source: §I · p. 1
  • After correcting for tests across five tasks, prompt style had a statistically significant effect on only two, and many mean-score differences came from a minority of outlier runs.The authors caution that a single summary score can therefore mislead.Source: §I · p. 1
  • Against human IOI contestants, the AI exceeded the average contestant on one task and fell short on two, remaining below gold-medalist level throughout.The comparison is presented as descriptive context.Source: §I · p. 1

A. AI in Competitive Programming

Prior work has studied AI competitive-programming ability, prompt effects on accuracy, and code-generation prompting, but these lines of work leave prompt-style effects on runtime speed unmeasured in this setting. This study compares prompt styles across accuracy and speed and places AI accuracy alongside human IOI performance.

A. AI in Competitive Programming

  • Earlier Olympiad benchmarks compared AI with human contestants, while other competitive-programming studies measured AI problem-solving without comparing prompt styles.Prior reported results ranged from AlphaCode’s average ranking in the top 54.3% of competitions with more than 5,000 participants to gold-medal performance at IOI 2024; another benchmark found frontier models solved none of the hard problems human experts could solve.Source: A. AI in Competitive Programming · pp. 1–2

B. Prompting Techniques for Code Generation

  • Code-generation prompting includes self-planning and Structured Chain-of-Thought, and runtime-efficiency prompt effects had also been examined on HumanEval and LeetCode.Those standard-benchmark studies reported only limited gains from prompting for efficiency.Source: B. Prompting Techniques for Code… · p. 2

C. Our Contribution

  • Prompt-style comparisons in competitive programming had focused on accuracy, while runtime-efficiency prompting was studied on standard coding benchmarks and human comparisons did not vary prompts.This study extends prompt comparison to runtime efficiency and contextualizes accuracy against human IOI contestants.Source: C. Our Contribution · p. 2

A. Benchmark Selection

  • The benchmark comprises five competitive-programming tasks, with three measuring accuracy and two measuring speed.Source: A. Benchmark Selection · p. 2

B. Accuracy Benchmarks

The study uses three subtask-scored IOI problems to measure solution accuracy and two NP-hard optimization problems to measure runtime speed. It evaluates nine distinct prompt styles, ranging from concise restatements to internally structured planning and reasoning.

B. Accuracy Benchmarks

  • The IOI accuracy benchmarks are Prisoner Challenge, Message, and Migrations, each scored through subtasks that award partial points for different solution quality.Their relatively low constraints reduce time-limit issues, helping isolate accuracy; those same constraints make them unsuitable for speed measurement.Source: B. Accuracy Benchmarks · p. 2

C. Speed Benchmarks

  • Maximum Clique Size and Number Partitioning are NP-hard optimization tasks selected to expose runtime differences across prompt styles.The tasks ask for the largest clique in a graph and the smallest difference between two group sums, respectively.Source: C. Speed Benchmarks · p. 2

A. Prompt Styles

  • Prompts 1–3 use minimal compression, a formal statement, or definitions of symbols and variables before a formal restatement.Source: A. Prompt Styles · p. 2
  • Prompts 4–6 use imperative commands, a requirements checklist, or a competitive-benchmark framing.Source: A. Prompt Styles · p. 2
  • Prompts 7–9 use an educational explanation, internal algorithm planning, or staged computational reasoning before coding.The planning and structured-reasoning styles adapt prior approaches to a single-shot setting, with the reasoning performed internally rather than emitted as intermediate output.Source: A. Prompt Styles · p. 2

1) Prompt generation:

The study generated one self-contained prompt for each of nine styles on each of five tasks, then ran each prompt nine times through GPT-5.5. Accuracy and speed were evaluated with task-specific grading procedures designed to reward correct performance.

1) Prompt generation:

  • GPT-5.5 generated one prompt in each of nine styles for each of five tasks, yielding 45 prompts, with reasoning effort set to high.Each prompt was self-contained, avoided contest and task names, and instructed the solver to output only valid C++20 code.Source: 1) Prompt generation · p. 3
  • Because each style had only one generated prompt per task, the comparisons concern those particular prompts, not every possible prompt in each style.Prompts could also differ in length and content, so effects attributed to style may partly reflect those differences.Source: 1) Prompt generation · p. 3

2) Solution generation:

  • Each of the 45 prompts was sent to GPT-5.5 nine times at medium reasoning effort, producing 405 solutions across the five tasks.Each request was a separate API call without conversation history; reasoning effort was not set higher because of response-generation time.Source: 2) Solution generation · p. 3

3) Accuracy evaluation:

  • Accuracy solutions were graded on oj.uz using each IOI task’s official scoring system and task-specific grading included in the solver prompt.The grading system was included to discourage optimization for the wrong objective.Source: 3) Accuracy evaluation · p. 3

4) Speed evaluation:

  • For speed tasks, each solution ran on identical, increasing-size test cases under a one-second limit, and its largest completed input size was normalized to a score out of 100.Wrong answers, runtime errors, and compilation errors scored zero; a baseline supplied correct answers and a ceiling for comparison.Source: 4) Speed evaluation · p. 3
  • Speed tests used one case per input size, ran solutions in a fixed style-and-iteration order, and stopped each run at the first case exceeding the time limit.The baseline had a 10-second limit per case and was used to establish correct answers and a non-tied ceiling, not to compete on speed.Source: 4) Speed evaluation · p. 3

5) Statistical analysis:

This section specifies the statistical tests, human reference scores, and run structure used to analyze the prompt-style evaluation.

5) Statistical analysis:

  • Kruskal-Wallis tests compared prompt-style score ranks within each task, with Holm–Bonferroni adjustment across five tests.The analysis also used Spearman rank correlation for mean accuracy versus mean speed and reported ε2 = H/(n −1) as an effect size.Source: 5) Statistical analysis · p. 3

C. Human Baseline

  • Human reference points were the mean scores of all IOI contestants and gold medalists, whose scores share the AI’s 0–100 scale.The human and AI scores were earned under different conditions.Source: C. Human Baseline · p. 3

D. Data Recording

  • Prompts, generated solutions and outputs, baseline solutions, and grading scripts for regenerating speed test cases from fixed seeds are publicly available.Results and solver code were saved independently for each prompt, task, and iteration.Source: D. Data Recording · p. 3

A. Overview

  • Each of five tasks used nine prompt styles with nine iterations each, for 405 runs total.The overview identifies the accuracy, consistency, speed, and human-comparison analyses reported in later sections.Source: A. Overview · p. 4

B. Accuracy Results

No prompt style dominated across the accuracy tasks: Prompt 6 led on Prisoner Challenge and Message, while Prompt 4 led on Migrations. Prompt effects also varied in statistical support, with many mean-score differences driven by a minority of unusual runs.

B. Accuracy Results

  • Prompt 6 had the highest mean on Prisoner Challenge and Message, the highest median on Prisoner Challenge, and tied for the highest median on Message.On Migrations, its median was 20, matching most prompts, and its second-highest mean came from two high-scoring runs.Source: B. Accuracy Results · pp. 4–5, Table III · p. 4, Table IV · p. 5
  • Prompt 4 had the highest mean score on Migrations and was the only prompt with a median above 20.01, but prompt style had no significant effect on that task.Five of Prompt 4’s nine runs scored above 20.01, versus 23 of the other prompts’ 72 runs combined; its lead may be chance and warrants more runs.Source: Table III · p. 4, B. Accuracy Results · p. 5, Table IV · p. 5
  • Most prompts tied on median scores, and a minority of unusually high- or low-scoring runs accounted for much of the spread in means.Prompt style affected Prisoner Challenge (p = 0.012) and Message (p = 0.029), but not Migrations (p = 0.53); after Holm adjustment, only Prisoner Challenge remained significant (p = 0.048).Source: B. Accuracy Results · p. 5, Table V · p. 5

C. Speed Results

Speed rankings varied with both the summary statistic and the task. Some mean advantages reflected avoiding failed runs rather than faster typical runs.

C. Speed Results

  • Prompt 3 ranked highest on mean averaged across both optimization tasks, but was tied for third on Maximum Clique Size median and second on Number Partitioning median.Fig. 2 shows mean speed performance averaged across the two tasks; Table VII reports median scores.Source: C. Speed Results · p. 5, Figure 2 · p. 6
  • Maximum Clique Size showed no significant prompt-style effect (p = 0.97); every style had runs completing N = 87 or less and N = 183 or more.The passage attributes the ranking to run-to-run variation rather than prompt-style differences.Source: C. Speed Results · p. 5
  • Prompt 3 was one of only two prompts with no zero scores across both tasks, while Prompt 6 also had no zeros but ranked fourth by mean.The authors say Prompt 3’s mean lead reflected avoiding failures more than being fastest.Source: C. Speed Results · pp. 5–6
  • Number Partitioning median scores ranged from 85.71 to 91.84, with six prompts at 87.76 and typical runs completing N = 42–45.Table VII reports medians over nine iterations for the maximum input completed within the one-second limit.Source: C. Speed Results · p. 6, Table VII · p. 6
  • Number Partitioning showed a prompt-style effect (p = 0.008, Holm-adjusted p = 0.039), but fixed grading order may have introduced thermal throttling.The authors treat this result as tentative; Prompts 8 and 9 were graded last, and scores declined with grading position.Source: C. Speed Results · p. 6
  • On Number Partitioning, mean rankings were largely decided by which prompts avoided failed runs that scored zero.Prompts 1, 2, 4, and 5 each had one zero-scoring iteration.Source: C. Speed Results · p. 6

D. Consistency Across Task Types

Prompt rankings did not transfer reliably between accuracy and speed, and mean differences often reflected outlier runs. Human comparisons varied by task and were descriptive rather than head-to-head.

D. Consistency Across Task Types

  • Prompt 6 led accuracy by both mean and median, while speed leaders varied by statistic and task; average accuracy and speed rankings were uncorrelated (ρ = −0.17, p = 0.67).The study reports no evidence that a prompt’s accuracy ranking predicts its speed ranking.Source: D. Consistency Across Task Types · p. 6
  • After Holm adjustment, prompt style had a significant effect on two of five tasks; many mean differences reflected outlier iterations, though smaller effects cannot be ruled out.Before adjustment, effects were significant on Prisoner Challenge, Message, and Number Partitioning; nine runs per prompt limited detection of smaller effects.Source: D. Consistency Across Task Types · p. 6

E. AI Performance Relative to Human Contestants

  • Prior assessments differed: OpenAI reported gold-medal-level IOI 2024 performance for o3, while a medalist-led benchmark found frontier models solved none of its hardest human-expert problems.The latter study attributed strong results to implementation precision and tool augmentation rather than superior reasoning.Source: E. AI Performance Relative to Human… · p. 6
  • The AI exceeded the average contestant only on Prisoner Challenge and remained below gold-medalist level on all three IOI tasks.Fig. 3 compares mean scores for all contestants, gold medalists, and the AI solver averaged across nine prompt styles.Source: E. AI Performance Relative to Human… · p. 6, Figure 3 · p. 7
  • The average contestant scored 36.84–38.45 across tasks, while the AI mean ranged from 24.88 on Migrations to 67.04 on Prisoner Challenge.The AI’s standing changed sharply by task; with only three tasks, the proposed prior-exposure explanation cannot be separated from other task differences.Source: E. AI Performance Relative to Human… · p. 7
  • Human comparisons were not head-to-head: contestants had five hours and up to 50 submissions per task, while the AI made one submission per iteration without feedback.The authors frame these scores as descriptive context because the evaluation conditions differed substantially.Source: E. AI Performance Relative to Human… · p. 7

VI. LIMITATIONS

The study’s conclusions are bounded by its prompts, tasks, model, and evaluation setup. The authors propose broader and more controlled evaluations to test whether the patterns generalize.

VI. LIMITATIONS

  • One generated prompt represented each style per task, so results concern those particular prompts and may partly reflect differences in prompt length or content.The authors also note that all three IOI tasks predate GPT-5.5’s documented knowledge cutoff.Source: §VI · p. 7

VII. FUTURE WORK

  • Future evaluations could use multiple prompts per style, randomized speed-test order, more test cases and iterations, additional tasks, models, and languages.The authors also propose testing post-training tasks and combining prompt styles.Source: §VII · p. 7

VIII. CONCLUSION

  • Across five tasks, prompt effects were detected on some but not others, and the most effective prompt depended on task type and task; no style consistently outperformed the others.The conclusion says the findings support matching prompt style to the objective within this study’s limits.Source: §VIII · pp. 7–8

APPENDIX A PROMPT STYLE DESCRIPTIONS

The appendix describes nine prompt styles supplied to a prompt-generation model, which produced one concrete prompt per style for each task.

APPENDIX A PROMPT STYLE DESCRIPTIONS

  • Prompts 1–3 used minimal lossless compression, a formal numbered specification, and clear symbol definitions followed by a compact formal restatement.Appendix B lists the generated Number Partitioning prompts; the repository contains prompts for every task.Source: Appendix A · p. 8
  • Prompts 4–6 used direct commands, a compact requirements checklist, and competitive-benchmark framing.The benchmark style frames the goal as achieving the strongest solution under task requirements.Source: Appendix A · p. 8
  • Prompts 7–9 used gradual teaching, plan-first algorithm design, and structured computational stages before implementation.The plan-first style emphasizes algorithm, data structures, invariants, and target complexity; the structured-reasoning style emphasizes computation flow and component interactions.Source: Appendix A · pp. 8–9

APPENDIX B PROMPTS GENERATED FOR NUMBER PARTITIONING

The appendix presents Number Partitioning prompts in distinct styles while retaining the same exact two-group objective and C++20 output constraints. The styles vary in how they frame the task, evaluation conditions, and reasoning instructions.

Prompt 2: Official statement style

  • The official-statement prompt presents the input, two-group objective, output, example, and evaluation rules as numbered requirements.The rules specify a 1-second limit, increasing-N tests, partial credit after TLE, and zero credit for WA or RTE.Source: Appendix B · p. 9

Prompt 3: Symbol-definition style

  • The symbol-definition prompt names N, A1 through AN, the two groups, their sums, and D before requiring the minimum D over all partitions.It also states the exactness, time-limit, scoring, and output requirements.Source: Appendix B · p. 9

Prompt 4: Imperative directive style

  • The imperative prompt gives direct instructions to read the values, assign each to one group, minimize the difference, and print the result.It also requires exactness and describes the evaluation behavior and implementation constraints.Source: Appendix B · p. 9

Prompt 5: Structured checklist style; Prompt 6: Competitive benchmark style

  • The checklist prompt organizes input, operation, objective, output, example, environment, evaluation behavior, and output restrictions into a list.The benchmark-framed prompt emphasizes exact solutions, performance on large N, and consequences of TLE, WA, and RTE.Source: Appendix B · p. 9

Prompt 7: Educational teaching style

  • The educational prompt explains the partition objective through placing positive integers into two boxes and making their sums as close as possible.It retains the exactness, efficiency, C++20, and scoring constraints.Source: Appendix B · p. 9

Prompt 8: Plan-first algorithm design style

  • The plan-first prompt asks the solver to identify an exact algorithm, correctness invariants, and time and memory targets before implementation.It also asks for internal stages that cover parsing, reachable group splits, minimizing the difference, and optimizing for large N under a 1-second limit.Source: Appendix B · pp. 9–10
Loading 2610.30223v1…