PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoCode: LLMs as Problem Setters for Competitive Programming

Shang Zhou, Zihan Zheng, Kaiyuan Liu, Zeyu Shen, Zerui Cheng, Zexing Chen, Hansen He, Jianzhu Yao, Huanzhi Mao, Qiuyang Mang, Tianfu Fu, Beichen Li, Dongruixuan Li, Wenhao Chai, Zhuang Liu, Aleksandra Korolova, Peter Henderson, Natasha Jaques, Pramod Viswanath, Saining Xie, Jingbo Shang

arXiv:2510.12803v1cs.SEcs.AIcs.CLcs.PL

TL;DR

Competitive-programming problem setting demands algorithmic depth and reliable edge-case coverage, yet synthesized test datasets can misjudge solutions. AutoCode uses a closed-loop test-generation framework and dual verification to generate tests and new problems. It reports more than 98% agreement with official judgments and novel problems validated by experts.

  • Problem

    Competitive-programming problems require deep algorithmic understanding and rigorous test coverage, while synthesized datasets can misjudge solutions through false positives and false negatives.

    Source: §1 · p. 2

  • Method

    AutoCode combines a Validator-Generator-Checker test pipeline with dual verification that compares efficient and brute-force solutions for generated problems.

    Source: §1 · p. 3, §3 · p. 4

  • Results

    More than 98% agreement with official judgments across over 7,500 problems accompanies successful production of novel problems validated by expert programmers.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The authors argue that problem-generation ability should be tested alongside problem solving, particularly for advanced programming tasks.

    Source: §1 · p. 1

    The authors acknowledge limitations to the problem-generation pipeline and identify them as avenues for future research.

    Source: A8. Limitations and Future Work · p. 27

Abstract

from arXiv · show

Writing competitive programming problems is exacting. Authors must: set constraints, input distributions, and edge cases that rule out shortcuts; target specific algorithms (e.g., max-flow, dynamic programming, data structures); and calibrate complexity beyond the reach of most competitors. We argue that this makes for an ideal test of general large language model capabilities and study whether they can do this reliably. We introduce AutoCode, which uses multiple rounds of validation to yield competition-grade problem statements and test cases. On held-out problems, AutoCode test suites approach 99% consistency with official judgments, a significant improvement over current state-of-the-art methods like HardTests, which achieve less than 81%. Furthermore, starting with a random seed problem, AutoCode can create novel variants with reference and brute-force solutions. By cross-verifying these generated solutions against test cases, we can further filter out malformed problems. Our system ensures high correctness, as verified by human experts. AutoCode successfully produces novel problems judged by Grandmaster-level (top 0.3%) competitive programmers to be of contest quality.

1. Introduction

The introduction argues that competitive-programming problem setting is a demanding test of LLM capability because it must expose shortcuts and rigorously cover edge cases. AutoCode combines a multi-role test-generation pipeline with dual verification to generate and vet novel problems.

1. Introduction

  • The authors argue that testing LLM problem-generation ability matters alongside problem solving, particularly for advanced programming tasks and future integration into software development.They cite the need to validate model-generated code with high fidelity.Source: §1 · pp. 1–2
  • Contest problem setting must expose shortcuts and corner cases while assessing algorithm design, data structures, and complexity trade-offs.Verifying the broad space of possible solutions and providing sufficient test coverage are necessary challenges.Source: §1 · pp. 1–2
  • Existing synthesized datasets can have high false-positive and false-negative rates, letting inefficient greedy solutions pass small random tests and malformed inputs penalize valid solutions.The authors connect these dataset errors to distorted evaluation and polluted reinforcement-learning data.Source: §1 · pp. 2–3
  • AutoCode’s Validator-Generator-Checker framework generates test cases, validates constraints, and checks solution correctness; dual verification compares efficient and brute-force solutions.New problems are accepted only when outputs match brute-force ground truth across test cases.Source: §1 · p. 3
  • Over 91% consistency with official judgments on 7,538 problems exceeds existing methods’ 72–81% range.The benchmark covers a large set of existing problems.Source: §1 · p. 3
  • 94% of problems passing automated verification were correct, and elite programmers judged generated problems high-quality and original enough for official contests.The generation process uses dual verification and expert vetting.Source: §1 · p. 3

2. Related Works

Related work covers coding benchmarks, test-generation methods, and solver or data-centric approaches. AutoCode combines test legality and adversarial coverage with reference-versus-brute-force verification to generate and certify new problems.

2. Related Works

  • Coding benchmarks assess interview, competitive-programming, and domain-specific tasks [13] [14] [15] [16], while other work strengthens verdict reliability through rule- or LLM-driven test generation [2–4, 17–20].Solver and data-centric approaches instead scale solution search or curate verified corpora [21] [22] [23] [24].Source: §2 · p. 3
  • AutoCode unifies test legality and adversarial coverage with dual verification to certify new problems, addressing static-benchmark contamination and under-constrained tests.The dual-verification protocol compares reference and brute-force solutions.Source: §2 · p. 4

3. Test Case Generation

AutoCode builds test data through a Validator-Generator-Checker loop, with an Interactor extension for interactive problems. Its components filter invalid inputs, broaden coverage, assess outputs, and distinguish correct solutions from mutated ones.

3. Test Case Generation

  • The framework validates inputs, generates test cases, and checks contestant outputs against reference outputs; interactive tasks add an Interactor for multi-turn verdicts.It distinguishes test cases (input-answer pairs) from test data, which also includes the Checker and Interactor programs.Source: §3 · p. 4

3.1. Validator

  • A Validator is selected from three LLM-generated candidates by their accuracy on 40 cases: 10 valid inputs and 30 near-valid illegal inputs.The selected program rejects inputs that violate the problem’s explicit constraints.Source: §3.1 · p. 4

3.2. Generator

  • The Generator combines small-data exhaustion, randomized and extreme inputs, and TLE-inducing cases to cover corner cases, implementation faults, and complexity weaknesses.A Validator filters invalid inputs before test cases are deduplicated by signature.Source: §3.2 · p. 5

3.3. Checker

  • The Checker is selected from three candidates using 40 cases that pair valid inputs with plausible, potentially mistaken outputs and expected verdicts.Each candidate is evaluated against the reference output and expected verdict; the best-performing Checker joins the final test data package.Source: §3.3 · pp. 5–6

3.4. Interactor

  • For interactive tasks, the system selects an Interactor that passes the reference solution while rejecting the greatest number of slightly incorrect mutant programs.Mutants are created through small logical edits, and candidate Interactors are tested in simulated interactions.Source: §3.4 · p. 6

4. Benchmarking Test Case Robustness

AutoCode’s tests achieve high agreement with official judgments on both a 7,538-problem benchmark and a more challenging set of recent Codeforces problems. The evaluation also reports error distributions and ablations of framework components.

4. Benchmarking Test Case Robustness

  • The primary benchmark contains 7,538 problems from four existing datasets and excludes interactive problems; its average difficulty is slightly below a typical Codeforces round.The second benchmark contains 720 recent rated Codeforces problems and is unfiltered.Source: §4 · p. 7
  • 91.1% consistency on 7,538 problems, while existing methods do not surpass 81.0%; AutoCode’s FPR is 3.7% and FNR is 14.1%.The paper reports these FPR and FNR values as approximately 50% lower than the current state of the art.Source: §4 · pp. 7–8
  • For 85.6% of problems in the 720-problem benchmark, verdicts are consistent with official judgments; 97.1% have three or fewer inconsistencies.Each problem is evaluated using 33 submissions generated by different LLMs.Source: Figure 2 · p. 8
  • 98.7% consistency on 720 recent, unfiltered Codeforces problems, including complex interactive tasks, where prior methods could not be evaluated.The benchmark was created to test the system under more challenging, real-world conditions.Source: §4 · pp. 7–8
  • Ablations show changes in consistency, FPR, and FNR when individual components are removed from the full framework.Removing prompt optimization drops consistency from 98.7% to 98.0% and raises FNR to 2.9%; removing exhaustive or random/extreme generation raises FPR to 1.7% or 1.6%, respectively. Removing Validator or Checker selection tests raises FNR to 1.3% or 1.4%, respectively.Source: §4 · p. 8

5. Problem Generation

AutoCode adapts seed problems into new programming challenges and uses dual verification to check their correctness. Expert evaluations find useful and creative outputs, while also revealing limits in LLM novelty and self-assessment.

5. Problem Generation

  • Human expert filtering found 61.6% suitable for LLM training and 3.2% suitable for ICPC/IOI-level contests.Levels 2–6 are correct and usable; Level 1 questions are wrong.Source: Figure 3 · p. 9
  • AutoCode adapts a random Codeforces seed rated below 2200, generating a new problem plus efficient and brute-force solutions that are cross-checked on test data.The brute-force solution acts as an initial ground truth, followed by another full test-generation cycle for the reference solution; this filters 27% of error-prone problems and raises reference-solution correctness from 86% to 94%.Source: §5 · p. 9
  • After filtering, over 80% of generated problems were judged suitable as training data and 23% as involving novel or creative designs.The paper reports these proportions alongside the expert grading criteria.Source: §5 · p. 9
  • About 4.2% of generated problems are solvable by other models or humans even though the generating model fails to solve them.The authors describe these logically sound problems as a potential source for model self-improvement.Source: §5 · p. 9
  • Generated problems tend to recombine existing algorithmic knowledge and established frameworks rather than introduce entirely new problem models or reasoning strategies.They more often test specific algorithmic knowledge or complex implementation than clever design or subtle observation.Source: §5 · p. 10
  • Adapted problems become about 334 Elo harder on average; novel ones gain 498, versus 108 for non-novel ones, and quality is highest with moderately difficult seeds.Very hard seeds leave little room for effective changes, while very easy seeds can remain below high-quality difficulty even after adaptation.Source: §5 · p. 10
  • o3–human correlations are 0.07 for quality and 0.11 for novelty, while difficulty correlates with human quality up to 0.60 and LLM-predicted difficulty has a proxy correlation around 0.18.Human experts and LLMs show high within-group quality consistency, but LLM self-evaluation alone is inadequate.Source: §5 · pp. 10–11, Figure 4 · p. 11

6. Conclusion

AutoCode combines a multi-role LLM framework with dual verification to generate test cases and novel programming problems. It reports high agreement with official judgments and expert-validated novel problems, while identifying weaknesses in LLM authorship and self-evaluation.

6. Conclusion

  • AutoCode reduces false positives and false negatives on more than 7,500 problems and recent Codeforces benchmarks, achieving over 98% agreement with official judgments.The conclusion also reports state-of-the-art test-generation reliability.Source: §6 · p. 11
  • A closed-loop framework combines Validator-Generator-Checker and Interactor roles with dual verification to produce test cases and novel contest-quality problems.The framework is presented as extending prior methods beyond test generation to new problem creation.Source: §6 · p. 11
  • Expert programmers validate the generated novel problems, while the analysis finds that LLMs recombine algorithmic knowledge but struggle with new reasoning paradigms and flawless sample design.Source: §6 · p. 11
  • Predicted difficulty and difficulty gain serve as proxy signals for problem quality, which the authors identify as a scalable path toward self-play.Source: §6 · p. 11

Appendix

The appendix lists supplementary material on preliminaries, generated-problem examples, input validity, mutant programs, grading, the interactor, and limitations.

Appendix

  • Appendix A1 provides preliminaries and definitions, while A2 and A3 give examples of a contest-grade generated problem and a problem the LLM cannot solve.Source: Appendix · p. 15
  • Appendices A4–A6 cover valid and near-valid illegal inputs, incorrect mutant programs, and the human-expert grading policy.Source: Appendix · p. 15
  • Appendix A7 gives simulator pseudocode for the interactor, and A8 covers limitations and future work.Source: Appendix · p. 15

A1. Preliminaries

The preliminaries section introduces judgment-process background, key terminology, and the AutoCode framework’s core components.

A1. Preliminaries

A1.1. Competitive Programming Solution Judgment

Competitive-programming judging runs a solution against test cases and returns one verdict. Acceptance requires passing every case within the time and memory limits.

A1.1. Competitive Programming Solution Judgment

A1.2. Key Terminologies

The section defines platform and difficulty terminology alongside components used to generate and evaluate competitive-programming test cases. These components include specialized judging for interactive problems and nonstandard outputs.

A1.2. Key Terminologies

  • Codeforces is a competitive-programming contest platform whose problem difficulty is often used as a benchmark.Source: A1.2. Key Terminologies · p. 15
  • Codeforces Elo ratings reflect contestants’ performance; ratings ≤2000 are easy to medium, (2000, 3000] medium to hard, and ≥3000 very hard.Source: A1.2. Key Terminologies · p. 15
  • A Generator produces test inputs, with diverse cases covering edge, average, and worst-case scenarios that challenge incorrect algorithms.Source: A1.2. Key Terminologies · p. 16
  • A Validator checks whether test cases satisfy the problem’s stated constraints.Source: A1.2. Key Terminologies · p. 16
  • A Checker compares contestant output with correct answers, including custom rules for multiple valid solutions, floating-point tolerances, or flexible formats.Source: A1.2. Key Terminologies · p. 16
  • An Interactor conducts real-time query-and-response exchanges with a solution that does not receive all input at once.Source: A1.2. Key Terminologies · p. 16

A2. Example of LLM Generated Problem: Row–Column Portal

The example asks for the fewest row and column flips needed to turn some allowed sub-rectangle of a binary grid into a portal, or to report impossibility.

A2. Example of LLM Generated Problem: Row–Column Portal

Input

This section presents problem specifications, examples, and a faulty mutant, along with the dimensions used to assess problems. It also defines a rubric for solvability, correctness, quality, novelty, and difficulty, and describes a simulator procedure and limitations requiring human judgment and further research.

A3. Example of LLM Generated Problem that Itself Cannot Solve: IT Restaurants — Distance-2 Rivalry

  • The restaurant variant forbids different brands at adjacent nodes and the same brand at distance two, then asks for optimal restaurant-count pairs using both brands.The output lists all pairs maximizing the total count, sorted by the first count; if no valid placement uses both brands, it prints 0.Source: A3. Example of LLM Generated Problem… · pp. 19–20, Output · p. 20

Explanation of the new rules

  • The restaurant rules couple siblings: an occupied node permits at most one occupied child, while an empty node permits at most one child of each brand, invalidating the earlier subset-sum approach.The passages say a new DP is needed.Source: Explanation of the new rules · p. 20

Examples

  • The examples give a path placement with three restaurants and totals (2, 1) or (1, 2), while a star yields exactly (1, 1).For the star, the center is empty and at most one leaf hosts each brand.Source: Examples · p. 21, Output · p. 21

A4. Near-Valid Test Case Example: Perpendicular Segments time

  • The perpendicular-segments task requires integer-coordinate segments inside an X-by-Y box, each of length at least K, whose containing lines meet at a right angle.The input guarantees that a solution exists; the segments themselves need not intersect.Source: A4. Near-Valid Test Case Example:… · p. 21

A4.1. NEAR-VALID TEST CASE EXAMPLE

  • A near-valid test pair differs only in K: one 4×4 case admits perpendicular diagonals, while the other is impossible because its required length exceeds the box’s 5.657 diagonal.The longer threshold is 6, so even a single qualifying segment cannot fit.Source: A4.1. NEAR-VALID TEST CASE EXAMPLE · p. 22

A5. Example of incorrect mutant programs: Guess The Tree

  • The interactive tree task reconstructs a secret tree from queries returning a node minimizing the difference between its distances to two queried nodes, with at most 15n queries.The input bounds n by 1000 and the sum of n by 1000; the hack format supplies the hidden tree’s n−1 edges.Source: A5. Example of incorrect mutant… · p. 23, Input · p. 23, Hacks · p. 23

A5.1. Why it’s Wrong

  • A subtly broken tree-reconstruction mutant compares a node index with the found node’s component label instead of comparing their component labels.It can pass the sample yet fail on hidden cases.Source: A5.1. Why it’s Wrong · pp. 24–25

A6. Human Expert Grading Policy

A6.1. Solvability

  • The solvability criterion asks whether a deterministic solution meets the time and space constraints, with outcomes of yes, no, or unknown.Source: A6.1. Solvability · p. 25

A6.2. Solution Correctness

  • Solution correctness assesses whether the reference solution’s core approach is sound; minor implementation errors may remain, while fundamental flaws and indeterminate cases are distinguished.Source: A6.2. Solution Correctness · pp. 25–26

A6.3. Quality Rating

  • The 1–5 quality rubric ranges from severely flawed (1) and low quality (2), through moderate (3) and high (4), to competition-grade excellent (5).It considers clarity, originality of approach, and implementation difficulty; high and excellent ratings require a clear, solvable problem, with excellent problems also elegant, novel, appropriately challenging, and free of contrived traps or rote patterns.Source: A6.3. Quality Rating · p. 26

A6.4. Novelty

  • Novelty is judged as yes when a problem is new even to experienced setters, no when it duplicates or closely resembles existing problems, and unknown when prior similarity is uncertain.Source: A6.4. Novelty · p. 26

A6.5. Difficulty

  • Difficulty is recorded on the Codeforces rating scale as a reference and does not directly affect the overall rating.Source: A6.5. Difficulty · p. 26

A6.6. Overall Problem Rating

  • The composite rating requires a problem to meet the requirements of its assigned level and every lower level; level ≥0 includes all evaluated problems.Source: A6.6. Overall Problem Rating · p. 26

A7. Simulator in Interactor

  • The interactor simulator initializes protocol state, limits, and a deterministic RNG, checks messages during the session, judges final answers, and returns accepted or rejected.Source: A7. Simulator in Interactor · p. 27

A8. Limitations and Future Work

  • LLM quality judgments correlate little with human experts’ scores, so AutoCode still needs human annotation to distinguish high-quality, novel problems from merely technically correct ones.Closed-source models and immature reinforcement-learning frameworks impede self-improvement; generated problems tend to recombine existing algorithms, and verification filters out valid problems the model cannot solve.Source: A8. Limitations and Future Work · pp. 27–28

Related papers

Loading 2510.12803v1…