CodeContests+: High-Quality Test Case Generation for Competitive Programming

• Public platforms often withhold test cases, limiting the use of available problems and solutions for constructing RL training datasets.Source: §1 · p. 1
• Generated cases can miss tricky or large-scale inputs, while constraint-violating cases can cause false positives and false negatives.Source: §1 · p. 2
• The paper proposes a Generator-Validator system: a generator creates diverse cases, and a validator checks constraints and returns errors for revision.Source: §1 · pp. 2–3
• CodeContests+ retains CodeContests’ problem set with generated replacement tests, evaluated for TPR and TNR using 1.72 million labeled solutions.Source: §1 · p. 3
• At matched TPR and TNR thresholds, CodeContests+ yields twice as many effective problems as CodeContests, and a 32B model shows a clear RL training advantage.The authors report these findings as evidence of better test-case quality.Source: §1 · p. 3

• Manual cases in MBPP and HumanEval are few and have insufficient coverage, while expert-built cases target problem characteristics for only some LiveCodeBench problems.Source: §2 · p. 3
• Mutation methods scale automatically, but may fail complex constraints, introducing incorrect cases and high FNR.The paper notes that mutation was also used to construct CodeContests.Source: §2 · p. 3
• Direct LLM input generation may violate constraints and is limited to small cases by the context window, precluding outputs such as million-vertex graphs.Source: §2 · pp. 3–4

3 The Generator-Validator Agent System

• The LLM-based Generator and Validator address test-case coverage and correctness by generating inputs and checking them against problem constraints.Invalid cases and validator comments are fed back to the generator for revision.Source: §3 · p. 4, Figure 2 · p. 4

3.1 Generator

• The generator agent analyzes constraints and likely solution errors, then produces a testlib generator and roughly 20 commands spanning sizes and special cases.Using the same command and generator produces identical cases across platforms; seeds can vary to scale the number of cases.Source: §3.1 · pp. 4–5

3.2 Validator

• A validator program checks whether each input satisfies problem constraints and can report violated constraints and their locations.Source: §3.2 · p. 5
• Sample inputs and runtime or compilation feedback help catch validator errors, but overlooked constraints lack an automatic supervision method and can leave some incorrect data.The paper says this process detects most validator errors.Source: §3.1 · p. 5, §3.2 · p. 6

4.1 Data Cleaning

• The authors removed low-quality CodeContests problems, including those lacking data ranges, with unclear requirements, or incorrect sample formats, reducing the total problem count from 13,610 to 11,690.In Sections 4.4 and 4.5, the authors focused on test-case quality and selected the same problem subset for both datasets.Source: §4.1 · pp. 6–7

4.2 Generated Test Cases using G-V Agent System

• CodeContests+ replaces the original tests with G-V generators and validators; its preprocessed 1x–5x versions average 25–98 test cases per problem.The generated cases passed validators; only 0.79 million of 1.18 million original CodeContests test cases passed validation (67.1%).Source: §4.2 · p. 7

4.3 Customized Checkers for Multiple-Answer Problems

• A Checker Agent supplies customized checkers for multiple-answer problems, which earlier datasets such as CodeContests lacked.The checker determines whether an output is correct for a given input.Source: §4.3 · p. 7

4.4 Test Case Quality Verification

• Using 1.72 million labeled submissions, the study treats test cases as binary classifiers and uses TPR for correctness and TNR for coverage.The evaluation sampled correct and incorrect submissions for each problem, selecting 10,166 problems with at least 10 of each when available.Source: §4.4 · pp. 7–8
• Over 4,000 CodeContests problems have TPR ≤0.1 and TNR ≥0.9, while similar cases are less prominent with the proposed agent system.The paper attributes the CodeContests pattern mainly to incorrect tests and missing custom checkers for multi-solution problems.Source: §4.4 · p. 8
• With one-quarter as many test cases, CodeContests+ 1x yields over 80% more qualified problems than CodeContests; CodeContests+ 5x yields almost twice as many.The comparison counts problems exceeding TPR and TNR thresholds.Source: §4.4 · p. 8, Figure 3 · p. 8
• Increasing CodeContests+ test cases from 1x to 5x improves overall evaluation accuracy and the number of problems with TPR and TNR ≥0.9.The paper also provides per-problem TPR and TNR so users can set quality-based filtering thresholds.Source: §4.4 · pp. 8–9

4.5 Test Case Quality Matters in RL Training

• CodeContests+ training yields consistent LiveCodeBench gains over CodeContests across difficulty levels, especially Easy and Medium.Table 3 reports Pass@1 across difficulty levels.Source: Table 3 · p. 9, §4.5 · p. 9

5 Conclusion and Future Work

• CodeContests+ enhances the original CodeContests dataset with better test cases produced by the Generator-Validator agent system.The system uses public problem data to construct test cases for competitive programming problems.Source: §5 · p. 9
• The authors report that CodeContests+ test cases are significantly higher quality than CodeContests test cases and that CodeContests+ has substantial advantages in RL training.Source: §5 · p. 9
• More than 100,000 problems have publicly available statements and ground-truth solutions, which the authors estimate their system can help fully utilize as a data foundation for improving LLM reasoning and coding.Source: §5 · p. 10

Appendix

• A submission is a contestant's program, which a judging system evaluates and assigns a verdict such as Accepted, Wrong Answer, or Time Limit Exceeded.Source: A Terms and Definitions · p. 14
• A test case checks whether a submission is correct and usually consists of test input and test output.Source: A Terms and Definitions · p. 14
• Test input is fed to the contestant's program, and its output is compared with the ground-truth data.Source: A Terms and Definitions · p. 14
• Test output is the correct answer for a test input; for problems with multiple correct solutions, it typically represents one possible answer.Source: A Terms and Definitions · p. 14
• A validator is a program that checks whether input data is correct.Source: A Terms and Definitions · p. 14
• A generator is a program that creates test input, including input too large or complex to handcraft.Source: A Terms and Definitions · p. 14
• A checker determines whether a contestant's output is correct, using simpler comparisons or more complex logic for problems with multiple solutions.Source: A Terms and Definitions · p. 14

B Checker

• Many contest problems permit multiple valid outputs, so character-by-character comparison cannot judge them reliably and checker logic is needed.The paper notes that approximately one quarter of Codeforces problems are multi-solution problems.Source: B Checker · pp. 14–15
• Checkers may need custom logic for multiple solutions, floating-point tolerances, or ignoring extra spaces and line breaks; they can also return detailed error feedback.That feedback can identify output differences or error locations and may support future Code LLM reflection.Source: B Checker · p. 15
• The checker agent selects a built-in checker for single-solution problems and implements custom checking logic for problems with multiple valid solutions.Built-in options include integer, floating-point, yes/no, and general full-text checkers.Source: B Checker · p. 15
• The authors test checker implementations against sample input and output, then return failures to the agent for reflection and correction.Some checkers still remain incorrect, particularly for harder multi-solution problems.Source: B Checker · p. 15

C Evaluation as a Service (EaaS): A Cloud Architecture for Large-Scale Code Evaluation

• Code evaluation is an RL-training bottleneck, with computational overhead comparable to the LLM's parameter updates.Source: C Evaluation as a Service (EaaS): A… · p. 15
• The evaluation service runs on a cluster with 25,000 CPU cores: 8,000 judging cores and 17,000 execution cores.The cluster comprises 2,000 4c16g judging pods and 8,500 2c4g execution pods.Source: C Evaluation as a Service (EaaS): A… · p. 16
• Each judging pod handles an entire code request, while each execution pod runs the code on one test input and returns its output for checking.Source: C Evaluation as a Service (EaaS): A… · p. 16
• Execution pods run exclusively on two host cores for isolation and consistent performance, while judging pods use higher concurrency.Source: C Evaluation as a Service (EaaS): A… · p. 16

D.1 Demo: Validator

• The validator treats segment crossings that create interior overlap as proper intersections, but permits shared endpoints and collinear boundary overlap.Collinear segments are checked for bounding-box overlap without treating that contact as area intersection.Source: §D.1 · pp. 16–17
• The validator rejects polygons that are not simple, have nonpositive area, or are not ordered counterclockwise.It also checks for duplicate vertices and nonadjacent edge intersections.Source: §D.1 · pp. 19–20
• Two polygons are invalid when they share interior area, while meeting only at a boundary or point is allowed.The check uses strict point-in-polygon tests and proper edge crossings to detect area overlap.Source: §D.1 · pp. 19–20

D.2 Demo: Generator and Checker

• The generator accepts n, m, k, and a type parameter, and produces one test case with k distinct important-square coordinates.The shown validation requires positive dimensions, n*m <= 200, and 1 <= k <= min(n*m, 7).Source: §D.2 · p. 21
• Its type modes vary flower fills and place buildings in corner, line, block, or random patterns to expose different corner cases.The code also shows uniform fills of 1 or 1000 flowers per square.Source: §D.2 · pp. 21–22
• The checker is for a problem with multiple valid answers and reads the garden input, important squares, and the jury’s minimum total sum.The supplied excerpt introduces the checker but does not show its validation logic.Source: §D.2 · pp. 24–25
• The checker rejects rows with incorrect lengths or invalid characters and verifies that the total matches the number of X squares.These checks compare the computed total with the plan’s stated total.Source: §D.2 · p. 25
• A total above the jury minimum is rejected, while a lower total triggers judge failure and an equal total passes the minimum check.The checker reports acceptance with the minimal total.Source: §D.2 · pp. 25–26
• The checker requires every important square to be covered with concrete and connected to the other important squares.Source: §D.2 · pp. 25–26

D.3 Case Study: Problems with Stronger Test Cases than Official Test Cases

• The excerpted implementation stores minimum, maximum, and answer values in each segment-tree node and combines child nodes by aggregating those values.Its merge operation takes the minimum of the minima, maximum of the maxima, and minimum of the answers.Source: §D.3 · p. 26
• The update function returns when the query range is disjoint from a segment or its stored minimum is at least x; for a fully covered segment whose maximum is below x, it calls relax(x, id, st).Source: §D.3 · p. 27
• The submission passes the official test cases, but on an agent-constructed input it outputs 2 when the correct answer is 3.The authors identify the submission as incorrect.Source: §D.3 · pp. 26, 28