Beyond the Verdict: Agentic Code Judging with Verifiable Feedback

1 INTRODUCTION

• The task asks a verifier to judge one candidate and, when it suspects a fault, produce a candidate-specific counterexample whose legality and effect can be checked.Official judges provide verdicts but usually no failing input or explanation, and finite submission budgets limit how freely that feedback can be requested.Source: §1 · p. 2
• 40.61% to 62.67% overall counterexample success and 71.41% to 87.41% USACO-JUDGE mean binary verdict accuracy followed training; IOI refinement scored 385.46 expected points, above gold.Under the same 50-submission cap, the expected ARBITER score crosses the gold threshold in round 4 while online-judge feedback remains below it.Source: Figure 1 · p. 1, §1 · p. 2
• USACO-JUDGE evaluates accepted and unaccepted programs from human experts and multiple LLMs against official USACO 2026 tests.Source: §1 · p. 2
• ARBITER is a 35B MoE agentic verifier trained with reinforcement learning, with reported results in verification, code selection, and refinement.Source: §1 · p. 2

2 RELATED WORK

• Existing feedback approaches include solution and step scoring, generative explanations, and tool-using verifiers; ARBITER connects verdicts for selection with diagnoses and counterexamples for refinement.Source: §2 · p. 2
• A predicted verdict is not an executable witness, and a disagreement input alone does not establish which program is wrong; a legal input that breaks one candidate supports its rejection.Source: §2 · p. 2
• The Easy and Hard UOJ-BENCH hacking splits contain only faulty programs, so they cannot test recognition of accepted code; USACO-JUDGE includes both classes.Source: §2 · pp. 2–3

3 VERIFICATION TASK

• Figure 2 presents the verifier’s answer before judge-side evaluation, whose results are not revealed during the verifier’s episode.Source: Figure 2 · p. 3
• The verifier can use a generic sandbox to construct and run tests, then submit a verdict, natural-language diagnosis, and deterministic Python generator; the generator is empty for Accepted and Compile Error.Source: §3 · p. 3
• A legal generated input that breaks the candidate establishes failure, but failure to find one does not establish acceptance; counterexample success is a checkable proxy, not proof of every diagnosis claim.Source: §3 · p. 3

4.1 TRAINING DATA

• The 20,000-program training set draws from 6,145 problems, excludes evaluation problems, and contains 15% accepted programs.A separate validation set contains 400 candidates from 100 hard problems with no overlap with training or test problems.Source: §4.1 · p. 4

4 TRAINING ARBITER

• The training policy samples multi-turn sandbox episodes and optimizes them with GRPO; token-budget truncations receive zero reward, while wall-clock-aborted episodes are excluded from updates.ARBITER starts from Qwen3.6-35B-A3B and predicts a verdict, diagnosis, and counterexample generator.Source: §4 · p. 3, §4.2 · p. 4

4.2 AGENTIC REINFORCEMENT LEARNING

• Unaccepted-verdict agreement earns small intermediate rewards, while a legal input that breaks the candidate earns 0.75 and matching its failure type adds 0.05.A verified unaccepted prediction totals 1.00; the diagnosis itself receives no correctness reward.Source: §4.2 · p. 4, Figure 3 · p. 4
• On the hard split, a false-positive penalty of λ = 0.5 raises Accepted precision from 77.78% to 80.98% while lowering recall from 76.67% to 62.86%.The final ARBITER uses λ = 0.Source: §4.2 · p. 4, Figure 4 · p. 5

4.3 EFFICIENT TRAINING THROUGH LENGTH CONTROL

• Length control shortens responses, reduces rollout timeouts, and avoids the late decline in mean reward seen without it.Because continuous penalties can curtail exploration, the authors alternate penalized and unpenalized phases.Source: §4.3 · p. 5, Figure 5 · p. 5

5.1 UOJ-BENCH

• UOJ-BENCH contains only faulty programs, so it tests counterexample discovery but cannot test whether a verifier recognizes accepted code.A verifier predicting unaccepted for every candidate could achieve perfect binary verdict accuracy without finding a counterexample.Source: §5.1 · p. 5

5.2 USACO-JUDGE

• USACO-JUDGE contains 900 C++17 candidates for 39 held-out USACO 2026 problems, with accepted and unaccepted programs spanning Bronze through Platinum.Candidates come from five LLMs and five human competitive programmers; each tier uses a 1:2 accepted-to-unaccepted ratio.Source: §5.2 · pp. 5–6, Figure 6 · p. 6
• Every USACO-JUDGE label is an actual official-test verdict, retained only when rejudging reproduces the candidate’s original score.Validators and custom checkers are verified end to end, and multiple runs check that judge variability does not change candidate status.Source: §5.2 · p. 6

6.1 VERIFICATION QUALITY

• Training raises overall hack success from 40.61% to 62.67%; on hard splits, verdict accuracy rises from 67.43% to 83.40% on UOJ-BENCH and from 65.34% to 84.50% on USACO-JUDGE.The resulting 35B ARBITER outperforms non-agentic models as large as 1.6T parameters; on USACO-JUDGE, it has the highest reported hack success among evaluated open-weight models on Easy and Hard and highest reported Hard verdict accuracy.Source: §6.1 · p. 7, Table 1 · p. 7
• Repeated sampling improves all three Qwen3.6-35B-A3B variants, while one ARBITER sample exceeds eight no-tools samples on every split.Figure 7 reports Hack@k using eight samples per candidate, with k denoting the size of the uniformly chosen subset.Source: §6.1 · p. 7, Figure 7 · p. 7
• In a 400-diagnosis audit, counterexamples succeed for 71.6% of correct diagnoses and 10.3% of incorrect or vacuous ones; 82.2% locate a real bug’s cause and 54.0% suggest a fix.Three competitive programmers independently agreed on all 400 initial labels.Source: §6.1 · p. 7, Figure 8 · p. 8

6.2 CODE REFINEMENT

• At Best@8, V + D scores above Online Judge for both solvers at every difficulty tier, and the V + D versus V + X comparison favors diagnosis as repair guidance.Adding counterexample-generator source X to V + D brings no consistent gain.Source: §6.2 · p. 8, Table 2 · p. 8
• 385.46 expected points with ARBITER-guided GPT-OSS-120B exceeds the 361.12 gold threshold under the official 50-submission cap per problem, versus 334.6 with online-judge feedback.The six-problem IOI 2026 case study counts submissions per problem; ARBITER reviews use no slots, while each online-judge feedback query uses one.Source: §6.2 · p. 8

6.3 CODE SELECTION

• Across all eight settings, one ARBITER call per candidate has higher reported pass@1 than the strongest published baseline; eight calls improve every setting further.With the 30B solver and a pool of 64, ARBITER scores 82.6 versus 73.7 on LiveCodeBench and 43.6 versus 34.9 on OJBench; eight calls move selection closer to the perfect-selection upper bound.Source: §6.3 · p. 9, Table 3 · p. 9

• On USACO-JUDGE, candidate-weighted verdict accuracy rises from 71.41% to 87.41%, and overall counterexample success rises from 40.61% to 62.67%.The conclusion also reports that ARBITER leads reported code-selection results and improves refinement over evaluated baselines.Source: §7 · p. 9
• ARBITER combines candidate verification and counterexample-grounded diagnosis.Source: §7 · p. 9
• In the IOI 2026 case study, ARBITER-guided GPT-OSS-120B reaches 385.46 expected points under the official 50-submission cap per problem, above the 361.12 gold threshold.Source: §7 · p. 9

• The verifier can run Python and shell commands in a generic sandbox, but cannot access official tests, reference solutions, validators, checkers, or counterexample-checking results.The final submit call commits a verdict, diagnosis, and counterexample generator.Source: A EVALUATION INFRASTRUCTURE AND… · p. 13
• The judge evaluates candidates on hidden tests and checks generated inputs for legality and failure against the reference solution and output checker.A counterexample succeeds only when it is legal and the candidate fails on it; these checks provide training labels and rewards and evaluate benchmarks, but are not exposed during verifier episodes.Source: A EVALUATION INFRASTRUCTURE AND… · p. 13
• Candidates run in isolated sandboxes under problem-specific time and memory limits on a deployment with up to 30 parallel judger containers.The UOJ service and database run in separate Docker containers from the judger pool.Source: A EVALUATION INFRASTRUCTURE AND… · p. 13
• Submissions, references, validators, and checkers are compiled as C++17 with -O2, while counterexample generators may use Python.This compilation configuration matches the source contests; generators produce inputs rather than judged contest solutions.Source: A EVALUATION INFRASTRUCTURE AND… · p. 13

B.1 TRAINING DATA COMPOSITION

• The 20,000-program training pool is grouped by score band and judge verdict; its 76–99 near-miss band contains wrong answers, time-limit, memory-limit, and runtime failures.A time-limit failure in that band is labeled TLE.Source: §B.1 · p. 13, Table 4 · p. 13

B.2 UOJ-BENCH RE-HOSTING

• UOJ-BENCH evaluation re-hosts each target problem with its validator, reference solution, and checker, and a generated hack succeeds when a valid input makes the candidate fail or its output is rejected.The released candidates are replayed to reproduce the original outcomes.Source: §B.2 · p. 14

B.3 USACO-JUDGE VALIDATORS AND CUSTOM CHECKERS

• Because USACO’s release omits validators for arbitrary generated inputs, the authors wrote validators for all 39 problems; six problems also require custom output checkers.Those six accept multiple correct outputs or ask for a witness; the other 33 use built-in token comparators.Source: §B.3 · p. 14, Table 5 · p. 14

B.4 USACO-JUDGE CONSTRUCTION PIPELINE

• USACO-JUDGE’s 900 candidates are drawn from C++17 programs generated by five LLMs and written by five human competitive programmers, with quotas split by score and verdict.Candidates are deduplicated, selected for diversity, and rejudged on UOJ before inclusion.Source: §B.4 · p. 14
• The hard tier contributes 630 candidates from 21 problems and the easy tier 270 from 18, with each tier using a 1:2 ratio of accepted to unaccepted programs.Programs are chosen by score-band quotas, with a diversity procedure that uses farthest-first selection and penalizes candidates from already represented authors.Source: §B.4 · pp. 14–15
• Program distance combines token 5-gram, AST-subtree, feature-vector, and per-test runtime signals; missing parse or timing terms are dropped and remaining weights rescaled.The weights are 0.25, 0.30, 0.20, and 0.25, respectively.Source: §B.4 · p. 15
• The benchmark retains candidates only when UOJ rejudging matches the sampling score, and accepted candidates are judged several times to catch runtime-driven verdict changes.Source: §B.4 · p. 15

C TRAINING DETAILS

• Each GRPO update samples 2,048 trajectories, uses a 65,536-token context cap and at most 30 tool calls per episode, and enables the length penalty only during selected step intervals.The penalty is active during steps [90, 110), [160, 180), and [230, 250).Source: C TRAINING DETAILS · p. 15

D.1 HACK SUCCESS AND VERDICT ACCURACY ARE COMPLEMENTARY

• Hack success changes only from 59.8% to 60.9% while verdict accuracy falls from 74.8% to 57.8% between steps 79 and 179.Fewer skipped attacks offset more rollout aborts; the length-controlled run limits abort growth and retains higher verdict accuracy.Source: §D.1 · p. 15

D.2 OUTPUT TOKENS AND ESTIMATED COST

• ARBITER averages 35.0K–46.9K output tokens per candidate, 22%–33% fewer than its agentic Qwen3.6-35B-A3B baseline, while overall hack success rises from 40.61% to 62.67%.Mean output cost is $0.0045 per verification, versus $0.0063 for agentic Qwen3.6-35B-A3B and $0.0298 for agentic DeepSeek-V4-Flash-Preview; Table 6 estimates output-token cost.Source: §D.2 · pp. 15–16, Table 6 · p. 16

D.3 VERIFICATION FUNNEL

• The verification funnel conditions execution, validity, and success rates on reaching the preceding stage, while UOJ-BENCH’s unaccepted-only pool precludes accepted precision and recall there.For each split, Attempt measures the share of unaccepted candidates for which a counterexample generator is submitted.Source: §D.3 · p. 16
• Attempt rises by 8–23 percentage points and conditional counterexample success by 7–13 points across four splits, as training improves both attack coverage and conversion.Execution and input validity exceed 94.7% on every split; accepted-program decisions also improve on the hard split.Source: §D.3 · p. 16

D.4 INTERACTION TURNS

• ARBITER averages 14.2–18.7 interaction turns per candidate, only 1.2–2.1 more than Qwen3.6-35B-A3B and fewer than agentic DeepSeek-V4-Flash-Preview on every split.All three systems operate at a similar interaction scale.Source: §D.4 · p. 19, Table 9 · p. 19

D.5 EXAMPLE ACCEPTED TRAJECTORY

• In an accepted-program example, ARBITER returns Accepted after four turns and identifies the invariant that unaffected windows need not be recomputed as cell values increase.The diagnosis also reports O(QK^2) time and agreement with differential testing.Source: §D.5 · p. 19, Table 10 · p. 19

D.6 BREAKING OFFICIALLY ACCEPTED SUBMISSIONS

• Two submissions scoring 100 on official tests were shown to fail on the deployed judge: one by time limit and one by wrong answer.The examples demonstrate failures in officially accepted submissions, not a general rate of such failures.Source: §D.6 · p. 19
• A one-group input with 200,000 cows triggers 4 × 10^10 cleanup-loop iterations and a time-limit failure in Hoof, Paper, Scissors Triples.The candidate’s quadratic postprocessing remained after it solved the main geometric counting problem.Source: §D.6 · pp. 19–20
• For Min Max Subarrays II, a 14-byte input with minimum zero exposes a sentinel collision: the candidate drops a valid constraint and prints -1.The candidate uses zero both as a legal constraint value and to mean no minimum constraint was recorded.Source: §D.6 · p. 20

E SELECTION DETAILS

• Selection ranks candidates by their Accepted counts, and ARBITER accepts 88.81% and 88.28% of correct programs while rejecting 96.36% and 93.67% of unaccepted programs from the 30B and 235B solvers, respectively.False acceptance can place an incorrect program in the top group; this occurs for 3.64% and 6.33% of unaccepted candidates, respectively.Source: E SELECTION DETAILS · p. 21, Table 13 · p. 21
• Calls two through eight add 0.9–3.4 pass@1 points; one call exceeds the published Agentic Verifier baseline in all eight settings.For pools of 64, OJBench improves through eight calls, while most LiveCodeBench gains arrive within four; smaller pools plateau earlier.Source: E SELECTION DETAILS · p. 21, Figure 9 · p. 22

F.1 SUBMISSION PROTOCOL

• The refinement protocol budgets at most 50 judged submissions per problem; the 50 initial trajectories per target are internal candidates and consume no submission slots.Round 0 is the shared, already judged starting program and is excluded from the budget of additional submissions.Source: §F.2 · p. 22, §F.1 · p. 22

F.2 IOI 2026 PROTOCOL AND SCORE ACCOUNTING

• ARBITER-guided refinement raises expected score from 240.49 before refinement to 385.46 by round 4, above the 361.12 gold cutoff, under the 50-submission-per-problem cap.Online-judge feedback reaches 334.6 under the same cap; the policies’ inference compute is not matched.Source: §F.2 · pp. 22–23

F.3 BINARY VERDICT QUALITY DURING REFINEMENT

• Refinement results use programs selected by ARBITER’s predicted verdicts without correcting false acceptances or false rejections after evaluation.The reference Accepted rates are 68.6% for GPT-OSS-120B and 46.2% for Qwen3.6-35B-A3B.Source: §F.3 · p. 23, Table 14 · p. 23

F.4 DIAGNOSIS QUALITY AND THE COUNTEREXAMPLE PROXY

• 134 of 163 correct bug diagnoses identify specific code, and 88 also state the behavior needed for a fix.Of 400 audited diagnoses, 332 were correct and 65 incorrect; three were vacuous.Source: §F.4 · pp. 23–24
• 71.6% of correct diagnoses with counterexamples break the candidate, versus 10.3% of incorrect or vacuous diagnoses, supporting replay success as a mechanically verifiable proxy.The analysis excludes candidates that do not compile; the diagnosis audit covers GPT-OSS-120B, while counterexample replay covers both solver models.Source: §F.4 · p. 24

F.5 DIAGNOSIS FOR REFINEMENT AND COUNTEREXAMPLES FOR VERIFICATION

• Counterexample-generator source X is non-empty in 27.4% of GPT-OSS-120B reviews and 50.8% of Qwen3.6-35B-A3B reviews.When X is empty, omitting it adds no feedback content within that review; trajectories can differ after revision paths diverge.Source: §F.5 · p. 25
• On the Hard split, diagnosis D improves over generator source X by 5.9 and 6.7 points for Qwen3.6-35B-A3B and 6.5 and 3.3 points for GPT-OSS-120B under Best@1 and Best@8, respectively.V + D outperforms V + X in ten of twelve settings; V + D and V + D + X perform similarly across twelve settings, with V + D scoring higher in nine.Source: §F.5 · p. 25

G.1 VERDICT-ONLY REWARDS ENCOURAGE MAJORITY-CLASS GUESSING

• Verdict-only reward leaves validation accuracy at 56%–58% as tool use falls, consistent with exploiting the training pool’s 85% unaccepted majority.Mean tool calls fall from 4.9 to below one, while rollouts with no tools rise from 45% to above 70%.Source: §G.1 · p. 25

G.2 FALSE-POSITIVE PENALTY

• On hard problems, the larger false-positive penalty raises precision from 77.78% to 80.98% while lowering recall from 76.67% to 62.86%.The matched runs establish the direction of the trade-off, not a universally optimal setting.Source: §G.2 · p. 25, Table 18 · p. 25

G.3 LENGTH DIAGNOSTICS

• Without length control, median response length grows from 22.6k to 29.5k tokens over 50 steps as aborts rise from 13.6% to 26.6%.With the penalty active every step, shortening from 17.2k to 12.9k coincides with reward falling from 0.777 to 0.755 and hack success from 62.3% to 59.1%; alternating penalty phases preserves room for longer reasoning.Source: §G.3 · p. 26

H.4 REFINEMENT PROMPTS

• Each refinement condition reviews its own current candidate, and the conditions differ in which of verdict V, diagnosis D, and generator X they show the solver.The verdict is always shown because it determines the Best@1 stopping decision; complete solutions are required at every round so unchanged code retains its score.Source: §H.4 · p. 28