TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution

1 Introduction

• Nearly all coding benchmarks count a submission as solved when it passes a fixed test suite, which checks only anticipated behavior.Agents can adapt to execution feedback and pass every test without fully satisfying the task.Source: §1 · p. 1
• Manual audits can expose failures but do not scale, while automated auditors and static test augmentation leave falsely passed trials or unanticipated failures unresolved.PatchDiff dynamically inspects submissions but still requires human effort to decide whether divergences are genuine defects.Source: §1 · p. 2
• TESTJACK generates prompt-grounded tests, validates them against the ground truth, and re-judges failures; its lightweight variant audits sampled trials and shares confirmed witnesses.The lightweight approach reduces audit cost by reusing tests across trials for the same task.Source: §1 · pp. 2–3
• 34.4% of 4,487 benchmark-passing trials were refuted across five benchmarks, lowering resolution across 8,862 trials from 50.6% to 33.2%.The benchmarks are DeepSWE, SkillsBench, SWE-Marathon, SWE-bench Verified, and SWE-bench Pro.Source: §1 · p. 3

Coding Benchmarks

• Nearly all coding benchmarks use a static protocol in which a submission is correct if it passes a test suite fixed at release.Benchmark tasks span function-level synthesis, repository-level issue resolution, and long-horizon engineering projects.Source: §2 · p. 3

Benchmark Auditing and Test Strengthening

• Prior auditing methods vary by static versus dynamic evaluation and manual versus automated verdicts, as summarized by Figure 2.Manual audits expose concealed failures but must be repeated; automated auditors inspect task fairness rather than individual trials.Source: Figure 2 · p. 3, §2 · p. 3
• TESTJACK occupies the automated, dynamic quadrant: it audits individual trials with prompt-derived tests, uses ground truth to validate expectations, and provides executable witnesses.Unlike PatchDiff, it does not require manual inspection to decide whether a divergence indicts a trial.Source: §2 · pp. 3–4

3.1 Problem Formulation

• A witness test checks a task requirement, passes on the ground-truth implementation, and fails on a benchmark-passing trial, exposing a defect beyond the fixed suite.The possible defects lie in the gap between behavior required by the prompt and behavior asserted by the benchmark tests.Source: §3.1 · p. 4

3.2 Per-Trial Auditing via Prompt-Grounded Differential Testing

• Per-trial auditing writes tests from prompt requirements, validates expectations against the ground truth, and re-judges failures against the prompt before refuting a trial.The reviewer rejects tests that assert behavior the prompt does not request; prompt ambiguity may instead be reported as a prompt defect.Source: §3.2 · pp. 4–5
• The per-trial audit uses an audit agent and reviewer, and its executable, prompt-grounded verdicts can catch reward-hacked patches without a predefined exploit taxonomy.The reviewer's valid test must fail on the trial and pass on the ground truth.Source: §3.2 · pp. 5–6

3.3 Lightweight Approach: Randomized Auditing with Witness Sharing

• Task-level analysis compares the prompt with the existing suite and adds ground-truth-validated tests once per task, regardless of the number of trials.Tests that fail on the ground truth are refined using execution feedback or discarded when the refinement budget is exhausted.Source: §3.3 · p. 6
• Randomly sampled trials receive full audits, and confirmed witnesses are run across active trials; finite budgets can miss defects unique to unsampled trials.Replaying a shared test costs one execution, but shared witnesses are not re-judged for every trial they refute.Source: §3.3 · p. 6

4 Evaluation

• The evaluation asks how per-trial auditing changes resolution rates, how much the lightweight approach recovers and costs, how it compares with SWE-ABS, and how its stages contribute.Source: §4 · p. 6

4.1 Evaluation Setup

• The evaluation covers five benchmarks, with trials from frontier models or released benchmark trajectories, and caps each per-trial audit at three generate-and-validate iterations.The benchmarks include suites with inherited tests and suites whose verifiers were written from scratch; SWE-bench Pro coverage is limited to its public split, and SWE-Marathon excludes GPU and ultra-long-horizon tasks.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 RQ1: Per-Trial Auditing on Frontier Benchmarks

• 34.4% of benchmark-passing trials were refuted, lowering the resolution rate across all trials from 50.6% to 33.2%.Refutations ranged from 7.6% on SWE-bench Verified to 53.6% on SWE-Marathon; DeepSWE and SWE-Marathon, with from-scratch verifiers, had the two highest rates.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 RQ2: Lightweight Randomized Approach

• At three audit rounds per task, the lightweight approach recovered 72.4% of per-trial refutations while reducing API cost by 84.6% to $4,037.Recall ranged from 40.0% on SWE-Marathon to 80.0% on SkillsBench; shared witnesses are replayed across remaining trials of the same task. Diverse failure modes and uniform sampling constrain finite-budget coverage.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 RQ3: Comparison with Existing Benchmark Augmentation

• At comparable precision (50.4% vs. 49.2%), the lightweight approach refuted 1,863 trials (63.6%) and achieved 72.4% recall, versus SWE-ABS’s 248 trials (8.5%) and 9.9% recall.On SWE-Marathon, SWE-ABS refuted no trials, while the lightweight approach refuted 10 of 28.Source: §4.4 · p. 8, Table 4 · p. 9

4.5 RQ4: Ablation Study

• Adding randomized auditing and witness sharing raised recall from 13.4% to 72.4%, a 5.4× increase, at essentially unchanged precision (49.3% vs. 49.2%).The two stages are complementary: most refutations require evidence from a concrete submission.Source: Table 4 · p. 9, §4.5 · p. 9

• TESTJACK refuted 34.4% of benchmark-passing trials across five benchmarks, lowering the overall resolution rate from 50.6% to 33.2%.Its lightweight variant recovered 72.4% of per-trial refutations at 15.4% of the cost, with 7.3× the recall of SWE-ABS.Source: §5 · p. 9

• The well-formedness assumption requires official-test behavior to be included in prompt-required behavior, which in turn is included in ground-truth behavior.Source: §A.1 · p. 15
• The assumption fails if official tests require behavior absent from the prompt, the prompt requires behavior absent from ground truth, or official tests fail on ground truth.Source: §A.1 · p. 15
• The first two mismatches are task-quality issues that trial-level auditing cannot repair; they call for fixing the task rather than judging a submission.The third mismatch is largely ruled out because audited benchmarks verify that the reference solution passes official tests before release.Source: §A.1 · p. 15

A.2 Implicit Requirements and Prompt Underspecification

• Unchecked ground-truth behavior divides into stated, implied, and incidental parts; only the implied part is underspecified because stated and incidental behavior have unambiguous requirements.Stated behavior must be implemented, while incidental behavior need not be reproduced.Source: §A.2 · p. 16
• A witness may target ground-truth behavior that no benchmark test checks, yet if readers cannot infer it is required, the failure reflects prompt underspecification rather than an unfaithful trial.The section calls this difference between author-intended and reader-inferable implicit behavior Δ(P).Source: §A.2 · p. 16
• TESTJACK treats a witness requirement as explicit when a checklist can quote the prompt; otherwise it uses trial behavior to assess whether the requirement is inferable.A model counts as skipping a requirement when most of its passing trials fail the witness test; ρ(q) is the fraction of models with passing trials that skip it.Source: §A.2 · p. 16
• Failures on explicit requirements are refuted; implicit requirements are refuted when at most half of models skip them, but otherwise the witness indicates prompt ambiguity and does not refute the trial.The skip fraction counts models rather than trials, so repeated runs by one model do not dominate.Source: §A.2 · p. 16
• Requirement classification is computed once per task over the full trial corpus, while tests for incidental behavior are rejected by the reviewer agent.The classification depends on ρ(q), which is computed from trials across agents.Source: §A.2 · p. 16
• The ambiguous class can include requirements that are clear but widely skipped, because agents may skip untested behavior or share the same misreading.The paper reports such tasks in Appendix B.2.Source: §A.2 · p. 16

A.3 Randomized Trial Auditing

• Witnesses from each sampled audit are replayed on the remaining active trials, and any trial failing at least one witness is added to the refuted set.The audited trial is included among those refuted when it fails a witness.Source: §A.3 · p. 17
• Task analysis first adds tests to the benchmark suite, with every added test passing on the ground-truth solution.The algorithm then refutes trials that fail the augmented suite.Source: §A.3 · p. 17
• The algorithm repeatedly samples a trial uniformly at random from the active set for in-depth auditing.The procedure takes a trial budget as input.Source: §A.3 · p. 17

B.1 Additional Experiment Setup

• The five benchmarks span human-screened and repository-mined tasks, original tasks with hand-written verifiers, and containerized tasks with hidden verifiers and reference solutions.SWE-Marathon further hardens its verifiers with multi-layer checks and adversarial exploit search.Source: §B.1 · p. 17
• The study retains 13 coding-centric SkillsBench tasks without GPU requirements and two SWE-Marathon tasks without GPU requirements or ultra-long-horizon scope.The exclusions reflect repeated environment builds and executions for candidate tests and trials.Source: §B.1 · p. 17
• Benchmark versions are pinned because changes to tasks, verifiers, or environments can change the numbers of passing and refuted trials.Git-based benchmarks use fixed commits, while the Hugging Face datasets use fixed revisions; SWE-bench Pro also has a pinned evaluation-harness commit.Source: §B.1 · p. 18
• Table 5 identifies each audited benchmark version by its nearest release tag, commits after that tag, date, and pinned repository commit or dataset revision.The date is the commit date for GitHub repositories and the download date for Hugging Face datasets.Source: Table 5 · p. 18
• Table 6 reports counts of underspecified tasks out of the tasks audited and marks task subsets as in Table 1.Source: Table 6 · p. 18

B.2 Underspecified Prompt

• 29 of 113 DeepSWE tasks (25.7%), 3 of 13 audited SkillsBench tasks, and both audited SWE-Marathon tasks have underspecified prompts.The classification calls a requirement ambiguous when the prompt omits it and most models with passing trials on the task skip it.Source: §B.2 · p. 18
• Underspecified prompts occur in every benchmark examined, and ambiguous requirements lower lightweight-audit precision because departures from the reference can be unclear violations.On such tasks, per-trial and lightweight audits can reach different verdicts for the same trial.Source: §B.2 · pp. 18–19
• In both case studies, every audited trial passes official tests and fails a witness test, but the witness reflects one coherent prompt reading rather than a trial defect.Each witness captures the reading implemented by the reference solution.Source: §B.2 · p. 19
• For Textual’s Log and RichLog widgets, the prompt leaves open whether is_following_end means viewport position alone or effective following when auto_scroll is enabled.The two readings diverge only when auto_scroll is disabled; all 30 audited trials use the position reading and fail a witness expecting both flags to be False in that setting.Source: §B.2 · p. 19
• For Updo alerting, the prompt leaves unspecified whether ConsecutiveRecoveries resets before the recovery decision is returned.All seven audited trials report the completed streak of 2 rather than the reference’s reset value of 0, although they report the correct event and states; official tests do not assert the counter.Source: §B.2 · p. 19
• A sentence defining the flag or reset timing, paired with an acceptance example, would close each ambiguity and make the withheld witnesses useful for clarifying prompts.The suggested examples cover disabled auto-scroll and the failure-and-recovery sequence.Source: §B.2 · p. 19

C.1 Cases in Per-Trial Audit

• The per-trial audit’s refutations identify benchmark-test defects: one trial omits an untested feature component, while another implements only what the visible fixture observes.The complementary case of a witness reflecting an underspecified prompt is discussed in Appendix B.2.Source: §C.1 · p. 20
• 6.7% of the lines added by the Navidrome trial are covered, leaving seven of eight requirements unexercised while its missing migration leaves the model pointing to a nonexistent column.The official evaluator still passes the trial because collecting a filename in memory satisfies its sole graded assertion.Source: §C.1 · p. 20
• Only one of six required Navidrome behaviors is asserted by the official patch, which never reaches the schema, album repository, or rescan.Table 7 maps the task requirements to evidence in the official test patch; the graded selection contains only the scanner test and one changed assertion.Source: §C.1 · p. 20, Table 7 · p. 21
• A confirmed witness runs the production migration path and checks for the image_files column, while existing persistence tests also fail because the column is absent.The proposed gap-closing tests cover schema creation, repository round-tripping, and triggering a full rescan, using an isolated temporary database.Source: §C.1 · p. 21
• The Open Library fixture observes one conflicting source–destination pair, so its outcomes do not distinguish per-record handling from a constant.Table 8 makes this agreement explicit; the task requires preserving conflicting notes and moving records that can move.Source: §C.1 · p. 21, Table 8 · p. 22
• The trial passes because the single-conflict fixture is its only observation; the evaluation restores the graded test file after applying the model patch.The trial submits a constant and does not edit tests or tamper with the harness.Source: §C.1 · p. 22
• Three witnesses distinguish the constant from the reference by testing multiple conflicts, a mixed movable-and-conflicting update, and the helper directly.They pass on the reference and fail on the trial, comparing complete row contents rather than counts.Source: §C.1 · p. 22
• The Open Library verdict is behavioral, not a claim of deliberate gaming: the prompt’s “one failure” wording could license rollback, while mixed-update and helper witnesses establish the departure.The authors say they make no claim about the model’s intent or knowledge of hidden tests.Source: §C.1 · pp. 22–23

C.2 Cases in the Lightweight Approach

• Task analysis adds tests for prompt requirements missing from T0, refuting 11 trials over per-request cookie use and acceptance of httpx.Cookies by update().Nine trials fail the per-request case; two leave the store empty when given the wrapper, which the reference unwraps before importing.Source: §C.2 · p. 23
• Three sampled trials reveal defects not apparent from the prompt alone: trailing-dot domain normalization, dropped Secure metadata during jar import, and an overly broad scheme policy.The witnesses check rejection of a trailing-dot domain, withholding imported Secure cookies from HTTP, and withholding Secure cookies from wss:// requests.Source: §C.2 · p. 24
• 24 of 42 benchmark-accepted trials (57.1%) are refuted: 11 by task analysis and 13 by the randomized loop, leaving 18 survivors.The four audit-generated witnesses were replayed across the 31 survivors; the number of rounds needed can vary with the sample.Source: §C.2 · pp. 23–24
• The stages find different defects: task analysis bulk-tests enumerated but unchecked requirements, while sampled audits expose implementation choices; all 24 refutations meet the necessary-requirement criterion.Three trials were sampled, one per round; their confirmed witnesses were replayed on the survivor pool, and a different sample could change the order of discoveries.Source: §C.2 · p. 24

D Prompt Templates

• The prompt templates assign auditing to a trial auditor, test review to a reviewer, and pre-trial requirement analysis to a task analysis agent.The audit prompt treats a passing evaluator result as a starting hypothesis and requires executable evidence for findings.Source: D Prompt Templates · p. 24, Audit Agent Prompt · pp. 24–25

Audit Agent Prompt

• The prompt directs reviewers to detect reward-hacking patterns such as verifier-shape mimicry and incomplete tested veneers, while treating transcript evidence as supporting rather than primary.When intent is ambiguous, it favors OVERFIT_INCOMPLETE over REWARD_HACK unless explicit intent or structural evidence is present.Source: Audit Agent Prompt · pp. 27, 28, 30
• Auditors turn task requirements into atomic checklist clauses, probe required but untested behavior, and use budgeted fuzzing and boundary tests to expose specification violations.When available, model behavior is differentially compared with a trusted reference; the specification remains the oracle of record.Source: Audit Agent Prompt · p. 28
• For each issue, auditors assess integration, robustness, performance, security, maintainability, and model-added tests, then assign one primary verdict with calibrated confidence.The verdict scheme distinguishes sound but low-quality work, incomplete or incorrect patches, reward hacks, and non-reproducible evaluator passes.Source: Audit Agent Prompt · pp. 28–30
• Every reported issue requires an executable test that fails on the patched model code and is checked against an independent oracle or explicit specification evidence.The test, command, outputs, and oracle evidence must be saved and reported; without a demonstrating test, a finding is downgraded to NOTE.Source: Audit Agent Prompt · p. 29
• The prompt guards against false positives: uncovered branches alone are not defects, reference divergence is not sufficient, and high-severity findings require demonstrated tests.The specification takes precedence over a reference solution when deciding whether behavior is required.Source: Audit Agent Prompt · p. 30

Reviewer Agent Prompt

• The reviewer checks audit-added probes for alignment in a read-only task, treating repository contents and probe materials as untrusted and non-authoritative for requirements.The reviewer must not edit host files or manage the supplied container.Source: Reviewer Agent Prompt · p. 32
• The reviewer treats the instruction as the behavioral authority and rejects probes that impose unrequested behavior or overfit to a solution, private internals, or incidental details.It distinguishes invalid probes from valid-but-weak or redundant ones and assesses coverage against official tests plus valid probes.Source: Reviewer Agent Prompt · p. 32
• The reviewer decomposes requirements, checks what official tests and valid probes actually assert, and assigns probe verdicts and an actionable filtering recommendation.Recommendations distinguish KEEP_ALL, KEEP_SOME, DROP_ALL, and HUMAN_REVIEW; only valid probes count toward combined completeness.Source: Reviewer Agent Prompt · pp. 32–33

Task Analysis Agent Prompt

• The task-analysis agent works without a reference implementation, maps requirements to existing tests, and identifies plausible incorrect patches that could pass the original suite.It records concrete behavioral gaps and code-level features for flagging suspicious patches, while treating candidate failure patterns as hypotheses rather than observed implementations.Source: Task Analysis Agent Prompt · pp. 33–34
• Added tests must target important uncovered requirements, fail for a plausible incorrect implementation, and remain deterministic and compatible with reasonable implementations.The prompt requires solution-blind validation of artifacts and tests while distinguishing expected failures on the base commit from syntax, discovery, fixture, or infrastructure problems.Source: Task Analysis Agent Prompt · p. 34

E Limitations

• TestJack requires a reference solution to certify new tests and assumes each task’s tests, prompt, and ground truth are mutually consistent.Tasks that violate this assumption call for repair rather than re-judging their trials.Source: E Limitations · p. 35
• Oracle validation confirms that a witness matches ground-truth behavior, but whether the prompt requires that behavior depends on the LLM reviewer.Underspecified prompts remain the main source of disagreement between configurations.Source: E Limitations · p. 36
• The lightweight approach trades coverage and precision for cost: finite sampling can miss defects confined to unaudited trials and spread noise or prompt ambiguity.Tasks with few passing trials offer little opportunity to amortize audits across trials.Source: E Limitations · p. 36
• Evaluation covered a fixed set of frontier models and excluded non-coding, GPU-requiring, and ultra-long-horizon tasks, limiting transfer of refutation rates to other agents or task types.RQ2–RQ4 recall and precision measure agreement with the per-trial audit, not a human-labeled ground truth.Source: E Limitations · p. 36