PaperlayerView exact revision on arXiv

Source-linked AI summary

Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

Junyu Guo, Shangding Gu, Ming Jin, Javad Lavaei

arXiv:2610.01023v1cs.SEcs.AIcs.CL

TL;DR

Coding-agent patches can look plausible while omitting required behavior, leaving reviewers to judge correctness from evidence that may not expose what is missing. The paper evaluates reviewers with execution-labeled traces and controlled cases, then compares official execution evidence with a cascade of independently checked evidence. Official evidence improves both review rates for most reviewers, while the deployable cascade remains a stress test.

  • Problem

    Plausible coding-agent patches can omit required behavior, and reviewers need evidence that reveals whether the issue was actually solved.

    Source: §1 · p. 1

  • Method

    The paper evaluates fixed reviewers on execution-labeled traces and controlled omissions, comparing unchecked evidence with official execution evidence and a cascade using independently checked evidence.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    With frozen official-evidence formats, five of six reviewers improved both rates on 122 held-out traces, and GPT-OSS-120B and GPT-4.1 classified all traces correctly.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Official execution evidence shows that weak reviewers can be reliable when decisive checks are available, while current generated checks still fall short.

    Source: §8 · p. 7

    The strongest result depends on official evidence unavailable in deployment, and the cascade remains a stress test rather than a reliable deployed reviewer.

    Source: §7 · p. 7

Abstract

from arXiv · show

Coding agents can return plausible patches that omit required behavior. These failures are hard to review because long traces and confident summaries often hide what was missed. We ask when a nominally weaker reviewer can reliably decide whether a patch solves its issue. We study 411 execution-labeled traces from three agents and 101 controlled cases. On 154 GPT-5.4 traces, structured but unchecked evidence raises both defect catch and over-rejection. We then provide official execution evidence as an upper-bound diagnostic. After choosing and freezing one of two formats per reviewer, five of six reviewers improve both rates on 122 held-out traces; two classify every trace correctly. Reviewer size is not a consistent predictor of quality. Because official tests are unavailable in deployment, we also evaluate a frozen cascade with patch-caused static errors and generated tests that first fail on the unpatched repository. On 121 scored held-out GPT-5.4 traces and 59 Gemini traces, its coverage is 0.89 and 0.86, risk is 0.33 and 0.26, catch is 0.76 and 0.80, and over-rejection is 0.66 and 0.67. Most false rejections occur when unresolved cases reach the reviewer. Official execution evidence shows the potential of weak review when decisive checks are available. Producing equally reliable checks without official tests remains the main bottleneck.

1 Introduction

Plausible coding-agent patches can hide omissions, so the paper asks when a weaker reviewer can make reliable decisions from checkable evidence. Its results distinguish the potential of official execution evidence from the still-limited performance of deployable checks.

1 Introduction

  • In 154 GPT-5.4 traces, official tests labeled 101 patches defective; omissions were the largest defect class and were hard to catch by inspection.Long traces and confident summaries can make plausible steps obscure missing behavior.Source: §1 · p. 1
  • Groundability asks whether an independent check can answer a disputed question about missing behavior without raising the same alarm on correct patches.Examples include sound static analysis and runnable tests.Source: §1 · p. 1
  • On 122 held-out traces, five of six reviewers improved both catch and over-rejection with frozen official-evidence formats; GPT-OSS-120B and GPT-4.1 classified all traces correctly.The evidence-format choices were made on a separate 32-trace design set; official execution evidence is an upper-bound diagnostic, not a deployable input.Source: §1 · p. 1
  • On held-out GPT-5.4 and Gemini traces, the frozen cascade reached coverage 0.89 and 0.86, but over-rejection was 0.66 and 0.67, respectively.The cascade is a high-coverage stress test, not yet a reliable deployed reviewer.Source: §1 · p. 2

2 Related Work

  • Unlike prior generated-test work that helps repair systems select patches, this work gives tests to an independent reviewer only after they fail on the unpatched repository.The independent generator and fail-on-base check are intended to provide behavioral evidence without a formal specification.Source: §2 · p. 3

3 Problem Formulation

The formulation separates patch correctness from whether the evidence lets a reviewer judge that correctness. It defines evidence types, checkable generated tests, and the rates used to evaluate review decisions.

3 Problem Formulation

  • Correctness is whether a patch passes the hidden oracle, while reviewability measures how well the evidence-reviewer pair matches execution-based ground truth.A correct patch may be hard to review without checkable evidence, while a defective patch may be easy to review when a failing check exposes the defect.Source: §3 · p. 3
  • The formulation distinguishes agent testimony, organized but unchecked structured evidence, and grounded evidence backed by an independently run check.Examples of grounded evidence include sound static analysis and test runs.Source: §3 · p. 3
  • A generated test is retained only if it fails on the unpatched repository because expected behavior is missing, then it is run on the patched repository.Failures caused by setup or environment errors are discarded; this check uses neither official hidden tests nor the reference patch.Source: §3 · p. 3
  • Catch, miss rate, and over-rejection are class-specific decision rates; uncertain decisions count as abstentions, while coverage and risk measure decided traces and their errors.Catch alone is insufficient because rejecting every patch can produce high catch; groundability requires an independent check that can determine whether a disputed requirement holds.Source: §3 · p. 3
  • Main reviewer comparisons use temperature 0.8 and five votes, with majority decisions and ties marked uncertain; design choices were fixed before evaluating remaining partitions once.Labels, official test identifiers, and construction information are excluded from deployable prompts.Source: §3 · p. 3

4 A Reviewability Benchmark

The benchmark combines execution-labeled agent traces with controlled cases that isolate known omissions. Its groups support different evaluations, from real defect detection to correct acceptance and no-change handling.

4 A Reviewability Benchmark

  • Execution labels are fixed by running official fail-to-pass and pass-to-pass suites on patches, while hidden test identifiers and content stay out of prompts.Labels distinguish acceptance from rejection due to real omission, regression, or no code change.Source: §4 · p. 4
  • The corpus includes 154 GPT-5.4 core traces, 59 Gemini transfer traces, and two accept/no-change sets; 101 of 154 core traces were defective, with omissions the largest bucket at 76.The real traces span ten Python repositories plus nine non-Python languages; accept/no-change sets contain correct submitted patches or no submission.Source: §4 · p. 4
  • Table 1 separates core and transfer sets with real defects from accept/no-change sets containing no defective submitted patches.The latter sets measure correct acceptance and no-change handling, not defect catch.Source: Table 1 · p. 4, §4 · p. 4
  • The 101 controlled cases comprise 37 correct reference patches, 34 known-missing-piece ablations, and 30 naturalistic pilot cases.Ablations remove one load-bearing piece: a used import, implementation behind a kept interface, or branch where the code still runs; partial-logic omissions are hardest because there is no syntax signal.Source: §4 · p. 4
  • Reviewers see agent artifacts, organized but unchecked requirements and possible missing work, or evidence backed by an independent check.The upper-bound condition uses official execution results; the deployable setting uses generated tests that pass the fail-on-base check.Source: §4 · p. 4

5 Findings

Official execution evidence improves the catch–over-rejection trade-off for most tested reviewers, while unchecked or automatically produced evidence remains limited. The findings also show that reviewer size alone does not reliably predict review quality.

5.1 Raw Agent Evidence Is Unreliable

  • Patch-only review accepts all nine design-set omissions for Llama-8B, while full raw traces do not give any of six reviewers the best tested catch–over-rejection trade-off.GPT-4.1 catches only 0.27 of omissions from the patch, and Qwen-32B catches 0.17 from the full trace.Source: §5.1 · p. 5

5.2 Structured Evidence Has a Trade-Off

  • On real traces, structured evidence raises catch but also over-rejection; GPT-4.1 catches 0.91 of defects while rejecting 0.86 of acceptable patches.Adding a construction-known missing item raises partial-logic catch from 0.53 to 1.00 for GPT-4.1 and from 0.07 to 1.00 for Llama-8B, but does not show an automatic pipeline can find that item.Source: §5.2 · p. 5

5.3 Official Checks Can Break the Trade-Off

  • Five of six reviewers improve both defect catch and over-rejection with design-selected official evidence on 122 held-out traces; GPT-OSS-120B and GPT-4.1 classify all correctly.Qwen-2.5-72B is the exception: its catch rises, but its over-rejection does not improve.Source: Table 2 · p. 5, §5.3 · p. 5
  • On 59 Gemini traces, all three reviewers catch all 44 defective patches, and GPT-OSS-120B and GPT-4.1 classify every trace correctly.Llama-8B rejects one of 15 acceptable patches.Source: §5.3 · p. 5

5.4 Generalization Across Languages and Agents

  • On accept/no-change sets, grounded evidence restores 49, 61, and 61 correct accepts and correctly rejects at least 135 of 137 no-change submissions per reviewer.The sets contain no real omissions; remaining grounded errors include small-model misreadings of official execution outcomes.Source: §5.4 · p. 6

5.5 Automatic Grounding Remains Limited

  • Automatically produced structural evidence catches missing imports but only 0.27 of partial-logic omissions; semantic suggestions without independent checks retain the false-alarm problem.Generated tests provide a check only when they run and target the right behavior.Source: §5.5 · p. 6

6 Testing a Deployable Review Cascade

A frozen cascade combines patch-caused static errors and generated tests that first fail on the unpatched repository, leaving unresolved cases to a reviewer who may abstain. On held-out sets it achieves high coverage, but risk and over-rejection remain substantial, with most false rejections arising at the reviewer stage.

6 Testing a Deployable Review Cascade

  • The cascade checks no-change submissions first, rejects only patch-caused static or execution errors, keeps generated tests that fail behaviorally on the base repository, and allows abstention.A generated-test failure rejects mechanically; a pass invokes the frozen accept arm, while unresolved cases reach the reviewer.Source: §6 · p. 6, §6.2 · p. 6

6.1 Design-set results and failed hypotheses

  • On the 32-trace design set, the cascade reaches coverage 0.91, risk 0.24, and catch 10/11, with 18 traces finishing before the five-vote reviewer stage.This count excludes test-generation cost, and no tested deployable baseline has both at least as much coverage and no higher risk.Source: §6.1 · p. 6
  • Faithful Docker reruns recover no decisions from 18 failed tests, model-aware harness repair moves generated-test decisions only from 13 to 16/32, and tests sometimes require too much or too little.No generated test passes on the agent patch while failing on the reference patch; tests generated without the patch perform worse.Source: §6.1 · p. 6

6.2 Run-once test-set evaluation

  • Stages 0–2 decide 36/121 held-out traces; among the 85 reaching the reviewer, it abstains on 13 and causes 19 of 21 false rejections.A failing checked test has reject precision 0.82, while a passing test has accept precision 0.30, so one pass cannot establish correctness.Source: §6.2 · p. 7

6.3 Design implications

  • The cascade tests how much of the official-evidence upper bound can be recovered without hidden tests, but current limits are producing decisive checks and calibrating decisions without one.Review systems should use patch-caused failures directly, send unresolved cases with relevant context to the reviewer, and retain uncertainty.Source: §6.3 · p. 7

6.2 Run-once test-set evaluation

  • On held-out GPT-5.4 and Gemini sets, defect catch is 0.76 and 0.80 and over-rejection 0.66 and 0.67; at coverage 0.89 and 0.86, no tested deployable baseline combines at least that coverage with no higher risk.Lower-risk baselines abstain on more traces; paired results show the measured advantage is coverage, not lower risk or higher decided-trace accuracy.Source: §6.2 · p. 7, Figure 3 · p. 7

7 Limitations

The real-defect evaluation covers only Python GPT-5.4 and Gemini sets, while accept/no-change sets contain no defects. The strongest result relies on official evidence unavailable in deployment, and the cascade remains a stress test rather than a reliable deployed reviewer.

7 Limitations

  • Real-defect results are limited to Python GPT-5.4 and Gemini sets; accept/no-change sets contain no defects, and the evidence-format design set comes from Django.The cascade scores 121/122 held-out traces, has risk 0.33/0.26 and over-rejection 0.66/0.67, and remains a stress test; call counts exclude test generation, so the paper makes no total-cost or general agentic-review claim.Source: §7 · p. 7

8 Conclusion

Official execution evidence indicates that weak reviewers can be reliable when decisive checks are available, while reviewer size alone is not a consistent predictor. The evaluated generated-check cascade still has high over-rejection.

8 Conclusion

  • Official execution evidence shows weak reviewers can be reliable when a decisive check is available.Reviewer size is not a consistent predictor of review quality.Source: §8 · p. 7
  • The generated-check cascade has held-out over-rejection of 0.66, and one passing test cannot establish correctness.The paper prioritizes reliable checks and safer handling of unresolved cases over larger reviewers alone.Source: §8 · p. 7
  • Parameter count does not consistently predict review quality in the heterogeneous reviewer ladder.Source: §8 · p. 7

S1 Extended Related Work

Related work studies weak-model monitoring of stronger systems and automated code review, often relying on narrative evidence. This paper focuses on execution-backed checks and explicit abstention for coding-agent patch review.

AI control and trace monitoring

Automated code review

S2 Worked Examples from the Benchmark and Pipeline

The worked examples trace how the cascade and reviewers handle omissions, generated tests, static checks, and grounded evidence. They illustrate both decisive checks and residual errors from under-specified probes or evidence misreading.

S2.1 The cascade, step by step (schematic)

S2.2 Why omissions defeat skim review (django-11905)

S2.3 A generated test with a fail-on-base check (django-10914)

  • A generated test is retained after failing on the unpatched base; its pass on the agent patch triggers a mechanical accept matching ground truth.In this example, the test checks the default file-upload permission behavior.Source: S2.3 A generated test with a… · p. 11

S2.4 A weak reviewer deciding on grounded evidence

S2.5 A probe under-specification failure (django-12308)

S2.6 What the reviewer sees: three conditions, one trace (php-cs-fixer-8064)

  • On pooled coverage sets, three reviewers move from 12/5/7 correct accepts with packet evidence to 49/61/61 with grounded evidence.The grounded packet states that both official suites passed for the submitted patch.Source: S2.6 What the reviewer sees: three… · p. 12

S2.7 No code change under the turn budget (Claude, astropy-13977); S2.8 A Stage-1 intermediate: patch-attributable static flag (django-11564)

S2.9 Evidence misreading, reject direction (Llama-8B, php-cs-fixer-8064); S2.10 Evidence misreading, accept direction (Llama-8B, sympy-14317)

S3 Full-Length Artifacts

This section reproduces full-length artifacts that let readers follow the review pipeline from prompts and evidence packets through probes and decisions. The artifacts illustrate how evidence misreadings, probes beyond the issue, mechanical labels, and environment-shaped trace failures affect interpretation, and how grounded execution evidence can expose omissions.

S3.1 The reviewer prompt (verbatim template)

  • The reviewer template prioritizes execution evidence and warns against rejecting on generic risk flags, yet earlier misreadings occur despite those instructions.It tells reviewers to use only the supplied evidence and not infer hidden labels.Source: S3.1 The reviewer prompt (verbatim… · p. 13

S3.2 A complete non-grounded packet (php-cs-fixer-8064)

  • The php-cs-fixer packet reports no blocker-level evidence, but its validation is insufficient and official correctness is unknown, so it supports uncertainty rather than confident acceptance or rejection.Its attempted tests fail because required executables are unavailable; the packet says that missing or environment-blocked validation alone is not a reject reason.Source: S3.2 A complete non-grounded packet… · pp. 14, 16, 17, 18

S3.3 A complete generated probe (django-10914)

  • The generated probe targets the affected storage-backend code path but does not simulate a full HTTP upload cycle, and its self-reported limitation is withheld from the reviewer.The generator sees only the issue and patch.Source: S3.3 A complete generated probe… · p. 18

S3.4 A complete five-vote decision (Llama-8B, grounded, php-cs-fixer-8064)

  • All five Llama-8B votes rejected an acceptable patch despite citing passed official suites; four asserted failure, and one rewrote the cited evidence.The issue text narrating pre-patch failure was the trigger.Source: S3.4 A complete five-vote decision… · pp. 18–19

S3.5 An over-specified probe, outcome matrix (django-12497)

  • Cross-executing a probe on the gold patch detects over-specification because a probe that fails the reference solution demands too much.On the design set, this class accounted for the probe-driven share of cascade over-rejection.Source: S3.5 An over-specified probe, outcome… · p. 19

S3.6 A mechanical label for no code change (astropy-13977)

S3.7 A complete agent trace (GPT-5.4, django-11905)

  • The django-11905 trace looked diligent, but the official suite failed on truthy non-booleans, confirming a real omission despite environment-shaped test failures.The agent read the relevant files and wrote a plausible fix and test; its own test runs failed because pytest collected zero items.Source: S3.7 A complete agent trace (GPT-5.4,… · pp. 19, 21

S3.8 One case, one reviewer, four evidence forms (django-11905)

  • Patch-only review accepted a real omission, while grounded execution evidence exposed it; non-grounded review also rejected correctly but over-rejected 0.755 of acceptable core-set patches.For this trace, the grounded reviewer rejected after citing the independently executed failing test, and the same condition had 0.000 over-rejection on the core set.Source: S3.8 One case, one reviewer, four… · p. 21

S4 Additional Cascade Results

The cascade’s held-out errors are concentrated in the residual reviewer stage, while generated-test failures after a fail-on-base check provide stronger rejection evidence than passing tests provide for acceptance. A tool-using reviewer had lower risk on commonly decided traces in a limited comparison, at substantially higher prompt-token use.

S4.1 Cascade error breakdown

  • 19 of the cascade’s 21 held-out false rejections came from the residual reviewer stage; most unresolved traces reached that stage.Stages 0–2 decided 36/121 held-out traces and 24/59 Gemini traces before reviewer decisions.Source: S4.1 Cascade error breakdown · p. 21
  • 0.82 reject precision followed a generated test that still failed on the patch after passing the fail-on-base check; pass precision was 0.30.A passing generated test is weaker evidence than an official suite because one test covers less behavior.Source: S4.1 Cascade error breakdown · p. 21
  • Among decided traces, catch was 0.82 and 0.92, while over-rejection was 0.84 and 0.77, on held-out and Gemini sets, respectively.These rates condition on traces where the cascade decided.Source: S4.1 Cascade error breakdown · p. 22

S4.2 Tool-using reviewer baseline

  • 0.36 and 0.11 risk for GPT-4.1 versus 0.46 and 0.33 for the cascade on commonly decided held-out and Gemini subsets, respectively.GPT-4.1 decided 0.55 and 0.45 of those sets, used 58–84 times as many prompt tokens, and the comparison used one sample versus five cascade votes.Source: S4.2 Tool-using reviewer baseline · p. 22

S5 Reproducibility Details

The reproducibility details specify the computing environment, models, metrics, evaluation and statistical procedures, and final hyperparameters. They report that additional grounded-review votes improved the three-reviewer mean, whereas non-grounded conditions changed little.

S5 Reproducibility Details

  • The section records infrastructure, models, metric definitions, run counts, statistical procedures, and final hyperparameters for reproducibility.Source: S5 Reproducibility Details · p. 22

S5.1 Computing infrastructure

  • Execution labeling, probe runs, and cascade runs used CPU-only HPC nodes and benchmark-provided Apptainer environments; agents and reviewers ran through provider APIs.The nodes had two AMD EPYC Milan processors and 256 GB RAM, with no GPUs.Source: S5.1 Computing infrastructure · p. 22

S5.2 Models and access routes

  • GPT-5.4, Gemini-2.5-Pro, and Claude-Sonnet-5 were audited; the official test suite, not a model, served as the labeling oracle.The passage also lists reviewer models and the probe generator.Source: S5.2 Models and access routes · p. 22

S5.3 Evaluation metrics

  • Catch, miss rate, and over-rejection measure class-specific outcomes; coverage and selective risk account for abstentions and decisions.Catch alone is insufficient because rejecting every trace would produce high catch.Source: S5.3 Evaluation metrics · p. 22

S5.4 Number of runs and variation

S5.5 Statistical testing

  • Cascade comparisons used paired non-parametric tests on commonly decided traces, with coverage differences reported separately from risk comparisons.The procedure included paired permutation tests, paired bootstrap intervals, and McNemar discordant-pair counts.Source: S5.5 Statistical testing · p. 22

S5.6 Final hyperparameters

  • The cascade kept generated tests only after behavioral failure on the base repository; patched-repository probe failures triggered rejection, and Llama-3.1-8B reviewed residual cases.Main reviewer conditions used five votes at temperature 0.8.Source: S5.6 Final hyperparameters · p. 22

S6 The Qwen-2.5 Exception: Replication and Dissection

The Qwen-2.5-72B over-rejection exception persists across frozen renderings and calibrated prompts. Audits trace most over-rejections to decisions that contradict the reviewer’s own cited evidence.

S6 The Qwen-2.5 Exception: Replication and Dissection

  • Qwen-2.5-72B was rerun after the freeze because it was the only reviewer without a decoupling grounded rendering.The rerun was registered as a post-hoc diagnostic on 154 traces, with k=5 and temperature 0.8.Source: S6 The Qwen-2.5 Exception: Replication… · pp. 22–23
  • 41/53 over-rejection under clean rendering and 28/53 under oracle rendering (frozen: 41/53 and 26/53, within the 95% CI) replicate the exception.The result is described as neither run-to-run noise nor a one-off harness artifact.Source: S6 The Qwen-2.5 Exception: Replication… · p. 23
  • 40/53 over-rejection for Qwen-2.5-72B under clean rendering and 45/53 under oracle rendering persist with explicit instructions to treat official evidence as decisive.Qwen-2.5-32B also over-rejects 44/53 under clean rendering; the failure persists across tested checkpoints and prompt variants.Source: S6 The Qwen-2.5 Exception: Replication… · p. 23
  • 173 of 198 audited over-rejection instances contradict the reviewer’s own citation, while 24 cite selectively and one requests manual review.The audited instances identify evidence-to-decision miscalibration as the mechanism.Source: S6 The Qwen-2.5 Exception: Replication… · p. 23
  • In one example, Qwen asserts failure while citing evidence that official suites passed, and treats environment-blocked agent commands as decisive against instruction.The authors recommend mechanically consuming decisive grounded evidence where possible and screening reviewers for evidence-to-decision calibration before deployment.Source: S6 The Qwen-2.5 Exception: Replication… · p. 23

Related papers

Loading 2610.01023v1…