PaperlayerView exact revision on arXiv

Source-linked AI summary

PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix

Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara, Martin Vechev

arXiv:2610.35987v1cs.SEcs.AIcs.LG

TL;DR

Whether coding agents can find the bugs maintainers would repair across whole repositories remains an open question. PROBE benchmarks this task using maintainer-fixed bugs as a partial reference and evaluates ten models. Agents recover few reference bugs, and their findings differ systematically from maintainers’ fixes even though many unmatched findings are real bugs.

  • Problem

    Whether agents can find bugs maintainers would fix across whole repositories remains unexplored, while the complete set of bugs in a repository is unknown.

    Source: §1 · p. 1

  • Method

    PROBE benchmarks whole-repository bug discovery against bugs found in repository snapshots and fixed by maintainers, and evaluates ten frontier models with a multi-pass audit harness.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Agents’ bug distribution differs substantially from maintainers’: the best configuration reaches 20.8% recall, and none exceeds 8.1% precision against the reference set.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Agents generally find real bugs, but mostly not the bugs maintainers fix, and PROBE makes that gap measurable.

    Source: §1 · p. 2

    The reference set includes only bugs fixed within 18 months, and bugs left unfixed may be unimportant or may reflect maintainers’ limited awareness or capacity.

    Source: §6 · p. 9

Abstract

from arXiv · show

Proactive bug discovery is an important software-engineering problem. Even though coding agents are already widely deployed in this domain, their capabilities remain understudied. In particular, the challenging task of finding all bugs that human maintainers would repair across an entire repository has not been studied yet. To close this gap, we introduce PROBE, a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories, together with a method for matching candidate findings to this partial reference set. We evaluate coding agents across frontier models and find they all achieve a recall of below $21\%$ and a precision of less than $9\%$ with respect to the reference set. While stronger models increase recall substantially from $8\%$ to $17\%$, they do not improve precision. Surprisingly, despite the low precision, LLM and human evaluators still judge $85\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers. The practical relevance of bug discovery and the low recall of frontier models make this benchmark a valuable resource for both model users and developers.

1 INTRODUCTION

Whether coding agents can find the bugs maintainers would repair across whole repositories remains unexplored, despite growing interest in autonomous bug finding. PROBE measures this gap and shows that agents find few maintainer-fixed bugs, while their reported bugs differ systematically from maintainers’ fixes.

1 INTRODUCTION

  • PROBE uses bugs fixed by maintainers as a reference for whole-repository discovery, addressing a task prior work had not studied across entire repositories.Prior work pointed agents to proposed patches or known-flaw files, or focused on security vulnerabilities; the full set of repository bugs is unknown.Source: §1 · pp. 1–2
  • The benchmark covers 2,457 bugs fixed across 20 repositories, with an LLM-based matcher reaching 99% precision and 98% recall on expert-labeled pairs.The bugs were present in repository snapshots and fixed by maintainers over the following 18 months.Source: §1 · pp. 1–2
  • 20.8% is the best configuration’s recall across ten models, and no configuration exceeds 8.1% precision against the reference bugs.Stronger models and higher reasoning effort raise recall mainly by reporting more findings, while precision barely changes.Source: §1 · p. 1
  • 84% of findings outside the reference set were confirmed as real bugs, while even a perfect finder is estimated to reach only 26–47% precision.Maintainers continue fixing snapshot bugs after the reference window, so the reference set is incomplete.Source: §1 · p. 2
  • Agents report similar bug types, but their distribution differs from maintainers’: local correctness errors are 19% of agent findings versus 10% of fixes, while incomplete input handling is 8% versus 18%.Within the same bug type, GPT-5.6 Luna runs overlap with each other a median of 6.3 times more than with maintainer fixes.Source: §1 · p. 2

2 EVALUATING OPEN-ENDED BUG DISCOVERY

PROBE constructs a reference set from bugs fixed after a repository snapshot, then evaluates how that set and findings are matched. It also audits the reference set and estimates how incomplete it may be.

2.1 BENCHMARK CONSTRUCTION

  • The collection pipeline starts with a January 2025 snapshot, extracts fixes through July 2026, and excludes bugs introduced after the snapshot; unfixed bugs at the end are missing.It identifies bug-fix commits, extracts and anchors candidate bugs to the snapshot, checks their presence, synthesizes reports, and deduplicates defects.Source: §2.1 · pp. 2–3
  • The 20 repositories were manually selected for active development, diverse domains, permissive licenses, and limits on size and external dependencies.Repositories with major refactoring during the reference window were excluded.Source: §2.1 · p. 2

2.2 BENCHMARK OVERVIEW

  • The reference set contains 2,457 distinct bugs from 20 Python, TypeScript, and JavaScript repositories, spanning domains including parsing, rendering, math, and workflow orchestration.Each bug includes its incorrect behavior, snapshot location and code, severity, and suggested fix.Source: §2.2 · p. 3
  • A two-stage audit found that reviewers disagreed substantially: experts rejected 12 of 100 sampled bugs, with only six overlapping the agent’s rejections, so the full reference set was retained.The agent categorized 209 bugs (8.5%) as incorrect, and agreement barely exceeded chance (κ = 0.29).Source: §2.2 · p. 3
  • A fitted lognormal model estimates 5.2k–5.9k bugs fixed within five years and 7.2k–9.4k within ten, implying a 26–47% precision ceiling for a perfect finder of bugs humans would fix within those horizons.The estimate models fix rates for snapshot bugs and treats bugs discovered far in the future as irrelevant.Source: §2.2 · p. 4

2.3 EVALUATION

  • Evaluation reports recall, precision, and F1 against the reference set, plus cost per run; matching compares bug descriptions, code snippets, and line ranges.An LLM judge counts a pair as matching when resolving either bug’s root cause would also resolve the other.Source: §2.3 · p. 4
  • Across 5 runs, bug matching averaged 98.8% precision and 97.6% recall against 67 expert-validated matches.The rule-based filter retained 2.7% of within-repository pairs, including all matches; all removed pairs were also rejected by the LLM judge in a validation experiment.Source: §2.3 · p. 4
  • An independent panel assesses findings beyond the reference set by requiring judges to demonstrate the bug on a real user input; validity rate excludes inconclusive judgments.Disagreement in the first judging tier triggers review by a second tier.Source: §2.3 · p. 4

3 BUG FINDING HARNESS

The harness partitions each repository into file groups and audits them in separate passes, supplying shared project context and code relationships. Passes run in isolated, historically faithful environments without network access or revision history.

File Partitioning

  • The harness audits file partitions in separate passes while giving every model identical partitions and context documents.This design was proposed after whole-repository audits achieved below 1% recall.Source: §3 · p. 5
  • Each repository is partitioned into 5–90 passes, yielding 740 passes across 4,468 files in total.Complex files receive their own groups; other files are grouped by directory and split when they exceed a 60k-token budget.Source: §3 · p. 5

Prompt Construction

  • Each pass receives shared project context, target files, a call-graph excerpt, and snippets of callers and callees; findings must follow a JSON schema.The context document describes project purpose, architecture, public API, and supported inputs.Source: §3 · p. 5

Execution and Isolation

  • Passes run for up to three hours in separate Docker containers with historically faithful dependencies, without revision history, network access, or web-search tools.Source: §3 · p. 5

4 EXPERIMENTAL EVALUATION

Across ten models, agents recover only a small share of maintainer-fixed bugs, and their findings differ from those maintainers fix. The section also examines effort, validation, bug-type patterns, harness variants, and repeated runs.

4.1 AGENTS RECOVER FEW MAINTAINER-FIXED BUGS

  • Even the best run achieves less than 21% recall and at most 8.1% precision against the reference set.Some cheaper models exceed GPT-5.6 Sol (medium) in F1 at less than a fifth of its cost.Source: §4.1 · p. 5, Table 3 · p. 6
  • GPT-5.6 Luna recall rises from 5.3% to 17.3% as cost increases from $8 to $131, while precision stays constant.GPT-5.6 Sol recall rises from 11.4% to 20.8% as cost increases from $276 to $1,185; Luna (xhigh) strictly dominates Sol (medium) in cost, recall, and precision.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 AGENTS FIND DIFFERENT BUGS THAN MAINTAINERS FIX

  • The panel confirms 84% of findings that do not match the reference set as correct, so low reference precision mostly reflects real bugs outside that set.The panel confirms 96% of matching findings; a blind human grader agreed with the panel on 93% of findings neither judged inconclusive.Source: §4.2 · p. 6
  • Agent bug-type distributions are similar to one another but differ from maintainers’: agents over-report local faults, while maintainers more often fix incomplete input handling and public-API errors.Humans more often find bugs requiring specific environments (10.8% vs. 4.7%) or visual inspection (3.8% vs. 0.7%); agent findings more often require only a specific input (83.9% vs. 74.3%).Source: §4.2 · pp. 6–7, Figure 6 · p. 6
  • Within bug types, GPT-5.6 Luna runs overlap with one another a median of 6.3 times more than with the human reference; other models are more likely to find bugs found by more Luna runs.For bugs found in none versus all nine Luna runs, the likelihood another model finds a reference bug rises from 5% to 87%, compared with 12–17% under category-matched random sampling.Source: §4.2 · p. 7, Figure 7 · p. 7, Figure 8 · p. 7

4.3 FURTHER ANALYSIS

  • The vanilla single-session setting has under 1% recall; adding passes without the full context yields intermediate recall and precision, but lower F1 and higher cost than the full harness.Hound achieves less than half the full harness’s recall at lower precision and 8.6× the cost; agentic-pbt achieves under 1% recall at good precision.Source: §4.3 · pp. 7–8, Table 5 · p. 8
  • Across repeated runs, combining findings trades recall against precision: the Muse Spark 1.3 union reaches 26.7% recall, while the GPT-5.6 Luna intersection reaches 10.5% precision.Only 63% and 57% of findings in a run are also reported by another Muse Spark 1.3 and GPT-5.6 Luna run, respectively; higher-report-probability findings are more likely to match human findings.Source: §4.3 · p. 8, Figure 9 · p. 8
  • Contamination cannot be ruled out for GLM 5.3 Flash, Muse Spark 1.3, and DeepSeek V4.1 Flash, which were released after T1 without published knowledge cut-offs.For models with published cut-offs, the reference window contains those cut-offs and the authors find no evidence of contamination.Source: §4.3 · p. 8

5 RELATED WORK

Prior work addresses bug resolution and other maintenance tasks, but the authors identify general repository-wide bug discovery as an unevaluated benchmark setting. Existing detection work is narrower, focusing on particular files, patches, or security and specialized bug classes.

5 RELATED WORK

  • The authors identify no prior benchmark evaluating general repository-wide bug discovery, despite work on bug resolution, refactoring, test generation, and optimization.Prior bug-detection studies often direct agents to known patches or affected files, or focus on security vulnerabilities.Source: §5 · p. 8
  • Unlike patch- or file-focused tasks, PROBE asks agents to report all defects across an entire repository.SWR-Bench uses pull requests, while Active-SWE supplies affected files for concrete historical bugs.Source: §5 · pp. 8–9
  • Existing repository-level methods target vulnerabilities or specific bug classes, whereas this work considers general software bugs beyond security vulnerabilities.Examples include null-pointer dereferences, memory leaks, use-after-free, missing defensive patterns, and vulnerability-related dependencies.Source: §5 · p. 9

6 LIMITATIONS

PROBE’s reference set captures bugs maintainers fixed within an 18-month window, across a restricted repository sample and under a code-focused audit method. These boundaries constrain what the benchmark measures and which bugs the evaluated agents can surface.

6 LIMITATIONS

  • The 18-month reference window excludes bugs not fixed within that period, while unfixed bugs may still matter; estimated 5–10-year precision ceilings are 26–47%.Recall likely carries over to bugs fixed later, and the reference set reflects maintainer priorities rather than a random bug sample.Source: §6 · p. 9
  • PROBE covers 20 repositories in Python, JavaScript, and TypeScript, excluding memory-safety bugs and codebases exceeding 200k lines.The authors identify extension to more languages and repositories as future work.Source: §6 · p. 9
  • Other harnesses could produce different results, and evaluated agents primarily inspect code although maintainer-fixed bugs more often require a specific environment, output inspection, or timing.The authors suggest exercising software as users do as a future direction; the evaluated harness is identical across models, outperforms single-session and prior methods, and allows each pass repository-wide access.Source: §6 · p. 9

7 CONCLUSION

PROBE benchmarks whole-repository bug discovery against bugs maintainers fix. Across ten models, agents recover few reference bugs, and their findings differ systematically from maintainer fixes despite most being real bugs.

7 CONCLUSION

  • 20.8% maximum recall across ten models, with low precision; stronger models and higher reasoning effort mostly produce more findings rather than better ones.An independent panel of LLM judges confirms that most findings are real bugs.Source: §7 · p. 9
  • Agent-found bugs disproportionately involve local faults such as wrong arithmetic or comparisons, while human-fixed bugs more often involve incomplete input handling and public API errors.The conclusion also notes unhandled degenerate values among agent-found bugs.Source: §7 · p. 9
  • PROBE is the first benchmark for whole-repository bug discovery using bugs fixed by maintainers as its reference.The authors state that the benchmark makes progress toward measuring discovery of bugs that matter to humans.Source: §7 · p. 9

A BENCHMARK DETAILS

The appendix identifies the repository inventory and describes the fields recorded for each reference bug.

A.1 THE REPOSITORIES

  • Table 6 lists the 20 PROBE repositories by GitHub name, grouped by language and sorted by reference-bug count.The table covers repositories across the benchmark’s languages.Source: §A.1 · p. 13
  • Table 6 reports each repository’s audited files and kLOC, audit passes, snapshot tests, reference-window commits and bug-fixing commits, and resulting bugs.Files and kLOC count audited non-test source files; the commit and bug counts cover the 18-month reference window.Source: Table 6 · p. 13

A.2 BUG STRUCTURE

  • Each reference-bug record includes incorrect behavior, a primary snapshot location, surrounding code, severity, and a suggested fix.The main text abbreviates repository, file, and line range as the primary code location, and title and description as the description.Source: §A.2 · p. 13, Table 7 · p. 13

A.3 ESTIMATING THE TOTAL BUG POOL

The section models bug-fix positions in bug-fixing commits to estimate how many snapshot bugs maintainers may eventually fix. A held-out timing check supports the fitted curves, but extrapolated totals remain highly sensitive to assumptions about the unobserved tail.

A.3 ESTIMATING THE TOTAL BUG POOL

  • The model measures time in bug-fixing commits, which factors out repository activity that varies substantially across calendar months.Pooled monthly counts range from 175 to 378, a factor of 2.16.Source: §A.3 · p. 14
  • The conditional likelihood uses only the shape of observed fix positions within each repository’s window, with parameters shared across repositories.Conditioning cancels each repository’s unknown number of snapshot bugs; continuous densities approximate the half-integer commit positions.Source: §A.3 · p. 14
  • The authors compare exponential, Weibull, and lognormal fix-position families, whose hazard and tail assumptions differ.The exponential assumes a constant fixing hazard; Weibull permits decreasing or increasing hazard, and lognormal permits a much longer tail.Source: §A.3 · p. 15
  • The held-out diagnostic predicts 928, 957, and 1,008 late fixes against 1,049 observed, with underprediction of 11.6%, 8.8%, and 3.9% for the exponential, Weibull, and lognormal fits.The check tests when bugs are fixed, not how many bugs exist; no formal confidence band is shown because same-commit fixes may be dependent and parameters are estimated from the data.Source: §A.3 · p. 15
  • For a horizon of y years, extrapolation assumes the observed fixing pace continues; eventual totals sum the fitted curve’s full tail.The observed 2,445 bugs are a lower bound, but the data provide no finite upper bound because never-fixed bugs are invisible and the holdout cannot measure the missing population.Source: §A.3 · p. 15

A.4 BUG MATCHING AND DEDUPLICATION

The appendix describes how findings and reference bugs are matched and deduplicated, then how repository files are assigned to audit passes. The matching filter retains nearly all confirmed duplicates in held-out pairs, while the deterministic planner groups files using connectivity, size, and directory structure.

A.4 BUG MATCHING AND DEDUPLICATION

  • The six-signal similarity filter sends a pair to the judge when any signal reaches its threshold, retaining 472 of 474 duplicates in held-out pairs.Thresholds were chosen to minimize judge calls while retaining all 217 confirmed duplicates in 3,770 finding–reference pairs from ten repositories.Source: §A.4 · p. 16, Table 10 · p. 16
  • Reference-bug and repeated-run matches are merged as connected components, while recall counts matched reference bugs and precision counts findings matched at least once.The procedure is many-to-many; reference-bug merging requires a shared root cause, and repeated-run merging appends suggested fixes to descriptions.Source: §A.4 · pp. 15–16
  • The matching validation reaches 98.5% precision and recall with the filter off, and later filtered runs agree with the first run on at least 99.8% of pairs.All errors concern three debatable pairs about whether one fix resolves both findings.Source: §A.4 · p. 16

B PASS PLANNING

  • The deterministic pass planner uses a static call graph, file sizes, and the directory tree, applying the same parameters so each snapshot yields the same plan.It covers non-test source files and skips 197 generated migration files in prefect.Source: B PASS PLANNING · p. 17
  • The file score weights cross-file connectivity most heavily, while individual passes are assigned by a score threshold subject to a 15% file cap.A cross-file call partner weighs as much as 60 lines; remaining files are batched by directory and oversized batches are split to stay within a 60k-token budget.Source: B PASS PLANNING · p. 17
  • Individual passes include up to 40 callers and callees from other files, while batches include up to 25 of each.Calls within a batch are omitted because both files are already included in the prompt.Source: B PASS PLANNING · p. 17

C.1 RESULTS ON THE FIVE-REPOSITORY SUBSET

On the five-repository subset, Claude Opus 5 reaches 21.2% recall at 10.0% precision, comparable in recall to GPT-5.6 Sol (medium).

C.1 RESULTS ON THE FIVE-REPOSITORY SUBSET

  • 21.2% recall at 10.0% precision: Claude Opus 5 is level with GPT-5.6 Sol (medium) and has more than twice Claude Sonnet 5’s recall.The comparison covers five repositories and 755 reference bugs; other configurations rank roughly as they do across all repositories.Source: §C.1 · p. 17

C.2 RECALL AND PRECISION PER REPOSITORY

Recall and precision vary substantially across repositories, and the section also describes the sampling and grading behind the finding-validity analysis.

C.2 RECALL AND PRECISION PER REPOSITORY

  • Tables 12 and 13 break recall and precision down by repository, while Table 11 reports recall, precision, and F1 for Claude Opus 5’s five audited repositories.Those five repositories contain 755 reference bugs; the repeated-run cells for Muse Spark 1.3 and GPT-5.6 Luna (medium) are means.Source: §C.2 · p. 17, Table 11 · p. 18
  • Repository-level recall averages range from about 3% on sqlglot to over 40% on liquidjs and jsonschema, while precision ranges from 0.6% on jose to 22.9% on pypdf.TypeScript recall averages 25.4%, compared with 10.8% for Python and 12.0% for JavaScript; repository rankings are consistent across configurations.Source: §C.2 · p. 17
  • Bug count does not explain repository recall differences (Spearman ρ = −0.27), and configurations rank repositories consistently (Kendall’s W = 0.75).Precision rankings are even more consistent (W = 0.92).Source: §C.2 · p. 17

C.3 VALIDITY OF FINDINGS

  • The validity analysis sampled findings in proportion to each run’s repository findings, used opaque identifiers, and cost $772 ($0.86 per finding).The author’s blind verdicts were compared with the panel’s on 180 findings graded by both; reported agreement excludes uncertain or inconclusive verdicts.Source: §C.3 · p. 18

C.4 CONTAMINATION

The fix-rank analysis finds no evidence that agents favor earlier- or later-fixed bugs after correction. The appendix also documents the construction of bug categories and differences between agent and human bug-type distributions.

C.4 CONTAMINATION

  • The fix-rank test compares matched bugs’ normalized fix-time ranks with a repository-weighted null distribution, using a Kolmogorov–Smirnov statistic and within-repository label permutations.The null treats each repository’s matched bugs as a uniformly random subset; p-values are adjusted across runs with Holm’s correction.Source: §C.4 · p. 18
  • After Holm correction, no run shows evidence of matching bugs preferentially by fix time; only DeepSeek V4.1 Flash had an unadjusted p < 0.05.Its unadjusted result was p = 0.008, D = 0.076, but its adjusted p was 0.204.Source: §C.4 · p. 18
  • The test can reveal contamination only when training data includes some, but not all, fixes in the reference window; seven published model cut-offs fall within that window.The listed cut-offs range from January 2025 to May 2026; three models have no published cut-off and were released after T1.Source: §C.4 · p. 18

D.1 CONSTRUCTION

  • The 33-category scheme was induced from human-fixed bugs and independently validated agent findings, then applied to every configuration using frozen categories.The construction combined 2,444 human-fixed bugs with 3,205 confirmed findings, counting 233 matches once; categories were proposed from a 2,546-bug sample and merged when small unless irreducible.Source: §D.1 · p. 22

D.2 CATEGORIES

  • Table 18 and Table 19 list the 33 bug types, grouped into six families.Category definitions also identify neighboring categories for borderline cases, although those clauses are omitted from the tables.Source: §D.2 · p. 22, Table 18 · p. 23, Table 19 · p. 24

D.3 BUG-TYPE DISTRIBUTION

  • For all ten bug types in Table 20, the human share falls outside the range across individual runs, so no single configuration drives the agent–human distribution gap.The table reports mean agent shares and ranges across runs, with gaps measured in percentage points.Source: §D.3 · p. 25, Table 20 · p. 25

D.4 TRIGGER

Agents’ findings concentrate on bugs triggered by specific inputs, while maintainer-fixed bugs more often require environmental, human, or timing conditions. The paper also models repeated runs as sampling bug findings with stable probabilities.

D.4 TRIGGER

  • Agent findings concentrate on bugs triggered by specific inputs, while human-fixed bugs more often require a specific environment, output inspection, or timing, concurrency, or load.For each of those latter trigger levels, every agent run falls below the human share.Source: §D.4 · p. 25, Table 21 · p. 26

E A MODEL OF REPEATED BUG DISCOVERY

  • A stable-beta process is fitted to repeated Muse Spark 1.3 and GPT-5.6 Luna runs to model finding overlap and extrapolate the union across many runs.The model reproduces both the many findings reported once and the stable core of findings.Source: E A MODEL OF REPEATED BUG DISCOVERY · p. 26, Figure 12 · p. 27

E.1 MODEL

  • The model’s setting distinguishes the historical reference bugs H from all bugs U and allows reachable agent findings to include false positives.A defect counts as a bug if a maintainer would fix it when presented with the defect and an available fix.Source: §E.1 · p. 26
  • The model treats each deduplicated finding as a feature, with each run recording whether it reported that finding.Conditional on the feature probabilities, reporting indicators are independent across runs and findings; probabilities need not sum to one, so a run can report multiple findings.Source: §E.1 · p. 26
  • The stable-beta model assigns reporting probabilities to findings through a three-parameter process with mass α, concentration c, and discount σ.Its intensity is an expected atom count rather than a normalized probability density; larger c reduces expected pairwise sharing at fixed α and σ.Source: §E.1 · pp. 26–27

E.2 FIT TO REPEATED RUNS

This section specifies the reproducibility, snapshot-based bug classification and extraction, candidate auditing, and finding-validation procedures, including prompts and taxonomy assignment. It also describes fitting the finding-frequency spectrum across repeated runs and the implications of the fitted process for the accumulated union.

E.2 FIT TO REPEATED RUNS

  • The stable-beta process is the only compared model not rejected by the frequency-spectrum fit (p = 0.36 and p = 0.064).The comparison uses deduplicated findings reported in at least one run; the binomial and Beta-binomial alternatives do not capture both frequent rediscovery of a stable core and continued discovery of rare findings.Source: §E.2 · p. 27, Table 22 · p. 28
  • The fitted stable-beta discounts are 0.26 and 0.29, and estimated masses of 7,772 and 2,644 closely match mean per-run counts of 7,770 and 2,636.The Beta-binomial fit reaches its lower bound a = 0.01, while GPT-5.6 Luna runs six to ten increase the distinct-finding union from 5,689 to 7,472, contrary to the binomial model’s small implied pool.Source: §E.2 · p. 27, Table 23 · p. 28

E.3 UNBOUNDED DISCOVERY AND FALSE POSITIVES

  • With positive fitted discount, distinct findings grow unbounded, so the false-positive fraction of the deduplicated union tends to one when the repository has finitely many bugs.This is not evidence that historical non-matches are false positives, and the incomplete reference set cannot provide the upper bound on the bug universe needed for a finite-budget numerical lower bound; individual runs may repeatedly report real bugs. Requiring findings to recur in at least n runs selects higher minimum reporting probabilities, trading recall for precision.Source: §E.3 · p. 28

F PROMPTS

  • Appendix F prints PROBE prompt templates grouped by pipeline stage or experiment, with runtime values named and output schemas appended when a stage constrains its output.The printed prompts are templates rather than individual runtime instances.Source: F PROMPTS · p. 29

F.1.1 SNAPSHOT ENVIRONMENT (PYTHON)

  • The Python snapshot-environment brief requires a reproducible Docker image with fully pinned pre-cutoff dependencies and a test suite that runs to completion.It also directs the agent to remove upstream Git history so post-snapshot fixes cannot be read, and to adapt test paths, dependencies, and flags to the repository.Source: §F.1.1 · pp. 29–30

F.1.2 SNAPSHOT ENVIRONMENT (TYPESCRIPT/JAVASCRIPT)

  • The TypeScript/JavaScript counterpart requires a reproducible Docker test environment with package versions pinned to the snapshot lockfile and none released after the cutoff.The brief is for a repository at its snapshot commit and runs the suite through environment/run_tests.py.Source: §F.1.2 · p. 31
  • Frozen installs from committed lockfiles preserve snapshot dependency versions, while removing upstream Git history prevents access to post-snapshot fixes.The setup also requires pinning the Node release and avoiding floating image tags.Source: §F.1.2 · pp. 31–32
  • Test wrappers should emit a parseable TOTAL line; bypass chained lint, coverage, or browser steps when needed so counts reflect the runnable unit suite.The guidance accepts longer in-suite timeouts on the throttled build VM and requires noting skipped suites and why.Source: §F.1.2 · p. 33
  • For suites taking over about 10 minutes, validate the environment on a representative subset but configure the image to run the full suite.The full run can be deferred, provided the subset-only validation is recorded.Source: §F.1.2 · p. 33

F.2.1 BUG-FIX COMMIT CLASSIFIER

  • A commit is a bug fix when any part corrects previously buggy behavior; dependency updates, tests, docs, tooling, and behavior-preserving refactors do not qualify by themselves.The classifier excludes reverts and changes that fix bugs only in dependencies, while directly patched vendored code counts.Source: §F.2.1 · pp. 33–34
  • Mixed commits count as bug fixes if they include a genuine repair, even when most changes are unrelated; edge-case handling qualifies only when prior behavior was broken.Confidence should reflect certainty that a real bug was fixed.Source: §F.2.1 · p. 34
  • The classifier uses file-change statistics and a commit diff truncated at the configured budget’s head and tail.Source: §F.2.1 · p. 34

F.2.2 BUG EXTRACTOR

  • The extractor defaults to one logical bug per commit, splitting only clearly independent root causes and not multiple symptoms of the same defect.It rejects commits that do not fix a behavioral bug and localizes each emitted bug to parent-side lines, using an insertion point when the fix only adds lines.Source: §F.2.2 · p. 35

F.2.3 BUG PRESENCE CHECK

  • The presence check evaluates candidates against the snapshot and drops one only when it establishes the defect was not already present.The agent works in a read-only checkout with later history available and can run the snapshot’s code or inspect Git history to resolve whether a candidate existed then.Source: §F.2.3 · p. 36
  • The presence check guards against temporal errors, where a fix addresses code or a call path introduced after the snapshot, and false absence claims caused by partial windows.It requires evidence from the snapshot and history; the prompt identifies these two errors as decidable through repository checks.Source: §F.2.3 · p. 37
  • A drop must name and cite one of four snapshot-code grounds: a postdated precondition, deliberate rejection, absent capability, or code absent from the snapshot.If none can be established, keep the record; the stated grounds concern facts about the snapshot, not the commit or PR's description.Source: §F.2.3 · pp. 37–38
  • When no concrete ground for dropping is established, keep with low confidence rather than treating uncertain equivalence or packaging as proof that no bug exists.The instructions explicitly reject “this changes no observable behavior” as a drop ground and require noting the close rival resolution.Source: §F.2.3 · pp. 38–39
  • Search the whole snapshot when pinned ranges miss the described defect, and do not infer absence from ranges, shifted lines, or renamed identifiers.A repo-wide grep hit can establish presence; only a repo-wide grep with no load-bearing-name occurrence supports code_absent.Source: §F.2.3 · p. 38
  • Use the fix diff to assess what the project treated as required, but establish whether the repaired code existed and misbehaved using evidence from the snapshot.The diff is against a later revision and may contain code, tests, or reproducers absent from the snapshot.Source: §F.2.3 · p. 38
  • Commit prefixes, merge history, SZZ status, and issue or PR labels are clues for choosing checks, not grounds for dropping a record.An ancestry answer is the strongest available evidence for whether a repaired precondition existed by the snapshot.Source: §F.2.3 · pp. 38–39
  • The call is valid for the project’s declared target, so the candidate is dropped as a capability absent only from an out-of-scope embedded runtime.The fix changes one call site while leaving two equivalent uses untouched.Source: §F.2.3 · pp. 40–41

F.2.4 BUG REPORT SYNTHESIZER

  • Bug reports describe a surviving defect as present at the snapshot, identify its primary candidate, and frame purely additive fixes as omissions rather than faulty visible code.Titles and descriptions use present tense, cite snapshot identifiers and lines, and avoid referencing the fix.Source: §F.2.4 · pp. 41–42

F.2.5 DEDUPLICATION JUDGE

  • Records count as duplicates only when they describe the same root-cause flaw; judges prioritize mechanism, trigger, and faulty behavior over overlapping snippets or line ranges.Different mechanisms or lines within the same function indicate distinct defects.Source: §F.2.5 · pp. 42–43

F.3.1 FALSE-POSITIVE AUDITOR

  • The false-positive auditor may drop a reference bug only on an established, admissibly cited ground, using snapshot execution and repository history while excluding commit, issue, and PR text as evidence.A reproducer that passes at the snapshot directly supports the conclusion that the defect is absent there.Source: §F.3.1 · p. 43
  • Every required check must have its own citation to command output, and a `none` verdict requires the temporal test, identifier search, pinned-line review, and a realizable input.A check without its own citation, or supported only by paraphrase, invalidates the verdict.Source: §F.3.1 · pp. 44–45
  • When multiple grounds apply, prioritize `temporal`, `not_defect`, `wrong_location`, then `wrong_description`, distinguishing post-snapshot code, non-bugs, misplaced pointers, and contradicted harms.Temporal grounds include code or triggering preconditions added after the snapshot; non-defect grounds include deliberate rejection or absent promised capability.Source: §F.3.1 · pp. 44–45
  • A defect inside the pinned unit remains `none` even if outside the pinned lines, while a defect in another file or outside that unit is `wrong_location`.For pins outside a function, method, or class, the defect must be within 10 lines in the same file to count as located.Source: §F.3.1 · p. 44
  • The fix diff shows what the project treated as required, not what existed at the snapshot; prove snapshot presence or absence repo-wide rather than inferring it from the pinned range.Snapshot tests can establish a declared surface or a realizable input that the code mishandles.Source: §F.3.1 · p. 45
  • The false-positive auditor record template captures repository and snapshot identifiers, fix commits, pinned locations and spans, a description, and a suggested fix.The suggested fix is part of the record, not evidence.Source: §F.3.1 · pp. 46–47

F.4.1 MATCHER JUDGE

  • The Gemini 3.5 Flash matcher judge scores recall and precision by deciding whether findings and reference bugs describe the same defect; Step 6 uses the stricter variant.Duplicates include bugs resolved by one fix or a subset bug automatically resolved by fixing the larger bug.Source: §F.4.1 · p. 47

F.5.1 REPOSITORY CONTEXT DOCUMENT

  • A one-time Claude Opus 5 exploration pass creates a repository context document, mounted for every downstream per-file finding and validation pass.The document guides reachability and expected-behavior decisions and supplies cross-module context that agents cannot reconstruct from a single file.Source: §F.5.1 · pp. 47–48
  • The repository-context document is requested as a structured report, with intentional deviations and unsupported features identified as load-bearing for validation.The instructions call for factual, specific context and say to omit inapplicable sections rather than pad the report.Source: §F.5.1 · p. 50

F.5.2 AUDIT PASS INSTRUCTIONS

  • Targeted passes prioritize broad bug discovery, provide repository and call-graph context, and leave validation to adjudicate findings while discouraging unsupported or non-runtime reports.Auditors are instructed to check caller-callee contracts systematically and report plausible findings with calibrated confidence.Source: §F.5.2 · pp. 50–51

F.5.3 VANILLA AUDIT PROMPT

  • The vanilla prompt audits each repository in one session with only audit instructions and the output schema, omitting pass decomposition, repository context, source, and call graph.An empty findings array is explicitly valid.Source: §F.5.3 · p. 53

F.5.4 PASSES-ONLY AUDIT PROMPT

  • The passes-only prompt follows the targeted passes but supplies only target-file paths, omitting repository context, source code, and call-graph material.Auditors are told to read the listed files from the checkout.Source: §F.5.4 · p. 53
  • The passes-only audit prompt requires agents to leave repository files unchanged and return findings in a specified JSON schema.An empty findings array is permitted.Source: §F.5.4 · p. 53

F.6.1 PANEL CONTEXT DOCUMENT

  • The panel context document guides judges on reachability and expected behavior, separating out-of-scope, intentional, and unsupported-feature rejections.It directs context authors to document the public API, supported scope, error handling, and relevant deviations or unsupported features.Source: §F.6.1 · p. 54

F.6.2 PANEL JUDGE

  • Panel judges independently verify one finding against repository code and a proof of concept, then return a structured verdict.They are instructed to calibrate confidence, verify rather than search for new bugs, and use a repository context document when available.Source: §F.6.2 · p. 56
  • Judges should verify source, callers, project scope, and any prefilled validation claims, then design a minimal PoC that tests the likeliest false-positive explanation.A finding's concrete realistic scenario should explain how the triggering input arises; inability to build a PoC alone is not grounds for rejection.Source: §F.6.2 · pp. 57–59
  • G1 requires a public or documented path that actually reaches the buggy code with an input a real caller or user could plausibly supply.Check callers and guards rather than inferring reachability from a function signature or importability; reject as unreachable when no realistic path exists and the function is not public API.Source: §F.6.2 · p. 57
  • G2 treats wrong output, crashes, hangs, corruption, or security breaches on supported input as bugs, but excludes unsupported inputs and clean cosmetic error handling.Divergence from an external reference counts only when it causes wrong results within the project's supported subset, and the reference should be verified.Source: §F.6.2 · p. 57
  • Confirmed findings require mechanism evidence and both gates: a realistic path to the buggy code and wrong behavior on supported input.A runnable PoC through a public path is preferred; if it is infeasible, code reasoning may support confirmation only with Low confidence.Source: §F.6.2 · p. 58
  • Rejections use a reason such as code-mismatch, unreachable, intentional, error-mode-only, out-of-scope, or style-only; severity and confidence are assessed independently.Severity is assigned only to confirmed findings, while confidence applies to every verdict and should reflect how well the evidence supports it.Source: §F.6.2 · pp. 58–59
  • Confidence is anchored to evidence: Medium requires no unverified link, High requires active falsification or mechanically forced outcomes, and reasoning-fallback confirmations cannot exceed Low.The protocol also calls for downgrading confidence when specified evidentiary shortcomings apply.Source: §F.6.2 · pp. 59–60
  • Judges verify the cited source and reasoning, test findings through a public entry point with a falsification-oriented PoC, and apply an explicit reachability-and-behavior gate.If a PoC is genuinely infeasible, reasoning may be used instead, but the confirmation is capped at Low confidence.Source: §F.6.2 · p. 60

F.7.1 CATEGORY INDUCTION

  • The taxonomy induction uses a stratified sample of historical human-fixed and agent-found bugs to define mutually exclusive categories by programming root cause, not symptom.The instructions distinguish commonly conflated fault types and require categories to meet size and coverage criteria, with rare types merged unless irreducible.Source: §F.7.1 · p. 62

F.7.2 THIN-CATEGORY MERGE

  • After all bugs are assigned, categories below the size floor are merged into the closest survivor unless conceptually irreducible, and assignments are remapped without re-attribution.Near-duplicate categories must merge regardless of size, while genuinely distinct mistakes should not be merged merely to meet the floor.Source: §F.7.2 · p. 62

F.7.3 BUG-TYPE ATTRIBUTION

  • Claude Sonnet 5 used the same prompt to classify the taxonomy corpus and each evaluated configuration’s findings, so Figure 6 distributions use one taxonomy.Source: §F.7.3 · p. 63
  • For each bug, the procedure assigns one fault-type category and one numbered level per fixed dimension, with each key included exactly once and no invented IDs.Source: §F.7.3 · p. 63
  • Classification follows the stated defect’s root cause, favors the more specific category and lower-numbered fitting dimension level, and uses 0 only when information or categories are insufficient.The code snippet is context rather than the defect itself.Source: §F.7.3 · p. 63

Related papers

Loading 2610.35987v1…