PaperlayerView exact revision on arXiv

Source-linked AI summary

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

Chenyang Yang, Hao He, Ti-Chung Cheng, Jenna Butler, Emerson Murphy-Hill, Carmen Badea, Christian Bird

arXiv:2610.13668v1cs.SE

TL;DR

Existing code-review benchmarks commonly compare AI reviews with human comments, which can be arbitrary, narrow, or accepted through social dynamics rather than serving as reliable ground truth. CRBugBench mines human-verified bugs from repository history and pairs each bug with its introducing PR to create review tasks. Agents are improving rapidly, but still struggle with complex changes and bugs requiring project context, particularly in proprietary repositories.

  • Problem

    Existing code-review benchmarks commonly compare AI reviews with human comments, which can be arbitrary, narrow, or accepted through social dynamics rather than serving as reliable ground truth.

    Source: §1 · p. 1

  • Method

    CRBugBench mines human-verified bugs from repository history and pairs each bug with its introducing PR to create review tasks.

    Source: §1 · p. 2

  • Results

    Agents are improving rapidly, but still struggle with complex changes and bugs requiring project context, particularly in proprietary repositories.

    Source: §6 · p. 19

  • Takeaways & Limitations

    The findings support retaining human oversight and adapting review effort, repository context, and agent configurations to project needs.

    Source: §6 · p. 19

    The repository selection favors mature, active, popular projects with detailed textual evidence and reproducible environments, and may not represent other software development settings.

    Source: §4.2 · p. 11

Abstract

from arXiv · show

With the advances of coding agents, software developers are spending more time reviewing code than writing code. As human review velocity falls behind agentic code generation, it is tempting to build automated review agents towards a future of "lights-off" software factory with little to no human supervision. To assess the practicality of such a fully autonomous software factory, we construct CRBugBench, a new benchmark that evaluates whether review agents detect human-verified bugs. We build a data pipeline that mines bugs from repository history, and pair each bug with its introducing PR to form a review task. CRBugBench consists of 1,000 review tasks across 98 open-source GitHub repositories, and 800 additional tasks from proprietary industry repositories at one large company. We then use the benchmark to study agent review quality: We found that SOTA review agents do well: the best (claude-opus-5 with copilot-cli) catches 71.9% of bugs within its top-5 comments, up from 47.7% (opus-4.6) in five months of model evolution. Yet agents remain less effective on complex changes and bugs requiring non-local reasoning, with sharp performance drops on proprietary industry repositories where project context is more often needed. To improve bug detection, we found scaling compute helps most through prompt changes: exploration-nudging prompts gain up to 5.8% performance at 10% higher cost. From these findings, we derive concrete guidance for developers to adopt review agents - scaling review effort to PR complexity, steering agents by project priorities, supplying repository knowledge, dispatching models by PR characteristics, and monitoring quality as models evolve.

1 Introduction

The introduction argues that existing review benchmarks rely on human comments that may not represent review ground truth, and introduces CRBugBench to evaluate detection of human-verified bugs. Its findings show improving agent performance alongside important weaknesses in complex, context-dependent, and proprietary-repository tasks.

1 Introduction

  • CRBugBench pairs human-verified bugs from addressed review threads or later bug-fix PRs with their introducing PRs, yielding 1,800 tasks across open-source and industry repositories.The benchmark spans 98 GitHub repositories and two large industry repositories.Source: §1 · p. 2
  • Within the top five comments, claude-opus-5-xhigh catches 79.6% of bugs found before merge and 64.2% found after merge; bug-catching capability rose from 47.7% (opus-4.6) to 71.9% (opus-5).The reported performance increase spans February to July 2026.Source: §1 · pp. 2–3
  • On proprietary industry repositories, review agents drop up to 25.5% in performance, partly because those tasks more often require project context beyond the diff.The proprietary repositories are industry-scaled and not part of the models’ training data.Source: §1 · p. 3
  • Agents are less effective on larger, more heterogeneous PRs, bug-fixing PRs, non-functional or less severe bugs, and bugs requiring non-local reasoning.These differences indicate that review quality varies with PR and bug context.Source: §1 · p. 3
  • claude-opus-5 executes code in 9.7% of turns versus 6.9% for gpt-5.6-sol; half of their runs end by turns 35 and 17, respectively.Agents spend most of their turns reading and searching code, but rarely execute it.Source: §1 · p. 3
  • Agents may trust misleading PR descriptions or code comments, or fail to account for repository-wide behavior and external-library semantics.The authors identify these as limitations of current frontier models.Source: §1 · pp. 3–4
  • Exploration-nudging prompts improve performance by 5.8% at 1.10x cost, compared with 4.7% at 5.55x cost for multi-agent setups.Increased reasoning effort can raise performance from 36% to 57% at 23.22x cost.Source: §1 · p. 4
  • The authors recommend scaling review effort to PR complexity, steering agents by project priorities, supplying repository knowledge, dispatching models by PR characteristics, and monitoring quality as models evolve.They also recommend building repository-specific benchmarks with CRBugBench’s methodology.Source: §1 · p. 4

2 Background and Related Work

Modern code review is lightweight and asynchronous, but its outcomes and comment usefulness vary. Existing evaluation approaches have limits, motivating CRBugBench’s focus on detecting human-verified bugs in introducing pull requests.

2.1 Modern code review

  • Modern code review is lightweight and asynchronous, and the usefulness of review comments depends on their technical content and context.Larger changes tend to receive a lower proportion of useful comments; CRBugBench focuses on bug finding while leaving subjective or domain-specific aspects to human reviewers.Source: §2.1 · p. 4

2.2 Benchmarking code review

  • Comment-matching benchmarks can penalize valid findings reviewers missed, while LLM-based evaluation is limited by evaluator reliability and does not measure historical confirmed-bug recovery.CRBugBench instead evaluates whether agents identify defects verified by human-authored changes, including bugs caught or missed during the original review.Source: §2.2 · pp. 4–5

2.3 Automating code review

  • Automation has progressed from static analysis and learned edits to agentic review, while prior studies report faulty feedback and no improvement in high-severity detection or review time.The paper evaluates end-to-end bug detection and characterizes conditions associated with failures to inform human-oversight allocation.Source: §2.3 · p. 5

3 Data Collection

The data pipeline selects established repositories, mines human-verified bugs, links them to inducing changes, and builds review tasks. The resulting benchmark spans public repositories and large proprietary industry repositories, with task and bug attributes for analysis.

3 Data Collection

  • The pipeline covers repository selection, human-verified bug identification, and construction of review-agent task environments.Figure 3 summarizes the process of identifying bugs from human comments and bug-fixing pull requests, tracing them to inducing PRs, and creating containerized tasks.Source: §3 · p. 5, Figure 3 · p. 6

3.1 Repository selection

  • The authors selected established, active repositories across five programming languages and sampled one repository per application domain and language, yielding 106 repositories across 31 domains.Selection filtered popular repositories by PR and contributor counts, engineering focus, recency, and activity.Source: §3.1 · pp. 5–6

3.2 Bug identification

  • Bug identification combines review-comment bugs and post-merge issue bugs, then traces defects to inducing PRs and filters noisy or poorly described pairs.SZZ linking and LLM quality checks reduced 1,168 candidate pairs to 736; further screening produced 689 fix–inducing PR pairs. A sample audit found 37 of 40 tasks valid (92.5%).Source: §3.2 · pp. 6–7

3.3 Task creation

  • Tasks use repository snapshots at the inducing PR’s base commit, provide PR context and changes, and score findings against ground truth by majority vote from three LLM judges.The judges achieved almost perfect human-LLM agreement (κ=0.89) and across-LLM agreement (Fleiss’ κ=0.95); recall@k measures detection within the agent’s top-k findings.Source: §3.3 · p. 7

3.4 Benchmark characteristics

  • The public benchmark contains 1,000 tasks across 98 repositories, with repository, change, and bug attributes collected or annotated for downstream analysis.The annotations include change type, bug impact, severity, and visibility; reported validation agreement ranges from substantial to almost perfect depending on attribute.Source: §3.4 · p. 8, Figure 4 · p. 8

3.5 Proprietary benchmark construction

  • Applying the same data pipeline to two industry monorepositories on enterprise collaboration and productivity platforms yielded 800 additional review tasks from repositories averaging >100k source files, versus 3k for open-source repositories.Source: §3.5 · p. 9

4 Experiments

The experiments evaluate review agents’ bug-detection performance across models and task conditions, and examine agent trajectories and harness, prompt, and orchestration choices. They also assess test-time compute approaches, including effort, prompting, and multi-agent setups, finding comprehensive prompting more cost-effective and performant than multi-agent review.

4.1 Research Design

  • The study compared large and smaller models using a pinned copilot-cli harness and official review prompt, measuring recall, cost, latency, and review length.It also used logistic regression for task-condition associations, instrumented tool trajectories, and varied reasoning effort, harness, prompt, and orchestration.Source: §4.1 · pp. 9–11

4.2 Threats to Validity

  • The evaluation measures recovery of known bugs rather than overall review usefulness, and its observational regressions estimate associations rather than causal effects.The authors note possible benchmark ground-truth errors, stochastic agent behavior, unreliable LLM judges, and limits to generalizing beyond mature, popular projects and the studied repositories.Source: §4.2 · p. 11

4.3 How well do review agents detect PR bugs (RQ1)?

  • 71.9% of bugs were caught within claude-opus-5’s top five findings, up from 47.7% for claude-opus-4.6, at $2.97 per task.Humans alone caught 50%, while combining human and agent findings caught 82.1%; performance decreased by up to 25.5% on internal repositories.Source: §4.3 · pp. 11–13, Table 1 · p. 12

4.4 How are task conditions correlated with review agent performance (RQ2)?

  • Detection odds fell for larger (OR=0.591), heterogeneous (OR=0.795), and bug-fixing PRs (OR=0.521), while language and repository size showed no significant effects.The authors suggest review attention is diluted by larger or mixed changes, and bug-fixing PRs may divert attention toward checking the fix itself.Source: §4.4 · pp. 13–14
  • Detection favored diff-visible (OR=2.621) and severe bugs (OR=1.906), while usability bugs were harder (OR=0.454); industry tasks had fewer diff-visible bugs than GitHub tasks.Visibility differences explained 20% of the industry performance gap. Opus-5’s gains narrowed on refactors and other PR types; humans were better at maintainability issues, while agents were better at severe bugs and reliability issues.Source: §4.4 · pp. 14–15

4.5 Where do agents spend time and how do they fail (RQ3)?

  • Agents devoted 75% of turn mass to reading and searching but 8% to running code; 95.6% inspected defect files, 80.2% reported findings, and 56.2% hit bugs.Half of gpt-5.6-sol runs were over by turn 17, while claude-opus-5 reached its half-life at turn 35.Source: §4.5 · p. 15

4.6 How do agent design choices affect bug-detection performance (RQ4)?

  • gpt-5.6-luna’s detection rose from around 36% to 57% at maximum reasoning effort, at 23.22x the cost; comprehensive prompting improved gpt-5.6-sol by 5.3% at similar cost.Multi-agent setups improved detection by 2.8% on average at 3.68x the cost, and comprehensive prompting configurations dominated them on the cost-performance frontier.Source: §4.6 · p. 16, Table 2 · p. 16
  • Prompt changes help models unevenly; as encouraged behaviors enter model training, earlier harness designs may become outdated and need periodic review.For claude-opus-5, prompt improvements were much smaller or absent compared with earlier models.Source: §4.6 · pp. 16–17
  • Copilot-cli outperforms claude-code, but claude-opus-5 with claude-code sharply raises costs, partly alongside more frequent sub-agent use.The cited trajectories show 9.0 sub-agents per run for claude-opus-5 with claude-code, versus 0 for copilot-cli and 0.1 for claude-code with opus-4.8; Anthropic documentation says Opus delegates more readily than prior models.Source: §4.6 · p. 17
  • Comprehensive prompting is more cost-effective and performant than multi-agent review, while effort, prompting, and multi-agent setups can all benefit agents.The passage does not provide quantitative comparisons for these findings.Source: §4.6 · p. 17

5 Discussion

Review agents are not ready to replace human reviewers, especially for large proprietary repositories and changes needing project context. The discussion recommends adapting review effort and agent support to the PR, while identifying research directions for more effective human-agent collaboration.

5.1 When can a software factory be lights-off?

  • Lights-off operation is supported only for narrowly scoped, low-consequence PRs; consequential changes still require an accountable human’s final decision.The authors recommend calibrating deployment using repository-specific, continuously monitored evidence rather than aggregate benchmark performance alone.Source: §5.1 · p. 17

5.2 Recommendations for best practices when adopting review agents

  • Review effort should scale with PR complexity, using cheaper configurations for small homogeneous changes and stronger models and careful human review for large heterogeneous ones.The authors also suggest splitting unrelated changes into smaller PRs and reserving human attention for project values and long-term maintainability.Source: §5.2 · p. 18
  • Custom instructions can steer agents toward project priorities such as usability, portability, backward compatibility, or maintainability.Agents tend to focus on more severe, functional bugs and may be misdirected by surface cues.Source: §5.2 · p. 18
  • Repository-specific instructions and knowledge bases can help address agents’ difficulty with project context, surrounding files, and runtime evidence.These difficulties are often relevant in large proprietary repositories.Source: §5.2 · p. 18
  • Teams should periodically evaluate deployed configurations and adapt review harnesses as models and codebases evolve.The authors suggest constructing custom benchmarks for teams’ own repositories using their data-construction method.Source: §5.2 · p. 18

5.3 Research opportunities

  • Future agents could maintain repository memory from engineering artifacts and use retrieval to seek relevant project information.Research opportunities also include systems that speed human inspection by highlighting high-risk or uncertain code, and agents that communicate calibrated uncertainty.Source: §5.3 · pp. 18–19

6 Conclusion

The paper introduces CRBugBench and finds that review agents are improving but still struggle with complex changes and project context, particularly in proprietary repositories. Its findings support human oversight and adapting review effort, context, and configurations to project needs.

6 Conclusion

  • Review agents are improving rapidly but still struggle with complex changes and bugs requiring project context, particularly in proprietary repositories.The conclusion supports retaining human oversight while adapting review effort, context, and agent configurations to project needs.Source: §6 · p. 19
  • Comprehensive prompting improves bug detection more cost-effectively than multi-agent setups, though benefits vary across models.Source: §6 · p. 19
  • CRBugBench enables empirical study of when and how code review agents detect human-verified bugs.Source: §6 · p. 19

Related papers

Loading 2610.13668v1…