PaperlayerView exact revision on arXiv

Source-linked AI summary

Code Review Agent Benchmark

Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, Abhik Roychoudhury

arXiv:2603.23448v3cs.SEcs.AI

TL;DR

As AI-generated code grows, human review remains a bottleneck, and it is difficult to assess whether automated reviewers identify the issues human reviewers raise. The paper introduces c-CRAB, which evaluates reviews using executable tests derived from human feedback; current agents identify only about 40% of benchmark issues, while often raising different concerns from humans.

  • Problem

    As AI-generated code increases, human review capacity remains a bottleneck, and it is unclear whether automated tools identify the same issues human reviewers raise.

    Source: §1 · p. 1

  • Method

    c-CRAB converts human review feedback into executable tests and uses them to evaluate whether review agents identify the underlying issues.

    Source: §1 · p. 2

  • Results

    Current systems successfully identify only about 40% of the issues captured in c-CRAB, and AI-generated reviews often focus on different aspects than human reviews.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The differing review perspectives may support future human-AI collaborative review workflows.

    Source: §1 · p. 2

    The benchmark is constructed from SWE-CARE and may not represent the broader software ecosystem.

    Source: §7 · p. 10

Abstract

from arXiv · show

Software engineering agents have shown significant promise in writing code. As AI agents permeate code writing, and generate huge volumes of code automatically -- the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases -- the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today -- the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex. Our c-CRAB dataset is systematically constructed from human reviews -- given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. Firstly, the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews -- indicating the potential for human-agent collaboration for code review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents -- remains to be investigated.

1 Introduction

As AI-generated code increases, human review capacity remains a bottleneck, while existing review evaluation methods can mistake differences in wording or location for differences in issue detection. c-CRAB evaluates review agents by turning human feedback into executable tests; current agents identify only about 40% of benchmark issues, and their reviews often address different aspects from human reviews.

1 Introduction

  • As AI code generation expands, human reviewers have not scaled at the same pace, creating a quality-assurance bottleneck for submitted pull requests.Existing evaluations often compare comment wording, although valid reviews can describe the same issue differently and human comments may include discussion or subjective suggestions.Source: §1 · p. 1
  • c-CRAB converts human review feedback into executable tests that serve as oracles for whether an agent identifies the underlying issue.The evaluation has a separate coding agent revise the PR based on review comments, then executes the curated tests on the revised code.Source: §1 · p. 2
  • Current systems identify only about 40% of the issues captured in c-CRAB, and AI reviews often focus on different aspects from human reviews.The authors describe the perspectives as complementary and suggest they may support future human-AI collaborative review workflows.Source: §1 · p. 2

2 Background and Related Work

AI-based review tools analyze pull requests using learned patterns and contextual information, but code-review benchmarks vary in context and in what counts as an evaluation oracle. c-CRAB uses executable tests derived from human feedback rather than comment similarity or issue localization alone.

2.1 Automated Code Review Tools

  • AI-based review tools use LLMs and supplied code or repository context to identify potential issues in pull requests.Some tools also filter false positives or rank severity before presenting summary or inline comments; this paper focuses on AI-based tools.Source: §2.1 · p. 2

2.2 Benchmarks for Automated Code Review

  • Earlier benchmarks often pair comments with localized code changes, while newer ones include full pull requests and repository context.Most benchmarks still use natural-language human comments as ground truth; c-CRAB instead uses dynamic program behavior witnessed by tests generated from human reviews.Source: §2.2 · p. 2

2.3 Evaluation Methods for Code Reviews

  • Comment-similarity metrics, localization, production metrics, and LLM judges each have limits; c-CRAB evaluates underlying issues with executable tests in reproducible environments.Text similarity can miss equivalent concerns phrased differently, localization omits issue substance, production metrics require access to production, and LLM judges can be unstable or prompt-sensitive.Source: §2.3 · pp. 2–3

3 Motivation

The section argues that review evaluation should measure whether tools capture human-identified issues, not merely whether their comments resemble human wording. Its examples show why similarity and issue-only judging can miss differences in review intent, while c-CRAB tests whether feedback leads to behaviorally correct fixes.

3 Motivation

  • The motivating comparison identifies textual overlap and embedding similarity as evaluation approaches that do not establish whether a review identifies a meaningful code issue.The section introduces c-CRAB as a test-based alternative to those methods.Source: §3 · p. 3, Table 2 · p. 3, §3.1 · p. 3

3.1 Comparison with Similarity Metrics

  • Similarity metrics can score reviews poorly despite a shared concern because their wording differs and embedding similarity remains sensitive to phrasing.The example describes low BLEU-4, ROUGE-L, and chrF scores and only moderate embedding similarity.Source: §3.1 · pp. 3–4

3.2 Comparison with LLM-as-a-Judge

  • A review may identify the same defect as a human reviewer yet suggest a different fix, so matching the issue alone does not capture review intent.The section contrasts a proposed length-limit mitigation with the human reviewer’s TextField replacement.Source: §3.2 · p. 4

3.3 Our benchmark: c-CRAB

  • c-CRAB evaluates review feedback by whether a coding agent’s revision passes a test encoding the human-identified issue.The test-based approach can distinguish a review that proposes a different fix from one that satisfies the expected behavior.Source: §3.3 · p. 4

4 Methodology

c-CRAB evaluates code review comments by whether they lead to verifiable improvements, rather than by their resemblance to human wording. Each benchmark instance pairs a pull request with tests derived from human review feedback.

4 Methodology

  • c-CRAB assesses review comments by whether they lead to behaviorally verifiable improvements in code, not by how closely they match human wording.The methodology section introduces the benchmark design and its evaluation procedure.Source: §4 · p. 4, §4.1 · p. 5

4.1 c-CRAB Overview

  • Each benchmark instance pairs a pull request with tests derived from human feedback, and a review is useful when its guidance leads to a revision that passes the corresponding test.Tests are constructed to fail on the original patch and pass once the identified issue is resolved.Source: §4.1 · p. 5

4.2 Benchmark Curation

Benchmark curation turns pull-request descriptions, patches, and human review comments into executable tests in a reproducible environment. The pipeline filters comments, synthesizes and refines tests, and validates that a coding agent can resolve each retained instance.

4.2 Benchmark Curation

  • The curation pipeline takes the PR description, candidate patch, and human comments as inputs, then constructs benchmark instances with tests derived from reviews.Raw discussions can be noisy, and reliable test execution requires an environment reproducing repository dependencies and runtime conditions.Source: §4.2 · p. 5

4.2.1 Review Filtering.

  • An LLM classifier filters for comments that identify specific, actionable, objectively verifiable issues, using a manually annotated gold set to refine its prompt.Two authors independently label 100 sampled comments and resolve disagreements through discussion.Source: §4.2.1 · pp. 5–6

4.2.2 Execution Environment Construction.

  • Each PR receives an isolated Docker environment with operating-system and software dependencies installed and the repository preconfigured.For historic PRs with underspecified dependencies, an agent may audit requirements and revise the installation script.Source: §4.2.2 · p. 6

4.2.3 Converting NL Comments to Tests.

  • For each retained comment, an LLM synthesizes a test from the comment, diff hunk, and before-and-after repository states; execution traces guide revisions until it passes before/after validation or the attempt budget ends.The required behavior is failure on the original state and success after the issue is addressed.Source: §4.2.3 · p. 6

4.2.4 Validation with Coding Agents.

  • A coding agent receives the human comment and PR context without seeing the generated test; instances are retained only if it can produce a test-passing revision within a fixed attempt limit.This validation is intended to distinguish review-feedback quality from limitations of the coding agent used during evaluation.Source: §4.2.4 · p. 6

4.3 Benchmark Usage

c-CRAB evaluates review feedback by whether it guides a coding agent to resolve issues captured in human-review tests. Its instance-level metric is the pass rate of validated tests, while aggregate pass rate measures alignment with human-identified issues, not additional comments.

4.3.1 Workflow.

  • Review comments guide coding agent A to revise a PR patch, which is then tested against the benchmark’s associated tests.A test passing after revision indicates the feedback guided the agent to resolve an issue originally identified by a human reviewer.Source: §4.3.1 · pp. 6–7

4.3.2 Evaluation Metric.

  • Instance-level performance is the fraction of validated tests passed by the patch produced when coding agent A follows a tool’s review findings.The test set is T_i and the resulting patch is P̂_i.Source: §4.3.2 · p. 7
  • Aggregate pass rate reflects capture of human-identified issues, but c-CRAB does not directly evaluate other valuable comments that humans did not identify.The authors discuss the quality of these additional comments separately.Source: §4.3.2 · p. 7

4.4 Benchmark Implementation

c-CRAB is built from pull requests with the required metadata and uses a curation pipeline to produce validated executable tests. The resulting benchmark contains 184 PR instances and 234 validated review comments, and sampled tests generally align with human-identified concerns.

4.4 Benchmark Implementation

  • The benchmark is built on SWE-CARE because it supplies PR instances with commit metadata, but the curation pipeline can use any PR with the required metadata.Source: §4.4 · p. 7

4.4.1 Implementation of the Curation Pipeline.

  • The implementation uses GPT-5.2 for review filtering and comment-to-test conversion, allows up to three test-generation attempts with execution feedback, and uses Claude Code with Sonnet-4.6 for validation.Source: §4.4.1 · p. 7

4.4.2 Benchmark Statistics.

  • The curation pipeline yields 184 PR instances and 234 validated review comments, each associated with at least one executable test.The benchmark is a carefully curated subset of the original data.Source: §4.4.2 · p. 7, Table 4 · p. 7
  • The benchmark’s 234 tests include behavioral tests that execute code and structural tests that inspect source text, patterns, or API surfaces.Source: §4.4.2 · p. 7

4.4.3 Test Suites Quality Assurance.

  • Two authors independently judged tests for 50 sampled instances; their agreement was 84%, with disagreements involving partial coverage or implementation-specific overfitting.They reviewed each test alongside its human comment and relevant diff, consulting the original PR when needed.Source: §4.4.3 · p. 7

5 Results

Across c-CRAB, automated review tools capture fewer human-identified issues than human reviewers, although many automated comments are useful and address different concerns. Their comment distributions and success in capturing human concerns vary across issue categories.

5.1.1 Evaluated Tools.

  • The evaluation runs Claude Code, Codex, Devin Review, and PR-Agent on 184 PR instances, with Claude Code using a fixed Sonnet-4.6 backend for validation and review evaluation.The experiments ran on Ubuntu 20.04.Source: §5.1.1 · pp. 7–8, §5.1.2 · p. 8

5.2 RQ1: Evaluation Scores on c-CRAB

  • 20.1%–32.1% pass rates for automated tools contrast with 100% for human reviewers; together, the tools passed 97 of 234 tests (41.5%).The tests encode concerns raised by human reviewers, and Figure 5 shows overlap among passed tests across the four tools.Source: Table 5 · p. 8, §5.2 · p. 8, Figure 5 · p. 8
  • Most sampled automated review comments were judged useful, so the lower pass rates indicate weaker alignment with human concerns rather than necessarily poor review quality.The authors manually assessed 92 comments from six randomly selected PRs.Source: §5.2 · p. 8

5.3 RQ2: Categories of Review Comments

  • For RQ2, two authors first inspected human reviews to define categories, then independently categorized comments and resolved differences through discussion.Source: §5.3 · pp. 8–9
  • Automated tools produce comment-category distributions unlike humans: security, efficiency, compatibility, and error handling are comparable, while design, documentation, and maintainability appear less often.They raise robustness and testing concerns more frequently; this can offer complementary suggestions but may increase false-positive burden.Source: Figure 6 · p. 9, §5.3 · pp. 9–10
  • Across tools, human concerns about robustness and functional correctness have higher pass rates, while documentation and design have lower rates; maintainability ranges from 7.9% to 27.0%.The authors suggest localized patch issues may be easier to capture, while the latter categories require broader repository context and project-specific conventions.Source: Figure 7 · p. 9, §5.3 · p. 10

6 Actionable Insights

The findings support treating automated reviewers as complements to humans and point to repository context as a route to closer alignment. Executable tests also offer an objective evaluation target and a possible training signal.

6 Actionable Insights

  • Automated review tools should complement rather than replace humans, since they achieve lower pass rates and often raise different issue types.They emphasize robustness and testing while missing categories humans often identify.Source: §6 · p. 10
  • Repository-specific context may help tools identify maintainability, design, and documentation issues that automated reviewers rarely raise.The paper points to architecture documents, review history, naming conventions, code patterns, and agent-facing instructions as useful context.Source: §6 · p. 10
  • Executable tests provide an objective optimization target and a potential training signal for reviews that lead to verifiable code improvements.This evaluates review comments through execution-based signals rather than relying solely on textual similarity or subjective LLM-as-a-judge assessments.Source: §6 · p. 10

7 Threats to Validity

The evaluation has internal, external, and construct validity constraints. The authors describe checks to mitigate these risks, while noting that the dataset may not represent the broader software ecosystem.

7 Threats to Validity

  • The coding agent that revises patches may affect outcomes, so the evaluation reflects both review quality and agent capability.The authors validate instances with human comments and use the same coding agent in validation and evaluation; retained instances are solvable.Source: §7 · p. 10
  • The 184 PR instances and 234 verifiable oracles from 56 repositories may not represent the broader software ecosystem.The authors state that the pipeline can be applied to new PR instances to extend the dataset.Source: §7 · p. 10
  • An LLM-based filtering classifier may retain borderline comments, and manual review labeling involves human judgment.Mitigations include a strict fail-then-pass test-generation criterion, validation with original human comments, and independent labeling by two authors.Source: §7 · p. 10

8 Conclusion

The paper presents c-CRAB as a test-based benchmark for evaluating automated code review tools amid growing code-generation capacity and persistent review-evaluation challenges. Its results show limited issue coverage and differences from human reviews, suggesting complementarity and motivating future work on context and human-centered workflows.

8 Conclusion

  • Code review remains a bottleneck as AI-based code generation becomes increasingly easy and scalable.Evaluating the effectiveness of proposed automated review tools remains an open challenge.Source: §8 · p. 10
  • Current state-of-the-art review tools identify only a fraction of the issues captured in c-CRAB.The benchmark converts human review comments into executable tests.Source: §8 · p. 10
  • Automated reviews often highlight different code aspects from human reviews, suggesting that the two are complementary.The paper identifies integrating tools into human-centric workflows as a future research direction.Source: §8 · p. 10
  • The paper identifies improving tools' ability to capture relevant codebase context as a future research direction.Source: §8 · p. 10

Disclaimer

The authors state that the views and conclusions are theirs alone and do not represent SonarSource’s official policies or endorsements.

Disclaimer

  • The authors state that the paper's findings are independent and are not an evaluation of SonarSource product quality.Source: Disclaimer · p. 11

Related papers

Loading 2603.23448v3…