PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects

Hanyang Guo, Xunjin Zheng, Zihan Liao, Hang Yu, Peng DI, Ziyin Zhang, Hong-Ning Dai

arXiv:2509.14856v3cs.SE

TL;DR

Existing code-review benchmarks assess isolated subtasks with limited context and narrow metrics, leaving repository-level review evaluation underrepresented. CodeFuse-CR-Bench combines rich repository-level instances with rule-based and model-based evaluation. Its study establishes baselines for state-of-the-art LLMs on comprehensive CR.

  • Problem

    Existing CR research fragments the review process, omits PR, issue, and repository context, and often relies on metrics that do not assess substantive review quality or location.

    Source: §1 · p. 1

  • Method

    CodeFuse-CR-Bench provides 601 Python review instances from 70 projects with multifaceted context, evaluated through rule-based checks and model-based judgments.

    Source: §1 · pp. 2–3, §4.1 · p. 9, §4.2 · p. 11

  • Results

    The empirical study establishes baselines and reports state-of-the-art LLM capabilities and limitations on comprehensive CR under real-world task complexity.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper presents repository-level, comprehensive CR evaluation as a more realistic setting for studying automated code review.

    Source: §1 · p. 2, §7 · p. 18

    Because CodeFuse-CR-Bench is based on Python, it cannot evaluate comprehensive CR in other programming languages such as Java or C++.

    Source: §6 · p. 18

Abstract

from arXiv · show

Automated code review (CR) is a key application for Large Language Models (LLMs), but progress is hampered by a "reality gap": existing benchmarks evaluate models on isolated sub-tasks using simplified, context-poor data. This fails to reflect the holistic context-rich nature of real-world CR. To bridge this gap, we introduce CodeFuse-CR-Bench, the first comprehensiveness-aware benchmark for repository-level CR evaluation. CodeFuse-CR-Bench comprises 601 high-quality instances from 70 Python projects covering nine Pull-Request (PR) problem domains, where each instance provides rich, multi-faceted context including the associated issue, PR details, and repository state, enabling end-to-end evaluation. Beyond superficial metrics, we also propose a novel evaluation framework that combines rule-based checks for location and syntax with model-based judgments of review quality. We present the first large-scale assessment of state-of-the-art LLMs on this comprehensive CR task. Our results establish crucial baselines and reveal that (1) no single LLM dominates all aspects of CR; (2) Gemini 2.5 Pro achieves the highest comprehensive performance; and (3) different LLMs exhibit varying robustness to redundant context. These findings highlight the necessity of holistic, multi-dimensional evaluation and provide actionable insights for advancing truly intelligent yet practical CR assistants.

1 Introduction

The introduction identifies a comprehensiveness gap: existing code-review evaluation fragments tasks, limits context, and uses narrow metrics. CodeFuse-CR-Bench and its evaluation framework target repository-level, end-to-end review assessment.

1 Introduction

  • Existing CR benchmarks often omit repository context and split review into subtasks, while common text-similarity metrics miss substantive quality and comment placement.Table 1 compares benchmark context coverage; the authors identify task fragmentation, context poverty, and evaluation narrowness as limitations.Source: §1 · pp. 1–2, Table 1 · p. 2
  • CodeFuse-CR-Bench provides 601 instances from 70 real-world Python projects, with PR, issue, CR, and repository-level context for each review task.The table caption describes context including PR and issue details, commit history, and the complete patch.Source: §1 · p. 2, Table 1 · p. 2
  • The evaluation framework combines rule-based checks with model-based judgments to assess review location, similarity, quality, relevance, and correctness.The rule-based component extracts paths, line numbers, and comments; model-based evaluators include a reward model and LLM-as-a-judge.Source: §1 · pp. 2–3
  • The authors conduct a large-scale assessment of state-of-the-art LLMs on comprehensive CR, establishing baselines and examining their capabilities and limitations.The study is presented as an empirical evaluation using the benchmark and framework.Source: §1 · p. 3

2 Related Work

Prior work studies automated code-review subtasks and related software-engineering tasks, while issue-solving benchmarks provide upstream context. The authors position their benchmark as an evaluation of comprehensive CR rather than isolated tasks.

2.1 CR-related Tasks

  • Prior CR work automates discrete stages, including review-comment generation and code revision, rather than evaluating the entire review workflow.Trans-Review and T5-Review automate code revision before or after review, while AUGER generates review comments (Li et al., 2022a).Source: §2.1 · p. 3
  • Issue-solving benchmarks assess upstream tasks such as resolving repository issues, question answering, fault localization, and code editing.SWE-Bench evaluates issue resolution in Python repositories; FAUN-Eval covers QA, fault localization, and code editing (Hu et al., 2024).Source: §2.1 · pp. 3–4
  • The authors identify existing approaches and benchmarks as focused on CR subtasks, and introduce a comprehensiveness-aware benchmark and metrics for full CR tasks.They also report an empirical study of state-of-the-art LLMs using the benchmark.Source: §2.1 · p. 4

2.2 LLM for Software Engineering Tasks

  • Related software-engineering studies examine LLMs in code generation, HRI code generation, vulnerability repair, and summary evaluation, alongside persistent task-specific challenges.The cited discussion includes challenges in spatial reasoning, constraint adherence, and summarization in logic programming languages.Source: §2.2 · p. 4

3 CodeFuse-CR-Bench Benchmark

CodeFuse-CR-Bench is a repository-level Python code-review benchmark built from manually selected real-world tasks. Its 22-field instances combine task, PR, review, and repository context, while its construction process filters and labels candidate tasks before manual annotation.

3.1 Benchmark Overview

  • Each CR instance has 22 structured fields spanning basic, PR-related, CR-related, and repository-level context.The fields support review of the change intent, review feedback, and broader project context.Source: §3.1 · pp. 4–5

3.2 Benchmark Construction Pipeline

  • The benchmark construction pipeline has five stages: repository selection, PR filtering, PR classification, feature labeling, and manual selection and annotation.These stages are described as the construction process for the benchmark.Source: §3.2 · p. 5, Figure 3 · p. 5

3.2.1 Repository Selection; 3.2.2 PR Crawling & Attribute-based Filtering

  • Repository selection retains active, mature Python projects, while PR filtering requires merged PRs with at least one closing-issue reference.The repository criteria include at least 1,500 PRs and maintenance activity in the preceding year.Source: §3.2.1 · pp. 5–6, §3.2.2 · p. 6

3.2.3 PR Classification

  • Weighted heuristic rules rank commits within each PR, after which comments tied to changed lines or resolved, outdated, or collapsed threads identify high-quality review instances.High-, medium-, and low-impact rules receive weights of 3.0, 2.0, and 1.0, respectively.Source: §3.2.3 · pp. 6–7

3.2.4 Feature Labeling

  • Each task is labeled for problem domain, implementation difficulty, and review effort, using existing repository labels when available or Qwen3-235B-A22B judgments otherwise.The resulting filtered pool contains 40,124 task instances.Source: §3.2.4 · p. 7

3.2.5 Manual Selection & Annotation

  • Manual screening removes noisy or unsuitable instances, then two experienced Python developers assess quality and select 601 instances from 70 projects.The screening excludes, among other cases, issue statements shorter than 40 words and instances with specified external references or noise comments.Source: §3.2.5 · p. 8

3.3.1 Comprehensiveness-aware Benchmark

  • CodeFuse-CR-Bench combines basic, repository-level, PR-related, and CR-related information to support repository-level review.The authors position this comprehensiveness-aware design as a closer simulation of real-world code review.Source: §3.3.1 · pp. 8–9

3.3.2 Strict Filter Process

  • The strict collection process targets mature, active projects, merged PRs with traceable development intent, and representative commits reflecting review collaboration and focused changes.Commit-selection heuristics also exclude merge commits that do not reflect development intent.Source: §3.3.2 · p. 9

4 Evaluation Metric Design

The evaluation framework combines model-based judgments of review quality with rule-based checks of formal correctness and similarity. It is designed to assess comprehensive code review beyond method-level semantic metrics.

4.1 Model-base Evaluation

  • Model-based evaluation uses LLMs to assess review quality, simulating human judgments of semantic quality and usefulness.The framework combines reward-model and LLM-as-a-judge scores.Source: §4.1 · p. 9, §4.1.2 · p. 11

4.1.1 Reward Model

  • The reward model is trained on GitHub review data and uses Qwen3-8B to produce a relevance score for a review given its query and patch.The data contains 174,661 positive and 114,458 negative examples; repositories used to select benchmark tasks are filtered out.Source: §4.1.1 · p. 10
  • Reward-model training combines a pairwise-ranking objective with auxiliary binary cross-entropy classification of review relevance.Positive and negative reviews are sampled for the same query and patch, with code context and diff hunks.Source: §4.1.1 · p. 10

4.1.2 LLM-as-a-Judge

  • LLM-as-a-judge scores reviews across Functionality, Quality, Style, and Documentation, averaging scores over five dimensions for each perspective.The dimensions are Correctness, Relevance, Clarity, Consistency, and Language.Source: §4.1.2 · p. 11, Figure 5 · p. 11

4.2 Rule-base Evaluation

  • Rule-based evaluation extracts file paths, line numbers, and comments, then applies location, semantic-similarity, and defect-match rules.It targets formal correctness and superficial similarity.Source: §4.2 · p. 11

4.2.1 Location Similarity

  • Location similarity weights exact file-path matching at 70%, line-number accuracy at 15%, and diff-hunk similarity at 15%.Same-hunk predictions receive full diff-hunk credit; nearby offsets can also receive partial credit.Source: §4.2.1 · pp. 11–12

4.2.2 Semantics Similarity

  • Review-comment semantic similarity is measured with BLEU-4, which compares n-gram overlap between generated and reference comments.The paper uses BLEU-4 as its semantic-similarity measure.Source: §4.2.2 · p. 12

4.2.3 Defects Match

  • Defect matching selects each prediction’s highest location-and-semantics match to a ground-truth defect, then uses matched defects to calculate precision, recall, and F1.The overall rule-based score combines defect-match, location-similarity, and semantics-similarity scores.Source: §4.2.3 · pp. 12–13

5 Experiment Design

The experiments address overall comprehensive-review performance, performance across PR types, the contribution of context, and reward-model performance. The benchmark contains 601 Python review instances from 70 projects, with mostly medium-difficulty tasks and moderate review effort.

5 Experiment Design

  • The experiments ask how LLMs perform overall and across PR types, how context contributes, and how the reward model performs.These are the four research questions guiding the experiments.Source: §5 · p. 13

5.1 Benchmark Distribution

  • The benchmark has 601 instances from 70 Python projects; bug fixes are the largest domain, and most tasks have medium implementation difficulty and review effort of 3.The instances span dates from 2016-11-06 to 2025-07-07; the top 17 projects account for 55.41% of instances.Source: §5.1 · p. 13

5.2 Studied LLMs

The study evaluates both open- and closed-source LLMs and examines retrieval strategies for supplying repository context. Because entire repositories cannot fit in model inputs, the authors retrieve relevant files using BM25 or an oracle-based approach.

5.2 Studied LLMs

  • The model set includes three open-source and four closed-source LLMs selected as mainstream models used in software-engineering studies.The evaluated models are DeepSeek-v3.1, Kimi-K2-0905-preview, Qwen3-235B-A22B, Claude-Sonnet-4-20250514, Gemini 2.5 Pro, GPT-4o, and GPT-5.Source: §5.2 · p. 14

5.3 Studied Context Acquisition Strategies

  • BM25 retrieves code files similar to the problem statement, with top-1, top-3, and top-5 retrieval sizes compared in RQ3.The paper motivates retrieval because full repositories are too large to provide as context.Source: §5.3 · p. 14
  • The oracle-based strategy retrieves changed files from both the target commit and related commits in the same PR as a broad context baseline.The selection combines diffs from the base commit to the target and merged commits.Source: §5.3 · p. 14

5.4 Experimental Setting

Gemini 2.5 Pro achieves the highest comprehensive code-review performance, while model strengths differ across evaluation dimensions. Experiments average three runs and use prompts containing issue, code-context, and patch information.

5.4 Experimental Setting

  • Results are averaged over three runs, and inputs exceeding a model’s context window after file removal receive a score of zero.The prompt includes the problem statement, code context, and patch to review, with the requested review format also specified.Source: §5.4 · p. 14, Figure 8 · p. 15

5.5 RQ1: What is the LLMs’ performance of comprehensive CR in CodeFuse-CR-Bench benchmark?

  • GPT-5 scores best on model-based evaluation but has a lower comprehensive score than Gemini, while Claude-Sonnet excels in formal correctness.GPT-5’s comprehensive score is 41.96.Source: §5.5 · pp. 15–16
  • Gemini 2.5 Pro achieves the highest comprehensive CR performance by balancing model-based and rule-based scores.Table 5 presents LLM performance scores under the benchmark.Source: Table 5 · p. 15, §5.5 · p. 16

5.6 RQ2: What is the LLM’s performance of comprehensive CR in different PR types?

Gemini 2.5 Pro leads comprehensive code review across all nine PR problem domains. Its results suggest strong performance across varied code changes and effective use of domain-specific context.

5.6 RQ2: What is the LLM’s performance of comprehensive CR in different PR types?

  • Gemini 2.5 Pro achieves the highest comprehensive score in all nine PR problem domains, consistently outperforming every other evaluated LLM.The authors report that this pattern demonstrates superior generalization and context utilization across diverse CR tasks.Source: §5.6 · p. 16
  • Its margin over the second-best model ranges from 5.93% to 28.17%, with scores of 58.86 in DU, 55.83 in TC, and 53.35 in DE.The authors identify DU, TC, and DE as particularly strong domains.Source: §5.6 · p. 16
  • The authors attribute this across-domain performance to generalization and effective use of domain-specific context, including test requirements, security constraints, and formatting rules.They characterize the model as robust and adaptable to heterogeneous code review practices.Source: §5.6 · p. 16

5.7 RQ3: How does different contextual information contribute to CR performance?

Gemini 2.5 Pro leads across the evaluated context configurations, including oracle-based and BM25 retrieval settings. Its top-1 retrieval score nearly matches its oracle-based score, and its performance varies little across retrieval depths.

5.7 RQ3: How does different contextual information contribute to CR performance?

  • Gemini 2.5 Pro achieves the highest comprehensive score across all oracle-based and BM25-based context configurations.The comparison is presented in Table 7, covering comprehensive scores under different context settings.Source: §5.7 · p. 16, Table 7 · p. 17
  • 52.24 under BM25 top-1 is 0.13 below Gemini 2.5 Pro’s oracle-based score of 52.37.The authors describe the top-1 result as nearly on par with the oracle-based setting.Source: §5.7 · p. 16, Table 7 · p. 17
  • Gemini’s scores are 52.24, 51.45, and 51.63 for BM25 top-1, top-3, and top-5, respectively, while GPT-5 and GPT-4o fluctuate more with context size.The authors characterize Gemini’s retrieval-depth performance as stable and contrast it with other models’ greater sensitivity to context size and redundant information.Source: §5.7 · pp. 16–17, Table 7 · p. 17

5.8 RQ4: What is the performance of the reward model?

The reward model performs well on the binary review-quality classification task and exceeds the tested general-purpose LLMs. Its performance remains similar when context information is removed.

5.8 RQ4: What is the performance of the reward model?

  • The reward model achieves 75.03% accuracy and an 80.64% F1 score in classifying code reviews.The authors report these results as evidence of its ability to distinguish high-quality from low-quality reviews.Source: §5.8 · pp. 17–18
  • Without context information, the reward model achieves 74.30% accuracy and 79.57% F1, compared with 75.03% accuracy and 80.64% F1 with context.The authors state that review context, location, and patch still provide substantial signal for quality assessment when context is removed.Source: §5.8 · p. 17
  • The reward model outperforms Kimi-K2-0711-preview and Gemini 2.5 Pro on this binary classification task, which the authors use to support its suitability as an evaluator.Kimi achieves 43.68% accuracy and 49.58% F1; Gemini achieves 51.30% accuracy and 61.36% F1.Source: §5.8 · p. 17

6 Threats to Validity

The authors identify threats from prompt selection and stochastic LLM outputs, as well as limits to generalizing beyond the benchmark’s language, problem domains, and evaluated models.

6 Threats to Validity

  • The prompt results may depend on the selected phrasing, since the authors chose the best-performing format among several tried but did not cover all available formats.They note that the selected prompt may not be optimal.Source: §6 · p. 18
  • LLM output randomness can affect results; the authors mitigate this by averaging three repeated experiments.The passage identifies hyperparameters such as temperature as a source of variation.Source: §6 · p. 18
  • The Python-based benchmark cannot evaluate comprehensive code review in other languages such as Java or C++, and its nine problem domains may not cover every CR type.The authors plan to expand language and domain coverage.Source: §6 · p. 18
  • Limited resources prevented experiments on all available LLMs, which may slightly affect how representative the results are of actual development scenarios.Source: §6 · p. 18

7 Conclusion

The conclusion presents CodeFuse-CR-Bench as a comprehensiveness-aware benchmark intended to address the gap between existing evaluation and real-world code review. It pairs the benchmark with comprehensive evaluation and an empirical study of state-of-the-art LLMs.

7 Conclusion

  • CodeFuse-CR-Bench is designed to address the lack of comprehensiveness in benchmarks and approaches detached from real code-review scenarios.Source: §7 · p. 18
  • The paper pairs the benchmark with an evaluation metric intended to enable comprehensive code-review evaluation.Source: §7 · p. 18
  • The authors conduct an empirical study exploring state-of-the-art LLM performance in real-world code review.Source: §7 · p. 18
  • The authors state that the benchmark encourages automatic code-review solutions with immediate applicability in open-source software development.Source: §7 · p. 18

Related papers

Loading 2509.14856v3…