PaperlayerView exact revision on arXiv

Source-linked AI summary

From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng

arXiv:2608.27442v1cs.SEcs.AIcs.CL

TL;DR

Existing code-review benchmarks largely assess single-round feedback, missing the multi-round evolution and tracking of defects in realistic reviews. MCR-Bench evaluates these capabilities across 2,269 multilingual tasks, finding that mainstream LLMs perform modestly and degrade as review rounds increase.

  • Problem

    Existing code-review benchmarks largely assess isolated or single-round feedback, lacking evaluation of defect evolution and lifecycle tracking across successive review rounds.

  • Method

    MCR-Bench is a multilingual benchmark of 2,269 real-world multi-round review tasks annotated with fine-grained defects and cross-round lifecycle states.

  • Results

    Mainstream LLMs show modest defect-identification and lifecycle-state-tracking performance, with accuracy degrading as the number of review rounds increases.

  • Takeaways & Limitations

    MCR-Bench provides a more realistic foundation for evaluating and developing multi-round, state-aware LLM-based code-review systems.

  • Takeaways & Limitations

    Because proprietary enterprise repositories are generally inaccessible, MCR-Bench may not fully represent all aspects of enterprise-scale development.

Abstract

from arXiv · show

In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.

1 Introduction

Code review is essential for software quality but remains labor-intensive, while existing LLM-oriented benchmarks largely simplify it into static, single-round decisions. MCR-Bench addresses this gap with a state-aware benchmark for realistic multi-round review and evaluates LLM defect identification and lifecycle tracking.

  • Code review supports functional correctness and long-term maintainability, but its labor-intensive nature has motivated efforts to automate it with LLMs.
  • Existing benchmarks mostly reduce code review to single-round, static decisions, overlooking iterative developer–reviewer interactions and continued defect tracking.
  • Nearly half of code changes involve multiple review rounds, with review time increasing from 0.33 days to 5.3 days and 31.3 days as rounds increase.The reported durations correspond respectively to single-round, 2–6 rounds, and >6 rounds.
  • MCR-Bench comprises 2,269 real review tasks across Python, Java, JavaScript, TypeScript, and C#, enabling state-aware multi-round code-review evaluation.It is presented as the first benchmark dedicated to multi-round code review.
  • Its multi-stage construction pipeline performs state-aware defect annotation, cross-round extraction and lifecycle tracking, followed by manual cross-validation for annotation quality.The pipeline follows a “local detection first, global tracking later” strategy.
  • Experiments show that current LLMs have modest performance on MCR-Bench, particularly in defect identification, motivating analysis of their capability boundaries.The evaluation focuses on defect identification, lifecycle state tracking, and common error patterns.

2 Related Work

Existing code review benchmarks have progressed from localized code changes to complete pull-request contexts, but most still model review as a single-round static task. This paradigm omits the iterative review–revision–revalidation cycles and defect-state tracking central to real-world review.

  • Localized Code-Change Benchmarks: Earlier benchmarks evaluate localized code changes at the function/method, diff-hunk, or commit/change level.Examples include Trans-Review, AutoTransform, and T5-Review at the function/method level.
  • Pull-Request-Level Benchmarks: Recent benchmarks evaluate code review at the pull-request level using complete PR contexts rather than isolated code changes.SWR-Bench is constructed from manually verified GitHub pull requests and provides full project context with structured ground-truth issues.
  • Limitations of Static Review: Most existing benchmarks remain single-round and static, failing to capture iterative review, revision, and revalidation cycles in real-world code review.This limitation obscures multi-round challenges such as tracking defect states across Open, Resolved, and Reopened transitions.

3 MCR-Bench Construction

Section 3 presents MCR-Bench as a multilingual, multi-round code review benchmark built from filtered real-world repositories and structured around evolving review context and defect lifecycle annotations. Its construction combines staged data curation, automated state-aware annotation, and consensus-based quality control.

  • Benchmark Scope: MCR-Bench contains 2,269 structured review tasks spanning five widely used GitHub languages.The selected languages are Python, Java, JavaScript, TypeScript, and C#.
  • Task Representation: Each task combines PR-related information with annotated ground truth for evaluating multi-round review interactions.Inputs include task instructions, static PR information, current-round code diffs, and accumulated review discussions and interactions.
  • Ground-Truth Annotation: Ground truth is organized into Defect Cards that specify each defect and record its lifecycle state at the current review round.Defect specifications include natural-language descriptions and precise locations, while dynamic status captures states such as New and Open.
  • Construction Pipeline: The construction pipeline comprises language and repository selection, pull request collection, LLM-based state-aware annotation, and manual cross-validation.Repository filters target mature, active, collaborative, PR-centric, permissively licensed, and non-forked projects.
  • State-Aware Annotation: Annotation uses a divide-and-conquer process that detects candidate defects within individual rounds before consolidating them across rounds and tracking their lifecycles.This design reduces the difficulty of processing long, iterative review histories in a single pass.
  • Quality Control: Agreement reached 0.87, with disagreements escalated to a third annotator and resolved through joint examination until consensus.Only tasks passing consensus-based verification were accepted into the benchmark.

4 MCR-Bench

MCR-Bench is a multi-round, state-aware benchmark containing 2,269 manually validated tasks across five programming languages, with structured defect annotations and rich contextual coverage. Its design captures realistic interaction trajectories, defect characteristics, and severity levels for detailed code-review analysis.

  • Dataset composition: MCR-Bench contains 2,269 manually validated task instances spanning five programming languages with relatively even representation.Java contributes 556 instances (24.50%), followed by C# with 454 (20.01%), TypeScript with 440 (19.39%), Python with 410 (18.07%), and JavaScript with 409 (18.03%).
  • Defect statistics: Tasks contain an average of 2.37 defects, a median of 2, and up to 13 defects, with most containing between one and four.This moderate defect density is intended to support recall evaluation without excessive contextual complexity obscuring review signals.
  • Defect taxonomy and annotations: The benchmark organizes defects into Functional and Evolvability groups using a fine-grained taxonomy extended with testing, security, and privacy categories.Each defect is structured by category, severity, and round-specific lifecycle state, enabling analysis across defect characteristics and review rounds.
  • Interaction rounds: Every task includes at least two review rounds, with three-round tasks the largest group at 40.86%.Four-round and five-round tasks account for 26.52% and 17.38%, respectively, and most tasks involve at least three interaction rounds.
  • Benchmark design: MCR-Bench is presented as the first benchmark to formulate code review as dynamic, multi-round PR-level interaction rather than a primarily single-round task.It captures iterative review behavior with an average of 3.8 rounds and supports broad evaluation across lower- and higher-severity defects.

5 Experimental Design

The experiments evaluate mainstream LLMs and automated code-review baselines on multi-round code review, examining performance, evolution across rounds, failure causes, and comment quality. A pre-study compares automatic metrics with human judgments and selects LLM-Hit-Judge with GPT-5.2-pro for subsequent evaluation.

  • Research Questions: The study addresses four questions covering general performance, performance evolution across rounds, failure causes, and review-comment quality.The questions examine mainstream LLMs, false positives and false negatives, and comments from pure prompting and ACR baselines.
  • Studied LLMs: The evaluation includes seven representative LLMs spanning commercial and open-source model families.The commercial models are GPT-5.2, Claude-Haiku-4.5, and Gemini-3-Flash; the open-source models are DeepSeek-V3.2, Qwen3-Max, GLM-4.7, and Kimi-k2.
  • Studied ACR Baselines: PR-Agent and Hybrid-Review serve as representative ACR baselines for agent-based and static-analysis-assisted review generation.PR-Agent analyzes pull-request diffs and context, while Hybrid-Review uses static-analysis warnings to guide generated comments.
  • Metric Selection: The pre-study samples 10% of generated comments and uses four senior developers to annotate Human Hit Rate for metric comparison.The annotators each have more than five years of programming experience, and automatic metrics are assessed against these judgments using Quadratic Weighted Kappa.
  • Metric Selection: ROUGE-L and BLEU-4 obtain QWK scores of 0.21 and 0.27, whereas LLM-based metrics achieve substantially higher agreement with human judgments.The reported LLM-based metrics reach a peak QWK value of 0.73, motivating the adoption of LLM-Hit-Judge with GPT-5.2-pro.

6 Experimental Results

The experimental results report performance across programming languages and overall results using Precision, Recall, and F1.

  • Table 5 presents performance across programming languages and overall results for Precision, Recall, and F1.

6.1 General Performance (RQ1)

Mainstream LLMs show limited performance on multi-round code review, both in defect identification and lifecycle-state tracking. Performance varies by model, language, defect category, and severity, while state-prediction errors reveal weaknesses in cross-round context integration.

  • Defect identification: Mainstream LLMs achieve only modest defect-identification performance, with the best F1 score slightly above 0.55.The results indicate persistent challenges in defect identification and contextual reasoning under multi-round review settings.
  • Defect identification: Claude Haiku 4.5 and GPT-5.2 generally perform best, whereas Qwen3-Max and Kimi-K2 make conservative predictions with low recall.Python and JavaScript generally yield higher F1 scores, while TypeScript and C# produce pronounced recall drops for most models.
  • Defect identification: PR-level baselines achieve generally low F1 scores and consistently low recall, indicating difficulty capturing defects whose evidence evolves across discussions and revisions.PR-Agent generally achieves higher overall F1 than Hybrid-Review.
  • Lifecycle state prediction: Claude Haiku 4.5 reaches lifecycle-state prediction accuracy close to 80%, while GPT-5.2 and DeepSeek V3.2 reach around 70%.These accuracies are calculated only on correctly identified defects; Python and Java generally outperform JavaScript, TypeScript, and C#.
  • Defect sensitivity: Defect-identification hit rates vary substantially across categories, with E.1.2, F.3, and F.4 near or above 0.50 while E.2 and E.3.1 remain lower.Clearer semantic cues make some categories easier to recognize, whereas implicit logic and lower-severity issues are easier to overlook.
  • Lifecycle state prediction errors: The largest lifecycle-state error is predicting Resolved as New (38.29%), followed by Open as New (22.06%) and Resolved as Open (18.20%).These errors reflect insufficient use of historical context and difficulty distinguishing partially addressed defects from fully resolved ones; Reopened errors occur less often.

6.2 Performance Evolution across Rounds (RQ2)

Across review rounds R2–R10, most LLMs experience performance degradation as interaction depth and contextual information increase, making defect identification and lifecycle-state tracking more difficult. The degradation differs across models, with GPT-5.2 showing stronger stability in deeper interactions than several other evaluated models.

  • Performance Evolution across Rounds (RQ2): The analysis begins at Round 2 because Round 1 lacks the historical antecedents needed to observe defect evolution and state transitions.The reported performance analysis therefore covers later multi-round dynamics rather than the initial phase.
  • Performance Evolution across Rounds (RQ2): Performance generally degrades as review rounds increase from R2 to R10, indicating that longer interactions make defect identification and state tracking more difficult.Most models remain relatively stable in earlier rounds such as R2–R4 before degrading as contextual information accumulates.
  • Performance Evolution across Rounds (RQ2): Model robustness varies substantially: Claude Haiku 4.5 performs strongly early but declines gradually, with a pronounced drop at R10.DeepSeek V3.2, Gemini 3 Flash, and GLM-4.7 show similar patterns, with relatively low F1 scores in later rounds.
  • Performance Evolution across Rounds (RQ2): GPT-5.2 demonstrates stronger stability during deeper multi-round interactions than the other described models.The passage contrasts GPT-5.2 with models whose performance remains relatively low in later rounds.

6.3 Root Cause Analysis (RQ3)

The root causes of LLM errors in multi-round code review differ between false positives and false negatives. False positives are dominated by state–temporal misalignment, while false negatives primarily reflect cross-round forgetting and failures in semantic and long-range reasoning.

  • Root Cause Analysis: Together, the error patterns reveal limitations in program semantic modeling, long-range reasoning, and independent verification.False positives represent over-reviewing, whereas false negatives represent defect omission or prematurely stopped tracking.
  • FP Error Analysis: False positives mainly arise from State–Temporal Misalignment (32.5%), the largest and most critical root cause.LLMs may rely on early comments or outdated code after defects are fixed, labeling them as new or reintroducing them.
  • FP Error Analysis: Over-reviewing accounts for 27.8% of false positives, as LLMs mistake optional improvements or advisory suggestions for actual defects.This reflects biased defect judgment and difficulty regulating review strictness when evidence is insufficient.
  • FN Error Analysis: Cross-round Defect Forgetting is the most prevalent false-negative cause (25.1%), causing unresolved defects to be prematurely dropped across successive rounds.Attention shifts toward newly introduced changes or discussions, disrupting continuous tracking of defects identified earlier.
  • FN Error Analysis: False negatives also stem from Long-range Dependency Miss (23.4%) and Semantic Defect Blindness (22.3%), reflecting limits in deeper semantic and system-level reasoning.These failures involve cross-file interactions, system-level effects, and specification-level reasoning beyond surface patterns.

6.4 Comment Quality (RQ4)

Comment quality is evaluated with ClearCRC across relevance, informativeness, and expression, revealing substantial differences across generation settings and backbone models. Pure prompting and PR-Agent generally outperform Hybrid-Review on average, while each setting exhibits distinct quality profiles.

  • Evaluation framework: ClearCRC evaluates generated review comments on Relevance, Informativeness, and Expression, with Avg. denoting the average across these dimensions.Relevance measures alignment with the code change and review issue under discussion.
  • Overall results: Pure LLM prompting and PR-Agent generally achieve higher average comment-quality scores than Hybrid-Review, although the best setting varies by backbone model.Table 12 reports ClearCRC-based quality results across generation settings and backbone models.
  • Dimension-level profiles: Pure LLM prompting often has high Expression but uneven Informativeness, whereas PR-Agent is more balanced for stronger backbones and Hybrid-Review scores lower in most settings.The findings show that fluent comments are not always sufficiently informative, and comment quality complements defect-identification accuracy.

7 Discussion

MCR-Bench enables realistic evaluation of defect evolution across multi-round code review, while the findings motivate lifecycle-aware methods and identify threats to validity. These threats concern model coverage, filtering and annotation, scenario representativeness, and evaluation-metric coverage.

  • Benchmarking Multi-round Dynamics: MCR-Bench supports state-aware evaluation of defect identification and lifecycle tracking across realistic multi-round review scenarios.It moves beyond single-round evaluation by modeling defect evolution across review rounds.
  • Implications for Future Automated Code Review Research: Future automated review methods should account for performance drops, defect forgetting, temporal misalignment, cross-round memory, historical context, and lifecycle-aware state tracking.The discussion highlights these issues as requirements for designing methods around multi-round review.
  • Implications for Future Automated Code Review Research: Pipeline complexity or fluent comments do not necessarily improve defect detection, motivating joint consideration of defect identification, state tracking, and comment quality.This implication follows from the reported ACR baseline and ClearCRC results.
  • Internal Threats: Internal validity threats include limited LLM selection, possible exclusion of difficult instances through consistency filtering, and dependence on human annotation.The study uses representative closed-source and open-weight models and filtering to improve annotation reliability, but results may not generalize to all current or future LLMs.
  • External Threats: MCR-Bench may not fully represent enterprise-scale development or cover all evaluation metrics because proprietary repositories are generally inaccessible.The benchmark instead emphasizes large, actively maintained repositories reflecting industrial coding practices and collaboration workflows.

8 Conclusion

MCR-Bench introduces a realistic benchmark for evaluating LLMs in multi-round code review by modeling cross-round defect identification and lifecycle state transitions from practical review workflows.

  • 8 Conclusion: MCR-Bench evaluates LLM abilities in defect discovery and tracking under continuous review settings, addressing the limitations of prior single-round studies.The benchmark models cross-round defect identification and defect lifecycle state transitions from practical review workflows.
Loading 2608.27442v1…