Source-linked AI summary
SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation
Zhengran Zeng, Ruikai Shi, Keke Han, Yixin Li, Kaicheng Sun, Yidong Wang, Zhuohao Yu, Rui Xie, Wei Ye, Shikun Zhang
TL;DR
Existing ACR benchmarks often use isolated code units, limited context, and evaluation metrics that do not directly establish whether important issues were identified. SWR-Bench provides manually verified PRs with full project context and an objective ground-truth-based evaluation, and its results show that current systems underperform while multi-review aggregation improves issue detection.
Problem
Many existing benchmarks evaluate isolated changes with limited project context and metrics that may not reveal whether code-review comments identify crucial issues.
Source: §1 · p. 2Method
SWR-Bench uses 1,000 manually verified GitHub PRs with full project context and an LLM evaluator that checks generated reviews against structured ground-truth issues.
Source: §1 · p. 3Results
Current ACR systems underperform and detect functional errors better than non-functional issues, while multi-review aggregation increases issue-detection F1 by up to 43.67%.
Source: §1 · p. 3, §7 · p. 20Takeaways & Limitations
SWR-Bench offers a more realistic platform for evaluating ACR and provides findings and an aggregation strategy relevant to practical ACR research and development.
Source: §7 · p. 20The evaluation’s external validity is bounded by the selected LLMs, ACR tools, and project quality; the study used 12 popular, well-maintained open-source GitHub projects.
Source: §6 · p. 20
Abstract
from arXiv · showhide
Automated Code Review (ACR) is crucial for software quality, yet existing benchmarks often fail to reflect real-world complexities, hindering the evaluation of modern Large Language Models (LLMs). Current benchmarks frequently focus on fine-grained code units, lack complete project context, and use inadequate evaluation metrics. To address these limitations, we introduce SWRBench , a new benchmark comprising 1000 manually verified Pull Requests (PRs) from GitHub, offering PR-centric review with full project context. SWRBench employs an objective LLM-based evaluation method that aligns strongly with human judgment (~90 agreement) by verifying if issues from a structured ground truth are covered in generated reviews. Our systematic evaluation of mainstream ACR tools and LLMs on SWRBench reveals that current systems underperform, and ACR tools are more adept at detecting functional errors. Subsequently, we propose and validate a simple multi-review aggregation strategy that significantly boosts ACR performance, increasing F1 scores by up to 43.67%. Our contributions include the SWRBench benchmark, its objective evaluation method, a comprehensive study of current ACR capabilities, and an effective enhancement approach, offering valuable insights for advancing ACR research.
1 Introduction
Existing code-review benchmarks often evaluate isolated changes with limited context and metrics that may not reflect whether important issues were identified. SWR-Bench addresses this with PR-level reviews, full project context, and an objective evaluation method, then uses the benchmark to assess and improve current systems.
1 Introduction
- Existing benchmarks often assess isolated code changes with limited context and rely on text similarity or LLM ratings that may not establish whether crucial issues were identified.Some studies use human annotation, but the paper notes that this approach is expensive and difficult to scale.Source: §1 · pp. 2–3
- SWR-Bench contains 1,000 manually verified GitHub PRs with full project context and evaluates reviews by checking whether they cover issues in a structured ground truth.Its PR-centric setup targets end-to-end review, including identifying areas for review, pinpointing issues, and generating meaningful comments.Source: §1 · p. 3
- Current ACR systems underperform, detect functional errors better than non-functional issues, and improve with multi-review aggregation, which raises issue-detection F1 by up to 43.67%.The proposed strategy synthesizes feedback from multiple review sources by filtering valid points and discarding erroneous suggestions.Source: §1 · p. 3
- The proposed objective LLM-based evaluation method agrees with human judgment at approximately 90%, and the study finds reasoning-focused LLMs perform better at code review.The paper also reports that existing tools are more adept at detecting functional errors, such as bugs, than non-functional issues such as outdated documentation.Source: §1 · p. 3
2 Background and Related Work
The paper frames code review as peer scrutiny of PR changes that yields actionable change-actions, including both functional and evolutionary improvements. It traces ACR from static analysis through LLM-based tools, then identifies gaps in public benchmarks’ scope, context, and evaluation.
2.1 Modern Code Review
- A change-action is an actionable directive that locates an issue, explains it, and proposes a solution; it may span multiple locations and address non-functional improvements as well as defects.The taxonomy distinguishes evolutionary changes from functional changes and further divides them into 11 types.Source: §2.1 · p. 4
- Modern code review proceeds from reviewer scrutiny of a PR to a report of change-actions, which developers verify and address by updating the PR.The paper focuses on generating review comments as a prerequisite for end-to-end review automation and downstream code refinement.Source: §2.1 · p. 4
2.2 Automatic Code Review
- ACR has progressed from rule-based and static-analysis tools through machine-learning methods to LLM- and agent-based approaches, while its real-world effectiveness remains an open question.The paper notes that earlier approaches can be rigid or struggle with deep intent and complex context.Source: §2.2 · pp. 4–5
2.3 Code Review Benchmarks
- Public code-review benchmarks have used method-level triplets or diff hunks, often with BLEU, Exact-Match, or simple LLM ratings rather than PR-level review with complete project context.Some newer benchmarks add related source code but still lack the complete project environment.Source: §2.3 · pp. 5–6
- Public benchmarks lack PR-level scope, comprehensive context, and reliable evaluation, while proprietary benchmarks with manual assessment are difficult to scale and reproduce.These limitations motivate a publicly available, context-rich, scalably evaluable benchmark.Source: §2.3 · p. 6
3 SWR-Bench
SWR-Bench is constructed from real-world GitHub PRs and paired with ground-truth-referenced LLM evaluation. It combines filtering, sampling, and manual verification to create a balanced, deliberately challenging benchmark.
Step 1: Source Data Collection and Initial Filtering
- SWR-Bench draws historical PRs from 12 open-source Python projects and collects their metadata, diffs, discussions, and statuses.PRs without review comments, with more than 10 commits, or involving rebases were excluded in initial filtering.Source: Step 1: Source Data Collection and… · pp. 6–7
Step 2: LLM-based change-actions Verification and Classification
- Gemini-2.5-Pro extracted change-actions from PR timelines, and only PRs with consistent results across three requests advanced.The consistent PRs were categorized as Change-PRs or Clean-PRs.Source: Step 1: Source Data Collection and… · p. 7
Step 3: Quality Enhancement through Filtering and Sampling
- Clean-PRs were stratified-sampled to match Change-PRs on key statistics, reducing the usefulness of coarse metrics as shortcuts.This intentionally departs from the natural data distribution to make the benchmark more challenging.Source: Step 3: Quality Enhancement through… · p. 8
Step 4: Manual Verification and Refinement
- Five experienced graduate students independently annotated each PR in pairs, resolved disagreements through discussion, and excluded PRs with only trivial actions.The final labels classify change-actions into 11 types; trivial actions are formatting- or documentation-only changes.Source: Step 4: Manual Verification and… · p. 8
Dataset Statistics
- 500 Change-PRs and 500 Clean-PRs make up the final SWR-Bench dataset.The balanced composition is intended to support evaluation of false positive rates.Source: Dataset Statistics · p. 8
- Each instance records PR metadata, pre-first-review ground-truth actions for Change-PRs, and a codebase checkpoint for reconstructing the relevant project state.The checkpoint supports agent-based tools that require an interactive environment.Source: Dataset Statistics · p. 8
- Change-PRs and Clean-PRs have similar average numbers of modified files and lines of code.After filtering out PRs with only trivial actions, functional changes account for 31.8% of SWR-Bench, compared with less than 15% in Raw-PRs.Source: Dataset Statistics · pp. 8–9
3.2 Evaluation Methodology
- The evaluation LLM extracts and type-labels predicted change-actions, then matches them semantically against ground-truth actions to calculate precision, recall, and F1.The objective matching approach is validated through manual verification.Source: §3.2 · pp. 9–10
4 Evaluation and Study
The section introduces a research question about the reliability of the proposed evaluation methodology.
4.1 Research Questions
- RQ1 asks how reliable the proposed evaluation methodology is.Source: §4.1 · p. 10
• RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench?
This section evaluates mainstream ACR tools and LLMs on SWR-Bench, examining their review performance and the effects of aggregating multiple reviews. It also considers aggregation across model sizes, report counts, cost, and time overhead.
RQ1: How reliable is the proposed evaluation methodology?
- For change-action Hit identification, Cohen’s Kappa ranged from 70.6 to 86.7 across human and LLM pairs; LLM-Hit-Judge aligned with human preferences at Kappa 52.8–62.0.LLM-Hit-Judge agreement was comparable to interhuman agreement (56.3–62.6), versus 31.0–44.9 for LLM-Score-Judge and -35 to -41 for BLEU; Gemini-2.5-Flash was selected for later experiments at about $1.57 per full benchmark evaluation.Source: Figure 6 · p. 11, RQ1: How reliable is the proposed… · pp. 11–12
• RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench?
- PR-Review with Gemini-2.5-Pro achieved F1 19.38%, while PR-Review’s overall-F1 was 18.73%, the highest across tools; high false-positive rates limit practical use.Some tool–model combinations produced over 7 false positives per PR on average, and four ACR techniques had precision below 10%.Source: Table 4 · p. 13, RQ1: How reliable is the proposed… · p. 13
4.2.2 Automated Code Review (ACR) Baselines.
- PR-Review outperformed prompting and agent baselines, while hunk-level Code-Reviewer and Llama-Reviewer scored F1 6.13% and 7.22%, respectively.The authors attribute hunk models’ weakness to their focus on isolated code units rather than holistic pull-request context; they also report that PR-Review’s prompt refinements reduce false positives.Source: RQ1: How reliable is the proposed… · pp. 13–14
RQ1: How reliable is the proposed evaluation methodology?
- For PR-Review, recall fell from 38.35% at N=1 to 8.88% at N≥5 as PRs contained more change-actions, while precision stayed roughly between 29.63% and 44.12%.PR-Review also detected functional changes more effectively: F.2 Logic reached F1 26.20%, whereas the best evolutionary type, E.3.1 Organization, reached 16.45%.Source: RQ1: How reliable is the proposed… · p. 15
- PR-Review’s false positives most often reflected lack of contextual understanding (48%), followed by over-sensitivity to modification (17%), vague feedback (16%), and lack of domain knowledge (13%).The authors connect these errors to difficulties understanding developer intent, code context, and project-specific knowledge.Source: RQ1: How reliable is the proposed… · p. 16
4.2.1 Large Language Models (LLMs).
- Reasoning-enhanced models generally performed better; Qwen-2.5-R1-14B reached F1 15.95%, compared with a maximum F1 of 13.76% for standard Qwen-2.5 models.Qwen-2.5-R1-7B was an exception due to output-formatting issues, and the highest overall-F1 among evaluated PR-Review/LLM combinations was 19%.Source: RQ1: How reliable is the proposed… · pp. 16–17, Table 8 · p. 17
RQ3: How can the performance of automated code review tools be improved?
- Only 36 successfully identified change-actions overlapped across different LLMs, and only 27 overlapped across five runs of the same LLM, indicating unstable detection.This variability motivated Multi-Review, which aggregates independent review reports into one report using an additional LLM call.Source: Figure 7 · p. 17, RQ3: How can the performance of… · p. 18, Figure 8 · p. 18
- With Gemini-2.5-Flash, Self-Agg at n=10 raised F1 to 21.91% (a 43.67% increase) and Recall to 30.44% (a 118.83% increase).The approach aggregates multiple independent reports from the same model or different models; the reported larger recall gain indicates improved identification of true defects, while further precision improvement remains a priority.Source: RQ3: How can the performance of… · p. 18
- F1 improves by 26.13% to 14.67% for Qwen-Chat-7B Self-Agg (n=10) and by 19.25% to 16.41% for Qwen-Chat-32B Self-Agg (n=10).Their aggregated results approach the baseline PR-Review performance of some larger commercial LLMs.Source: RQ3: How can the performance of… · p. 19
- F1 reaches 23.84% with Gemini-2.5-Pro Self-Agg (n=5), a 23.01% improvement over its single-pass baseline.This is the highest reported F1; performance gains diminish beyond n=5 while costs continue to scale linearly.Source: RQ3: How can the performance of… · p. 19, Figure 9 · p. 19
- F1 of 20.48% from Gemini-2.5-Flash Self-Agg (n=5) exceeds the Gemini-2.5-Pro single-pass baseline F1 of 19.38% at lower API cost ($3.68E-03 vs. $5.86E-03).The passage presents this as evidence that multiple runs of a smaller, cheaper model can be more cost-effective than one run of a larger model.Source: RQ3: How can the performance of… · p. 19, Figure 9 · p. 19
- Initial drafts take 38.75 seconds per PR and are parallelizable, while aggregation takes 4.67, 8.76, 12.80, and 24.35 seconds for n=1, 3, 5, and 10, respectively.The passage characterizes the aggregation overhead as low and the approach as practical for real-world deployment.Source: RQ3: How can the performance of… · p. 19
5 Discussion and Implications
The discussion recommends objective fact matching against ground truth and dedicated code-critique reasoning. It also identifies false-positive reduction and aggregation of multiple reports as practical priorities.
5 Discussion and Implications
- Evaluate generated reviews through objective fact matching against predefined ground truth; text similarity and subjective LLM scoring do not reliably reflect semantic quality or human preferences.The discussion recommends verifying whether generated reviews identify the predefined issues.Source: §5 · p. 19
- Code review requires logical deduction, counterfactual reasoning, and cross-file dependency analysis, unlike code generation’s pattern completion.The discussion links this distinction to the underperformance of models optimized primarily for code generation and better results from reasoning-enhanced models.Source: §5 · pp. 19–20
- Many false positives stem from insufficient contextual understanding and missing project-specific conventions, limiting practical adoption by burdening developers with invalid alerts.The discussion suggests integrating historical repository data to learn implicit coding norms.Source: §5 · p. 20
- Because functional changes are detected more reliably than evolutionary ones, the discussion recommends making functional defects mandatory and evolutionary suggestions optional.Source: §5 · p. 20
- Aggregating independent review reports can mitigate missed defects across stochastic LLM runs, and multiple runs of smaller models can match or exceed one run of a larger, more expensive model.Source: §5 · p. 20
6 Threats to Validity
The authors identify implementation and manual-verification accuracy, the selection of tools and projects, and the reliability of the evaluation method as validity threats. They describe code review, representative selections, and human validation as mitigations.
6 Threats to Validity
- Internal validity may be affected by implementation bugs and manual-verification accuracy; the authors report reviewing the code and making it publicly available for independent verification.Source: §6 · p. 20
- External validity depends on the selected LLMs, ACR tools, and project quality; the study selected representative tools and 12 popular, well-maintained open-source GitHub projects.Source: §6 · p. 20
- Construct validity depends on the accuracy and reliability of the objective LLM-based evaluation method, which the authors checked through human validation showing high consistency with expert evaluations.Source: §6 · p. 20
7 Conclusions
The conclusion presents SWR-Bench as a full-context benchmark for realistic PR review and reports underperformance by current LLM-based systems alongside stronger functional-error detection. It also reports improved F1 scores from multi-review aggregation.
7 Conclusions
- SWR-Bench combines 1,000 real-world pull requests, full project context, and objective LLM-based evaluation; the study found current LLM-based ACR systems generally underperform, with better aptitude for functional errors.Source: §7 · p. 20
- The authors proposed and validated a multi-review aggregation strategy that significantly boosts F1 scores.Source: §7 · p. 20
Data Availability
- The replication package, containing source code and experiment scripts, is available at the anonymous repository.Source: Data Availability · p. 20