AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context

Introduction

• Existing benchmarks’ reliance on raw historical PR review comments as ground truth leaves issue coverage incomplete and limits assessment of models’ ability to uncover potential defects.The paper identifies this as a barrier to faithfully characterizing performance in real-world code review scenarios.Source: §1 · p. 1
• 80 senior software engineers reviewed 2,145 comments generated by two ACR systems across six LLMs, supplementing 391 real-world GitHub PR review comments.The annotators each had at least two years of industry experience; the resulting hybrid corpus is intended to improve coverage of latent issues.Source: §1 · p. 2
• AACR-Bench provides complete repository-level dependency information across 10 mainstream programming languages to address context and language-diversity limitations.The paper argues that existing cross-file benchmarks are predominantly single-language, limiting generalizability and potentially introducing language-specific structural biases.Source: §1 · pp. 1–2
• AACR-Bench is presented as the first multilingual, repository-level context-aware benchmark for LLM-enabled ACR, alongside a comprehensive evaluation of mainstream LLMs.The paper says its evaluation reveals new insights into mainstream LLMs’ ACR capabilities.Source: §1 · p. 2

2.1. Automatic Code Review Based on LLM

• Early LLM-based ACR includes T5 and CodeReviewer pretrained on diff–comment pairs, Llama-Reviewer’s parameter-efficient fine-tuning, and a feedback-driven Data Flywheel.Source: §2.1 · p. 2
• Later ACR methods use preference optimization, static analysis for latent defects, and retrieval-augmented generation for project-level semantics.Source: §2.1 · p. 2
• Agent-based ACR extends reasoning through multi-agent collaboration and dialectical debate to produce more comprehensive, objective reviews.Source: §2.1 · p. 2

2.2. Code Review Benchmarks

• Existing benchmarks trade off context and language coverage: CodeReviewer is diff-level, SWR-Bench and CodeFuse-CR-Bench are Python-only, and ContextCRBench is file-level.Source: §2.2 · p. 2
• AACR-Bench is introduced as the first multilingual ACR evaluation dataset with repository-level context, targeting more realistic production evaluation.Source: §2.2 · p. 2

3.1. Overview of AACR-Bench

• AACR-Bench evaluates end-to-end ACR at PR granularity across 200 PRs and 1,505 comments from 50 repositories spanning 10 languages.Methods traverse each PR’s diff hunks; generated comments are matched against the ground truth to calculate metrics such as precision, recall, and F1-score.Source: §3.1 · p. 3, Figure 1 · p. 3

Selection of programming languages and repositories

• Curators selected five repositories per language, filtered PRs for explicit issues and other quality criteria, and sampled 200 PRs across repositories, domains, and change sizes.For each PR, an LLM analyzed the revision with the most inline comments to extract confirmed defects from review threads and reorganize them as augmented review comments.Source: Selection of programming languages and… · p. 4

3.2. Dataset Curation

• The benchmark combines LLM-augmented human reviews with model-generated reviews, using six models and two generation frameworks before semantic deduplication and expert verification.This construction addresses noise and incompleteness in raw GitHub reviews and uses multiple models to mitigate single-model bias.Source: Selection of programming languages and… · p. 4, §3.3 · pp. 4–5

3.3. Comparisions with Existing Datasets

• 1,505 verified review comments yielded a 285% increase in issue coverage, and each comment was annotated for the context scope needed to detect its issue.The comments include 391 augmented from original reviews and 1,114 augmented by LLMs and human experts; 80 senior software engineers verified comments, with disagreements resolved by a six-member core team.Source: Selection of programming languages and… · p. 4, §3.3 · p. 5
• AACR-Bench provides repository-level context across 10 languages, bridging the gap between repository-context benchmarks limited to Python and multilingual datasets limited to narrower context.It includes PR metadata and complete code repositories.Source: §3.3 · p. 4, Table 2 · p. 4

4.1. Evaluation Settings

• The benchmark evaluation compares three open-source and two commercial models using no context, BM25, embedding retrieval, and Claude Code’s agent framework, with precision, recall, and F1.Similarity-based retrieval uses three contexts, while the agent chooses how many to retrieve; experiments use AACR-Bench’s complete dataset.Source: §4.1 · p. 5

4.2. Main Results

• The main-results evaluation treats contextual dependency as the practical paradigm for ACR and uses no-context performance chiefly as a comparison baseline.The experiments are designed to test AACR-Bench’s multi-language support, repository-level context, and broader defect exposure.Source: §4.2 · p. 5, §4 · p. 5

4.2.1. ACR PERFORMANCE BENCHMARKED WITH A MORE SUFFICIENT EXPOSURE TO DEFECTS

• Agent-based review trades higher precision for lower recall, while context retrieval does not improve performance uniformly across models.Claude-4.5-Sonnet in Agent mode reaches 39.90% precision and 10.10% recall, while GPT-5.2 falls to 9.90% precision and 2.99% recall in Agent mode. Claude’s F1 drops from 14.46 with no context to 9.98 with BM25, whereas DeepSeek-V3.2 performs best with BM25 (F1 15.59) and Qwen-480B-Coder with Embedding (F1 14.36).Source: §4.2.1 · pp. 5–6, Table 3 · p. 6

4.2.2. CONTEXT LEVEL-WISE IMPACT ON ACR PERFORMANCE

• Non-agent methods detect fewer issues as the required context expands from diff to file to repository, whereas agents often improve at repository scope.The analysis reports recall only for context-level comparisons; under Agent, DeepSeek-V3.2 rises from 4.28% at diff level to 8.00% at repository level, though the low diff-level score may indicate missed local issues.Source: §4.2.2 · pp. 6–7, Table 4 · p. 7

4.2.3. LANGUANGE-WISE IMPACT ON ACR PERFORMANCE

• ACR performance varies substantially by language, with Claude-4.5-Sonnet’s F1 ranging from 0.247 on Python to 0.081 on TypeScript.The paper attributes this disparity partly to uneven training-data availability and hypothesizes that language structure also matters: GPT-5.2 scores 0.309 on C# versus 0.085 on C, and Qwen-480B-Coder scores 0.268 versus 0.104. The authors associate C’s lower performance with implicit dependencies and pointer, macro, and memory-management features.Source: §4.2.3 · p. 7, Figure 3 · p. 8
• Context effects also vary by language: retrieval or agent frameworks often hurt performance, but Claude-4.5-Sonnet improves on Go, Java, and C.For GPT-5.2, C# falls from 0.309 with no context to 0.095 in Agent mode, while Claude-4.5-Sonnet rises from 0.120 to 0.218 on Go, 0.142 to 0.241 on Java, and 0.106 to 0.189 on C.Source: §4.2.3 · pp. 7–8, Figure 3 · p. 8

5. Error Analysis

• Current models continue to suffer from knowledge errors during code review.Detailed case studies of incorrect model-generated review comments are provided in Appendix D.Source: §5 · p. 8
• Noisy data introduced by context retrieval is a significant factor in generating incorrect review comments.Source: §5 · p. 8

6. Conclusion

• AACR-Bench is a multilingual benchmark designed to evaluate ACR systems with repository-level context and measure their ability to use complex repo-level context.Source: §6 · p. 8
• Context granularity and retrieval methods significantly affect ACR performance, with effects varying by LLM, programming language, and usage paradigm, including agent use.The findings underscore the need for a benchmark that evaluates methods using complex repository-level context.Source: §6 · p. 8
• Comprehensive ground truth remains difficult because real-world software systems are complex and subjective; future work will expand the dataset and explore semi-automated methods to improve ground-truth quality.Source: §6 · pp. 8–9

Impact Statement

• AACR-Bench is presented as the first multilingual, repository-level context-aware ACR benchmark, characterizing current LLM capability boundaries and challenges.The study frames this as establishing a rigorous evaluation standard and charting a course for future ACR systems.Source: Impact Statement · p. 9
• Traditional generative approaches maximize recall but hallucinate frequently, while Agents localize defects precisely but provide incomplete coverage.The authors argue future systems should move beyond single-metric optimization toward hybrid architectures balancing precision and coverage.Source: Impact Statement · p. 9
• Retrieved context has different effects across languages, and irrelevant context can act as noise rather than improving review.The authors call for systems to adapt retrieval necessity and granularity to code characteristics and task types.Source: Impact Statement · p. 9
• RAG can improve global dependency understanding while its noise harms local diff-defect detection; Agents may tunnel into context and miss obvious local errors.The authors propose attention mechanisms integrating syntax-level checks with cross-file risk assessment.Source: Impact Statement · p. 9
• ACR performance depends on information utilization and reasoning robustness, since even relevant BM25- or embedding-retrieved code can degrade results without noise tolerance.The authors favor active auditing, in which Agents explore context, verify hypotheses through multi-turn interactions, and filter noise.Source: Impact Statement · p. 9

B. AACR-Bench Details

• Appendix Section B covers AACR-Bench construction, comparisons with existing benchmarks, and representative data samples.Source: A. Overview of the Appendix · p. 12

A. Overview of the Appendix

• The appendix also outlines Section C's experimental pipeline, models, hyperparameters, and fine-grained results, plus Section D's evaluation case studies.Source: A. Overview of the Appendix · p. 12

B.1. Detailed Dataset Curation Process

• The source pool comprised 50 high-activity repositories—five per language—across the ten programming languages selected from the StackOverflow Developer Survey 2025.Repositories ranked among the top 2,000 for both new stars and closed PRs during December 1, 2024–December 1, 2025, and were then ranked by new stars.Source: §B.1 · p. 12
• Manual verification of 350 domain classifications found 92.36% accuracy, with a 5.06% margin of error at the 95% confidence level.The classification used Qwen3-235B-A22B-Thinking-2507.Source: §B.1 · p. 13
• Filtering required English PR titles and descriptions, at most 1,000 changed lines, repository-language consistency, and at least two inline comments including one accepted constructive comment.The process collected 12,715 PRs created between December 1, 2024, and December 1, 2025, and selected the revision with the most review comments.Source: §B.1 · pp. 13, 15
• The curation process produced a final benchmark of 200 PRs, stratified by repository, problem domain, and PR size from 573 semantically verified PRs.The 573 PRs came from 3,328 automatically filtered PR items.Source: §B.1 · pp. 13, 15
• Augmentation analyzed 1,119 review threads to extract confirmed defects or discard comments without substantive issues, achieving 95% accuracy on 300 manually evaluated results.The reported 95% confidence-level margin of error was 4.74%.Source: §B.1 · p. 15
• To address incomplete issue annotation, six models generated comments through two review systems, followed by semantic deduplication and three-round human annotation by over 80 engineers.Two double-blind rounds by language-matched annotators were followed by core experts adjudicating conflicts; the workflow aimed for high coverage and accuracy.Source: §B.1 · pp. 17–18
• Only 11 comments were detected by three models, while many were found by a single model, supporting multi-model generation to improve issue coverage.The analysis compared generated and augmented comments by how many models detected them in the annotated ground truth.Source: §B.1 · p. 18, Table 8 · p. 18

B.2. Examples of Ground Truth

• The ground-truth examples section presents annotation-verified review comments with each repository, PR ID, target file, addressed diff hunk, comment, and analysis.Source: §B.2 · p. 18

C.1. Detailed Benchmark Process

• Each PR is an evaluation instance: methods review its diff hunks, and generated comments are compared with ground truth using Precision, Recall, and F1-score.A comment is Line Correct when its range overlaps a ground-truth comment, and Semantically Correct when line numbers overlap and the content is identical.Source: §C.1 · p. 18
• For non-Agent methods, the workflow synchronizes Base and Target versions, extracts diff hunks with GitPython, and optionally retrieves repository context from a Target-version code index.The PR title and description are always provided as context; repository code context depends on the experimental design.Source: §C.1 · p. 18
• Non-Agent methods must output the problematic diff fragment, whether the comment applies to the left or right side, and the review-comment content.Source: §C.1 · p. 18
• For Agent methods, Claude Code autonomously extracts diff hunks with Git and retrieves context, so setup typically needs no pre-built code index.The setup typically clones the repository and checks out the corresponding PR version locally.Source: §C.1 · p. 27
• Issue-category classification uses Qwen3-235B-A22B-Instruct-2507, and its prompt achieved 97% accuracy on AACR-Bench ground truth.Source: §C.1 · p. 27

• Tables 9 and 10 report additional experiment statistics, with Table 9 comparing model performance (%) across different types.Source: Table 9 · p. 27, §C.2 · p. 27

D. Case Study

• The authors identify model knowledge errors and noisy context introduced by retrieval as contributors to incorrect review comments.The cases examine comments under agent-based and similarity-based retrieval settings.Source: D. Case Study · p. 27

Case Study: Agent-GLM good comment

• Agent case studies include examples labeled as correct for GLM, DeepSeek, and Claude.One example is from elastic/elasticsearch PR #120546; the supplied passages do not provide the comments’ contents.Source: Figure 21 · p. 30, Case Study: Agent-GLM good comment · pp. 31–32

Case Study: Agent-Claude bad comment

• Figure 25 is captioned as a sample error case using an agent, and the supplied repository metadata identifies infiniflow/ragflow PR #7151 in rag/utils/infinity_conn.py.Source: Figure 25 · p. 34

Case Study: Agent-GLM bad comment

• For wavetermdev/waveterm PR #1755, the cited material identifies frontend/app/view/preview/directorypreview.tsx, and Figure 26 presents a sample error case using an Agent.Source: Figure 26 · p. 35

Case Study: Agent-DeepSeek bad comment

• Figure 27 captions a sample error case using an Agent for elastic/elasticsearch PR #124403, in server/src/main/java/org/elasticsearch/index/codec/tsdb/es819/ES819TSDBDocValuesProducer.java.Source: Figure 27 · p. 36

Case Study: bm25-Claude bad comment

• The supplied material identifies nextcloud/server PR #52846 in lib/public/AppFramework/App.php; Figures 28–31 are captioned as sample error cases using BM25/Embedding Retrieval.Source: Figure 28 · p. 37, Figure 29 · p. 38, Figure 30 · p. 39