Source-linked AI summary
AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context
Lei Zhang, Yongda Yu, Minghui Yu, Xinxin Guo, Zhengqi Zhuang, Guoping Rong, Dong Shao, Haifeng Shen, Hongyu Kuang, Zhengfeng Li, Boge Wang, Guoan Zhang, Bangyu Xiang, Xiaobin Xu
TL;DR
Existing ACR benchmarks provide limited evidence because they often lack multilingual repository-level context and rely on incomplete raw PR comments. AACR-Bench addresses these gaps with an expert-verified, AI-assisted benchmark, showing that context granularity and retrieval choices affect performance differently across models, languages, and usage paradigms.
Problem
Existing ACR benchmarks lack multilingual repository-level context and often use incomplete raw PR comments as ground truth, limiting comprehensive evaluation.
Method
AACR-Bench combines multi-model review generation, expert annotation, and complete repository-level dependencies across multiple programming languages.
Results
Context granularity and retrieval methods significantly affect ACR performance, with effects varying by LLM, programming language, and usage paradigm.
Takeaways & Limitations
AACR-Bench provides a broader basis for evaluating how ACR systems use complex repository-level context.
Takeaways & Limitations
Fully comprehensive ground truth remains difficult because real-world software systems are complex and subjective.
Abstract
from arXiv · showhide
High-quality evaluation benchmarks are pivotal for deploying Large Language Models (LLMs) in Automated Code Review (ACR). However, existing benchmarks suffer from two critical limitations: first, the lack of multi-language support in repository-level contexts, which restricts the generalizability of evaluation results; second, the reliance on noisy, incomplete ground truth derived from raw Pull Request (PR) comments, which constrains the scope of issue detection. To address these challenges, we introduce AACR-Bench a comprehensive benchmark that provides full cross-file context across multiple programming languages. Unlike traditional datasets, AACR-Bench employs an "AI-assisted, Expert-verified" annotation pipeline to uncover latent defects often overlooked in original PRs, resulting in a 285% increase in defect coverage. Extensive evaluations of mainstream LLMs on AACR-Bench reveal that previous assessments may have either misjudged or only partially captured model capabilities due to data limitations. Our work establishes a more rigorous standard for ACR evaluation and offers new insights on LLM based ACR, i.e., the granularity/level of context and the choice of retrieval methods significantly impact ACR performance, and this influence varies depending on the LLM, programming language, and the LLM usage paradigm e.g., whether an Agent architecture is employed. The code, data, and other artifacts of our evaluation set are available at https://github.com/alibaba/aacr-bench .
1. Introduction
AACR-Bench addresses incomplete issue annotations and restricted language coverage in existing ACR benchmarks by combining expert-verified augmentation with multilingual repository-level context. It supports broader evaluation of LLM-based ACR capabilities.
- Existing benchmarks often use raw historical PR comments as ground truth, limiting issue coverage and faithful assessment of latent-issue detection.
- Cross-file defects require repository-level context, but many context-aware benchmarks focus on a single language, limiting generalizability and potentially introducing language-specific bias.
- 80 senior software engineers reviewed 2,145 comments generated by two ACR systems across six LLMs, supplementing 391 real-world review comments.
- AACR-Bench provides complete repository-level dependency information across 10 mainstream programming languages.
- AACR-Bench combines multi-model generation with large-scale human annotation to improve issue coverage and establish a better ground-truth dataset.
2. Related Work
Prior ACR research spans pretrained, fine-tuned, preference-optimized, retrieval-augmented, and agent-based systems. As these methods diversify, the field needs standardized benchmarks that evaluate them systematically in realistic contexts.
- LLM-based ACR methods include pretrained models, parameter-efficient fine-tuning, iterative feedback, preference optimization, and static-analysis integration.
- Retrieval-augmented generation incorporates project-level semantics to address limited context in automated code review.
- Agent-based frameworks extend review reasoning through multi-agent collaboration and debate-style interactions.
- ACR benchmarking remains nascent, with datasets consisting mainly of individual-study resources and dedicated evaluation datasets.
- AACR-Bench introduces a multi-language, repository-level-context evaluation dataset intended to better reflect real-world production environments.
3. The AACR-Bench
AACR-Bench is a repository-level ACR benchmark built from multilingual PRs and expert-verified review comments. Its design combines broad repository context, hybrid comment sources, context-scope labels, and comparisons with existing benchmarks.
- Overview of AACR-Bench: The dataset contains 200 PRs and 1,505 fine-grained review comments from 50 repositories spanning 10 mainstream programming languages.
- Dataset Curation: Its comments combine model-augmented human reviews and fully model-generated reviews, with human expert verification supporting benchmark credibility.
- Overview of AACR-Bench: AACR-Bench evaluates end-to-end ACR performance by treating each PR as an evaluation unit and matching generated comments against 1,505 ground-truth items.
- Selection of programming languages and repositories: AACR-Bench covers 10 selected programming languages and filters PRs using language, size, and textual-quality criteria.
- Dataset Curation: The curation process uses six mainstream LLMs and two heterogeneous generation frameworks, followed by semantic deduplication and review by 80 senior software engineers.
- Benchmark Comparison: Unlike prior benchmarks, AACR-Bench combines multilingual coverage with repository-level context and annotates the context scope required by each review comment.
4. Experiments
The experiments assess how multilingual coverage, repository-level context, defect exposure, retrieval strategy, context scope, and programming language affect ACR performance. Results show strong dependence on the model, retrieval paradigm, context level, and language.
- The evaluation tests multi-language support, repository-level context, and more comprehensive defect exposure as core AACR-Bench features.
- ACR performance benchmarked with a more sufficient exposure to defects: Agent-based methods generate 0.08 ∼0.15 comments per patch, substantially fewer than traditional approaches.Claude-4.5-Sonnet reaches 39.90% Precision in Agent mode versus 8.70% with No context, but Recall falls to 10.10%.
- ACR performance benchmarked with a more sufficient exposure to defects: Retrieval is model-dependent: Claude-4.5-Sonnet reaches F1=14.46 without context, while BM25 lowers it to F1=9.98, a 31% decrease.DeepSeek-V3.2 performs best with BM25 at F1=15.59, whereas Qwen-480B-Coder peaks with Embedding at F1=14.36.
- Context level-wise impact on ACR performance: Non-Agent methods generally decline as context expands from Diff to File to Repo, whereas Agent methods often improve at Repo level.Under Agent, DeepSeek-V3.2 rises from 4.28% at Diff to 8.00% at Repo, and Qwen-480B-Coder rises from 4.49% to 5.94%.
- Language-wise impact on ACR performance: ACR performance varies substantially by programming language, with Claude-4.5-Sonnet strongest on Python, Java, Go, and C but weaker on TypeScript and PHP.Across frameworks, C# outperforms C; GPT-5.2 scores 0.309 on C# versus 0.085 on C in No context mode.
- Language-wise impact on ACR performance: Context can degrade performance in C#, C++, JavaScript, PHP, Python, Rust, and TypeScript, while C, Go, and Java remain stable or improve.Claude-4.5-Sonnet improves under Agent from 0.120 to 0.218 on Go, 0.142 to 0.241 on Java, and 0.106 to 0.189 on C.
5. Error Analysis
Error analysis finds that current models still produce incorrect review comments because of knowledge errors and noise introduced by context retrieval.
- Current models continue to suffer from knowledge errors during code review.
- Noisy data introduced by context retrieval is identified as a significant factor contributing to incorrect review comments.
6. Conclusion
AACR-Bench is a multilingual, repository-level benchmark whose evaluations show that context granularity and retrieval methods affect ACR performance differently across models, languages, and usage paradigms. The benchmark’s ground truth remains difficult to make fully comprehensive because real-world software is complex and subjective.
- AACR-Bench evaluates ACR systems with multilingual, repository-level context.
- Context granularity and retrieval methods significantly affect ACR performance, with effects varying by LLM, programming language, and usage paradigm.
- A fully comprehensive Ground Truth remains difficult to construct because real-world software systems are inherently complex and subjective.
Impact Statement
AACR-Bench frames practical ACR progress around precision-recall balance, adaptive context use, and integration of local and global reasoning. Its findings favor active auditing that can explore, verify, and filter context rather than passively ingesting retrieved snippets.
- Navigating the Precision-Recall Trade-off: Agent-based methods offer higher precision but less comprehensive defect coverage, while traditional approaches offer higher recall with more hallucinations.
- Towards Adaptive Context Awareness: Retrieved context does not uniformly improve ACR because its impact varies across programming languages and irrelevant context can introduce noise.
- Unifying Local and Global Perspectives: RAG can improve global dependency understanding while compromising local defect detection, whereas context tunneling can cause Agents to overlook local errors.
- From Passive Ingestion to Active Auditing: Reliable ACR requires active context exploration, hypothesis verification, and noise filtering rather than passive ingestion of pre-retrieved snippets.
B.1. Detailed Dataset Curation Process
AACR-Bench was curated through repository and PR filtering, multilingual sampling, review-comment augmentation, semantic deduplication, and expert annotation. The resulting benchmark combines broad data diversity with verified review issues and multi-model complementarity.
- Repository and language selection: The benchmark selected repositories spanning JavaScript, Python, TypeScript, Java, C#, C++, C, PHP, Go, and Rust.
- PR filtering: 12,715 Pull Requests were collected from 50 repositories before preprocessing and quality filtering.
- PR filtering: Filtering required English PR descriptions, at most 1,000 changed lines, language consistency, and at least two inline comments including one accepted constructive comment.
- Sampling: The final benchmark contains 200 PRs selected through stratified sampling by repository, problem domain, and PR size.
- Review-comment augmentation: LLM analysis augmented review threads by extracting confirmed defects and discarding comments without substantive issues.
- Expert annotation: More than 80 senior software engineers verified comment correctness, issue types, and context-dependency scope after multi-model generation and semantic deduplication.
- Annotation outcomes: Only 11 comments were detected by three models, while many were detected by a single model, supporting multi-model generation for broader issue coverage.
C.1. Detailed Benchmark Process
AACR-Bench evaluates ACR methods by scanning Pull Request diff hunks and matching generated comments against verified ground truth. It supports both standardized non-Agent workflows and autonomous Agent-based retrieval, with precision, recall, and F1-score as core measures.
- Evaluation setup: Each PR is an evaluation instance, and ACR methods generate comments while iterating through its changed diff hunks.
- Comment matching: Generated comments are matched to ground truth using line overlap for Line Correctness and line overlap plus identical content for Semantic Correctness.
- Evaluation metrics: Performance is measured with Precision, Recall, and F1-score against the PR’s ground-truth review comments.
- Non-Agent methods: Non-Agent methods clone repositories, synchronize Base and Target versions, extract diff hunks, and scan each hunk for review comments.
- Agent methods: Claude Code autonomously extracts diffs and retrieves repository context through tool use after checking out the relevant PR version locally.
- Issue categorization: Qwen3-235B-A22B-Instruct-2507 classified generated issue categories with 97% validated accuracy.
C.2. Additional Statistics
This section provides additional experimental statistics, including detailed performance comparisons across models and issue types.
- Table 9 reports detailed performance comparisons of models across different issue types.
- The section identifies Tables 9 and 10 as additional statistics from the experiment.
D. Case Study
This section presents case studies of correct and incorrect review comments generated under Agent-based and similarity-based retrieval settings. The cases include repository and change metadata, review outputs, retrieved context where applicable, and analyses of erroneous comments.
- Case Study: The case studies compare correct and incorrect review comments from Agent-based and similarity-based retrieval settings.The examples include Agent, BM25, and embedding-based cases.
- Case Study: Each case records the repository, pull request, file path, diff hunk, generated comment, and retrieved code context where applicable.
- Case Study: Current models continue to suffer from knowledge errors during code review.
- Case Study: The appendix includes Agent definitions and prompts, issue-classification prompts, and samples of correct and erroneous cases.
- Case Study: The examples cover repositories including Elasticsearch, Kestra, Ragflow, Cherry Studio, Waveterm, Hyperswitch, and Nextcloud.
- Case Study: Additional examples show correct and incorrect comments produced with BM25 and embedding retrieval.