PaperlayerView exact revision on arXiv

Source-linked AI summary

LAURA: Enhancing Code Review Generation with Context-Enriched Retrieval-Augmented LLM

Yuxin Zhang, Yuxia Zhang, Zeyu Sun, Yanjie Jiang, Hui Liu

arXiv:2512.01356v2cs.SE

TL;DR

Code review is time-consuming and knowledge-intensive, and existing review-generation approaches have overlooked useful change context and prior review knowledge. LAURA combines context augmentation, retrieved review exemplars, and systematic guidance for LLM-generated reviews. For both models, over 40% of comments were judged correct or at least helpful to developers.

  • Problem

    Existing review-generation approaches mainly use historical changes and comments, while important change context and prior review knowledge remain overlooked.

    Source: §I · p. 1

  • Method

    LAURA augments LLMs with pull-request and code context, retrieved review exemplars, and systematic prompt guidance, using a constructed dataset for retrieval and evaluation.

    Source: §I · pp. 1–2, A. Context Augmentation · p. 2, B. Review Exemplar Retrieval · p. 3

  • Results

    42.2% of LAURA-GPT comments and 40.4% of LAURA-DS comments were judged completely correct or at least helpful to developers, significantly outperforming SOTA baselines.

    Source: §I · p. 2

  • Takeaways & Limitations

    The study reports that all three components contribute to improved LLM performance, with their combination yielding the best results.

    Source: §VII · p. 11

    Experiments covered C, C++, Java, and Python, so generalization to other programming languages is unclear.

    Source: §V · p. 10

Abstract

from arXiv · show

Code review is critical for ensuring software quality and maintainability. With the rapid growth in software scale and complexity, code review has become a bottleneck in the development process because of its time-consuming and knowledge-intensive nature and the shortage of experienced developers willing to review code. Several approaches have been proposed for automatically generating code reviews based on retrieval, neural machine translation, pre-trained models, or large language models (LLMs). These approaches mainly leverage historical code changes and review comments. However, a large amount of crucial information for code review, such as the context of code changes and prior review knowledge, has been overlooked. This paper proposes an LLM-based review knowledge-augmented, context-aware framework for code review generation, named LAURA. The framework integrates review exemplar retrieval, context augmentation, and systematic guidance to enhance the performance of ChatGPT-4o and DeepSeek v3 in generating code review comments. Besides, given the extensive low-quality reviews in existing datasets, we also constructed a high-quality dataset. Experimental results show that for both models, LAURA generates review comments that are either completely correct or at least helpful to developers in 42.2% and 40.4% of cases, respectively, significantly outperforming SOTA baselines. Furthermore, our ablation studies demonstrate that all components of LAURA contribute positively to improving comment quality.

I. INTRODUCTION

Code review is time-consuming and knowledge-intensive, while LLMs lack important change context and systematic guidance. LAURA combines context augmentation, review exemplars, and guidance to improve generated reviews.

Motivation

  • LLMs may lack PR details, code context, and domain-specific review experience, as well as guidance on review scope and logical feedback structure.These omissions can lead to suboptimal review quality.Source: §I · p. 1

Contributions

  • LAURA combines context augmentation, review exemplar retrieval, and systematic guidance, supported by a carefully filtered dataset of PR, diff, and review information.The dataset contains 301,256 diff-comment-info series from 1,807 GitHub projects, with 384 manually annotated recent entries for evaluation and earlier entries used for retrieval.Source: §I · pp. 1–2

Results

  • 42.2% for LAURA-GPT and 40.4% for LAURA-DS were the rates of comments judged correct or at least helpful, with both outperforming their base models and CodeReviewer.LAURA also ranked among the top approaches on four of five LLM evaluation metrics, slightly trailing in brevity.Source: §I · p. 2

Contributions

  • The authors made the retrieval-augmented data publicly available to support future improvements in code review.Source: §I · p. 2

II. APPROACH DESIGN

LAURA augments code diffs with pull-request and code context because diffs alone may omit information needed to understand changes. The approach supplies relevant context to LLMs to support review generation.

II. APPROACH DESIGN

A. Context Augmentation

  • PR titles and bodies, commit messages, and changed file paths provide intent, rationale, change-specific detail, and clues about a file’s role.Source: A. Context Augmentation · p. 2
  • Surrounding code is included because some issues can only be identified by considering context beyond the diff.Source: A. Context Augmentation · p. 2

Review Exemplar Retrieval

LAURA enriches diffs with surrounding code and retrieves similar reviewed changes as exemplars. The augmented context and references are supplied to LLMs to inform review generation.

Context Augmentation

  • Context augmentation extends diffs with surrounding code because default diffs may provide insufficient context for identifying issues.The framework also gathers other relevant pull-request information and provides it alongside the enriched diff.Source: A. Context Augmentation · p. 3

B. Review Exemplar Retrieval

  • Review exemplar retrieval supplies LLMs with similar code diffs and their review comments as references for generating comments on the target diff.The approach uses cosine similarity over CodeT5+ embeddings to identify relevant examples.Source: B. Review Exemplar Retrieval · p. 3
  • For diffs of at most 2,048 tokens, LAURA may retrieve examples from both the same repository and language; larger diffs receive only a same-language example.When both sources return the same example, only that pair is supplied.Source: B. Review Exemplar Retrieval · p. 4

C. Systematic Guidance

LAURA uses prompt components to help LLMs understand the extended diff, focus on review-critical aspects, and reason through feedback in a structured way.

Diff example

  • An example and explanation of the extended diff format help LLMs interpret its added code context and line numbers.Source: C. Systematic Guidance · p. 4

Review guidelines

  • Review and code-suggestion guidelines define review scope, emphasize change-relevant aspects, and discourage redundant or meaningless suggestions.The prompt also specifies an output format intended to make responses more organized and readable.Source: C. Systematic Guidance · p. 4

Chain of thought

  • Chain of thought guides LLMs to understand the change, identify and explain issues, and then offer suggestions using contextual information.Source: C. Systematic Guidance · p. 4

III. EXPERIMENTAL DESIGN

This section outlines the study’s research questions and the construction and selection of a high-quality code-review dataset. The experiments compare LAURA with direct generation and CodeReviewer, and examine the contribution of its three components.

A. Research Questions

  • The study asks how LAURA compares with direct generation and CodeReviewer, and how much context augmentation, exemplar retrieval, and systematic guidance contribute.Both research questions use LLM and human evaluation.Source: A. Research Questions · p. 4

B. Dataset Construction

  • Existing datasets lack pull-request context and contain low-quality reviews, motivating a carefully filtered code-review dataset.The cited estimate is about 25% low-quality data overall and about 32% for CodeReviewer.Source: B. Dataset Construction · p. 4

1) Project Selection:

  • Projects were limited to C, C++, Java, and Python repositories with at least 2,500 stars and 500 pull requests, followed by manual exclusions.Tutorial-oriented and forked projects were removed.Source: 1) Project Selection · pp. 4–5
  • The final selection comprised 1,807 repositories: 228 C, 420 C++, 346 Java, and 813 Python projects.The selection followed repository activity filtering and manual review.Source: 1) Project Selection · p. 5

2) Data Collection:

The authors collected pull-request and code-review data, applied staged quality filtering, and built a time-partitioned retrieval database of review exemplars.

2) Data Collection:

  • PR records included review comments, commits, diffs, and file paths; complete diffs were fetched, and the 10-line rule retained cases where later commits changed code near earlier review locations.The collection also excluded cases where the PR author and reviewer were the same.Source: 2) Data Collection · p. 5

3) Data Filtering:

  • Rule-based filtering removed bots, invalid or irrelevant comments, and non-ASCII comments, leaving 817,833 series before LLM filtering.GPT-4o mini then assessed each comment with its corresponding diff for usefulness and clarity.Source: 3) Data Filtering · pp. 5–6
  • 0.914 accuracy and 0.606 recall were achieved by the LLM filter on reviewed samples; filtering left 371,831 series, and merging comments yielded 301,256.The authors manually reviewed 240 samples to assess the filter; they note some high-quality samples may be discarded.Source: 3) Data Filtering · p. 6

4) Retrieval System Construction:

  • For retrieval, 298,494 pre-December 26, 2024 diffs were embedded as 256-dimensional vectors, and cosine similarity selected review exemplars for test cases.The embedding model was codet5p-110m-embedding.Source: 4) Retrieval System Construction · p. 6

5) Evaluation Dataset Construction:

This section describes a manually annotated evaluation set, the comparison baselines, and the LLM-based evaluation procedure. The authors also explain why they supplemented LLM ratings with human assessment.

5) Evaluation Dataset Construction:

  • The evaluation set contains 384 manually validated high-quality samples dated after December 26, 2024, selected to reduce possible training-data leakage.The authors annotated 546 samples to obtain 384 valid ones; inter-rater Cohen’s kappa was 0.883.Source: 5) Evaluation Dataset Construction · p. 6

C. Baselines

  • CodeReviewer, ChatGPT-4o, and DeepSeek v3 were selected as baselines for evaluating LAURA’s performance improvements.CodeReviewer represents a traditional pre-trained model baseline.Source: C. Baselines · p. 6

D. Evaluation

  • Because BLEU and ROUGE are unsuitable for diverse LLM-generated comments and LLM ratings may miss real-world usefulness, the study uses both LLM and human evaluation.The human assessment addresses the stated limitation of LLM evaluation.Source: D. Evaluation · p. 6

1) LLM Evaluation:

  • ChatGPT-4o rated comments on five quality metrics using a 1–5 scale, with anonymized method sets compared for the main evaluation and ablations.The scoring guidelines were intended to calibrate ratings; the RQ2 evaluation was split into two sets due to input-length limits.Source: 1) LLM Evaluation · pp. 6–7

2) Human Evaluation:

This section defines human-review categories and quality scores, then describes the evaluator calibration and rating process.

2) Human Evaluation:

  • Human reviewers classified comments as Instrumental, Helpful, Uncertain, or Misleading, with uncertainty reflecting limits in understanding project-specific context.The authors caution that usefulness does not establish factual grounding or likely acceptance within a particular project.Source: 2) Human Evaluation · p. 7
  • I-Score measures Instrumental comments, IH-Score combines Instrumental and Helpful comments, and M-Score measures Misleading comments; higher is better for the first two and lower for M-Score.Misleading comments include factual errors, incorrect suggestions, or failures to address the diff.Source: 2) Human Evaluation · p. 7
  • Four experienced volunteers calibrated ratings together, then independently evaluated samples with source blinding; their initial independent ratings reached Krippendorff’s Alpha 0.804 (p <0.001).For the ablation evaluation, the four evaluators each handled 96 results.Source: 2) Human Evaluation · p. 7

E. Experimental Details

The experiments compare LAURA-guided generation with direct LLM generation and CodeReviewer, using LLM-based and human evaluations. LAURA-GPT and LAURA-DS outperform their base models, with examples illustrating how added context and exemplars support useful comments.

E. Experimental Details

  • Experiments ran each API entry independently with designed prompts and inputs; CodeReviewer was further fine-tuned on the custom dataset for 10 epochs.The API temperature was 0.5, selected through preliminary tests.Source: E. Experimental Details · pp. 7–8

A. RQ1: Effectiveness of LAURA

  • 42.2% of LAURA-GPT comments and 40.4% of LAURA-DS comments were rated instrumental or helpful; both outperform direct generation and CodeReviewer.LAURA-GPT also achieved the highest I-Score (20.0%) and lowest M-Score (0.8%); CodeReviewer had I-Score 3.4% and IH-Score 10.2%.Source: A. RQ1: Effectiveness of LAURA · p. 8
  • In the human-evaluation example, LAURA-GPT identifies string-truncation risk and recommends tests, while LAURA-DS detects the risk but omits the test suggestion.LAURA-GPT earned an “Instrumental” rating, LAURA-DS “Helpful,” and CodeReviewer “Misleading.”Source: A. RQ1: Effectiveness of LAURA · p. 8
  • Added code context highlights duplication in one case, while a retrieved exemplar helps DeepSeek v3 address redundant “displayName” in another.The paper describes these as two real-world cases of LAURA assisting review-comment generation.Source: A. RQ1: Effectiveness of LAURA · p. 8
  • The authors attribute direct ChatGPT-4o’s advantage over CodeReviewer, contrary to Tufano et al., to model advances, dataset differences, and evaluation methods.They note that their dataset may contain more long diffs, which may have overwhelmed pretrained models.Source: A. RQ1: Effectiveness of LAURA · p. 8

B. RQ2: Ablation Study of the Three Key Components of LAURA

Ablations indicate that context augmentation, review exemplar retrieval, and systematic guidance each contribute to comment quality. Their combined effects reinforce one another, rather than simply adding up.

B. RQ2: Ablation Study of the Three Key Components of LAURA

  • Removing only exemplar retrieval performs slightly better overall than the other ablations, despite its associated declines in human-evaluation scores.The authors report no significant difference between removing context augmentation and removing systematic guidance.Source: B. RQ2: Ablation Study of the Three… · p. 8
  • Removing any one LAURA component reduces LLM-evaluation metrics; removing context augmentation or systematic guidance lowers I-Score by 10.4%–11.7% and IH-Score by 11.0%–12.2%.Removing exemplar retrieval lowers I-Score by 9.1% for LAURA-GPT and 8.4% for LAURA-DS, and IH-Score by 10.5% and 8.4%, respectively.Source: B. RQ2: Ablation Study of the Three… · p. 8, Table V · p. 9
  • Context, exemplars, and precise guidance respectively provide relevant code information, reviewer experience, and direction for producing useful feedback.The ablation comparison removes each of these three components in turn.Source: B. RQ2: Ablation Study of the Three… · pp. 8–9
  • The components reinforce one another: systematic guidance helps LLMs interpret extended diffs, while extended diffs help them understand retrieved exemplars.The sum of individual component-removal drops exceeds the drop from removing all three components together.Source: B. RQ2: Ablation Study of the Three… · p. 9

V. THREATS TO VALIDITY

The section outlines internal, external, and construct-validity risks, then surveys code-review automation and the study’s line-level-diff focus.

V. THREATS TO VALIDITY

  • Potential data leakage, uncertain extended-diff context scope, and unexplored prompt designs threaten internal validity.The authors cannot fully rule out post-release model optimization using test data; too little context may miss details, while too much may add noise.Source: §V · p. 10
  • External validity is bounded by possible residual low-quality comments, evaluation on four programming languages, and omitted baselines.The study used C, C++, Java, and Python; the authors also note that broader evaluation across LLMs remains open.Source: §V · p. 10
  • Construct validity is limited by testing only ChatGPT-4o and DeepSeek v3, variable LLM-based metrics, and human evaluation focused on factual alignment.The practical usefulness of mismatched review comments therefore remains uncertain.Source: §V · p. 10

A. Code Review Automation

  • Code-review automation spans reviewer recommendation, review-necessity prediction, code-area identification, refinement, comment generation, and suggestion.Early reviewer-recommendation approaches used file paths, traceability graphs, load balancing, collaborative filtering, expertise and workload, cross-project collaboration, or composite features.Source: A. Code Review Automation · p. 10
  • T5CR, CodeReviewer, and CommentFinder represent model-based and retrieval-based approaches to review generation and related automation tasks.T5CR covers method-level pre-review optimization, comment generation, and post-review refinement; CodeReviewer handles line-level diffs across nine languages, while CommentFinder retrieves similar snippets and comments.Source: A. Code Review Automation · pp. 10–11
  • Prior LLM studies explored review automation, but the authors characterize their exploration as limited and focus this work on generating reviews from line-level diff hunks.Reported prior findings include ChatGPT-3.5 outperforming CodeReviewer in post-review refinement, while ChatGPT-3 lagged in review generation against three existing methods.Source: A. Code Review Automation · p. 11

B. Large Language Model

The section situates LAURA among LLM enhancement approaches and concludes that its three components improve review generation, with their combination yielding the best results.

B. Large Language Model

  • The paper applies LAURA to ChatGPT-4o and DeepSeek v3, two LLMs identified among notable models.The passage describes LLMs as evolving from statistical and neural models to models with over 10B parameters trained on vast datasets.Source: B. Large Language Model · p. 11
  • LLM enhancement approaches include richer input context, retrieval-augmented generation, and prompt engineering.The passage describes these as ways to support task understanding, incorporate external knowledge, and guide model actions through instructions.Source: B. Large Language Model · p. 11

VII. CONCLUSION

  • LAURA outperforms direct LLM use and substantially surpasses CodeReviewer; its three components each contribute, with their combination yielding the best results.The authors identify additional useful input contexts and better input integration as directions for further improvement.Source: §VII · p. 11
Loading 2512.01356v2…