Issue-Oriented Agent-Based Framework for Automated Review Comment Generation

1 Introduction

• Existing methods often focus on easy refactorings and perform poorly on bug fixes, testing, and logging, with dataset bias leaving complex issue types underrepresented.A single LLM also struggled to capture the nuanced logic of each category even when guided by a five-category taxonomy.Source: §1 · p. 2
• RevAgent replaces a single-model pipeline with five issue-category commentator agents that propose comments, a critic that selects an issue-comment pair, and category-tailored agent training.The critic is trained on a diversified dataset pairing similar changes with comments from multiple issue perspectives.Source: §1 · p. 3
• 12.90%, 10.87%, 6.32%, and 8.57% average improvements in BLEU, ROUGE-L, METEOR, and SBERT, respectively, over PLM- and LLM-based state-of-the-art baselines.The evaluation used Curev, a dataset of 20,000 review-comment instances across five issue types.Source: §1 · p. 3
• RevAgent achieves 60.20% overall issue-category identification accuracy and 21.69% for Bugfix, where existing approaches typically score below 5%.Ablations found that specialized commentators improve category-aligned comments and the critic is most important for issue-category identification.Source: §1 · p. 3
• Human evaluations of 1,920 samples found RevAgent comments helpful, readable, accurate, and aligned with intended categories, while its additional latency was 0.038 seconds per generation.The authors report that this latency trade-off remains practically acceptable; 48% of low-quality-comment failures involved missing project-specific knowledge.Source: §1 · p. 3
• The paper also evaluates open-source LLMs of different sizes to examine their effectiveness and scalability within RevAgent.The authors publicly release a replication package with code and resources to reproduce the results.Source: §1 · pp. 3–4

Automated Code Review

• Automated code-review research includes PLM- and LLM-based comment generation, fine-tuning, dataset augmentation, prompt engineering, and collaborative systems.Examples include CodeReviewer, LLaMA-Reviewer, CodeAgent, BitsAI-CR, and CodeDoctor.Source: §2 · p. 4
• Most existing comment-generation methods rely on a single model, which can struggle with diverse issue types and produce less precise, context-aware feedback.The paper says its approach follows the paradigm of generating an issue-specific comment for a code change.Source: §2 · p. 4

Agent-Based AI

• Agent-based AI research uses interactions among agents to pursue shared goals, with applications including requirement elicitation, code generation, and vulnerability detection.Prior software-engineering agents include persona-based elicitation and specialized agents for recalling examples, planning, code generation, and debugging.Source: §2 · pp. 4–5
• Code review poses distinct agent-design challenges because reviewers must interpret fine-grained diffs, infer issue types, and produce context-aware feedback.RevAgent couples category-specialist commentator agents with a critic that coordinates and selects a relevant comment.Source: §2 · p. 5

3 RevAgent

• The framework has Generation, Discrimination, and Training stages, with five issue-category perspectives feeding a critic that selects the most appropriate issue-comment pair.The critic evaluates candidates for correctness and severity; the agents are fine-tuned on task-specific data.Source: §3 · pp. 5–6

3.1 Generation Stage

• Five commentator agents independently analyze code changes from assigned issue perspectives and compete to produce accurate, contextually appropriate review comments.The prompt taxonomy includes Refactoring, Bugfix, Testing, Logging, Documentation, and Others; Others is excluded as noisy data.Source: §3.1 · p. 6

3.2 Discrimination Stage

• The critic scrutinizes candidates for correctness and severity, selecting the issue-comment pair that best fits the code change.Its judgments depend on the quality and diversity of the candidates produced by the commentators.Source: §3.2 · p. 7

3.3 Training Stage

• Commentators are fine-tuned on separate category-specific corpora using LoRA, which updates low-rank matrices while keeping pretrained parameters fixed.The paper describes this as reducing trainable parameters and resource consumption while preserving performance.Source: §3.3 · pp. 7–8
• Critic training combines each diff and gold comment with four non-ground-truth category candidates, retrieved using BM25 to provide code-relevant examples.The category corpora are disjoint, preventing candidate-comment duplication across categories.Source: §3.3 · pp. 8–9

Dataset

• Curev provides 20,000 curated review-comment instances, split 75% for training and 25% for evaluation with no train-test overlap.The source dataset has 176,613 multilingual samples across nine programming languages; Refactoring accounts for 69.3% of Curev.Source: §4 · p. 9

Implementation and Environment

• The experiments use open-source Qwen2.5-Coder, DeepSeek-Coder, and Llama3 model families and report average performances in Table 3.Training uses float16 precision for five epochs with batch size 64; temperatures are set to zero.Source: §4 · pp. 9–11, Table 3 · p. 11

Baselines

• The comparison includes CodeReviewer, LLaMA-Reviewer, TufanoLLM, and CodeAgent, with variants for category-and-comment prediction and a function-level CodeAgent adaptation.The study excludes unavailable systems and methods whose task or setting differs from its code-review generation comparison.Source: §4 · p. 10

Evaluation Metrics

• Comment generation is assessed with BLEU-4, ROUGE-L, METEOR, and SBERT, while Pred. Acc. measures issue-category prediction accuracy.The research questions cover baseline performance, component contributions, and human evaluation.Source: §4 · pp. 9–11

5.1 RQ1: Comparison with Baselines

• RevAgent exceeds LLaMA-Reviewer by 12.90% on BLEU, 10.87% on ROUGE-L, 6.32% on METEOR, and 8.57% on SBERT, with all metric comparisons statistically significant.The two-sided Wilcoxon signed-rank tests report p-values below 7.53e-05.Source: §5.1 · p. 12, Table 4 · p. 12
• Among similar-sized models, LLaMA-3 performs worst, while Qwen2.5-Coder 14B exceeds its 7B version by 0.39 BLEU, 0.54 ROUGE-L, 0.36 METEOR, and 0.39 SBERT points.Qwen2.5-Coder and DeepSeek-Coder show complementary strengths; the authors suggest model specialization may matter more than size among similarly sized models.Source: §5.1 · p. 12
• RevAgent achieves 21.69% Bugfix accuracy, while existing approaches typically remain below 5%.Across issue categories, it outperforms baselines on five metrics in most categories, while showing a Logging-category weakness on METEOR and SBERT linked to underrepresentation in critic training data.Source: §5.1 · pp. 13–14, Table 5 · p. 13

5.2 RQ2: Ablation Experiment

• The ablation section evaluates contributions from the three stages, analyzing training first and generation and discrimination afterward.The supplied section passage describes the analysis plan but does not provide its ablation results.Source: §5.2 · p. 14

• Removing supervised fine-tuning from both commentator and critic agents degrades performance, supporting fine-tuning’s role in category-specific comment generation and issue identification.Fine-tuned agents also outperform their few-shot counterparts when the commentator and critic are ablated separately.Source: RQ2.1: Effect of Training Stage · p. 15
• On Qwen2.5-Coder-14B, removing critic-agent fine-tuning yields higher prediction accuracy and comparable performance on the other three metrics.The authors attribute this exception to the model’s strong code understanding, while noting that fine-tuning remains important for smaller models.Source: RQ2.1: Effect of Training Stage · p. 15
• Removing Candidate Comment Retrieval can improve Qwen2.5-Coder-14B’s prediction accuracy, but smaller models perform worse without retrieved comments.The authors link this difference to the larger model’s higher-quality candidate comments, which help the critic focus on semantic distinctions; retrieved comments are generally better than candidates from smaller models.Source: RQ2.1: Effect of Training Stage · p. 15

RQ2.2: Effect of Generation and Discrimination Stages

• RevAgent outperforms single-fusion and multiple-comment-merging variants on most metrics, and the critic agent plays the most crucial role in overall effectiveness.The MSC variant trades lower BLEU and ROUGE-L for higher METEOR and SBERT on both Qwen2.5-Coder models.Source: RQ2.2: Effect of Generation and… · p. 16, Table 7 · p. 16

5.3 RQ3: Human Evaluation

• The human evaluation covered 1,920 comments from five approaches, rated by two Java developers using a reference-free five-point scale; inter-rater Cohen’s kappa was 0.74.The 384 sampled test cases were proportionally represented across issue categories.Source: §5.3 · p. 17
• RevAgent scores highest among baselines on Readability, Accuracy, and Category-Matching, beating LLaMA-Reviewer by 0.7, 0.8, and 0.7 points, respectively.Its lower standard deviations in Readability and Accuracy indicate more consistent performance across cases.Source: §5.3 · pp. 17–18, Table 8 · p. 17
• In the refactoring example, RevAgent identifies deeply nested conditions and suggests a feasible refactoring, while baselines miss the core issue; in the bug-fix example, it detects an abnormal return value.Source: Figure 2 · p. 18, §5.3 · p. 19

Inference Consumption Analysis

• RevAgent uses 8.68× the tokens of LLaMA-Reviewer and 4.47× those of TufanoLLM, but only 0.06× CodeAgent’s tokens; it adds 0.038 seconds per prediction versus LLaMA-Reviewer.It is 0.46× faster than TufanoLLM, and the authors characterize its performance-efficiency balance as favorable.Source: §6 · p. 19

Role of Issue Prediction in Review Comment Generation

• Correct issue predictions yielded BLEU 9.08, ROUGE-L 20.68, METEOR 13.50, and SBERT 49.25, versus 7.65, 17.79, 11.90, and 46.51 with mispredictions.The authors attribute the decline to misprediction selecting an ill-suited commentator agent, which can produce generic comments or miss the core issue.Source: §6 · pp. 19–20

Root Causes Behind Low-Quality Generated Comments

• Low-quality comments were attributed to missing business logic (22%), confusion over project-specific coding standards (48%), and limited contextual information (43%).The authors identify repository-level review incorporating business logic, project standards, and broader context as future needs.Source: §6 · p. 20

Internal Validity

• LLMs may have encountered test cases during training, although their suboptimal zero-shot performance suggests outputs are not solely memorized; minor critic positional bias may remain despite structured inputs.Inference temperature was set to 0 to stabilize outputs and support reproducibility.Source: §7 · p. 20

External Validity

• LLaMA-3.1-70B dataset annotations may contain errors, and Refactoring’s disproportionate test-set share may skew overall metrics.The authors report category-level results to highlight performance across issue types.Source: §7 · p. 20
• Fine-tuning covered only selected models under 14B parameters; proprietary models such as GPT-4o were excluded because of resource constraints.The authors describe the study as a foundation for incorporating larger and more advanced models.Source: §7 · p. 20

Construct Validity

• Human comment-quality judgments may be subjective, though two independent Java-experienced participants were assessed for consistency and comprehension using inter-rater agreement.Source: §7 · p. 21

8 Conclusion

• RevAgent outperformed state-of-the-art baselines in textual quality metrics and issue-category prediction accuracy on large-scale datasets.Source: §8 · p. 21
• Human evaluation further confirmed the quality and usefulness of RevAgent’s generated comments.Source: §8 · p. 21
• The authors report a favorable balance between performance and efficiency.Source: §8 · p. 21
• Future directions include finer-grained issue types, function- or class-level review, and external sources such as repository documentation or history.Source: §8 · p. 21