CodeAgent: Autonomous Communicative Agents for Code Review

1 Introduction

• CodeAgent models code review as collaborative interaction among diverse roles rather than a single input-output generation task.The framework simulates code-change authors, reviewers, and decision makers.Source: §1 · pp. 1–2
• The paper positions CodeAgent as an autonomous agent-based system for practical code review in software maintenance.Its evaluations span multiple review objectives rather than a single task.Source: §1 · p. 2
• The work contributes a dataset of 3 545 real-world code changes and commit messages for evaluating vulnerability, style, and revision tasks.The dataset contains relevant files and details in a self-contained format.Source: §1 · p. 2
• QA-Checker monitors conversations so questions and answers remain aligned with the original review objective, addressing prompt drifting in multi-agent systems.It can refine queries and realign answers with the intended objective.Source: §1 · p. 2
• CodeAgent outperforms GPT-4 and CodeBERT by 3 to 7 percentage points in vulnerabilities detected, exceeds ReAct by approximately 14% in inconsistency-recall, and improves Edit Progress by about 30%.These results cover vulnerability detection, format alignment, and code revision, respectively.Source: §1 · p. 2

2.1 Tasks

• CodeAgent covers consistency analysis, vulnerability analysis, format consistency, and code revision as its four code-review tasks.The tasks compare code changes with commit messages, identify introduced vulnerabilities, assess formatting alignment, and suggest rewrites.Source: §2.1 · pp. 2–3

2.2 Pipeline

• The pipeline proceeds through Basic Info Sync, Code Review, Code Alignment, and Document phases with role-specific conversations.The phases move from input recognition through review and revision to final synthesis.Source: §2.2 · pp. 3–4, Figure 2 · p. 4
• Each phase is decomposed into atomic two-agent conversations using instructor-assistant interaction, with QA-Checker supervising conversations to prevent digression.The architecture uses sequential phase-level organization and task-oriented conversation-level role-playing.Source: §2.2 · pp. 3–4

2.3 Role Card Definition

• The simulated team defines roles including User, CEO, CPO, CTO, Reviewer, and Coder, assigning them distinct responsibilities across review and revision.Reviewer analyzes code while Coder participates in review and processes revisions with other roles.Source: §2.3 · p. 4

2.4 Self-Improving CoT with QA Checker

• QA-Checker refines an initial question when an answer is inappropriate, appending instructions and repeating dialogue until the answer is acceptable or the turn limit is reached.The refinement combines the original question with additional instructions as q1 = CB(q0 + aai0).Source: §2.4 · pp. 4–5

3 Experimental Setup

• The evaluation covers qualitative and quantitative experiments across nine programming languages using four distinct metrics and multiple baseline systems.The compared approaches include CodeBERT, GraphCodeBERT, CodeT5, GPT, COT, and ReAct, alongside code-revision systems.Source: §3 · p. 5, §3.3 · pp. 5–6

3.1 Datasets

• The new dataset contains over 3,545 commits and 2,933 pull requests from more than 180 projects across nine languages, with merged and closed positive and negative samples for consistency and format detection.Source: §3.1 · p. 5

3.2 Metrics

• F1-Score and recall evaluate consistency and format detection, while Edit Progress measures the reduction in edit distance from erroneous code to predicted correct code.Hit Rate measures the proportion of confirmed vulnerabilities among issues found for vulnerability analysis.Source: §3.2 · p. 5

3.3 State-of-the-Art Tools and Models

• The comparison includes established code-review models such as Trans-Review, AutoTransform, T5-Review, CodeBERT, GraphCodeBERT, and CodeT5, plus GPT, COT, and ReAct.These systems represent code abstraction, pretrained code-text modeling, code-structure modeling, identifier awareness, generic generation, chain-of-thought reasoning, and reasoning-action interleaving.Source: §3.3 · pp. 5–6
• Table 2 evaluates vulnerability findings over 3,545 items using confirmed-rate and dataset-wide-rate measures, including a CodeAgent variant without QA-Checker.Source: Table 2 · p. 6, §3.3 · p. 6

4.1 Vulnerability Analysis

• CodeAgent confirmed 449 high-risk vulnerabilities from 483 potential findings across 3,545 samples, achieving a 92.96% Ratecr and a 12.67% Rateca.The vulnerability analysis used manual verification, with each sample checked by at least one researcher and one engineer.Source: §4.1 · p. 6
• CodeAgent’s confirmed vulnerability findings overlap with 212 CodeBERT, 317 GPT-3.5, and 334 GPT-4.0 findings, indicating broad coverage across model detections.Source: §4.1 · p. 7, Figure 4 · p. 7
• Removing QA-Checker substantially lowers CodeAgent’s vulnerability hit rates, demonstrating its critical role in the full system.Source: §4.1 · p. 7

4.2 Consistency and Format Detection

• For code-change and commit-message consistency, CodeAgent reaches 90.11% recall and 93.89% F1-Score on merged commits and 87.15% recall and 92.40% F1-Score on closed commits.These results correspond to average improvements of 5.62% in recall and 3.79% in F1-Score over other models.Source: §4.2 · p. 7, Table 3 · p. 7
• For format consistency, CodeAgent achieves overall averages of 89.46% recall and 94.07% F1-Score, improving over other models by 13.39% and 10.45%, respectively.The reported merged and closed results are also higher for CodeAgent across both metrics.Source: §4.2 · pp. 7–8, Table 4 · p. 8

4.3 Code Revision

• For code revision, CodeAgent obtains the highest Edit Progress of 37.6% on the T5-Review dataset and an average Edit Progress of 31.6% across datasets.Source: §4.3 · p. 8

Automating Code Review Activities

• Prior code-review automation addresses code-change anticipation, modification semantics, comment suggestions, vulnerability detection, and code consistency using specialized models and tools.Source: §5 · p. 8

Collaborative AI

• Collaborative AI research studies multi-agent cooperation, collective thinking, conversation curation, sociological phenomena, and cooperation efficiency.Source: §5 · p. 8
• The paper identifies a gap in integrating collaborative AI advances with structured software-engineering practices and presents its approach as addressing that challenge.Source: §5 · p. 8

6 Conclusion

• CodeAgent automates code reviews and addresses vulnerability detection, code-message consistency, code style, and code revision suggestions.Source: §6 · pp. 8–9
• The study reports that CodeAgent outperforms existing state-of-the-art solutions for code revision suggestions.Source: §6 · p. 9

Limitations

• The system’s generalizability across software environments and industries requires further validation and testing.Source: Limitations · p. 9
• The baseline testing may not fully capture performance in edge cases or complex scenarios, motivating more comprehensive evaluation.Source: Limitations · p. 9

Ethics Statements

• The study used publicly available data without human participants or personal or sensitive information.Source: Ethics Statements · p. 9

A Details of QA-Checker Algorithm

• Under twice differentiability, negligible remainder terms, positive-definite Hessians, and suitable learning rates, the analysis suggests convergence to an optimal question-answer pair.Source: A Details of QA-Checker Algorithm · pp. 13–14
• The QA-Checker refines questions with additional instructions when answer quality falls below a threshold, prompting agents to generate improved answers.Source: A Details of QA-Checker Algorithm · p. 14
• The quality assessment combines relevance, specificity, and coherence, with equal non-negative weights α, β, and γ summing to 1.Source: A Details of QA-Checker Algorithm · p. 14

B Complete Related Work

• The paper frames CodeAgent as integrating human code-review processes into multi-agent systems rather than relying only on natural-language collaboration.Source: B Complete Related Work · p. 15

C Experimental Details

• Experiments use at most 10 conversation rounds, with six defined roles and a dedicated QA-Checker prompt for maintaining topic relevance.Source: C Experimental Details · p. 15, §C.1 · pp. 15–16, Figure 5 · p. 16

C.2 Execute Time Across Languages

• Merged pull requests take approximately 44.92 seconds longer on average than closed pull requests across nine programming languages.Source: §C.2 · pp. 16–17, Figure 6 · p. 17
• The longer merged-PR time is attributed to potentially more rigorous review and more frequent contributor updates needed for integration into the main codebase.Source: §C.2 · pp. 16–17

D.2 Differences and Implications

• QA-Checker specializes in question-answer tasks, whereas recursive self-improvement systems support broader application ranges and systemic learning from diverse projects.Source: §D.2 · p. 17, §D.4 · p. 17

D.3 Importance of QA-Checker in Role Conversations

• In role conversations, QA-Checker focuses on accuracy, relevance, clarity, and contextually appropriate responses.Source: §D.3 · p. 17
• Its optimization-based refinement is described as efficient for adapting responses to evolving conversational nuances.Source: §D.3 · p. 17, §D.4 · p. 17

• CodeAgent provides consistency, vulnerability, and format analysis for code review, while QA-Checker can improve conversation generation between agents.Source: E Capabilities Analysis between… · p. 17

F Dataset

• CodeAgent supplements three earlier code-revision datasets, assigning them to CRB, CRA, and RCG tasks according to their available review artifacts.AutoTransformdata is excluded from CRA and RCG because it lacks review comments.Source: F Dataset · pp. 17–18
• The dataset spans consistency and format analysis with positive and negative samples grouped by merged and closed pull-request status across nine languages.Figures 7 and 8 visualize the distributions and comparative commit counts across the language and status categories.Source: F Dataset · p. 18, Figure 7 · p. 19, Figure 8 · p. 19
• The dataset covers over 3,545 commits and nearly 2,933 pull requests from more than 180 projects, compiled using a custom crawler targeting post-April 2023 repositories.Source: F Dataset · p. 18

I Algorithmic Description of CodeAgent Pipeline with QA-Checker

• The framework uses six characters and four phases: Basic Info Sync, Code Review, Code Alignment, and document finalization.The phases assign agents to language and modality analysis, review, revision, and documentation activities.Source: I Algorithmic Description of CodeAgent… · p. 19

G Key Factors Leading to Vulnerabilities

• The vulnerability analysis checks a broad catalog including input validation, authentication, sensitive-data exposure, injection, configuration, resource, and communication risks.The catalog also covers dependency, CSRF, deserialization, SSRF, deprecated-function, mobile, and cloud-configuration issues.Source: I Algorithmic Description of CodeAgent… · p. 20

I Algorithmic Description of CodeAgent Pipeline with QA-Checker

• QA-Checker evaluates answer relevance and accuracy, then adds instructions and repeats the question-answer cycle when the quality score falls below a threshold.The process updates Qi+1 = Qi + aai and requests a new response before returning the refined review document.Source: I Algorithmic Description of CodeAgent… · pp. 19, 21

J Detailed Performance of CodeAgent in Various Languages on VA task

• Python has the highest merged vulnerability count, while Go, Ruby, and C have lower counts; closed vulnerabilities are generally fewer than merged ones.C is an exception, with more closed than merged vulnerabilities.Source: J Detailed Performance of CodeAgent in… · p. 21, Table 10 · p. 21
• Rateclose generally exceeds Ratemerge across languages, while C++ records the highest Rateavg at 16.49%.For Python, Ratemerge is 14.00% and Rateclose is 18.16%.Source: J Detailed Performance of CodeAgent in… · p. 21

K More detailed experimental results on CA and FA tasks

• Detailed experimental results for CA are shown in Figures 9 and 10, while detailed experimental results for FA are shown in Figures 11 and 12.Source: K More detailed experimental results… · p. 21

L Case Study

• Table 10 maps specific programming languages to the figure numbers used for the case studies.This provides a lookup between language-specific examples and their corresponding visual analyses.Source: L Case Study · p. 21, Table 10 · p. 21

L.2 Difference of CodeAgent-3.5 and CodeAgent-4.0

• CodeAgent-4.0 shows no general code-review difference from CodeAgent-3.5, but handles longer inputs and logic reasoning more effectively.In one consistency-detection example, the reported output grows from 15k lines with CodeAgent-3.5 to more than 17.7k lines with CodeAgent-4.0.Source: §L.2 · p. 21

M Ablation study

• The ablation evaluates vulnerability analysis with and without QA-Checker because the full system’s performance is attributed mainly to the combination of chain-of-thought and QA-Checker.The comparison is conducted across nine programming languages.Source: M Ablation study · pp. 21, 24
• CodeAgent without QA-Checker identifies 1,057 merged Python cases, of which 140 are confirmed, yielding a Ratemerge of 13.25%.Table 11 also reports merged and closed vulnerability counts and rates across nine programming languages.Source: M Ablation study · p. 24, Table 11 · p. 28
• Removing QA-Checker generally lowers Ratemerge, Rateclose, and Rateavg, indicating different or more conservative vulnerability confirmation behavior.The overall language pattern remains similar, with higher merged counts in Python and lower counts in Go and Ruby.Source: M Ablation study · p. 24

N Cost statement

• CodeAgent-4 has higher query time and cost than CodeAgent-3.5 because of its increased complexity and capabilities.The paper also notes that integrating AI models and multi-agent systems may require specialized implementation knowledge.Source: N Cost statement · p. 24, Table 12 · p. 35

O Tool

• The authors provide a website for CodeAgent as an accessible tool interface.The website is illustrated in Figure 23.Source: O Tool · p. 35, Figure 23 · p. 35