PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study

Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, Xin Peng

arXiv:2309.08221v1cs.SE

TL;DR

Manual code review is costly, and the effectiveness of ChatGPT for review-based code refinement remains unclear. The paper evaluates ChatGPT against CodeReviewer using existing and newly constructed datasets, finding stronger results for ChatGPT on the new dataset while analyzing failure causes and mitigation strategies.

  • Problem

    Code review requires substantial manual effort, while ChatGPT’s effectiveness for automated refinement from review comments remains unclear.

  • Method

    The paper conducts an empirical study using CodeReview and a newly constructed high-quality dataset, comparing ChatGPT with CodeReviewer and analyzing underperforming cases.

  • Results

    22.78 EM and 76.44 BLEU: ChatGPT exceeds CodeReviewer’s 15.50 EM and 62.88 BLEU on the new dataset.

  • Takeaways & Limitations

    The study identifies underperformance related to domain knowledge, unclear locations, and unclear changes, and suggests improving review quality or using more advanced models.

  • Takeaways & Limitations

    The ChatGPT–CodeReviewer comparison assumes CodeReviewer’s selected hyperparameters achieve its best performance, despite different evaluation settings.

Abstract

from arXiv · show

Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given code reviews. To conduct the study, we select the existing benchmark CodeReview and construct a new code review dataset with high quality. We use CodeReviewer, a state-of-the-art code review tool, as a baseline for comparison with ChatGPT. Our results show that ChatGPT outperforms CodeReviewer in code refinement tasks. Specifically, our results show that ChatGPT achieves higher EM and BLEU scores of 22.78 and 76.44 respectively, while the state-of-the-art method achieves only 15.50 and 62.88 on a high-quality code review dataset. We further identify the root causes for ChatGPT's underperformance and propose several strategies to mitigate these challenges. Our study provides insights into the potential of ChatGPT in automating the code review process, and highlights the potential research directions.

1 INTRODUCTION

This study examines whether ChatGPT can automate code refinement from review comments, addressing the time and effort required for manual code review. It evaluates ChatGPT against established methods, analyzes underperformance, and proposes mitigation strategies.

  • Code review improves software reliability and maintainability but requires substantial manual effort, averaging approximately six hours per developer per week.
  • ChatGPT’s instruction-following ability and compatibility with review comments motivate its use for code refinement.Human reviews can function as prompts for refining code.
  • The study investigates ChatGPT’s comparative performance, underperformance causes, and possible mitigation strategies for review-based code refinement.
  • ChatGPT’s prompt and temperature settings affect Exact Match scores by up to 5% and 15%, respectively, with lower temperatures producing better and more stable results.Including the code review scenario in prompts also improves performance.
  • 22.78 EM and 76.44 BLEU: ChatGPT outperforms CodeReviewer’s 15.50 EM and 62.88 BLEU on the new high-quality dataset.
  • The paper contributes the first empirical study of ChatGPT for review-based code refinement, a high-quality dataset, and analysis of challenges with mitigation strategies.

2 BACKGROUND

The paper frames code refinement as the transformation of original code using natural-language review comments, focusing on an automated single-round review scenario. ChatGPT is positioned as a conversational model that returns revised code and reasoning.

  • A contributor submits original code, receives natural-language review comments, and produces revised code whose difference from the original is the code change.
  • The study focuses on a single review round in which models generate revised code from the review comment and original code.
  • ChatGPT is a GPT-3.5-series large language model trained with reinforcement learning from human feedback.
  • The proposed conversational approach supplies original code and review comments to ChatGPT and obtains revised code with reasoning about the modifications.

3 STUDY DESIGN

The study compares ChatGPT with CodeReviewer across existing and newly constructed code-review datasets while testing configuration effects, strengths, weaknesses, causes of failures, and mitigation strategies.

  • Study overview: The study combines the existing CodeReview benchmark with a new CodeReview-New dataset designed to address quality and possible training-data overlap concerns.
  • Research questions: RQ1 evaluates five prompts and five temperatures from 0 to 2 to measure how ChatGPT settings affect code-refinement performance.
  • Research questions: RQ2 compares ChatGPT with the state-of-the-art CodeReviewer on both CodeReview and CodeReview-New.
  • Research questions: RQ3 manually annotates 200 samples from each dataset by review quality and code-change type to characterize ChatGPT’s strengths and weaknesses.
  • Research questions: RQ4 analyzes 206 failed cases from RQ3 and studies whether better review quality or stronger models can mitigate ChatGPT’s errors.
  • Datasets: CodeReview-New applies strict filtering, including single-hunk code changes and exclusion of non-code changes, to improve dataset quality.
  • Datasets: The new dataset uses reviews collected from January 1, 2022 onward and includes repositories beyond those in CodeReview to reduce training-data overlap.
  • Models: ChatGPT is evaluated with default GPT-3.5-Turbo in a zero-shot setting, while CodeReviewer is fine-tuned using its training and validation data; GPT-4 is additionally used in RQ4.

4.1 RQ1 Impact of Prompts and Temperatures

This section evaluates how prompt design and temperature affect ChatGPT’s code-refinement performance, then selects a configuration for subsequent experiments. Lower temperatures and concise scenario descriptions generally produce better, more stable results, while overly detailed requirements can hurt performance.

  • 4.1.1 Setup.: Temperature controls output randomness and creativity, and the study tested values from 0 to 2 in increments of 0.5.The selected values were 0, 0.5, 1.0, 1.5, and 2.0.
  • 4.1.1 Setup.: ChatGPT was evaluated across five prompts and five temperature settings, using 500 randomly selected CodeReview test samples and ten repetitions per setting.The prompts varied instruction, context, input, output indicators, scenario descriptions, and requirement detail; repeated runs were averaged.
  • 4.1.1 Setup.: Prompt construction ranged from a simplest prompt to versions adding scenario descriptions, detailed requirements, concise requirements, or combinations of these elements.P2 added a scenario description, P3 detailed requirements, P4 concise requirements, and P5 combined scenario and requirement information.
  • 4.1.1 Setup.: Temperature 0 achieved the best performance for every prompt, while performance decreased significantly as temperature increased.The authors attribute the decline to higher-temperature outputs becoming more creative but less reliable for precise code generation.
  • 4.1.2 Results.: Under stable temperatures of 0, 0.5, and 1.0, P2 and P5 significantly outperformed the other prompts, indicating benefits from additional scenario descriptions.P3 performed worse than P4 and sometimes worse than the simplest prompt, P1.
  • 4.1.2 Results.: Additional requirement information helped relative to simpler prompts, but overly complex requirements harmed performance, possibly by making the prompts harder for ChatGPT to understand.P1 achieved higher EM-trim scores than P3 at all three stable temperature settings, while P4 generally outperformed P1.
  • 4.1.2 Results.: A replication on 1,000 training- and validation-set samples produced comparable EM and BLEU performance and reinforced the prompt-and-temperature conclusions.Temperatures above 1.5 were repeated twice; other settings were repeated ten times because of budget constraints.
  • 4.1.2 Results.: Statistical comparisons led the authors to select P2 for later experiments: it significantly outperformed P1, P3, and P4, while its EM-T difference from P5 was not significant.P2 was also significantly better than P5 on BLEU-T.

4.2 RQ2 Effectiveness of ChatGPT

ChatGPT is evaluated against CodeReviewer on CodeReview and newly constructed datasets, with analysis of why the models differ. ChatGPT shows stronger generalization on the unseen dataset but remains imperfect, and it can produce extra explanatory text alongside refined code.

  • Evaluation setup: ChatGPT and CodeReviewer are compared on CodeReview and two CodeReview-New subsets covering same-repository and different-repository reviews.The evaluation uses the best ChatGPT configuration identified earlier and reports sample counts alongside comparative results.
  • Error analysis: 2,283 cases showed correct ChatGPT responses where CodeReviewer failed, and 150 were manually analyzed to identify four root causes.The analysis examined cases from the new dataset in which the two systems differed.
  • Error analysis: CodeReviewer’s errors included inaccurate review understanding and over-deletion, including deletion of correct code or substantial code portions that required preservation.The reported causes include 34 cases of inaccurate review understanding and 62 cases of over-deletion.
  • Interpretation: Different model understanding appears central: CodeReviewer struggles with unclear reviews, whereas ChatGPT more accurately captures their underlying semantics.This interpretation is based on the manual root-cause analysis and the models’ differing behavior on unclear review content.
  • Limitations: ChatGPT often adds explanations around its refinements, helping users assess changes but requiring cleanup before code submission.The paper notes that code blocks enclosed by triple backticks may make this extra-text filtering relatively easy to automate.
  • Limitations: 22.78 EM-trim and 76.55 BLEU-trim show that ChatGPT’s generalization advantage on unseen data still falls short of expected effectiveness.The paper characterizes the remaining performance as limited despite ChatGPT outperforming CodeReviewer on the new dataset.

4.3 RQ3 Strengths and Weaknesses of ChatGPT

The study evaluates ChatGPT across review quality and code-change categories using manual annotations and compares performance across these conditions. ChatGPT performs best with highly relevant, concrete reviews and on refactoring, while documentation-and-code changes are especially difficult.

  • 4.3.1 Setup: The qualitative study manually annotates 400 samples—200 each from CodeReview and CodeReview-New—by review relevance, information level, and code-change category.The annotation study uses a 90% confidence level and a 5.8% confidence interval.
  • 4.3.1 Setup: Review relevance measures correspondence between comments and code changes using Not, Partial, and Perfect levels.Perfect relevance means the code changes strictly follow the review comment with clear correspondence.
  • 4.3.1 Setup: Comment information measures instruction sufficiency and clarity, ranging from vague questions to vague suggestions and concrete suggestions.Concrete suggestions explicitly request code additions or modifications, or identify code to remove.
  • 4.3.1 Setup: CodeReview-New contains more perfectly relevant and fewer irrelevant samples than CodeReview, indicating higher review quality under stricter filtering.It has 150 versus 135 perfectly relevant samples and 21 versus 36 irrelevant samples.
  • Review quality: ChatGPT performs best on perfectly relevant reviews and concrete suggestions, while low relevance and low information provide insufficient context for accurate predictions.Performance is lower for partial or irrelevant reviews and is similar for vague suggestions and vague questions.
  • Code-change categories: ChatGPT performs best on Refactor changes with 37.50% EM-trim and 83.58% BLEU-trim, but worst on Documentation-and-Code changes with 0% EM-trim and 64.09% BLEU-trim.The paper attributes the latter difficulty to simultaneously changing code and documentation while maintaining consistency.
  • Code-change categories: Overall, ChatGPT is strongest on refactoring and weaker on documentation and functionality refinement tasks.This summarizes the reported pattern across code-change categories.

4.4 RQ4 Root Causes Analysis and Mitigation

The analysis attributes ChatGPT’s underperformance mainly to missing domain knowledge, unclear modification locations, and unclear review instructions. Review revision and GPT-4 substantially improve resolution of unclear-location and unclear-change cases.

  • Root causes: 107 cases (51.94%) were attributed to missing domain knowledge, 44 (21.36%) to unclear locations, and 13 (6.31%) to unclear instructions.These were the principal causes within the incorrect-prediction category.
  • Mitigation strategies: The study considers improving review quality and using more advanced models as two mitigation directions.Proposed review improvements include more precise and explicit comments; advanced mitigation such as automatic review refinement remains future work.
  • Mitigation strategies: GPT-3.5 with mitigation resolved 24/32 (75%) unclear-location cases and 6/11 (54.54%) unclear-change cases.The strategies added location information, made comments more explicit, or used GPT-4.
  • Mitigation strategies: GPT-4 with mitigation resolved 31/32 (96.88%) unclear-location cases and 10/11 (90.91%) unclear-change cases.Without mitigation, GPT-4 performed close to GPT-3.5 combined with mitigation techniques.

5 IMPLICATIONS

The study discusses implications for developers and researchers. Developers should configure and validate ChatGPT carefully, while researchers can pursue better models and higher-quality code reviews.

  • Developers: Developers should configure language models carefully, ensure review quality, and validate generated refinements.Lower temperatures and concise prompts containing scenario information produced better and more stable results.
  • Researchers: Researchers can use the identified underperformance causes and mitigation strategies to guide improvements in language models and code reviews.Suggested directions include automatic generation of high-quality reviews, review refinement, and advanced model development.

6 THREATS TO VALIDITY

The study identifies threats involving benchmark and baseline selection, ChatGPT randomness, prompt settings, comparison settings, and manual annotation. The authors apply several mitigations, but some assumptions remain.

  • Study design: Benchmark and baseline selection may threaten validity despite using a state-of-the-art reference and a newly filtered test dataset.The new dataset is described as CRN with stricter filtering rules.
  • Study design: ChatGPT randomness was addressed by running each RQ1 setting ten times, but RQ2 used single runs because API access was costly.The authors present repeated RQ1 runs as providing more reliable and stable results.
  • Study design: Prompt wording and the comparison with CodeReviewer may affect results because optimal settings and hyperparameters were not established equivalently.The study tested prompts with varying complexity and specificity, but assumes CodeReviewer also used its best hyperparameters.
  • Manual annotation: Random sample selection and subjective manual annotation could threaten validity, although independent annotation, conflict resolution, and relatively high Cohen’s Kappa were used.Two co-authors annotated samples independently, with a third author resolving disagreements.

7 RELATED WORK

Prior automated code-review research covers code-change recommendation, review-comment generation, and code refinement. Existing models established strong baselines, but ChatGPT’s RLHF-based design and emergent abilities motivated dedicated evaluation.

  • Pre-trained models: Encoder-only, decoder-only, and encoder-decoder architectures are the main pre-trained model categories discussed for software engineering.GPT-3 is described as a decoder-only model with 175 billion parameters.
  • Automated code review: Automated code review is categorized into code-change recommendation, review-comment generation, and code refinement.The categories correspond to different stages of the code-review process.
  • Prior methods: CodeReviewer uses CodeT5 with four code-review pre-training tasks, and CodeT5 previously outperformed existing techniques in code-change recommendation and code refinement.The comparison includes Trans-Review, AutoTransform, and T5-Review.
  • ChatGPT evaluation: ChatGPT requires dedicated evaluation because RLHF and emergent abilities distinguish it from earlier code-oriented large language models.The paper presents this work as the first comprehensive empirical study of ChatGPT for code refinement.

8 CONCLUSION

The paper empirically investigates ChatGPT for automated code review, focusing on code refinement based on code reviews. It evaluates configurations across standard benchmarks and a newly collected dataset, identifying underperformance causes and potential mitigation strategies.

  • The study examines ChatGPT’s potential for automating code review through code refinement based on code reviews.
  • The evaluation varies ChatGPT configurations across standard code review benchmarks and a newly collected dataset.
  • The findings indicate promising potential for ChatGPT in code refinement while identifying causes of underperformance and potential mitigation strategies.
Loading 2309.08221v1…