Source-linked AI summary
Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT
Burak Yetiştiren, Işık Özsoy, Miray Ayerdem, Eray Tüzün
TL;DR
This study addresses the uncertain value of AI-assisted code generators by comparing their generated code across multiple quality dimensions. Using HumanEval and defined code-quality assessments, it finds ChatGPT most successful overall, while all tools perform worse without docstrings.
Problem
The value of AI-assisted code generators remains undetermined despite their popularity and potential, motivating systematic comparison across code-quality dimensions.
Method
The study compares GitHub Copilot, Amazon CodeWhisperer, and ChatGPT using 164 HumanEval problems and evaluates validity, correctness, security, reliability, and maintainability.
Results
ChatGPT was the most successful tool, generating correct solutions for 65.2% of HumanEval problems, while all tools produced fewer correct answers when docstrings were removed.
Takeaways & Limitations
The comparison provides practitioners with evidence about the strengths and weaknesses of these generators when selecting tools for coding tasks.
Takeaways & Limitations
The evaluation covers 164 HumanEval problems, and broader problem topics and a larger dataset could improve the study’s coverage.
Abstract
from arXiv · showhide
Context: AI-assisted code generation tools have become increasingly prevalent in software engineering, offering the ability to generate code from natural language prompts or partial code inputs. Notable examples of these tools include GitHub Copilot, Amazon CodeWhisperer, and OpenAI's ChatGPT. Objective: This study aims to compare the performance of these prominent code generation tools in terms of code quality metrics, such as Code Validity, Code Correctness, Code Security, Code Reliability, and Code Maintainability, to identify their strengths and shortcomings. Method: We assess the code generation capabilities of GitHub Copilot, Amazon CodeWhisperer, and ChatGPT using the benchmark HumanEval Dataset. The generated code is then evaluated based on the proposed code quality metrics. Results: Our analysis reveals that the latest versions of ChatGPT, GitHub Copilot, and Amazon CodeWhisperer generate correct code 65.2%, 46.3%, and 31.1% of the time, respectively. In comparison, the newer versions of GitHub CoPilot and Amazon CodeWhisperer showed improvement rates of 18% for GitHub Copilot and 7% for Amazon CodeWhisperer. The average technical debt, considering code smells, was found to be 8.9 minutes for ChatGPT, 9.1 minutes for GitHub Copilot, and 5.6 minutes for Amazon CodeWhisperer. Conclusions: This study highlights the strengths and weaknesses of some of the most popular code generation tools, providing valuable insights for practitioners. By comparing these generators, our results may assist practitioners in selecting the optimal tool for specific tasks, enhancing their decision-making process.
1 Introduction
The study addresses uncertainty about the value of AI-assisted code generators by systematically comparing GitHub Copilot, Amazon CodeWhisperer, and ChatGPT across five code-quality dimensions.
- The study evaluates code generators using Code Validity, Code Correctness, Code Security, Code Reliability, and Code Maintainability.
- The researchers compare GitHub Copilot, Amazon CodeWhisperer, and ChatGPT to assess generated-code quality.
- The research questions examine validity, correctness, security, reliability, maintainability, input effects, and tool evolution over time.
- The study aims to help users leverage code generators and help developers identify strengths and weaknesses relative to competing tools.
2 Background
The background distinguishes IDE-integrated code-generation tools and ChatGPT while describing their supported environments, inputs, and user-facing capabilities.
- GitHub Copilot: GitHub Copilot uses an IDE integration and the OpenAI Codex model to generate code.
- Amazon CodeWhisperer: Amazon CodeWhisperer generates recommendations from English comments and prior IDE code using models trained on Amazon and open-source data.
- Amazon CodeWhisperer: CodeWhisperer supports multiple IDEs and languages, includes a reference tracker, and can scan code for security issues.
- ChatGPT: ChatGPT is a language model capable of generating human-like responses, answering varied questions, admitting mistakes, challenging premises, and rejecting inappropriate requests.
- Tool comparison: The authors observed that Copilot and CodeWhisperer support IDEs and local files, while ChatGPT mainly provides one suggestion and cannot access local files.
3 Methodology
The study evaluates three code-generation tools on 164 HumanEval Python problems, using execution-based tests and SonarQube metrics, with additional experiments on input quality and tool versions.
- Dataset: The experiment uses HumanEval’s 164 Python problems, each containing a task ID, prompt, canonical solution, and unit tests.
- Experimental workflow: The researchers pass function prototypes and docstrings to the tools, then execute generated solutions against the associated tests.
- Code metrics: Code validity is binary and reflects syntax or execution errors, while correctness is measured as passed unit tests divided by all tests.
- Code metrics: Average code correctness is the sum of problem-level correctness scores divided by the problem count, with invalid generations scored as zero.
- Code metrics: SonarQube measures security vulnerabilities, reliability bugs, and maintainability through code smells and their associated technical debt.
- Additional experiments: Additional experiments remove docstrings, replace meaningful function names with “foo,” and compare different tool versions.
4 Results
ChatGPT produced the highest correctness rate, while validity was high for all three tools; the study also reports security, reliability, maintainability, input-quality, and version-comparison findings.
- Code Validity: 93.3% of ChatGPT generations, 91.5% of GitHub Copilot generations, and 90.2% of Amazon CodeWhisperer generations were valid.
- Code Correctness: 31.1% of Amazon CodeWhisperer generations were correct, 28.7% were incorrect, and 40.2% were partially correct.
- Code Security: The dataset produced limited security findings, and the authors state that broader problems and longer solutions could yield better security results.
- Code Maintainability: Average technical debt from code smells was 9.1 minutes for GitHub Copilot, 5.6 minutes for Amazon CodeWhisperer, and 8.9 minutes for ChatGPT.
- Tool evolution: Compared with older versions, GitHub Copilot’s correct rate rose from 28.7% to 46.3%, while CodeWhisperer’s rose from 24.4% to 31.1%.
5 Discussion
The tools generally produce valid code, but correctness varies: partially correct outputs are common, and performance depends on prompt complexity and practitioner feedback. Across security, reliability, and maintainability, the study identifies distinct strengths and weaknesses rather than a single superior generator.
- Code Validity: 93.3% of 164 problems produced valid code with ChatGPT, compared with 91.5% for GitHub Copilot and 90.2% for Amazon CodeWhisperer.
- Code Correctness: Partially correct generations occurred for 23.2% of GitHub Copilot problems, 40.2% of Amazon CodeWhisperer problems, and 22.6% of ChatGPT problems.
- Code Correctness: 60.4% of GitHub Copilot’s partially correct generations, 53.1% of Amazon CodeWhisperer’s, and 63.2% of ChatGPT’s exceeded 50.0% correctness.
- Code Correctness: 36 unique problems were solved correctly by ChatGPT, compared with 15 by GitHub Copilot and two by Amazon CodeWhisperer.
- Code Correctness: Continuous practitioner input is needed for better correctness, while longer and more complex prompts yielded lower Code Correctness scores.
- Security, Reliability, and Maintainability: All generators received the maximum security rating, but reliability weaknesses included unique bugs and maintainability smells such as improper naming and high cognitive complexity.The study therefore could not establish one generator’s superiority in reliable code, while all generators exhibited maintainability issues.
6 Threats to Validity
The study identifies threats arising from test coverage, tooling and prompting choices, nondeterministic and changing generators, evaluation design, and dataset and IDE scope.
- Conclusion Validity: Trivial outputs such as empty arrays or Boolean values can pass tests by chance without implementing the intended algorithm.This threat affects correctness assessments when such expressions happen to match expected outputs.
- Conclusion Validity: Test-case coverage may miss corner cases because HumanEval problems average 7.7 tests, with varying amounts across problems.The authors plan to improve test quantity and quality.
- Conclusion Validity: SonarQube found scarce vulnerability information, possibly because HumanEval solutions contain only a small number of lines.The same constraint may affect the security, maintainability, and reliability measurements.
- Internal Validity: One-shot prompting used function metadata, docstrings, and limited tests, so additional code-snippet hints could increase success rates.The experiment did not provide further intent-clarifying code to the generators.
- Internal Validity: The generators were nondeterministic and dynamically retrained, so later runs may produce different outputs and may not fully replicate these results.The authors generated each problem once and evaluated the saved code.
- Internal Validity: Copilot and CodeWhisperer can generate code automatically or through explicit commands, and the two methods produced different results for the same context.The authors avoided the latter method where possible to maintain consistency.
- Internal Validity: Line-by-line generation sometimes allowed previously generated lines to affect subsequent suggestions, while the tools controlled whether code was generated in blocks or lines.The experiment accepted the generation method selected by the tools.
- External Validity: The study used January 2023 tool versions, and later evaluations may differ because generator versions and underlying models change.Amazon did not provide a CodeWhisperer version number.
7 Related Work
Related work has largely evaluated GitHub Copilot or Codex for correctness, with fewer studies of ChatGPT and Amazon CodeWhisperer. This study addresses that gap by comparing three generators across correctness, validity, security, maintainability, and reliability.
- GitHub Copilot: GitHub Copilot research includes benchmark correctness comparisons, security analysis, user studies, and evaluations across multiple languages and understandability measures.Reported findings include no significant speed or success-rate improvement in one user study and 40% vulnerable generated programs in another.
- GitHub Copilot: Mastropaolo et al. studied whether semantic-preserving changes to natural-language Java method descriptions affect GitHub Copilot’s generated code.They compared generations from original, manually paraphrased, and automatically paraphrased descriptions.
- Research Landscape: Studies of ChatGPT and Amazon CodeWhisperer remain comparatively scarce, with CodeWhisperer having no comparable prior studies identified by the authors.The authors describe both tools as relatively new.
- ChatGPT: ChatGPT-related work examined bug fixing and software-testing responses, reporting performance similar to Codex on QuixBugs and 44% correct or partly correct testing responses.The testing study also reported 57% correct or partially correct explanations.
- Research Landscape: Prior research primarily assessed GitHub Copilot or Codex, usually emphasizing code correctness rather than multiple quality dimensions.Exceptions focused on security or practical usage performance.
- This Study: The present study compares GitHub Copilot, Amazon CodeWhisperer, and ChatGPT across code correctness, validity, security, maintainability, and reliability.The authors characterize it as the first study to compare these tools across all five dimensions.
8 Conclusion
The study compared ChatGPT, GitHub Copilot, and Amazon CodeWhisperer across multiple code-quality dimensions using HumanEval. ChatGPT achieved the highest success rate, while input quality and tool version influenced performance.
- ChatGPT generated correct solutions for 65.2% of HumanEval problems, with 22.6% partially correct and 12.2% incorrect.
- 9.1 minutes, 5.6 minutes, and 8.9 minutes were the average times to eliminate code smells in GitHub Copilot, Amazon CodeWhisperer, and ChatGPT outputs, respectively.
- When only function names and parameters were provided, all three tools produced lower percentages of correct answers than in their initial setup.ChatGPT and GitHub Copilot produced correct answers for 20%-22% of problems, while incorrect answers occurred for 50%-53%.
- With dummy function names, ChatGPT generated correct solutions for 61.6% of the examined problems, alongside 25.6% partially correct and 12.8% incorrect solutions.
- ChatGPT was the most successful tool and Amazon CodeWhisperer the least successful, while clearer problem descriptions improved the observed outcomes.The study also observed rapid improvement in Amazon CodeWhisperer and GitHub Copilot across newer versions.
- The study contributed a comparative analysis, a HumanEval-based code-quality assessment pipeline, and analyses of version-related performance improvements.The study’s data and code are available in a repository.