Source-linked AI summary
Is ChatGPT the Ultimate Programming Assistant -- How far is it?
Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, Tegawendé F. Bissyandé
TL;DR
The paper examines whether ChatGPT can be a useful programming assistant when evaluated on realistic, potentially unseen problems rather than possibly leaked benchmarks. It empirically studies code generation, program repair, and code summarization across two benchmarks, comparing ChatGPT with prior approaches. ChatGPT performs effectively on common programming problems, while struggling to generalize to new and unseen problems and showing sensitivity to prompt length.
Problem
Existing ChatGPT programming studies often rely on older public benchmarks that may have appeared in its training data, leaving its realistic capability on unseen problems insufficiently scoped.
Method
The study evaluates ChatGPT on code generation, program repair, and code summarization using two benchmarks, practical prompts, repeated requests, and comparisons with state-of-the-art approaches.
Results
ChatGPT performs effectively on common programming problems and dominates prior LLMs in code generation, but struggles to generalize to new and unseen problems.
Takeaways & Limitations
The findings provide actionable insights for adopting LLMs as programming assistants and for understanding ChatGPT’s practical software-engineering applications.
Takeaways & Limitations
Benchmark selection remains a threat to external validity, and ChatGPT’s random responses require repeated requests to reduce misleading conclusions.
Abstract
from arXiv · showhide
Recently, the ChatGPT LLM has received great attention: it can be used as a bot for discussing source code, prompting it to suggest changes, provide descriptions or even generate code. Typical demonstrations generally focus on existing benchmarks, which may have been used in model training (i.e., data leakage). To assess the feasibility of using an LLM as a useful assistant bot for programmers, we must assess its realistic capabilities on unseen problems as well as its capabilities on various tasks. In this paper, we present an empirical study of ChatGPT's potential as a fully automated programming assistant, focusing on the tasks of code generation, program repair, and code summariziation. The study investigates ChatGPT's performance on common programming problems and compares it with state-of-the-art approaches on two benchmarks. Among several findings, our study shows that ChatGPT is effective in dealing with common programming problems. However, our experiments also reveal limitations in terms of its attention span: detailed descriptions will constrain the focus of ChatGPT and prevent it from leveraging its vast knowledge to solve the actual problem. Surprisingly, we have identified the ability of ChatGPT to reason the original intention of the code. We expect future work to build on this insight for dealing with the open question of the oracle problem. Our findings contribute interesting insights to the development of LLMs for programming assistance, notably by demonstrating the importance of prompt engineering, and providing a better understanding of ChatGPT's practical applications for software engineering.
1 Introduction
AI and LLMs are being applied to automate code generation, program repair, and code summarization, but ChatGPT requires realistic evaluation on unseen programming problems. This study evaluates those tasks on common problems and benchmarks to characterize practical programming-assistant capabilities.
- AI techniques automate code generation, program repair, and code summarization to support programming efficiency, debugging, comprehension, maintainability, and reuse.
- Existing ChatGPT repair studies often use pre-2022 public benchmarks, which may have leaked into training and limit generalizability to unseen problems.
- The study evaluates ChatGPT as a fully automated programming assistant across code generation, program repair, and code summarization.
- The experiments target common programming problems and compare ChatGPT with state-of-the-art approaches on two benchmarks.
- The paper reports findings and actionable insights about ChatGPT’s performance and practical applications for software engineering.
2 Background and Related Work
The background describes LLM architectures and ChatGPT, then reviews code generation, program repair, and code summarization as established software-engineering tasks. Prior approaches include pretrained, neural, structural, and generative methods for producing or transforming code and explanations.
- Large-scale Language Models and ChatGPT: BERT uses masked language modeling in a transformer architecture to learn context-aware word representations for downstream software-engineering tasks.
- Large-scale Language Models and ChatGPT: GPT models use unidirectional causal prediction, with later versions scaling parameters and training data to generate context-aware content.
- Large-scale Language Models and ChatGPT: ChatGPT is a conversational GPT-3.5 and GPT-4 chatbot trained on extensive text and code, with instruction fine-tuning and RLHF for task adaptation.
- Code-related Tasks: Code generation creates complete programs or code snippets from higher-level descriptions such as natural language, models, or specifications.
- Code generation: Prior code-generation systems include PLBART, InCoder, CodeGen, and GitHub Copilot, spanning sequence-to-sequence, infilling, large-scale generation, and code-completion approaches.
- Code-related Tasks: Program repair automatically identifies and fixes software bugs or vulnerabilities, while code summarization generates concise explanations of code or functionality.
3 Study Design
The study evaluates ChatGPT as an automated programming assistant across code generation, program repair, and code summarization using two benchmarks and multiple experimental protocols. It addresses common programming problems, benchmark selection, data leakage, prompting, and response variability.
- Research questions: The study asks whether ChatGPT can generate correct and efficient code, repair diverse buggy programs, and identify code intention, including when code contains errors.
- Benchmark selection criteria: The evaluation uses Python because ChatGPT and descendant GPT-3 models are reported as especially capable in Python, while the study requires diverse programming problems and implementations.
- Datasets: The two benchmarks are LeetCode, covering diverse problems with difficulty levels and test cases, and Refactory, containing 2,442 correct and 1,783 buggy Python programs from five assignments.Refactory also provides instructor-designed tests, reference solutions, and assignment descriptions.
- Datasets: To mitigate possible data leakage, the study evaluates LeetCode problems from 2016–2020 and 2022 rather than constructing entirely new programming problems.The authors note that existing online information remains important for training language models.
- Application of ChatGPT: ChatGPT is applied through prompts for code generation and repair, with GPT-3.5 selected for experimentation and five independent requests used for each prompt.Performance is summarized with TOP-5, whether any of five attempts succeeds, and AVG-5, the average success rate across five attempts.
4 Experiments & Results
The experiments evaluate ChatGPT across code generation, program repair, and code summarization, using common programming problems and benchmark comparisons. Results show strengths on easier or well-specified tasks, but reduced generalization to unseen difficult problems and sensitivity to prompt length and code structure.
- Experimental scope: ChatGPT’s experiments cover code generation, program repair, and code-intention explanation across common programming problems and benchmark datasets.The study examines generation and repair performance, then evaluates whether ChatGPT can explain the intention of correct and incorrect code.
- Code generation: ChatGPT-generated correct code had a better overall time-complexity rank percentile than Codex-generated correct code, with both generally in the top 50%.The reported rank scores use lower values as better performance.
- Code generation: 33/40 easy problems were solved on the LeetCode 2022 dataset, while only 2 hard problems were solved.Correctness and time-complexity performance declined across difficulty levels, with larger reductions as difficulty increased.
- Code generation: Correct predictions used relatively shorter prompts than incorrect predictions for ChatGPT and Codex.The analysis excludes problem levels with too few solved cases for statistical significance and recommends clear, concise prompts.
- Code summarization: ChatGPT can infer the intentions of many incorrect programs similarly to correct programs and their problem descriptions, but multi-function code and minimal return-only implementations reduce similarity.For Unique Dates Months, intention identification succeeded for unique_day but not for unique_month and contain_unique_days.
5 Threats to validity
The study addresses validity threats from ChatGPT’s response randomness, benchmark selection, and extra natural-language output contaminating requested code.
- Threats to External Validity: ChatGPT’s response randomness can make conclusions based on a single request misleading.The authors requested five results to compute TOP-5 and AVG-5 metrics.
- Threats to External Validity: The study mitigates randomness partly by using five requests and a large repair dataset of 1783 code samples.The authors compare TOP-5 and AVG-5 performance metrics against state-of-the-art approaches.
- Threats to External Validity: Benchmark selection may threaten validity because available large datasets and potential data leakage constrain the choices.The benchmarks were selected to address both dataset availability and leakage prevention.
- Threats to Internal Validity: ChatGPT sometimes adds natural-language explanations despite requests for plain code, causing outputs to fail test suites.This processing issue is identified as a major internal-validity threat.
6 Conclusion
The study evaluates ChatGPT as a programming assistant across code generation, program repair, and code summarization. It finds strong performance on common problems but weaker generalization to unseen problems and sensitivity to prompt length, with implications for future software-engineering research.
- 6 Conclusion: The study evaluates ChatGPT on code generation, program repair, and code summarization as an assistant for programmers.Code generation uses 120 common LeetCode problems from the 2016–2020 and 2022 datasets.
- 6 Conclusion: ChatGPT dominates two prior LLMs on sampled code-generation problems but struggles to generalize to new and unseen problems.The conclusion reports both strong benchmark performance and limited generalization.
- 6 Conclusion: ChatGPT achieves competitive program-repair results against Refactory, a state-of-the-art semantic-based assignments repair tool.Generic problem descriptions unrelated to bug information make ChatGPT perform worse.
- 6 Conclusion: The findings are expected to impact software-engineering research and advance assistive technology for programmers.The authors specifically highlight actionable insights for further research.
- 6 Conclusion: All study artefacts are available in a public repository.The paper provides the repository URL for the released artefacts.