PaperlayerView exact revision on arXiv

Source-linked AI summary

Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues

Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D. Le, David Lo

arXiv:2307.12596v2cs.SE

TL;DR

Reliable and high-quality ChatGPT-generated code remains insufficiently characterized despite widespread use. This paper evaluates Java and Python solutions across LeetCode tasks, analyzes their correctness and quality issues, and tests feedback-based repair. The results show functional success alongside frequent quality problems, while self-repair improves code quality only partially.

  • Problem

    Formal evidence about the reliability and quality of ChatGPT-generated code remains limited, despite its use by professional developers, novices, and people without coding experience.

  • Method

    The study evaluates ChatGPT-generated Java and Python code for 2,033 LeetCode tasks, using test suites, static-analysis tools, qualitative categorization, and repair prompts.

  • Results

    66% of Python and 69% of Java programs are functionally correct, while generated code commonly contains wrong outputs, errors, style and maintainability issues, and performance inefficiencies.

  • Takeaways & Limitations

    Generated code requires quality assessment beyond functional testing because apparently correct solutions can still contain maintainability problems and other code-quality issues.

  • Takeaways & Limitations

    The findings may not generalize beyond 2,033 LeetCode tasks or Java and Python, which may not represent real-world software development or other languages.

Abstract

from arXiv · show

We systematically study the quality of 4,066 ChatGPT-generated code implemented in two popular programming languages, i.e., Java and Python, for 2,033 programming tasks. The goal of this work is three folds. First, we analyze the correctness of ChatGPT on code generation tasks and uncover the factors that influence its effectiveness, including task difficulty, programming language, time that tasks are introduced, and program size. Second, we identify and characterize potential issues with the quality of ChatGPT-generated code. Last, we provide insights into how these issues can be mitigated. Experiments highlight that out of 4,066 programs generated by ChatGPT, 2,756 programs are deemed correct, 1,082 programs provide wrong outputs, and 177 programs contain compilation or runtime errors. Additionally, we further analyze other characteristics of the generated code through static analysis tools, such as code style and maintainability, and find that 1,930 ChatGPT-generated code snippets suffer from maintainability issues. Subsequently, we investigate ChatGPT's self-repairing ability and its interaction with static analysis tools to fix the errors uncovered in the previous step. Experiments suggest that ChatGPT can partially address these challenges, improving code quality by more than 20%, but there are still limitations and opportunities for improvement. Overall, our study provides valuable insights into the current limitations of ChatGPT and offers a roadmap for future research and development efforts to enhance the code generation capabilities of AI models like ChatGPT.

1 INTRODUCTION

The paper formally examines the reliability and quality of ChatGPT-generated code, motivated by limited evidence and risks for both experienced and inexperienced users. It evaluates correctness, characterizes quality issues, and studies prompting strategies for mitigation.

  • The study addresses three questions: ChatGPT’s code-generation effectiveness, common generated-code issues, and whether prompting can repair those issues.
  • 2,033 LeetCode programming tasks form the benchmark, with generated Java and Python code evaluated against test suites and analyzed using static-analysis tools.
  • 66% of Python and 69% of Java programs are functionally correct, while performance varies with task difficulty, introduction time, and program size.
  • Performance drops up to five times on programming tasks introduced after January 2022, indicating limited adaptation to newer tasks.
  • 1,930 generated snippets exhibit code style or maintainability issues, including 53% of passing Java code and 37% of passing Python code.
  • The paper contributes a time-sensitive dataset, a characterization of quality issues, and an exploration of ChatGPT’s self-repairing capability.

2 BACKGROUND

ChatGPT’s code-generation promise is illustrated alongside concerns that functionally correct code may still contain smells, bugs, or vulnerabilities. A motivating example shows how apparently straightforward generated code can conceal quality problems.

  • ChatGPT’s training on extensive source-code repositories enables code generation from natural-language requirements, including for users without coding experience.
  • 2.2 Motivation: Figure 1 combines task descriptions, constraints, and templates in prompts for the “Statistics from a Large Sample” problem.
  • 2.2 Motivation: The motivating example demonstrates that ChatGPT-generated code may look correct while containing a bug that is difficult to detect.
  • 2.2 Motivation: A generated ten-line solution for “Frequency of the Most Frequent Element” declares and assigns an unused min_index variable, illustrating a code smell despite functional correctness.

3 STUDY SETUP

The study builds a time-aware LeetCode benchmark and evaluates zero-shot ChatGPT code generation across programming languages, task difficulty, and task introduction periods. The setup addresses limitations of smaller, older benchmarks lacking temporal metadata.

  • 3.2 Constructing Benchmark Dataset: Tasks are distributed across easy, medium, and hard difficulty levels, enabling analysis of performance by task complexity.
  • 3.2 Constructing Benchmark Dataset: The benchmark contains 2,033 LeetCode programming tasks, collected with task descriptions, templates, and public test cases for evaluating generated solutions.
  • 3.2 Constructing Benchmark Dataset: Existing benchmarks such as HumanEval and MBPP are smaller, released before 2021, and lack detailed temporal metadata for modern-model evaluation.
  • 3.2 Constructing Benchmark Dataset: The dataset focuses on Java and Python tasks because both languages are widely used and have large developer communities.
  • 3.3 The ChatGPT Model: ChatGPT-March-23 is evaluated with zero-shot prompting, using a model trained on data up to 2021.

4 RQ1: PERFORMANCE

ChatGPT’s code-generation performance varies significantly with task difficulty, task introduction time, program size, and programming language. Performance declines on harder and newer tasks and as generated programs become longer.

  • Experimental Design: Pass@1 measures whether ChatGPT’s first generated solution passes all test cases, with evaluation using deterministic generation at temperature 0.The study also tests performance differences across factors using the Mann-Whitney U test and Cliff’s Delta effect size.
  • Difficulty and Time: ChatGPT’s performance declines across task difficulty and time period for both Python and Java.Hard tasks had pass rates above half before June 2021 but nearly 0.1 in subsequent periods.
  • Program Size: The pass@1 rate decreases as generated code length increases in both languages.Python declines from 0.872 for programs under 10 lines to 0.265 for programs over 50 lines; Java declines from 0.838 for 10–20 lines to 0.478 for programs over 50 lines.
  • Summary: Task difficulty, introduction time, program size, and programming language significantly and substantially affect ChatGPT’s performance.The study identifies these factors as major influences on code-generation effectiveness.

5 RQ2: BUGS AND ISSUES

The study combines test-case evaluation, static analysis, and qualitative card sorting to characterize code-quality issues in ChatGPT-generated Java and Python programs. Wrong outputs and code style or maintainability issues are most prevalent, while issue patterns vary with task difficulty and programming language.

  • 5.1 Static Analysis: Static analysis supplements test-case results by detecting potential bugs, style issues, and maintainability concerns in functionally passing and failing code.The study uses PMD and Checkstyle for Java, and Pylint and Flake8 for Python, while excluding minor formatting and naming warnings.
  • 5.3 Quantitative Analysis: Clean code is more common among passed tasks than failed tasks, while cleanliness declines as task difficulty increases for both languages.For example, clean Java code falls from 54% for easy tasks to 45% for medium tasks and 33% for hard tasks.
  • 5.2 Open Card Sorting Discussion: Open card sorting categorizes issues using task metadata, generated code, test results, and static-analysis findings from a representative sample of 154 tasks.Cards could receive multiple categories because a single generated program could contain more than one issue.
  • 5.2 Open Card Sorting Discussion: Four issue categories emerge: Compilation & Runtime Errors, Wrong Outputs, Code Style & Maintainability, and Performance & Efficiency.Compilation and runtime errors prevent correct execution, whereas wrong outputs can occur even when code is syntactically correct and runs without errors.
  • 5.3 Quantitative Analysis: 1,082 of 4,066 programs exhibit wrong outputs, and 1,930 of 4,066 exhibit code style or maintainability issues.These are more prevalent than compilation or runtime errors and performance or efficiency issues.
  • 5.3.2 Analysis on Compilation & Runtime Errors: Execution errors reflect language-specific patterns: 45 of 97 Java errors are illegal-index errors, while Python has 27 type-mismatch errors versus 6 in Java.The findings identify distinct error distributions across languages and motivate language-specific debugging and detection approaches.

6 RQ3: REPAIR WITH PROMPTING

The study evaluates prompt-based repair of ChatGPT-generated code using simple feedback and feedback enriched with static-analysis or runtime-error details. ChatGPT repairs some issues, but ambiguous feedback and iterative fixes can limit gains or introduce new problems.

  • 6.1 Repairing Prompt Types: 20% to 60% is the reported range for ChatGPT’s fixed rate across code quality issues.The fixed rate is the proportion of issues resolved when the issue no longer occurs.
  • 6.1 Repairing Prompt Types: Static-analysis and runtime-error feedback is more effective for code style and maintainability, while simple feedback performs better for the remaining issues.The comparison covers both Java and Python.
  • 6.1 Repairing Prompt Types: Simple feedback may be more effective than static-analysis feedback or runtime errors for some issues, possibly because detailed feedback introduces noise.The passage links lower self-repair performance to abstract assertion feedback and limited guidance.
  • 6.1 Repairing Prompt Types: Detailed feedback can effectively assist self-repair, whereas ambiguous feedback may negatively affect ChatGPT’s performance.Runtime feedback such as “TIMEOUT” is often less specific, while static-analysis warnings can identify issue locations and solutions.
  • 6.1 Repairing Prompt Types: ChatGPT can introduce new code quality issues while generating fixes, so repairing one issue does not guarantee an issue-free revision.An example describes simple feedback producing an incorrect replacement solution, while more accurate feedback led to a correct loop change.
  • 6.2 Iterative Repairing: Four iterative feedback rounds were evaluated on 402 LeetCode tasks released after 2022, with iteration 0 representing the original generated code.Iterations 1–4 represent successive feedback rounds, ending when no quality issues remain.
  • 6.2 Iterative Repairing: Pass rates increased clearly during iterations 1 and 2 before stabilizing, demonstrating improved functional correctness through feedback integration.Java generation with simple feedback showed a small decline relative to other scenarios because the feedback lacked specific details.
  • 6.2 Iterative Repairing: Code without style and maintainability issues increased across all four iterative rounds, while overall iterative repairing was particularly effective with detailed feedback.The study summarizes this pattern as iterative repairing improving code quality over successive rounds.

7.1 Enhancing ChatGPT’s Code Generation and Self-Repair Capabilities

The paper examines prompt engineering as a way to improve ChatGPT’s code generation and self-repair. Its findings indicate that prompt quality and specificity influence repair effectiveness, while iterative gains eventually stabilize.

  • 7.1 Enhancing ChatGPT’s Code Generation and Self-Repair Capabilities: Prompt engineering uses carefully designed specialized prompts to improve interactions with large language models in software engineering tasks.Prior work cited in the paper reports benefits from adding examples or explicit semantic facts to prompts.
  • 7.1 Enhancing ChatGPT’s Code Generation and Self-Repair Capabilities: Self-repair effectiveness is significantly influenced by prompt quality and specificity, corroborating prior findings on prompt engineering.The paper reports this influence for ChatGPT’s repair of code issues.
  • 7.1 Enhancing ChatGPT’s Code Generation and Self-Repair Capabilities: Detailed static-analysis feedback and compiler feedback can enhance ChatGPT’s code repair capability over iterative interactions.Repeated interactions allow refinement based on user feedback, but performance stabilizes in later rounds.
  • 7.1 Enhancing ChatGPT’s Code Generation and Self-Repair Capabilities: Future work should establish interaction design patterns and benchmarks to advance conversational code generation efficiency and efficacy.The recommendation follows the observed stabilization of performance in later iterative rounds.

7.2 Lessons Learned

The study identifies recurring quality problems in ChatGPT-generated code and shows that performance depends on task and generation conditions. It also emphasizes feedback tailored to the language and issue type.

  • 7.2 Lessons Learned: ChatGPT-generated code exhibits compilation and runtime errors, wrong outputs, and maintainability problems.The study presents these issues as prevalent in AI-generated code.
  • 7.2 Lessons Learned: Task difficulty, task-established time, and program size significantly influence ChatGPT’s code-generation performance.The paper suggests that model improvements should account for these factors across task types.
  • 7.2 Lessons Learned: Self-repair effectiveness depends on feedback type, programming language, and the specific code quality issue.Static-analysis feedback works better for style and maintainability, while simple feedback is more effective for other issues.
  • 7.2 Lessons Learned: Further prompt optimization could improve the accuracy and reliability of ChatGPT-generated code.The paper identifies tailored feedback and prompt engineering as important for self-repair and code-generation quality.

7.3 Threats to Validity

The paper identifies threats to generalizability, experimental validity, and evaluation validity. These limitations arise from the task and language selection, possible analysis errors, test-suite incompleteness, and prompt variability.

  • 7.3 Threats to Validity: The study’s findings may not generalize beyond 2,033 LeetCode tasks, Java and Python, or the software-project types represented by that dataset.The authors suggest broader task sources, languages, and project types as future mitigations.
  • 7.3 Threats to Validity: Manual analysis and categorization may introduce bias, although three annotators conducted a sorting discussion to mitigate this risk.The authors also released analysis and categorization results for public verification.
  • 7.3 Threats to Validity: The study uses temperature 0 to reduce ChatGPT output variability and improve internal validity.This setting is intended to produce consistent outputs for the same input.
  • 7.3 Threats to Validity: Pass@1 treats a program as functionally correct when it passes all test cases, but incomplete test suites could miss bugs.The authors use LeetCode’s original test suite and consider this risk minimal.
  • 7.3 Threats to Validity: Different prompts can change generated-code quality, so consistent prompts were used across tasks while prompt optimization remains future work.The paper identifies prompt engineering as a construct-validity concern.

8 RELATED WORK

Prior work established strong LLM-based code-generation effectiveness, but ChatGPT’s generated-code quality remained a critical concern. This study addresses gaps in time-sensitive evaluation, systematic quality analysis, and repair strategies.

  • LLM-based approaches have advanced program understanding, analysis, and generation, with CodeX initiating a new era of LLM-based code generation.
  • ChatGPT generates code and solves programming problems with remarkable accuracy, yet generated-code quality remains a critical adoption concern for software companies.
  • This paper conducts the first time-sensitive ChatGPT code-generation evaluation and systematically analyzes quality issues and potential repair solutions.
  • Existing code-quality studies largely examine human-written code rather than AI-generated code.
  • Prior evaluations examined CodeX and GitHub Copilot on relatively few programming tasks, leaving ChatGPT less systematically studied.

9 CONCLUSION

The study finds that ChatGPT can generate functional code but often produces compilation or runtime errors, wrong outputs, maintainability problems, and performance inefficiencies. It also examines self-repair and feedback strategies, emphasizing context-aware feedback and code-quality awareness for more reliable AI-generated code.

  • ChatGPT can generate functional code for varied programming tasks, but its outputs often contain compilation errors, wrong outputs, maintainability problems, and performance inefficiencies.
  • The study explores ChatGPT’s self-repairing capabilities and evaluates feedback strategies for addressing generated-code quality issues.
  • The findings highlight the importance of context-aware feedback and code-quality considerations when using AI-driven code-generation tools.
  • The research offers direction for future efforts to enhance the reliability, efficiency, and maintainability of AI-generated code.
Loading 2307.12596v2…