PaperlayerView exact revision on arXiv

Source-linked AI summary

No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT

Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, Liang Feng Zhang

arXiv:2308.04838v2cs.SE

TL;DR

The paper systematically assesses ChatGPT-generated code across functional correctness, complexity, and security, including multi-round fixing. It finds stronger correctness on pre-2021 problems than later ones, weak direct fixing, generally preserved or increased complexity during fixing, and substantial vulnerability remediation.

  • Problem

    Assessing generated code remains essential because LLM-based code generation promises productivity gains while posing unresolved quality challenges.

  • Method

    The study prompts ChatGPT with LeetCode problems and CWE scenarios, extracts generated code, and tests correctness or vulnerabilities while evaluating complexity and multi-round fixing.

  • Results

    ChatGPT generated more functionally correct code for pre-2021 than post-2021 problems by 48.14% in average Accepted rate, while fixing was relatively weak and complexity generally stayed the same or increased.

  • Takeaways & Limitations

    ChatGPT-based code generation requires assessment across correctness, complexity, and security because fixing performance and generated-code quality vary across these dimensions.

  • Takeaways & Limitations

    The evaluation may not represent real-world coding because LeetCode problems are designed for practice and interviews and may omit domain-specific requirements and constraints.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated impressive capabilities across various NLP tasks. Additionally, LLMs are also highly valuable in supporting software engineering tasks, particularly in the field of code generation. Automatic code generation is a process of automatically generating source code or executable code based on given specifications or requirements, improving developer productivity. In this study, we perform a systematic empirical assessment to the quality of code generation using ChatGPT. We leverage 728 algorithm problems in five languages (i.e., C, C++, Java, Python, and JavaScript) and 18 CWEs with 54 code scenarios for the code generation task. Our evaluation encompasses a comprehensive analysis of code snippets generated by ChatGPT, focusing on three critical aspects: correctness, complexity, and security. We also specifically investigate ChatGPT's ability to engage in multi-round fixing process (i.e., ChatGPT's dialog ability) of facilitating code generation. By delving into the generated code and examining the experimental results, this work provides valuable insights into the performance of ChatGPT in tackling code generation tasks over the three critical aspects. Overall, our findings uncover potential issues and limitations that arise in the ChatGPT-based code generation and lay the groundwork for improving AI and LLM-based code generation techniques.

1 INTRODUCTION

The study motivates empirical assessment of ChatGPT-generated code because correctness, complexity, and security are not guaranteed. Its reported results show substantial variation across problem eras, languages, fixing rounds, vulnerabilities, and repeated generations.

  • Automatic code generation can improve developer productivity and reduce development time by generating source or executable code from specifications.
  • ChatGPT code generation requires assessment because generated code may be functionally incorrect, complex, or insecure.
  • 48.14% advantage in Accepted rate distinguishes ChatGPT’s results for problems before 2021 from problems after 2021 across different languages.
  • ChatGPT’s multi-round fixing generally preserves or increases code complexity, while its direct correction of erroneous functionality is relatively weak.
  • The study uses ChatGPT GPT-3.5 rather than GPT-4, limiting direct interpretation of its results for GPT-4.
  • More than 89% of vulnerabilities were successfully addressed during multi-round fixing, although vulnerabilities appeared in several algorithm and CWE language scenarios.

2 BACKGROUND

The paper describes LLMs and ChatGPT as language models capable of processing and generating human-like text, including code. ChatGPT’s dialog ability lets users refine generated code through conversation context and follow-up prompts.

  • LLMs use large-scale language datasets and deep-learning Transformer architectures to learn language patterns, relationships, and structures.
  • ChatGPT is implemented atop GPT-3.5 or GPT-4 and uses instruction tuning and reinforcement learning from human feedback to align with human expression.
  • Conversation history includes prompts, system instructions, and previous responses, enabling more coherent and informed responses during ongoing dialogue.
  • For code generation, users can submit multiple prompts to ask ChatGPT to fix functionally incorrect code after an initial response.

3 EMPIRICAL STUDY SETUP

The study evaluates ChatGPT code generation using LeetCode problems and CWE scenarios, with a workflow that generates, tests, and optionally revises code through dialog. Prompts are designed to simulate reasonable real-world usage rather than maximize performance.

  • 3.1 Data Collection: The evaluation combines LeetCode algorithm problems with CWE code scenarios to assess ChatGPT-generated code.
  • 3.1 Data Collection: A bubble-sort prompt illustrates the study’s code-generation interaction, in which ChatGPT produces Python code from a natural-language requirement.
  • 3.1 Data Collection: 728 LeetCode problems are divided into 354 published after 2021 and 374 published before 2021 because ChatGPT was trained on pre-2021 text data.
  • 3.1 Data Collection: The CWE dataset contains 18 CWEs with 54 scenarios, complementing LeetCode’s limited coverage of vulnerability classes.
  • 3.2 Methodology: The workflow sends a constructed prompt to ChatGPT, extracts code, tests correctness or vulnerabilities, and uses feedback for another round when testing fails.
  • 3.2 Methodology: The multi-round process uses conversation context and a round limit, while prompt design aims to provide sufficient information without optimizing for a specific dataset.
  • 3.2 Methodology: Token-limit handling adds necessary information and removes as little earlier dialog content as possible while preserving response space.

RQ1: Is the code generated by ChatGPT functionally correct?

The study evaluates ChatGPT’s functional code correctness on sampled LeetCode algorithm problems, separating problems before and after 2021 and testing generated submissions by judgment status. ChatGPT performs substantially better on pre-2021 problems, while correctness declines with difficulty and compile or runtime errors remain common.

  • Dataset and evaluation: 728 sampled LeetCode algorithm problems are divided into 374 before-2021 and 354 after-2021 problems, with matched difficulty distributions.Each generated solution is evaluated in five languages using LeetCode’s automatic judgment statuses.
  • Error patterns: 58% is the highest Wrong Answer rate for after-2021 problems, while Compile Error and Runtime Error rates each reach 16%.Wrong-answer snippets pass only 25% of their average 109 test cases; runtime errors include null dereferences, out-of-bounds access, and heap-buffer overflows.
  • Overall correctness: 15.38%, 19.37%, 20.17%, 23.93%, and 22.51% are the Accepted rates for C, C++, Java, Python3, and JavaScript on after-2021 problems.For these problems, acceptance is lower than for before-2021 problems across all listed languages.
  • Difficulty and temporal effects: 40.13%, 70.95%, and 89.80% are the Accepted rates for hard, medium, and easy before-2021 problems, respectively.The corresponding after-2021 acceptance rates are substantially lower, especially for medium and easy problems.
  • Overall correctness: 48.14% higher Accepted rate is achieved for before-2021 than after-2021 problems across languages.The authors attribute the split to ChatGPT’s training data being from before 2021.
  • Language effects: C++ at 44.75%, Java at 48.74%, Python3 at 50.00%, and JavaScript at 48.80% outperform C at 31.28% in overall Accepted rate.The authors report similar probabilities of compile or runtime errors across the problem periods, averaging 14.23%.

RQ2: How effective is the multi-round fixing process in improving code generation for functional correctness?

Multi-round fixing improves some generated code, but its effectiveness varies sharply by error type and problem difficulty. ChatGPT often preserves or transforms errors rather than reaching functionally correct code, especially when fixes require logical or algorithmic reasoning.

  • Wrong Answer: 157 Wrong Answer problem-language pairs were analyzed, but only 25 were fixed within five rounds; fewer than half of all pairs were fixed.Twelve of 13 easy problems were fixed, whereas medium and hard problems had fix rates below 30%.
  • Wrong Answer: Only 2 of 10 additionally tested unfixed Wrong Answer pairs were fixed within 10 rounds, while 8 still failed.The authors attribute persistent failures to difficulty understanding logical details and solving problems requiring complex reasoning.
  • Wrong Answer: Wrong Answer defects mainly involved M-L and Misaligned Algorithm subclasses, reflecting failures to grasp logical details or handle complex reasoning.The study categorizes Wrong Answer cases as WD, MCC, and MP from a logic perspective.
  • Compile Error: Most compile-error snippets could be fixed, with 70% fixed overall and 26% fixed to accepted code, but many compile errors were platform-related rather than genuine code failures.Constant-function errors accounted for 159/314 compile errors, while wrong method names, main redefinitions, and incompatible parameter types reflected LeetCode settings.
  • Runtime Error: ChatGPT fixed most runtime-error snippets, reaching 78% fixed and 28% accepted, but remaining failures were mainly tied to algorithm implementation details.The runtime-error analysis identifies overflow as predominant in static languages and type errors as predominant in dynamic languages.

RQ3: How complex is the code generated by ChatGPT?

ChatGPT-generated code has language- and difficulty-dependent complexity, with C generally most complex and Python3 least complex. Across difficulty levels, low complexity declines while high and very high complexity rise, and multi-round fixes often preserve or increase complexity.

  • Measurement: SonarQube and cccc measure cyclomatic and cognitive complexity for generated code before and after multi-round fixing.Cognitive complexity is reported for Java, Python3, and JavaScript because of tool-support limitations.
  • Complexity measures: Cyclomatic complexity counts linearly independent paths, while cognitive complexity measures how difficult code is to understand and reason about.The study evaluates both metrics because complexity affects readability, maintainability, testing difficulty, and potential errors.
  • Human comparison: Human-written code has more low complexity than generated code in Python3, while generated and human-written distributions otherwise closely resemble each other in C++ and Python3.For Python3, human-written code has 42.6% low complexity, six percentage points higher than generated code.
  • Difficulty effects: Low complexity decreases while high and very high complexity increase as problem difficulty rises, with moderate complexity changing little.The trend in ChatGPT-generated code is comparable to the trend observed in human-written code.
  • Multi-round fixing: More than half of complexity comparisons preserve the initial level during multi-round fixing, while increases generally occur more often than corresponding decreases.Fixes include changing recursive DFS to iterative DFS, modifying conditions, correcting type errors, and replacing algorithms such as brute force with dynamic programming.
  • Language differences: C is the most complex language, Python3 the least complex, and C++, Java, and JavaScript have comparable complexity distributions.Python3 has 44.5% low-complexity code and 8.5% very-high-complexity code, while C has the lowest combined low and moderate cyclomatic complexity at 37%.

RQ4: Is the code generated by ChatGPT secure?

ChatGPT produces mostly valid code, but substantial vulnerabilities remain, especially in C and in several security categories. Multi-round fixing addresses many vulnerabilities, although performance varies sharply by CWE and scenario.

  • Specific weaknesses: Missing NULL tests accounted for 91.8% of the remaining vulnerability queries, particularly in C and C++ code.Other queries occurred less frequently, but the authors still identify them as significant.
  • Overall security results: 2,983 valid snippets achieved a 99.07% average validity rate, and 994 of them, or 33.32%, were vulnerable.The valid snippets comprised 1,402 C snippets and 1,581 Python3 snippets.
  • Language differences: C contained 51.64% vulnerable valid snippets, compared with 17.08% in Python3.The authors report that this difference indicates greater security awareness is needed for ChatGPT-generated C code than Python3 code.
  • Vulnerability groups: Vulnerabilities appeared across overflow, data neutralization, input validation, unsafe memory operation, and unauthenticated access groups, while Information Leakage had no red scenarios.The evaluated scenarios therefore showed strongly different security outcomes across vulnerability groups and CWEs.
  • Multi-round fixing: 143 of 160 vulnerable snippets were fixed through multi-round interaction, with 30 scenarios fully fixable, four partially fixable, and two not fixable.The overall fixing rate exceeds 89%, but complete repair was not consistent across scenarios.
  • Successful repairs: Fixes were effective for many overflow and data-neutralization cases, including replacing sprintf with snprintf to prevent buffer overflow.Providing corresponding CWE information often helped ChatGPT repair vulnerable snippets.
  • Repair limitations: Input-validation and information-leakage repairs remained weak, including unfixed checks on external inputs and database errors still returned by exception handlers.All snippets in one CWE-20 scenario remained unfixed, and five CWE-200 scenario-2 snippets remained vulnerable.

RQ5: How does the non-deterministic output of ChatGPT affect code generation?

ChatGPT can generate different functionally correct, complex, or vulnerable programs from the same input, particularly at temperature 0.7. Multi-round fixing also varies across trials, even when temperature is set to 0.

  • One-round generation: At temperature 0.7, repeated generations for the same problem and language can produce different status rates, complexity levels, and security outcomes.For example, one JavaScript problem produced 50% accepted, 10% wrong-answer, 10% time-limit-exceeded, and 30% runtime-error results across ten trials.
  • Complexity variation: Different trials may use different algorithms and implementations, producing variation in both cyclomatic and cognitive complexity.For one Java problem, cyclomatic complexity ranged from low through high levels across trials, while cognitive complexity was concentrated in low and high levels.
  • Security variation: Security outcomes are also nondeterministic: one C problem produced vulnerable code in 42.85% of seven non-constant-function trials.The authors observe this security variation in the temperature-0.7 results.
  • Fixing variation: Multi-round fixing can yield different success rates for the same error and problem-language pair, including only two successful trials out of five for one C case.The cited C and Python3 examples each had only 40% successful trials at temperature 0.7.
  • Fixing complexity: Fixed-code complexity varies across trials regardless of temperature, ranging from low to very high levels in some repeated cases.This variation reflects different fixed-code outcomes rather than a single stable complexity level.
  • Fixing security: ChatGPT may or may not repair the same vulnerable code across trials, with examples showing inconsistent fixes at both temperatures 0.7 and 0.The paper reports this behavior for CWE-20 and CWE-190 scenarios.
  • Temperature effects: Setting temperature to 0 is proposed as a mitigation for one-round nondeterminism, but it does not eliminate variation during multi-round fixing.The study evaluates ten trials at temperature 0 for algorithm problems and CWE scenarios.

5 DISCUSSION

ChatGPT’s code-generation quality varies with problem familiarity and language, while multi-round fixing affects correctness, complexity, and security unevenly. Evaluation is also bounded by model, dataset, token, and sampling constraints.

  • Functional Correctness: 48.14% higher average Accepted rate for Bef. than Aft. problems indicates stronger functional correctness on earlier problems.The discussion attributes this pattern to possible familiarity with training data and reports weaker direct fixing of erroneous code.
  • Complexity: C, C++, Java, Python3, and JavaScript exhibit differing cyclomatic and cognitive complexity distributions.The cited tables report complexity for Bef. and Aft. problems across repeated trials.
  • Complexity: Multi-round fixing generally preserves or increases generated-code complexity, potentially making the code harder to understand.This complexity trend is reported alongside the broader fixing-process analysis.
  • Security: Vulnerabilities occurred in selected C, C++, Java, and Python3 scenarios, while multi-round fixing addressed more than 89% of vulnerable snippets.The discussion recommends combining ChatGPT with vulnerability-detection tools such as CodeQL.
  • Threats and Scope: ChatGPT’s non-determinism and token limitation can change or truncate generated code, affecting functional correctness, complexity, and security.The study varies temperature in selected evaluations and proposes limiting output tokens to simulate incomplete snippets.
  • Threats and Scope: The findings reflect GPT-3.5-turbo-0301, a closed-source model with unknown training data, and may not generalize to real-world coding tasks.LeetCode and CWE scenarios do not fully capture real-world complexity, diversity, or vulnerability classes.

6 RELATED WORK

Related work covers language-model architectures, code-generation methods, evaluation datasets, and prior studies of ChatGPT code quality. This paper distinguishes itself through a broader assessment across correctness, complexity, security, and multi-round fixing.

  • Language Models: Language models support tasks including translation, question answering, summarization, text generation, and code generation, commonly using Transformer architectures.The related-work discussion describes large parameter counts, extensive training corpora, and stacked encoder-decoder architectures.
  • Code Generation: Prior code-generation studies use recurrent networks, attention, pointer mechanisms, abstract syntax trees, Transformer-XL, and multitask learning.These approaches target code completion and structural modeling from large codebases.
  • Evaluation: EvalPlus combines a base evaluation dataset with LLMs and mutation techniques to generate and diversify test cases.This illustrates dataset and testing support for code-generation evaluation.
  • ChatGPT Code Quality: Earlier ChatGPT research examined correctness, maintainability, bug repair, and code-style repair in Java and Python using feedback information.The present study is described as differing through systematic analysis across multiple quality dimensions.

7 CONCLUSION

The paper presents a systematic assessment of ChatGPT-generated code across correctness, complexity, security, and multi-round fixing. Its findings show stronger performance on earlier problems, complexity growth during fixing, vulnerabilities in several settings, and high vulnerability-remediation rates.

  • Conclusion: ChatGPT is evaluated across functional correctness, cyclomatic and cognitive complexity, security, and multi-round fixing.The assessment covers code-generation behavior across these three quality dimensions and the dialog-based repair process.
  • Conclusion: 48.14% higher average Accepted rate for Bef. than Aft. problems accompanies relatively weak direct repair of erroneous code.The conclusion reports this comparison across different programming languages.
  • Conclusion: Multi-round fixing generally preserves or increases complexity, while vulnerable-code fixing addresses 100% and 89.4% of vulnerabilities in reported settings.The two vulnerability-remediation percentages correspond to the specific scenarios summarized in the conclusion.
Loading 2308.04838v2…