PaperlayerView exact revision on arXiv

Source-linked AI summary

No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT

Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, Liang Feng Zhang

arXiv:2308.04838v2cs.SE

TL;DR

How well ChatGPT generates code across correctness, complexity, and security, including iterative repair, remains an empirical question. This study systematically assesses those dimensions and finds that code quality varies across tasks, with multi-round fixes weak at achieving correctness but addressing more than 89% of vulnerabilities.

  • Problem

    The quality of ChatGPT-generated code across correctness, complexity, and security, including iterative fixing, calls for systematic empirical assessment.

    Source: §7 · p. 33

  • Method

    The study evaluates ChatGPT-generated code for correctness, complexity, and security, using iterative prompting, functional testing, and vulnerability analysis.

    Source: §3.2 · p. 3, RQ3: How complex is the code generated… · p. 17, §7 · p. 33

  • Results

    Across the studied dimensions, ChatGPT performed better on pre-2021 than post-2021 problems by 48.14% in average Accepted rate; multi-round fixing was weak for correctness, generally preserved or increased complexity, and addressed over 89% of vulnerabilities.

    Source: §1 · p. 2, §7 · p. 33

  • Takeaways & Limitations

    The assessment identifies potential issues in ChatGPT code generation across functional correctness, complexity, and security, with results also varying due to non-determinism.

    Source: §1 · p. 2

    LeetCode problems may not represent the complexity and diversity of real-world coding tasks, despite CWE scenarios supplementing the evaluation.

    Source: §5.3 · p. 32

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated impressive capabilities across various NLP tasks. Additionally, LLMs are also highly valuable in supporting software engineering tasks, particularly in the field of code generation. Automatic code generation is a process of automatically generating source code or executable code based on given specifications or requirements, improving developer productivity. In this study, we perform a systematic empirical assessment to the quality of code generation using ChatGPT. We leverage 728 algorithm problems in five languages (i.e., C, C++, Java, Python, and JavaScript) and 18 CWEs with 54 code scenarios for the code generation task. Our evaluation encompasses a comprehensive analysis of code snippets generated by ChatGPT, focusing on three critical aspects: correctness, complexity, and security. We also specifically investigate ChatGPT's ability to engage in multi-round fixing process (i.e., ChatGPT's dialog ability) of facilitating code generation. By delving into the generated code and examining the experimental results, this work provides valuable insights into the performance of ChatGPT in tackling code generation tasks over the three critical aspects. Overall, our findings uncover potential issues and limitations that arise in the ChatGPT-based code generation and lay the groundwork for improving AI and LLM-based code generation techniques.

1 INTRODUCTION

The paper evaluates ChatGPT-generated code for correctness, complexity, and security, including whether multi-round dialogue can fix code and how nondeterminism affects results. It studies 728 algorithm problems and 54 code scenarios across five languages.

Motivation

  • Because generated code’s correctness, complexity, and security cannot be guaranteed, the study systematically assesses ChatGPT and its multi-round fixing ability.The paper frames this analysis as a way to reveal potential issues and limitations in ChatGPT-based code generation.Source: §1 · p. 2

Our Study

  • The study evaluates code generation using 728 algorithm problems across five languages (C, C++, Java, Python, and JavaScript) and 18 CWEs with 54 code scenarios from LeetCode and, respectively.Its five research questions examine functional correctness, multi-round fixing, code complexity, security, and the effect of nondeterministic outputs.Source: §1 · p. 2
  • ChatGPT had a 48.14% Accepted-rate advantage on pre-2021 versus post-2021 problems, but multi-round fixing was relatively weak at achieving functional correctness.The study also reports that generated-code quality varies with ChatGPT’s non-deterministic outputs.Source: §1 · p. 2
  • ChatGPT-generated code had relevant vulnerabilities in C, C++, and Java algorithm scenarios and C and Python3 CWE scenarios; multi-round fixing addressed more than 89% of vulnerabilities.The authors characterize this fixing result as promising.Source: §1 · p. 2
  • Complexity distributions varied across languages, and multi-round fixing generally preserved or increased code-snippet complexity.Non-determinism also produced variations in code functional correctness, complexity, and security.Source: §1 · p. 2

2 BACKGROUND

The background presents ChatGPT, [31] as an instruction-aligned, dialogue-capable LLM and explains how prompts and conversation history support code generation and iterative fixes.

Background

  • LLMs learn from large text datasets using Transformer self-attention; GPT is decoder-based, BERT encoder-based, and Codex adapts GPT-3 to GitHub code, [29], [9].GPT-3 uses 175 billion parameters, and Codex underlies GitHub Copilot.Source: §2 · p. 2
  • ChatGPT, [31], implemented atop GPT-3.5 (or GPT-4), uses InstructGPT-style RLHF for conversational follow-ups; this study evaluates its default GPT-3.5 version.Its dialog ability lets users ask follow-up questions and request corrections to earlier responses.Source: §2 · p. 3
  • Prompts express users’ intent, while ChatGPT considers user messages, system instructions, and prior responses to maintain context across follow-ups.Conversation history supports more coherent responses informed by the ongoing dialogue.Source: §2 · p. 3
  • In software engineering, users can iterate through prompts to modify or fix code; the example requests bubble sort, then nonincreasing output, prompting a comparison-operator change.The example changes the comparison from > to < in the same conversation.Source: §2 · p. 3

3 EMPIRICAL STUDY SETUP

The study assesses ChatGPT-generated code using LeetCode problems and CWE scenarios, with testing and multi-round feedback used to evaluate or fix generated code. Its setup also specifies prompt principles, model settings, and the experimental environment.

3.1 Data Collection

  • The study uses 728 LeetCode problems and 54 code scenarios spanning 18 CWEs as its two code-generation datasets.The LeetCode set includes 354 problems published after 2021 and 374 published before 2021; each CWE has three scenarios.Source: §3.1 · p. 3

3.2 Methodology

  • Generated snippets are tested with LeetCode judgments or CodeQL and manual analysis, with failed code fed back to ChatGPT for fixing up to a round limit.The study calls this iterative interaction a multi-round fixing process; a one-round process has no fixing step.Source: §3.2 · pp. 3–4
  • Prompts aim to provide sufficient information and simulate real-world code-generation use, rather than optimize performance for specific prompts or datasets.The authors report drawing on online templates, including OpenAI Cookbook and PromptBase.Source: §3.2 · p. 4
  • Experiments use gpt-3.5-turbo-0301 at temperature 0.7, with a 4,096-token limit that can affect long multi-round conversations.When conversation history risks exceeding the limit, the strategy preserves task details and the most recent response while removing older dialog blocks; reported one-round prompt and response lengths stay below 2,400 and 800 tokens, respectively.Source: §3.2 · p. 4

3.3 Experiment Environment

  • Experiments run on Ubuntu 20.04 with an Intel i9-10900X CPU, 128 GB RAM, Python 3.10.9, and CodeQL 2.12.2.Source: §3.3 · p. 4

4 EXPERIMENT AND EVALUATION · 4.1 Functionally Correct Code Generation · RQ1: Is the code generated by ChatGPT functionally correct?

In one-round LeetCode tests, ChatGPT generated accepted code much more often for pre-2021 than post-2021 problems, with success also varying by difficulty and language.

RQ1: Is the code generated by ChatGPT functionally correct?

  • The study tested 374 sampled pre-2021 and 354 post-2021 algorithm problems, prompting for five languages once per problem-language pair.The pre-2021 sample was randomly selected with difficulty distribution matched to the post-2021 set; manual review found no cases where post-2021 problems were easy reformulations of pre-2021 ones.Source: RQ1: Is the code generated by ChatGPT… · pp. 4–5
  • LeetCode submission statuses measured functional correctness, with compile and runtime errors prioritized over wrong answers and timeouts.Because testing stops at the first failed test case, reported test-case pass rates are lower bounds; the analysis also used Wilcoxon rank-sum tests with Holm-Bonferroni correction and Cliff’s Delta.Source: RQ1: Is the code generated by ChatGPT… · p. 5
  • For post-2021 problems, accepted rates fell from 52.47% on easy problems to 13.90% on medium and 0.66% on hard problems.Wrong answers reached 58% across languages; their snippets passed only 25% of an average 109 test cases, while compile and runtime error rates also reached 16%.Source: RQ1: Is the code generated by ChatGPT… · p. 6
  • ChatGPT [12] averaged a 68.41% accepted rate on pre-2021 problems versus 20.27% on post-2021 problems, a 48.14% difference.The difference across five languages was significant (p-value 0.008; effect size 1).Source: RQ1: Is the code generated by ChatGPT… · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • For pre-2021 problems, accepted rates were 89.80% on easy, 70.95% on medium, and 40.13% on hard problems.Wrong answers fell to 17.03%, but compile and runtime error rates remained near 13%; time-limit-exceeded cases were 1.87%.Source: RQ1: Is the code generated by ChatGPT… · pp. 7–8
  • In a 50-problem sample, none of the generated solutions exactly matched ground-truth code token by token.Six pre-2021 easy or medium solutions were Type-2 clones; the authors characterize this clone finding as a lower bound because they could not check every ground-truth solution.Source: RQ1: Is the code generated by ChatGPT… · p. 8
  • Across both periods, accepted rates were 31.28% for C, 44.75% for C++, 48.74% for Java, 50.00% for Python3, and 48.80% for JavaScript.C was substantially lower than the other four languages.Source: RQ1: Is the code generated by ChatGPT… · p. 8
  • For both problem periods, functionally correct generated code had lower runtime and memory overheads than at least 50% of human solutions.The paper reports average human-ranking performance above 68% for runtime and 51% for memory on post-2021 accepted problems, and above 71% and 54%, respectively, on pre-2021 accepted problems.Source: RQ1: Is the code generated by ChatGPT… · pp. 7–8

4.2 Multi-round Fixing for Code Generation · RQ2: How effective is the multi-round fixing process in improving code generation for functional correctness?

This section assesses ChatGPT’s multi-round fixing of code with wrong-answer, compile-, runtime-, or timeout-error outcomes, including whether snippets are fixed or changed. It also describes remaining failures, such as logical or algorithmic flaws, language-specific correction limits, inefficiency, and conflicts introduced during fixing.

RQ2: How effective is the multi-round fixing process in improving code generation for functional correctness?

  • For wrong-answer and compile-error cases, the process used LeetCode feedback in a conversation of up to five fixing rounds, refreshing the prompt template when token limits threatened problem context.Wrong-answer cases began from sampled code and error categories; compile-error fixing excluded empty-body constant-function cases because fixing them would amount to regenerating the code.Source: RQ2: How effective is the multi-round… · p. 8, §4.2.1 · p. 10, §4.2.2 · p. 12

4.2.1 Code with Wrong Answer

  • Wrong-answer defects were dominated by Multi-hunk subclass M-L (51 pairs) and Misaligned Algorithm (62); the authors classify their underlying logic errors as WD, MCC, or MP.WD involves detail mistakes despite roughly correct logic; MCC uses a suitable algorithm but misses a key condition; MP misunderstands the problem and uses a misaligned algorithm. The authors identify weak handling of logical details and complex reasoning as reasons fixes fail.Source: §4.2.1 · pp. 8–11
  • Only 25 of 157 wrong-answer problem-language pairs were fixed within five rounds, and hard and medium pairs had fix rates below 30%.Of the 25 fixed pairs, 21 were fixed in one round; 12 of 13 easy problems were fixed. In a separate 10-round check of 10 unfixed pairs, only two were fixed.Source: §4.2.1 · pp. 10–11, Table 5 · p. 11, Table 6 · p. 11

4.2.2 Code with Compile Error

  • More than half of compile errors were categorized as unreal errors, especially empty function bodies; other platform-related classes exposed failures to follow requested signatures or templates.The authors report constant-function errors as 159/314 and note wrong method name, redefinition of main, and incompatible parameter types as platform-related categories; they identify generation stability and prompt alignment as concerns.Source: §4.2.2 · p. 11, Table 7 · p. 12
  • Multi-round fixing fixed 70% of compile-error snippets, with 26% fixed to acceptance; the authors attribute persistent errors to language-specific and cross-language issues.Of 115 fixed snippets, 40 were accepted: 30 in C, seven in C++, and three in Java. The authors suggest adding human knowledge to prompts as a potential remedy for unfixed errors.Source: Table 8 · p. 12, §4.2.2 · pp. 12–13

4.2.3 Code with Runtime Error

  • The runtime-error analysis manually classified all 194 snippets with runtime errors across C, C++, Java, Python3, and JavaScript using LeetCode’s error messages.This part provides the analysis setup but no runtime-fixing outcome.Source: §4.2.3 · p. 14
  • Of 143 runtime-error snippets fixed, 52 were accepted; 32 snippets retained errors and 9 changed errors, often because code parts conflicted during continued fixing.The 52 accepted snippets spanned C (23), C++ (12), Java (2), Python3 (10), and JavaScript (5); manual analysis attributed remaining runtime errors mainly to algorithm implementation flaws.Source: §4.2.3 · p. 15
  • Runtime errors were mostly overflow in static languages (105) and type errors in dynamic languages (21); Python3 and JavaScript also had 10 constant-function and 13 wrong-method-name snippets.For unfixed runtime errors, ChatGPT often recognized the error message but reproduced the same error; the authors suggest adding human knowledge to prompts.Source: §4.2.3 · pp. 14–15

4.2.4 Code with Time Limit Exceeded

  • Timeouts arose from aligned but inefficient implementations (AIAI), functionally correct but misaligned algorithms (CMA), and functionally incorrect algorithms (IA).AIAI examples included Euclidean-algorithm implementations using subtraction with O(n) complexity rather than modulo with O(log n); CMA often involved brute force, while IA could involve incorrect greedy logic.Source: §4.2.4 · pp. 15–16
  • 44 of 140 timeout snippets were fixed; among the 96 unfixed, 64 became W.A., 8 R.E., and 24 remained T.L.E.ChatGPT patched or rewrote AIAI and IA solutions and changed algorithms for CMA; human-provided inefficient locations and fixing suggestions were proposed for unresolved cases.Source: §4.2.4 · pp. 16–17
  • Fewer than 32% of snippets with W.A., C.E., R.E., or T.L.E. were fixed to A.; at least 70% of C.E., R.E., or T.L.E. snippets were fixed to A., W.A., or T.L.E., excluding T.L.E. → T.L.E.The findings identify factors behind errors and unfixed cases under multi-round fixing.Source: §4.2.4 · p. 17

4.3 Code Complexity · RQ3: How complex is the code generated by ChatGPT?

ChatGPT-generated code complexity varies by language and problem difficulty: C is most complex, Python3 least, and harder problems yield more high-complexity code. Multi-round fixing generally preserves or increases complexity levels.

RQ3: How complex is the code generated by ChatGPT?

  • The assessment uses SonarQube and cccc to measure cyclomatic and cognitive complexity in before- and after-fixing code; cognitive complexity is measured only for Java, Python3, and JavaScript.Cyclomatic complexity reflects independent paths and testing difficulty, while cognitive complexity concerns human understanding and reasoning.Source: RQ3: How complex is the code generated… · p. 17
  • 54.9% of C solutions have low or moderate cyclomatic complexity, versus more than 66% in each other language; Python3 has 44.5% low and 8.5% very high complexity.C++, Java, and JavaScript have more similar distributions across the four complexity levels.Source: Table 12 · p. 17, RQ3: How complex is the code generated… · pp. 17–18
  • More than 70% of generated code in each language has low or moderate cognitive complexity, with Python3’s low-complexity share at 42.6%.Java and JavaScript have similar distributions; Python3 has a higher low-complexity share and correspondingly lower moderate share than the other two languages.Source: Table 13 · p. 17, RQ3: How complex is the code generated… · p. 18
  • As problem difficulty increases, low complexity decreases while high and very high complexity increase, a trend comparable to human-written code.Python3 has the highest proportion of low and moderate complexity across difficulty levels, while C has the lowest; for hard problems, C has only 37% low or moderate cyclomatic complexity.Source: RQ3: How complex is the code generated… · pp. 18–19, Figure 21 · p. 19, Figure 22 · p. 19
  • For the same problem, most cross-language complexity combinations differ by at most one level; one-level differences exceed 50%, reaching 76.5% for cognitive complexity.All-same-level combinations are individually frequent but collectively below 50%, so most language combinations have differing complexity levels.Source: Table 17 · p. 20, Figure 23 · p. 20, Figure 24 · p. 20
  • Cross-language complexity differences arise from language-specific built-in libraries, different algorithms, and variations in how logic is implemented.For example, Python3 can use heapq for a min-heap, whereas C must generate the supporting code; snippets using the same algorithm can also differ in implementation concision.Source: RQ3: How complex is the code generated… · p. 20
  • In multi-round fixing, complexity levels remain unchanged in more than half of cases, and the process generally preserves or increases complexity.Fixes may adjust logic, add or modify conditions, or change algorithms; decreases can result from simplifying logic, deleting conditions, or switching algorithms.Source: RQ3: How complex is the code generated… · p. 21
  • C-generated code is most complex, Python3 least, and C++, Java, and JavaScript are comparable; C++ and Python3 complexity is slightly higher but nearly equal to human-written code.The conclusion also notes that Python3 has the highest probability of the lowest complexity level, while C has the lowest.Source: RQ3: How complex is the code generated… · pp. 18, 21

4.4 Security Code Generation · RQ4: Is the code generated by ChatGPT secure?

ChatGPT generated many valid but often vulnerable security-scenario snippets, with vulnerability rates varying by language, CWE, and scenario. Multi-round fixing addressed most sampled vulnerabilities overall, but results differed substantially across vulnerability groups.

RQ4: Is the code generated by ChatGPT secure?

  • In algorithm-problem snippets, MissingNullTest accounted for 91.8% of detected vulnerable snippets, while no CWE-416 or CWE-190 vulnerabilities were detected in C, C++, or Java.All sampled vulnerable snippets in this setting were fixed; detection used CodeQL on C, C++, and Java, limited to five CWEs and pointer- or memory-related issues.Source: RQ4: Is the code generated by ChatGPT… · pp. 21–23, Table 18 · p. 23
  • 2,983 valid snippets (99.07%) included 994 vulnerable snippets (33.32%); the vulnerable rate was 51.64% in C and 17.08% in Python3.The evaluation covered 18 CWEs and 54 scenarios; the C and Python3 samples contained 1,402 and 1,581 valid snippets, respectively.Source: Table 19 · p. 23, RQ4: Is the code generated by ChatGPT… · pp. 23–24
  • Across 54 code scenarios, 18 (33%) were green, 4 (7%) orange, 16 (30%) yellow, and 16 (30%) red; Overflow’s 12 scenarios included 7 red (5) and yellow (2), with the remainder green.All code snippets in CWE-787, CWE-190, and CWE-119 can be fixed; one CWE-78 scenario 1 snippet remained vulnerable to OS command injection.Source: RQ4: Is the code generated by ChatGPT… · pp. 24, 26
  • Input Validation had four red scenarios, though most others were below 10% vulnerable; fixing was weak, with none of the CWE-20 scenario 3 snippets fixed.CWE-22 scenario 1 was 100% vulnerable versus 43.1% for its Python3 web-app scenario 2; unfixed cases also missed input or output checks and image-type or size requirements.Source: RQ4: Is the code generated by ChatGPT… · pp. 24, 26
  • Unsafe Memory Operation performed poorly initially, with four of six scenarios red and all CWE-476 scenarios red; fixing resolved all CWE-476 snippets, but one CWE-416 snippet remained vulnerable.The remaining CWE-416 case still risked double-free; the algorithm-problem detection found no CWE-416 vulnerabilities in C, C++, or Java.Source: RQ4: Is the code generated by ChatGPT… · pp. 22, 24, 26
  • Unauthenticated Access had only one red scenario, but three of five sampled CWE-522 scenario 1 snippets remained vulnerable after fixing because they used one-time SHA-256 hashing.Most other scenarios in this group had vulnerability rates below 6%, except CWE-798 scenario 3 at 13%; adding a salt or bcrypt context reduced vulnerabilities in generation.Source: RQ4: Is the code generated by ChatGPT… · pp. 25–26
  • Information Leakage had no red generation scenarios, but five sampled CWE-200 scenario 2 snippets remained vulnerable after fixing because exception handlers still returned database errors.CWE-200 scenario 2 had a 23.6% vulnerability rate, despite the group’s generally favorable generation results.Source: RQ4: Is the code generated by ChatGPT… · pp. 25–26
  • 143 of 160 sampled vulnerable snippets were fixed (89.4%) after up to five rounds, but 30 scenarios were fully fixed, four partly fixed, and two not fixed.Fixing was strongest for Overflow, Data Neutralization Issues, Unsafe Memory Operation, and Unauthenticated Access, and weaker for Input Validation and Information Leakage.Source: Table 21 · p. 25, RQ4: Is the code generated by ChatGPT… · pp. 25–27

4.5 Non-determinism of ChatGPT · RQ5: How does the non-deterministic output of ChatGPT affect code generation?

ChatGPT’s one-round code generation varies across trials at temperature 0.7, while temperature 0 may mitigate that variation; multi-round fixes can vary at either temperature. Variation affects functional correctness, complexity, and security.

4.5 Non-determinism of ChatGPT

  • The study tested 18 algorithm problems across five languages in 10 trials at temperatures 0.7 and 0, plus 20 CWE scenarios and repeated multi-round fixes.The multi-round process fixed each sampled error or vulnerable snippet five times, with a maximum of five rounds.Source: RQ5: How does the non-deterministic… · p. 27

RQ5: How does the non-deterministic output of ChatGPT affect code generation?

  • At temperature 0.7, repeated generations for the same problem and language can have different correctness statuses; problem 2224 in JavaScript ranged from 50% accepted to 30% runtime errors.The remaining trials were 10% wrong answer and 10% time-limit exceeded; constant-function outputs were excluded from subsequent complexity and security evaluations.Source: RQ5: How does the non-deterministic… · p. 27
  • At temperature 0.7, complexity levels can differ between trials, reflecting changes in algorithms or implementations; problem 363 in Java had mixed cyclomatic and cognitive complexity levels.Its cyclomatic levels were 40.0% low, 50.0% moderate, and 10.0% high; cognitive levels were 40.0% low and 60.0% high.Source: RQ5: How does the non-deterministic… · p. 27
  • Security outcomes also varied at temperature 0.7: problem 2264 in C had a 42.85% CWE rate among its seven non-constant snippets.The reported vulnerability was MissingNullTest; the authors also note trial-to-trial variation in the vulnerability counts.Source: RQ5: How does the non-deterministic… · p. 27
  • At temperature 0, one-round correctness statuses were consistent across 10 trials, complexity was consistent except for problem 2532 in Python3, and sampled CWE scenarios had identical outputs.All generated C code for problems 304 and 2523 had MissingNullTest vulnerabilities; only problem 2532 in Python3 used different code structures across trials.Source: RQ5: How does the non-deterministic… · p. 27
  • Multi-round fixing outcomes for the same error code varied at both temperatures; at 0.7, only two of five fixes succeeded for problem 363 in C and problem 2124 in Python3.The trials sampled error snippets at each temperature and ran five fixes per snippet.Source: RQ5: How does the non-deterministic… · pp. 27–28
  • Multi-round fixes also varied in complexity and security at either temperature, as ChatGPT chose different patches and sometimes fixed or retained vulnerabilities.For example, complexity levels for problem 2532 in Python3 spanned low through very high at temperature 0; at 0.7, a CWE 20 scenario was fixed once, while at 0 a CWE 190 scenario was not fixed once.Source: RQ5: How does the non-deterministic… · p. 28
  • One-round generation varies across trials at temperature 0.7, whereas temperature 0 may mitigate non-determinism; multi-round fixes vary at both temperatures.The authors conclude that temperature 0 yields consistent one-round results in nearly all tested cases, but fixing results can still differ at temperatures 0.7 and 0.Source: RQ5: How does the non-deterministic… · pp. 27–28

5 DISCUSSION

The discussion highlights functional-correctness differences across problem familiarity and languages, alongside complexity, security, and nondeterminism concerns in ChatGPT-generated code. It also describes study limitations and threats to validity that constrain how broadly the findings can be interpreted.

5.1 Lessons Learnt and Insight

  • ChatGPT generates functionally correct code more often for problems seen before than for later problems, with C performing substantially worse than C++, Java, Python3, and JavaScript.The success probabilities for C++, Java, Python3, and JavaScript are described as close to one another.Source: §5.1 · pp. 28–29
  • Errors and failed fixes reflect limits in reasoning, generation stability, and attention to logical details or requirements such as provided method signatures.The authors suggest supplying missing details or algorithm logic, or designing an effective automatic-generation workflow, to support multi-round fixing.Source: §5.1 · p. 29
  • Multi-round fixing generally preserves or increases code complexity, while generated code often contains vulnerabilities that CWE information can help ChatGPT fix.The authors suggest combining ChatGPT with vulnerability detectors such as CodeQL to mitigate generated vulnerabilities.Source: §5.1 · p. 29
  • ChatGPT’s nondeterminism can change code correctness, complexity, and security; setting temperature to zero may mitigate variation only in one-round generation.Multi-round fixes can vary at both temperature settings, and prompt or hyperparameter strategies were outside this study’s scope.Source: §5.1 · pp. 29–30
  • When output exceeds the token limit, discarded content can leave incomplete code with compile or syntax errors, lower-or-equal complexity, and vulnerabilities that may remain or disappear.The proposed strategy discards as little early content as possible and supplements necessary information throughout the conversation.Source: §5.1 · pp. 30–31
  • The discussion contrasts code-generation models Codex [9] and CodeGen [67] with ChatGPT’s follow-up-answer capability supported by RLHF [31].The passage describes code-related models as autoregressively predicting tokens from prior context.Source: §5.1 · pp. 31–32

5.2 Limitations

  • The study’s findings are limited by ChatGPT’s closed-source status, unknown training data, and use of model version gpt-3.5-turbo-0301 at the time of the work.The authors therefore cannot map results to the model’s internal workings or determine whether evaluated problems appeared in its training data.Source: §5.2 · p. 32

5.3 Threats to Validity

  • Threats to validity include nonrepresentative LeetCode and CWE scenarios, lower-bound platform pass rates, possible CodeQL false positives, and security evaluation across only some languages.The study uses 728 LeetCode problems and 60 snippets per CWE scenario to address randomness, and limits multi-round fixing to five rounds.Source: §5.3 · p. 32

6 RELATED WORK

The related work traces language models and code-generation methods, then positions this study among prior evaluations. It distinguishes this work through its five-language analysis of correctness, complexity, security, and multi-round fixing.

Language Models

  • Transformer-based models include GPT and BERT; GPT-2 and GPT-3 expanded model size and introduced zero-shot and few-shot learning, while Codex underlies Copilot and ChatGPT builds on InstructGPT methods,, [29],,, [9],, [31],,.The passage also describes ELMo's multi-layer bidirectional LSTM architecture [74].Source: §6 · p. 32

Code Generation

  • Traditional code generation uses templates or rules and input-output specifications, while neural approaches learn from examples and codebases; recent LLMs generate code autoregressively,,,,,, [80],,, [9],, [67].ChatGPT, based on GPT-3.5, supports follow-up questions through RLHF, [31].Source: §6 · pp. 32–33

Evaluation on LLM-based Code Generation

  • Prior evaluations span code-generation benchmarks, APR, vulnerability repair, Copilot quality and security, attention alignment, and benchmark enhancement [85], [51],,, [87], [53],,,, [88], [25].ChatGPT code-quality issues and repair using feedback were also studied across Java and Python [89].Source: §6 · p. 33
  • Unlike prior ChatGPT code-quality work, this study assesses five languages, correctness, complexity, security, causes of incorrectness, and multi-round fixing [89].The authors frame this as a deeper and more realistic evaluation of iterative code generation.Source: §6 · p. 33

7 CONCLUSION

The study systematically assesses ChatGPT-generated code for correctness, complexity, and security, including multi-round fixing, and identifies limitations across these dimensions.

7 CONCLUSION

  • ChatGPT’s average Accepted rate for Bef. problems exceeded Aft. problems by 48.14%, while directly fixing erroneous code to functional correctness was relatively weak.Source: §7 · p. 33
  • Cyclomatic and cognitive complexity distributions varied across languages, and multi-round fixing generally preserved or increased code snippets’ complexity levels.Source: §7 · p. 33
  • Vulnerabilities appeared in algorithm scenarios for C, C++, and Java and CWE scenarios for C and Python3; multi-round fixing addressed 100% and 89.4% of vulnerabilities.Source: §7 · p. 33
  • ChatGPT’s non-determinism could cause generated snippets to vary in functional correctness, complexity, and security.Source: §7 · p. 33

Related papers

Loading 2308.04838v2…