PaperlayerView exact revision on arXiv

Source-linked AI summary

Bugs in Large Language Models Generated Code: An Empirical Study

Florian Tambon, Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, Giuliano Antoniol

arXiv:2403.08937v2cs.SEcs.AI

TL;DR

The study addresses the limited examination of bug patterns in LLM-generated code, which matters as code-generation tools are used in software work. It analyzes 333 bugs from three LLMs, develops a taxonomy, and surveys practitioners and researchers. The findings include patterns uncommon in human-written buggy code, and survey respondents report encountering the observed patterns.

  • Problem

    Bug patterns in LLM-generated code have received limited examination, despite the importance of understanding whether LLMs produce faults like human developers.

    Source: §1 · p. 2

  • Method

    The authors manually analyze a sample of 333 bugs from CodeGen, PanGu-Coder, and Codex, organize patterns into a taxonomy, and validate it with an online survey.

    Source: §1 · p. 3, §3.1.1 · p. 7

  • Results

    The taxonomy includes patterns described as uncommon in human-written buggy code, 64% of sampled bugs are not linked to prompt-information lack, and survey respondents mostly report encountering the observed patterns.

    Source: §8 · p. 37

  • Takeaways & Limitations

    The authors present the taxonomy and survey as initial steps toward characterizing LLM-generated-code bugs and as resources for quality-assurance research and tools.

    Source: §8 · p. 38

    The study covers Python functions and three examined LLMs, and manual sample collection and labeling may influence its results.

    Source: §7 · p. 35

Abstract

from arXiv · show

Large Language Models (LLMs) for code have gained significant attention recently. They can generate code in different programming languages based on provided prompts, fulfilling a long-lasting dream in Software Engineering (SE), i.e., automatic code generation. Similar to human-written code, LLM-generated code is prone to bugs, and these bugs have not yet been thoroughly examined by the community. Given the increasing adoption of LLM-based code generation tools (e.g., GitHub Copilot) in SE activities, it is critical to understand the characteristics of bugs contained in code generated by LLMs. This paper examines a sample of 333 bugs collected from code generated using three leading LLMs (i.e., CodeGen, PanGu-Coder, and Codex) and identifies the following 10 distinctive bug patterns: Misinterpretations, Syntax Error, Silly Mistake, Prompt-biased code, Missing Corner Case, Wrong Input Type, Hallucinated Object, Wrong Attribute, Incomplete Generation, and Non-Prompted Consideration. The bug patterns are presented in the form of a taxonomy. The identified bug patterns are validated using an online survey with 34 LLM practitioners and researchers. The surveyed participants generally asserted the significance and prevalence of the bug patterns. Researchers and practitioners can leverage these findings to develop effective quality assurance techniques for LLM-generated code. This study sheds light on the distinctive characteristics of LLM-generated code.

1 Introduction

The paper examines bugs in LLM-generated code for real programming tasks, where their characteristics had not been thoroughly studied. It develops and surveys a taxonomy of bug patterns across three LLMs, motivated by the possibility that these bugs differ from human-written-code bugs.

1 Introduction

  • The study addresses whether LLMs produce fault patterns like human developers, a question relevant because mutation testing depends on precise fault characterization.The authors note prior findings that LLMs may not generate the same vulnerabilities as humans and may attend to different parts of prompts.Source: §1 · p. 2
  • Codex’s unrequested sorting of extracted flags illustrates how apparently reasonable code can fail a task through a non-prompted feature.The example motivates the possibility that LLM bugs differ from human-made bugs and can escape notice without testing.Source: §1 · pp. 2–3
  • From 333 bugs generated by CodeGen, PanGu-Coder, and Codex, the authors inductively identify 10 bug patterns and validate them with a survey of 34 practitioners and researchers.Survey respondents reported encountering the patterns in proportions similar to those in the sample.Source: §1 · p. 3
  • The contributions include a taxonomy, survey validation, and a publicly available dataset intended to support replication and follow-on work.The paper presents these as resources for researchers and practitioners.Source: §1 · p. 4

2 Background

The background introduces the three code-generation LLMs used in the study and explains open coding, the qualitative approach used to develop categories from observed data.

2.1 LLMs for code generation

  • CodeGen, PanGu-Coder, and Codex are introduced as code-generation models with distinct training and interaction characteristics.CodeGen supports iterative, multi-turn synthesis; PanGu-Coder was trained in unsupervised and supervised stages; Codex underlies GitHub Copilot.Source: §2.1 · pp. 4–5

2.2 Open Coding

  • Open coding develops categories inductively by having reviewers label textual data, discuss differences, and iteratively revise a codebook.The paper describes two reviewers coding independently and a third reviewer helping resolve disagreements.Source: §2.2 · p. 5

2 Background

  • The study uses open coding to analyze bug characteristics and construct a taxonomy rather than beginning with a fixed set of labels.Open coding is also described as a qualitative method used in prior software-engineering studies to construct taxonomies.Source: §2.2 · pp. 5–6

3 Study Design

The study combines benchmark-based analysis of generated Python code with manual open coding and a survey of practitioners and researchers. Its design limits the sample to runnable CoderEval tasks and uses iterative reviewer discussion to construct and validate the bug-pattern taxonomy.

3 Study Design

  • The study proceeds from benchmark sampling to pilot coding, iterative labeling, taxonomy construction, and practitioner survey validation.The design addresses bug characteristics in real-world tasks and the relevance of the resulting patterns to practitioners and researchers.Source: §3 · p. 6

3.1.1 Data collection

  • The analysis uses CoderEval’s Python functions and docstring prompts, and treats generated samples that fail execution or unit tests as buggy.The authors restrict analysis to self-, standard-library-, or public-library-runnable code and sample 333 buggy fragments from 1,997 eligible samples.Source: §3.1.1 · pp. 7–8

3.1.2 Manual labeling and classification

  • Three reviewers use open coding to build a 10-pattern taxonomy through a pilot, independent labeling, conflict resolution, and iterative codebook revisions.Samples could receive multiple labels; the process took approximately 108 person-hours.Source: §3.1.2 · pp. 8–9

3.2 RQ2: To what extent are the identified bug patterns in LLM-generated code relevant for software practitioners and researchers working with LLMs?

  • The survey is designed to assess the relevance of the identified bug patterns among software practitioners and researchers working with LLMs.Participants were asked about pattern frequency and about diagnosing, complexity, and fixing.Source: §3.2 · p. 9, §3.2.2 · p. 11

3.2.1 Participants’ selection

  • Practitioner contacts come from collaborators on LLM-code repositories, while researcher contacts are identified through relevant Google Scholar papers.The team collected 200 practitioner and 182 researcher email addresses and sent the survey to 382 unique addresses.Source: §3.2.1 · p. 10

3.2.2 Survey with Practionners

  • The questionnaire gathers demographics and presents each bug pattern with a description and code example for Likert-scale feedback.The survey was anonymous, open for two weeks, and included optional comment fields.Source: §3.2.2 · pp. 10–11, §3.2.1 · p. 10

3.2.3 Analysis of Survey Results

  • Survey ratings are summarized with weighted averages, and Spearman rho compares reported pattern frequency with the sample’s pattern distribution.The comparison is intended to assess whether sample proportions correspond to practitioners’ reported experience.Source: §3.2.3 · pp. 11–12

4 Empirical Results

This section examines bug patterns and survey responses, characterizing their nature and distribution across CodeGen, PanGu-Coder, and Codex, as well as differences by task and dependency level. It also reports that survey respondents encounter these patterns and generally address them with manual effort.

4.1.1 Taxonomy

  • Misinterpretation accounts for 20.77% of buggy samples, and Missing Corner Case is second at 15.27%.Both percentages are of buggy code samples.Source: §4.1.1 · pp. 12, 15
  • The taxonomy distinguishes prompt misinterpretation, syntax errors, silly mistakes, prompt-biased code, and missing corner cases.These cover deviation from prompt intent, syntax faults, redundant or risky code, overreliance on examples or prompt terms, and overlooked cases; Figure 2 shows the full taxonomy and category percentages.Source: §4.1.1 · pp. 12–15, Figure 2 · p. 13
  • Other categories capture incorrect input types, nonexistent objects, and invalid attributes or module properties.The paper also identifies Incomplete Generation, covering absent, empty, or unfinished code, and Non-Prompted Consideration, for unrelated statements that cause errors.Source: §4.1.1 · pp. 15–17
  • Silly Mistake, Hallucinated Object, and Incomplete Generation rank third; Wrong Attribute ranks fourth, Non-Prompted Consideration fifth at 8.15%, and Wrong Input Type is least frequent.Incomplete Generation, Hallucinated Object, and Silly Mistake are tied at third; the passage gives no percentage for Wrong Input Type.Source: §4.1.1 · pp. 14–18

4.1.2 Analysis of the Bug Patterns

  • Some buggy samples receive multiple labels: 62.16% have one, 29.13% two, and 8.41% three, while one sample has four.Multiple labels reflect different bugs in a sample or overlap between categories.Source: §4.1.2 · p. 18
  • Misinterpretation was the most common pattern overall (20.77%); Codex most often had Missing Corner Case bugs, while PanGu-Coder and CodeGen most often had Misinterpretations.CodeGen had the most Silly Mistake and Syntax Error samples, Codex had the most Non-Prompted Consideration samples, and Hallucinated Object was PanGu-Coder’s third most common pattern.Source: §4.1.2 · pp. 18–19
  • Bug patterns were generally evenly distributed across tasks, but ambiguous prompts concentrated bugs in a few cases, including a task where 72% were Missing Corner Case bugs.Across two tasks, all buggy samples fell into one pattern; in the “force string” task, an additional newline made the prompt ambiguous about whether the input was a string or bytes object.Source: §4.1.2 · p. 20, Figure 3 · p. 21
  • Misinterpretation was the most frequent pattern at every runnable level, and the observed distributions did not show that dependency level affected most bug patterns.Missing Corner Case increased with runnable level, while Prompt-biased Code and Non-Prompted Consideration decreased; the authors found no similar trends for other patterns.Source: §4.1.2 · pp. 21–22

4.2.1 Participants’ demographics and experience with LLM bugs

  • Respondents reported often or always encountering LLM-generated code bugs (80%); 68% said encountered bugs were medium difficulty to fix, and 30% fixed them manually.The remaining participants combined manual inspection with re-prompting; none reported relying only on re-prompting based on error messages.Source: §4.2.1 · pp. 23–24, §4.2 · p. 23

4.2.2 Frequency of bugs patterns

  • All taxonomy categories were encountered by survey respondents; Wrong Attributes, Prompt-biased Code, and Missing Corner Cases scored above 3 for frequency, while Syntax Error scored 2.03 and Silly Mistake 2.35.Missing Corner Cases scored 3.23; reported pattern frequencies correlated with the sampled Codex distribution (ρ = 0.47), but discrepancies included more Prompt-biased Code and Wrong Attribute reports.Source: §4.2.2 · pp. 24, 26, Figure 4

4.2.3 Diagnosing, Complexity, and Fixing of bug patterns

  • Most patterns were considered easy to diagnose, but Missing Corner Cases were quite hard to diagnose because they can pass compilation yet fail unit tests.Respondents generally considered patterns low-complexity; Misinterpretation and Missing Corner Cases had the highest complexity scores and were more like mistakes developers might make.Source: §4.2.3 · pp. 26–27
  • Misinterpretation bugs were harder to fix than other categories, while Syntax Errors and Silly Mistakes were considered easy to fix; respondents said additional prompt information could help with Hallucinated Objects.Misinterpretations may require re-prompting or extensive manual work, and added information may not be enough when the generated code substantially misreads the task.Source: §4.2.3 · p. 27

4.2 RQ2: To what extent are the identified bug patterns in LLM-generated code relevant for software practitioners and researchers working with LLMs?

  • Survey respondents encountered the identified bug patterns; reported frequencies correlated most with Codex (ρ = 0.47), and Misinterpretation and Missing Corner Cases were harder to diagnose and fix.Respondents generally described bugs as easy to diagnose and fix, but said addressing them still required manual effort; these two patterns were more akin to human-made faults.Source: §4.2.2 · p. 24, §4.2.3 · p. 28

5 Lessons Learned

The lessons address both users and researchers: generated code can look correct while misleading users, and the authors identify practical directions for safer use and further research.

5.1 Lessons Learned for LLM Users

  • Limited prompts can leave models astray; users can re-prompt with details such as library versions or add examples and test cases.CoderEval tasks provide function signatures and docstrings, which may omit version requirements; specifying them can constrain generation.Source: §5.1 · pp. 28–29
  • Test generated code when possible, but subtle misinterpretations and missed corner cases may escape tests; trusting plausible-looking code can put novices at risk.Without tests, users must either trust the output or inspect it manually, which can be time-consuming.Source: §5.1 · pp. 29–30
  • When precise requirements demand extensive prompting and debugging, manual coding may be preferable; code completion or a kickstart can suit other tasks.Completion adds user-written context, while a kickstart can help when users do not know how to approach a task.Source: §5.1 · pp. 30–31

5.2 Lessons Learned for Researchers

  • Practical bug patterns can guide automatic program repair or complement prompts that ask models to check their generated code.Prior repair studies used simpler datasets and still missed mistakes, motivating analysis of bugs in practical tasks.Source: §5.2 · p. 31
  • Benchmarks and metrics tailored to code features, dependency levels, and bug detection could support fairer evaluation of testing approaches.The authors suggest adding bug patterns and other code-related features to existing benchmarks or creating new ones.Source: §5.2 · p. 31
  • Interpretability techniques that examine how prompt variations affect outputs could help diagnose bugs or inform potential fixes.Such techniques can analyze input-element contributions without requiring access to the model's internal architecture.Source: §5.2 · p. 32

6 Related Works

Prior work examined generated-code quality, robustness, and repair, but the authors distinguish this study by its bug-pattern taxonomy for practical tasks and its survey validation.

Programming-assistant studies

  • Copilot studies reported users struggled to detect or repair generated errors, while another found repair costs differed from human-written code.One study found participants preferred writing from scratch over repairing Copilot-generated bugs.Source: §6 · p. 32

Repair studies

  • Earlier repair work classified Codex bugs using human-bug fix categories and found Codex outperformed APR tools, whereas this study also observed non-human-like mistakes.The authors connect those differences to CoderEval's GitHub-mined tasks rather than competition snippets.Source: §6 · pp. 33–34

Prior bug and quality analyses

  • Studies of LeetCode quality, instance robustness, and prompt paraphrasing reported output errors or shifts, but did not thoroughly characterize bug patterns.The authors relate their Misinterpretation and Missing Corner Case findings to prior wrong-output observations while noting task differences.Source: §6 · pp. 33–34

Prior bug taxonomies

  • A Java study found generated simple, stupid bugs more often than correct code; the authors relate similar Python cases to Missing Corner Cases.That study also found comments could help the model generate fewer such bugs.Source: §6 · pp. 34–35
  • A prior code-translation study developed a bug taxonomy centered on translation-specific issues such as API-behavior mismatches.Its categories include removed source logic and behavior changes after replacing API calls.Source: §6 · p. 35

Novelty and scope

  • The authors position their work as the first systematic taxonomy of bugs in LLM-generated code from real-world CoderEval tasks, validated through a survey.They contrast this scope with prior work on programming-competition tasks.Source: §6 · p. 35

7 Threats to Validity

The authors identify threats to construct, internal, external, and conclusion validity, and describe steps taken to mitigate or bound them.

Construct Validity

  • Manual collection and labeling could influence results, while excluding above-threshold runnable samples risks losing patterns but avoids treating insufficient prompt information as model weakness.The authors used open coding, independent ratings, discussion to consensus, and a practitioner survey to address these concerns.Source: §7 · p. 35

Internal Validity

  • Manual inspection and CoderEval snippet sampling may introduce bias; independent labeling, statistical sampling, and survey validation were used to address these threats.The survey response rate was 8.9%, and respondents reported no new bug patterns.Source: §7 · p. 36

External Validity

  • The Python-only dataset and three examined LLMs constrain external validity; Python also omits some language-specific bug types, such as type-declaration and memory-management issues.The authors recommend extending the study to more models, functions, prompts, and programming languages.Source: §7 · p. 36

Conclusion Validity

  • Inspecting 333 buggy snippets from a pool exceeding 1,997 leaves potential for wrong or missing patterns, which the authors sought to limit through three models, independent labeling, and survey validation.A replication package is provided to support further study.Source: §7 · p. 37

8 Conclusion and Future Work

The study presents an initial taxonomy of LLM-generated code bugs and reports that these bugs differ from human-written code bugs. Its findings connect prompt ambiguity and stochastic generation to distinct bug patterns, with practitioner survey feedback supporting their relevance.

Conclusion

  • 64% of the sampled bug patterns, including Wrong Attribute and Hallucinated Object, were not linked to missing prompt information.The authors associate some patterns, including Hallucinated Object and Wrong Attribute, with stochastic generation; prompt ambiguity may lead to Missing Corner Case or Prompt-biased Code bugs.Source: §8 · p. 37
  • Survey respondents mostly encountered the observed patterns and rated Misinterpretation and Non-Prompted Consideration as difficult to diagnose and fix.The passage also reports that most patterns were relatively easy to diagnose and fix.Source: §8 · p. 37
  • LLM-generated code bugs differ from human-written bugs, including in Syntax Error, and the taxonomy includes Prompt-biased Code, Wrong Attribute, and Hallucinated Object patterns.The authors describe several of these patterns as not common in human buggy code.Source: §8 · pp. 37–38

Future Work

  • The findings form an initial taxonomy that may inform improvements to software-engineering tools based on human buggy-code characteristics and future code-generation research.The authors also identify potential relevance to computer-science education studies.Source: §8 · p. 38

A Reddit Post

The appendix documents the study's survey post on Reddit, which was shared through two relevant channels.

A Reddit Post

  • The survey questionnaire was posted on the LocalLLaMA and MachineLearning Reddit channels.Figure 5 shows the posted survey.Source: A Reddit Post · p. 44, Figure 5 · p. 44
  • The section records the survey post as part of the study's online outreach.The post appeared on two relevant Reddit channels.Source: A Reddit Post · p. 44
  • The appendix identifies Figure 5 as the posted survey on Reddit.The figure caption describes it as the posted survey.Source: Figure 5 · p. 44

B.1 The details of the questionnaire in the survey

The questionnaire had demographic and bug-pattern sections. It asked about participants' experience with LLM-generated code and gathered feedback on the study's taxonomy.

B.1 The details of the questionnaire in the survey

  • The survey used Google Forms and had two parts: participant demographics and feedback about bug patterns.Demographic items covered job title, LLM use, and non-Python programming languages.Source: §B.1 · pp. 44–45
  • Figure 6 presents the participant questionnaire's demographic questions, including job title, LLM use, and programming languages other than Python.The survey also asked about the frequency and complexity of mistakes in generated code.Source: §B.1 · p. 45, Figure 6 · p. 45
  • Participants rated how often they encountered mistakes and how difficult they were to fix, then described whether they fixed them manually, re-prompted, or combined both approaches.The first three questions used five-level Likert scales; respondents could also provide another fixing approach.Source: §B.1 · p. 45
  • The second survey part gathered participant feedback on aspects of each bug pattern in the taxonomy.Figure 7 gives an example question using a CoderEval task [6].Source: §B.1 · pp. 45–46, Figure 7 · p. 46

B.2 Survey Results

Table 4 reports survey validation ratings for bug-pattern frequency, diagnosis and fixing difficulty, and complexity.

B.2 Survey Results

  • The survey asks about bug-pattern frequency, difficulty diagnosing and fixing bugs, and bug complexity.Source: Table 4 · p. 47
  • Ratings run from 1, meaning never, easy, trivial, or low, to 5, meaning always, hard, complex, or high.Source: Table 4 · p. 47
  • Bold marks the highest number in each category for each bug pattern.Source: Table 4 · p. 47
Loading 2403.08937v2…