PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation

Ziyao Zhang, Yanlin Wang, Chong Wang, Jiachi Chen, Zibin Zheng

arXiv:2409.20550v2cs.SEcs.AIcs.CL

TL;DR

Research on code-generation hallucinations remains limited, although practical development involves more complex requirements than standalone-function generation. This study examines repository-level generation, establishes a hallucination taxonomy, and finds task-requirement conflicts most prevalent while RAG consistently improves performance across the studied LLMs.

  • Problem

    Research on code-generation hallucinations remains limited, although practical development involves more complex requirements than standalone-function generation.

    Source: §I · p. 1, B. Hallucinations in LLMs · p. 3

  • Method

    The authors manually annotate generated code from six mainstream LLMs on CoderEval, iteratively developing a hallucination taxonomy and analyzing hallucination patterns.

    Source: §I · p. 2

  • Results

    The taxonomy identifies three categories and eight subtypes, with Task Requirement Conflicts most frequent across models.

    Source: 3) Project Context Conflicts (24.56%) · p. 7

  • Takeaways & Limitations

    Repository-based retrieval of relevant code snippets consistently improves performance across all studied LLMs.

    Source: §I · p. 2

    The findings are scoped to Python and CoderEval’s 230 tasks, limiting generalizability to other programming languages and broader task sets.

    Source: §VII · p. 10

Abstract

from arXiv · show

Code generation aims to automatically generate code from input requirements, significantly enhancing development efficiency. Recent large language models (LLMs) based approaches have shown promising results and revolutionized code generation task. Despite the promising performance, LLMs often generate contents with hallucinations, especially for the code generation scenario requiring the handling of complex contextual dependencies in practical development process. Although previous study has analyzed hallucinations in LLM-powered code generation, the study is limited to standalone function generation. In this paper, we conduct an empirical study to study the phenomena, mechanism, and mitigation of LLM hallucinations within more practical and complex development contexts in repository-level generation scenario. First, we manually examine the code generation results from six mainstream LLMs to establish a hallucination taxonomy of LLM-generated code. Next, we elaborate on the phenomenon of hallucinations, analyze their distribution across different models. We then analyze causes of hallucinations and identify four potential factors contributing to hallucinations. Finally, we propose an RAG-based mitigation method, which demonstrates consistent effectiveness in all studied LLMs. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/LLMCodingHallucination

I. INTRODUCTION

The study examines hallucinations in repository-level code generation, extending beyond standalone-function evaluations to characterize their forms, distribution, causes, and mitigation. It reports a practical-code hallucination taxonomy and finds that task-requirement conflicts are most prevalent across models, while a retrieval-based approach consistently improves performance.

Motivation

  • Repository-level generation involves contextual dependencies that standalone-function evaluations miss, while existing practical benchmarks emphasize test-pass correctness rather than failure causes.The paper targets hallucinations in this more complex development setting.Source: §I · p. 1

Related work

  • Unlike Liu et al.’s study of standalone function and script generation [75], this work examines repository-level hallucinations through their phenomena, mechanisms, and mitigation.The prior study used HumanEval and DS-1000 and categorized hallucinations from a problem-presentation perspective.Source: §I · p. 1

Study design

  • The study applies manual open coding to CoderEval tasks and analyzes generated code from six mainstream LLMs to develop and examine a practical-code hallucination taxonomy.Researchers first annotate 10% of tasks to form a preliminary taxonomy, then iteratively label the remaining 90% while refining it.Source: §I · p. 2

Findings

  • Hallucinations comprise three major categories and eight subcategories; task-requirement conflicts are most prevalent across models, and four potential contributing factors are identified.The factors are training-data quality, intention-understanding capacity, knowledge-acquisition capacity, and repository-level context awareness.Source: §I · p. 2

Mitigation

  • A lightweight RAG approach retrieves repository code snippets relevant to each task as prompts and consistently improves performance across all studied LLMs.The retrieval library is built from the repository for each task, with snippets selected by similarity between the task description and code.Source: §I · p. 2

II. BACKGROUND & RELATED WORK · A. LLM-based Code Generation · B. Hallucinations in LLMs

The background motivates repository-level code generation by highlighting its dependence on repository-specific APIs and frames code hallucination as an underdefined problem in real-world development. It connects existing NLP hallucination concepts to the paper’s focus on software-engineering contexts.

A. LLM-based Code Generation

  • Realistic code generation relies on repository-specific APIs, including private APIs unavailable to language models, so many functions are not independent.Source: A. LLM-based Code Generation · p. 2
  • Dependent functions account for more than 70% of functions in the 100 most popular Java and Python GitHub projects.Source: A. LLM-based Code Generation · p. 2
  • CoderEval, ClassEval [60], and EvoCodeBench [61] test repository code in its environment, yet models perform poorly and tend to generate independent segments.These benchmarks use code snippets and descriptions collected from real repositories, with test cases checking correctness.Source: A. LLM-based Code Generation · p. 2

B. Hallucinations in LLMs

  • In NLP, hallucination describes generated content inconsistent with the input or expected environment, meaningless, or factually incorrect.Source: B. Hallucinations in LLMs · p. 2
  • NLP hallucinations are categorized as input-conflicting, context-conflicting, or fact-conflicting, involving deviation from the source, prior output, or established facts [69].Source: B. Hallucinations in LLMs · pp. 2–3
  • Prior code-generation work defined five hallucination types [8], but omitted real-world software-engineering factors such as environments, resources, external constraints, and code repositories.The paper addresses this gap with empirical study using real-development datasets and new hallucination types.Source: B. Hallucinations in LLMs · p. 3
  • Code hallucinations can reduce code quality and maintainability and cause unexpected errors or security vulnerabilities.Source: B. Hallucinations in LLMs · p. 3

III. EVALUATION SETUP · A. Dataset · B. Studied LLMs

The evaluation uses CoderEval tasks from real-world Python repositories and six mainstream LLMs spanning open- and closed-source models and different parameter sizes.

A. Dataset

  • CoderEval supplies 230 Python repository tasks, each with a natural-language description, ground-truth snippet, test cases, and project environment context.The benchmark is used to simulate practical development scenarios.Source: A. Dataset · p. 3

B. Studied LLMs

  • For code generation on the studied programming tasks, the study uses several mainstream LLMs covering both open-source and closed-source models and spanning various parameter sizes.The selected models are ChatGPT GPT-3.5-Turbo, CodeGen-350M-Mono, PanGu-α-2.6B [44], DeepSeekCoder-6.7B base [51], and CodeLlama-7b-Python-hf.Source: B. Studied LLMs · p. 3
  • The ChatGPT model used is GPT-3.5-Turbo.The paper describes ChatGPT as a versatile text-generation model with code-generation capabilities.Source: B. Studied LLMs · p. 3
  • For its generation task, the study uses the CodeGen-350M-Mono model and the PanGu-α-2.6B model; PanGu-α can perform code-generation tasks in multiple languages [44].Source: B. Studied LLMs · p. 3
  • The study uses the DeepSeekCoder-6.7B base model [51].The paper notes its performance among open-source models across programming languages and benchmarks.Source: B. Studied LLMs · p. 3
  • The study also uses CodeLlama-7b-Python-hf and StarCoder2-7B.CodeLlama ranges from 7 to 34 billion parameters; StarCoder2 offers three model scales.Source: B. Studied LLMs · p. 3
  • For each task, each LLM generates 10 code snippets using nuclear sampling at temperature 0.6, following CoderEval settings.Source: B. Studied LLMs · p. 3

C. Taxonomy Annotation

The authors used manual open coding [65] to develop a hallucination taxonomy, progressing from preliminary analysis to independent annotation of the remaining code snippets.

Initial Open Coding

  • Two authors tested and compared 1,380 generated snippets with ground truth after sampling 23 CoderEval tasks across six models and generating ten snippets per task.The 23 tasks represented 10% of the 230 CoderEval Python coding tasks.Source: C. Taxonomy Annotation · p. 3

Preliminary Taxonomy Construction

  • Annotators documented hallucination locations, discussed the code, and grouped similar phenomena into a preliminary taxonomy, allowing multiple hallucination types per snippet.The taxonomy described the types and their meanings in LLM-generated code.Source: C. Taxonomy Annotation · p. 3

Full Taxonomy Construction

  • Three experienced Python volunteers independently annotated the remaining snippets and documented uncovered hallucinations for discussion and taxonomy updates.Two annotators had more than ten years of programming experience, and one had four years.Source: C. Taxonomy Annotation · p. 3

IV. EVALUATION RESULTS · A. RQ1: Hallucination Taxonomy · 1) Task Requirement Conflicts (43.53%):

The evaluation addresses three research questions, and its hallucination taxonomy groups generated-code errors into requirement, factual-knowledge, and project-context conflicts. Task requirement conflicts account for 43.53% and include violations of functional and non-functional requirements.

IV. EVALUATION RESULTS

  • The evaluation-results section presents findings intended to answer the paper’s three research questions.The taxonomy discussion introduces the detailed hallucination types and points to a distribution figure, without reporting its values in the supplied passages.Source: §IV · p. 4, A. RQ1: Hallucination Taxonomy · p. 4

A. RQ1: Hallucination Taxonomy

  • The taxonomy identifies three primary categories and eight specific types: Task Requirement, Factual Knowledge, and Project Context Conflicts.These correspond, respectively, to input-conflicting, knowledge-conflicting, and context-conflicting hallucinations in the general domain.Source: A. RQ1: Hallucination Taxonomy · p. 4

1) Task Requirement Conflicts (43.53%):

  • In code generation, Task Requirement Conflicts occur when generated code does not align with task requirements; they comprise Functional Requirement Violation (36.66%) and Non-functional Requirement Violation (6.86%), totaling 43.52%.Source: 1) Task Requirement Conflicts (43.53%) · pp. 4–5
  • Functional Requirement Violations can involve wrong or missing functionality, causing logic bugs or runtime errors when requirements are misunderstood or incompletely understood.The paper gives PanGu-α’s omission of timezone-specific LocalTime handling as an example of a functional requirement violation.Source: 1) Task Requirement Conflicts (43.53%) · pp. 4–5
  • Non-functional requirements include security, performance, style, and code-smell concerns; violating them can create safety risks or increase project maintenance complexity.Examples include unsanitized inputs, inefficient loops, inconsistent conventions, overly complex functions, and excessive global variables.Source: 1) Task Requirement Conflicts (43.53%) · p. 5

2) Factual Knowledge Conflicts (31.91%):

Factual knowledge conflicts account for 31.91% of hallucinations and arise when generated code misapplies background, library/framework, or API knowledge.

Background Knowledge Conflicts

  • Background knowledge conflicts (8.82%) occur when generated code contradicts domain concepts or standards, potentially causing invalid code, logic bugs, compliance issues, or safety risks.An OCFL storage-root example shows generated code focusing on irrelevant aspects rather than the specification's required conformance declaration.Source: 2) Factual Knowledge Conflicts (31.91%) · p. 5

Library Knowledge Conflicts

  • Library knowledge conflicts (2.68%) arise when models misuse framework or third-party library capabilities, producing unexpected behavior or security risks.For a caching decorator task, DeepSeekCoder used asyncio, an asynchronous-processing framework, inappropriately.Source: 2) Factual Knowledge Conflicts (31.91%) · pp. 5–6, Figure 6 · p. 6

API Knowledge Conflicts

  • API knowledge conflicts (20.41%) include parameter errors, missing precondition checks, similar-but-wrong or deprecated APIs, and improper exception handling.These misuses can cause functional errors, runtime failures, instability, or reduced usability; CodeGen, for example, used a nonexistent year parameter for datetime.timedelta().Source: 2) Factual Knowledge Conflicts (31.91%) · p. 6, Figure 7 · p. 6

3) Project Context Conflicts (24.56%):

Project Context Conflicts (24.56%) arise when generated code violates project-specific facts, which are generally unavailable publicly. They comprise Environment, Dependency, and Non-code Resource Conflicts.

3) Project Context Conflicts (24.56%):

  • Project Context Conflicts account for 24.56% and concern project-specific facts, unlike Factual Knowledge Conflicts, which involve publicly accessible facts.The paper attributes these conflicts to LLMs lacking awareness of project-specific information and divides them into environment, dependency, and non-code resource types.Source: 3) Project Context Conflicts (24.56%) · p. 6
  • Environment Conflicts account for 0.94% and occur when generated code mismatches the development environment, such as using unsupported language features or nonexistent packages.The passage gives higher-version Python f-strings and an unavailable _lfu_cache package as examples.Source: 3) Project Context Conflicts (24.56%) · p. 6
  • Dependency Conflicts account for 11.26%; because 70% of functions are non-standalone, limited project context can lead LLMs to use nonexistent APIs, functions, attributes, or variables.These dependencies may be user-defined or imported; resulting errors include undefined variables and no-member errors.Source: 3) Project Context Conflicts (24.56%) · pp. 6–7
  • Non-code Resource Conflicts account for 12.36% and involve data, configuration, asset, or connection mismatches that can cause errors, display issues, failed operations, or security incidents.One example adds an “archive” directory to a target path, pointing to a nonexistent file and causing a program error.Source: 3) Project Context Conflicts (24.56%) · p. 7
  • The RQ1 taxonomy contains three main categories and eight subtypes, with Task Requirement Conflicts the most frequent category.Project Context Conflicts are one of the three main categories.Source: 3) Project Context Conflicts (24.56%) · p. 7

B. RQ2: LLM Comparison

Task Requirement Conflicts were the most prevalent hallucination type across the models, with CodeGen and StarCoder2 showing higher frequencies than other models. Factual and Project Context Conflicts occurred at approximately similar frequencies, while PanGu-α had the most Factual Knowledge Conflicts.

B. RQ2: LLM Comparison

  • Task Requirement Conflicts were most prevalent across models, with CodeGen and StarCoder2 showing notably higher frequencies than others.DeepSeekCoder and CodeLlama had the lowest occurrence of Task Requirement Conflicts.Source: Figure 11 · p. 8, B. RQ2: LLM Comparison · p. 8
  • Factual Knowledge Conflicts and Project Context Conflicts occurred at approximately similar frequencies across models, while PanGu-α had the most Factual Knowledge Conflicts.The paper suggests PanGu-α’s extensive training on Chinese corpora may have limited its exposure to English factual knowledge, including domain concepts.Source: B. RQ2: LLM Comparison · p. 8
  • Differences in Task Requirement Conflicts may relate to requirement-understanding ability, potentially influenced by model size or training corpora.DeepSeekCoder and CodeLlama were trained on diverse code and text corpora, whereas CodeGen and StarCoder2 were primarily trained on code-related data.Source: B. RQ2: LLM Comparison · p. 8

C. RQ3: Root Cause Analysis · 1) Training Data Quality: · 2) Intention Understanding Capacity:

The analysis identifies training-data quality and intention understanding as potential roots of hallucinations in practical code generation: flawed repository data can shape model outputs, while pattern-based generation can miss specific requirements.

C. RQ3: Root Cause Analysis

1) Training Data Quality:

  • Training corpora may include mismatched docstrings, inefficient or insecure implementations, misused APIs, outdated library guidance, and limited domain diversity.Such flaws can affect models’ inference capabilities.Source: 1) Training Data Quality · p. 8
  • Hallucinations can reflect flaws learned from training data, including unsafe API use; data quality may contribute to Task Requirement and Factual Knowledge Conflicts.The paper points to Figure 4 as an unsafe-API example and highlights the need for high-quality code-related training data.Source: 1) Training Data Quality · p. 8

2) Intention Understanding Capacity:

  • LLMs may generate from common training patterns rather than deeply interpreting specific user intentions, causing functionally or non-functionally inaccurate code [49] [48].The paper describes this as a limitation rooted in pattern recognition.Source: 2) Intention Understanding Capacity · p. 8
  • In the cited example, the task required handling LocalTime, but the generated code ignored it, illustrating incomplete interpretation of requirements.The paper points to Figure 3 for this example.Source: 2) Intention Understanding Capacity · p. 8
  • Poor understanding of task scope can leave complex logic or multi-step requirements partly addressed and cause failures on edge cases or specific business logic [47].Such snippets may appear correct while failing practical functional requirements.Source: 2) Intention Understanding Capacity · p. 8

3) Knowledge Acquisition Capacity: … B. RAG-based Mitigation

The paper links practical-code hallucinations to gaps in learned knowledge and access to repository context, motivating retrieval-augmented generation. Its mitigation builds repository-specific code corpora and retrieves relevant snippets for generation.

3) Knowledge Acquisition Capacity:

  • LLMs may learn incorrect knowledge, miss domain-specific facts, or lack information introduced after training, leading to incorrect or outdated code facts.The paper uses incorrect OCFL-related code as an example and motivates a mechanism such as RAG to update, correct, or supplement learned knowledge.Source: 3) Knowledge Acquisition Capacity · p. 8

4) Repository-level Context Awareness:

  • Feeding all repository context is impractical because token limits and quadratic computation growth constrain input size, while irrelevant context can distract from the coding task.The paper identifies static-analysis tools [78] and repository-level retrieval corpora [62] as approaches to improve project-context awareness.Source: 4) Repository-level Context Awareness · p. 9
  • The analysis identifies four possible contributors to hallucinations: training-data quality, intention understanding, knowledge acquisition, and repository-level context awareness.Deficiencies in any of these factors can lead to hallucinations in practical development scenarios.Source: 4) Repository-level Context Awareness · p. 9

A. Motivation

  • At inference, hallucinations can stem from misunderstanding requirements, lacking task-specific factual knowledge, or being unable to access necessary repository code and non-code resources.The authors propose providing relevant code snippets so LLMs can better understand requirements and gain factual and project-context awareness.Source: A. Motivation · p. 9

B. RAG-based Mitigation

  • The RAG corpus is built from repository source files using 20-line windows stepped by 2 lines, excluding ground-truth code and subsequent lines to prevent answer leakage.The procedure follows RepoCoder’s parameter settings [62].Source: B. RAG-based Mitigation · p. 9
  • Retrieval tokenizes the query and code snippets, ranks them by Jaccard similarity, and returns the top ten snippets as the LLM prompt.The method uses a sparse bag-of-words model and computes Jaccard similarity as intersection size divided by union size.Source: B. RAG-based Mitigation · p. 9

C. Evaluation · VI. DISCUSSION

On CodeEval, RAG improves Pass@1 across all six evaluated LLMs, though the reported gains are modest and the method is preliminary. The discussion calls for better hallucination detection and broader mitigation methods and knowledge sources.

C. Evaluation

  • RAG improves Pass@1 for all six evaluated LLMs over the Raw method, but the gains are modest and the study presents this as a preliminary pilot.The Raw method supplies docstrings and function signatures; RAG adds ten related code snippets retrieved from the repository library.Source: C. Evaluation · p. 9
  • Case studies show RAG resolving Task Requirement Conflicts for CodeGen and Project Context Conflicts for ChatGPT by using repository-consistent functions.CodeGen uses splitlines rather than replace; ChatGPT uses self.pop() rather than the nonexistent self.items.popitem() API.Source: C. Evaluation · pp. 9–10, Figure 12 · p. 10, Figure 13 · p. 10

VI. DISCUSSION

  • Static analysis and tests can reveal undefined variables, runtime errors, or test failures, but incomplete functionality and security issues may evade both and reach production.Such undetected hallucinations can contribute to unreliable software and severe security risks.Source: §VI · p. 10
  • LLM self-feedback localization approaches [71], [72] detect some hallucinations but depend heavily on model capabilities and cannot overcome training-corpus limitations.Source: §VI · p. 10
  • Future work should develop faster, more precise techniques to identify and localize hallucinations in LLM-generated code.Source: §VI · p. 10
  • RAG’s potential remains underexplored: repository-only retrieval provides insufficient information for some hallucinations, including background knowledge conflicts.The paper suggests augmenting retrieval with online search engines, API documents, and StackOverflow discussions.Source: §VI · p. 10
  • Input query refinement [4], [49] and multi-agent systems [73] could support iterative clarification, code generation, testing, and mitigation.This requires suitable agent–tool interaction protocols and prompting strategies for tools such as search engines and static analysis.Source: §VI · p. 10

VII. THREATS TO VALIDITY · VIII. CONCLUSION

The conclusion summarizes the study’s taxonomy, analysis of hallucination causes and distributions, and RAG-based mitigation; the threats section describes limits involving scope, annotation, and evaluation.

VII. THREATS TO VALIDITY

  • The taxonomy and root-cause exploration focus on Python, and extending the taxonomy to other programming languages remains future work.Source: §VII · p. 10
  • CoderEval contains 230 tasks; the authors used six LLMs and generated 10 snippets per task to obtain sufficient annotations.Source: §VII · p. 10
  • The taxonomy’s manual annotation lacked a formal inter-rater reliability measure, so annotators resolved discrepancies in meetings and one author reviewed all labels.Source: §VII · p. 10
  • To mitigate model bias during annotation, the authors mixed generation results from the six models before annotation.Source: §VII · p. 10
  • To address construct-validity threats in evaluating mitigation, the authors tested six models using CoderEval test cases, a standard method for evaluating generated-code correctness.Source: §VII · p. 10

VIII. CONCLUSION

  • The study constructs a taxonomy of hallucinations in practical code-generation scenarios through full manual analysis.Source: §VIII · p. 11
  • The authors discuss hallucination causes and distributions across different LLMs.Source: §VIII · p. 11
  • The authors implement a RAG-based hallucination mitigation approach and discuss potential mitigation approaches.Source: §VIII · p. 11

Related papers

Loading 2409.20550v2…