PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code

Fang Liu, Yang Liu, Lin Shi, Zhen Yang, Li Zhang, Xiaoli Lian, Zhongqi Li, Yuchi Ma

arXiv:2404.00971v3cs.SEcs.AI

TL;DR

This study addresses the limited understanding of hallucinations in LLM-generated code, where outputs may conflict with requirements, contain inconsistencies, or reflect inaccurate knowledge. It thematically analyzes generated code to build a taxonomy and examine hallucination distributions, causes, impacts, and prompt-based mitigation.

  • Problem

    Research has mainly examined hallucinations in natural-language generation, leaving code-specific hallucination types, distributions, causes, and impacts insufficiently understood despite deployment risks.

  • Method

    The study thematically analyzed 3,120 code samples from two benchmarks using open coding, developing a taxonomy with 3 primary categories and 12 specific hallucination types.

  • Results

    The analysis finds frequent code hallucinations, especially requirement-conflicting cases, and links them to incorrect functionality, poor readability, poor maintainability, and low efficiency.

  • Takeaways & Limitations

    The study provides a basis for evaluating code hallucinations and explores training-free prompt enhancement as a lightweight mitigation approach.

  • Takeaways & Limitations

    The empirical study targets Python and Java tasks from two code-generation benchmarks, limiting the stated scope of its generalizability.

Abstract

from arXiv · show

The rise of Large Language Models (LLMs) has significantly advanced various applications on software engineering tasks, particularly in code generation. Despite the promising performance, LLMs are prone to generate hallucinations, which means LLMs might produce outputs that deviate from users' intent, exhibit internal inconsistencies, or misaligned with the real-world knowledge, making the deployment of LLMs potentially risky in a wide range of applications. Existing work mainly focuses on investigating the hallucination in the domain of Natural Language Generation (NLG), leaving a gap in comprehensively understanding the types, causes, and impacts of hallucinations in the context of code generation. To bridge the gap, we conducted a thematic analysis of the LLM-generated code to summarize and categorize the hallucinations, as well as their causes and impacts. Our study established a comprehensive taxonomy of code hallucinations, encompassing 3 primary categories and 12 specific categories. Furthermore, we systematically analyzed the distribution of hallucinations, exploring variations among different LLMs and benchmarks. Moreover, we perform an in-depth analysis on the causes and impacts of various hallucinations, aiming to provide valuable insights into hallucination mitigation. Finally, to enhance the correctness and reliability of LLM-generated code in a lightweight manner, we explore training-free hallucination mitigation approaches by prompt enhancing techniques. We believe our findings will shed light on future research about code hallucination evaluation and mitigation, ultimately paving the way for building more effective and reliable code LLMs in the future. The replication package is available at https://github.com/Lorien1128/code_hallucination

I. INTRODUCTION

This study addresses the limited understanding of hallucinations in LLM-generated code by categorizing their types, distribution, causes, and impacts. It also preliminarily explores training-free prompt-enhancement strategies for mitigation.

  • Research gap: LLM-generated code hallucinations remain insufficiently understood regarding their types, causes, consequences, and distribution across models and benchmarks.Prior hallucination research has focused mainly on natural language generation, while code generation has stricter semantic and functional constraints.
  • Approach: The study analyzes 3,120 code samples from different LLMs on two widely used code-generation benchmarks using thematic open coding.The analysis categorizes hallucinations according to code-specific characteristics and examines their causes and impacts.
  • Contributions: The resulting taxonomy contains 3 primary categories and 12 specific hallucination types.The three primary categories are Requirement Conflicting hallucinations, Code Inconsistency hallucinations, and Knowledge hallucinations.
  • Mitigation: Prompting models to refine requirements or use chain-of-thought reasoning helps mitigate Requirement Conflicting hallucinations, while domain knowledge reduces Knowledge hallucinations.These preliminary mitigation approaches require no model-parameter updates.
  • Contributions: The paper systematically analyzes hallucination distributions across LLMs and code-generation tasks, along with their causes and impacts.These analyses are intended to clarify where hallucinations occur and how they affect generated code.

II. PRELIMINARY

The paper defines code hallucination as a direct semantic conflict between generated code and established facts. It distinguishes conflicts involving requirements, contextual code, and real-world knowledge while excluding indirect reasoning errors.

  • Terminology: Established facts derive from requirements, contextual code, or publicly available and verifiable real-world knowledge.Examples include input/output constraints, variable names or API usage, and mathematical laws.
  • Definition of Code Hallucination: Code hallucination is an explicit semantic conflict caused by failure to retain, recall, or process information accurately.The relevant information may come from user requirements, contextual code, or real-world knowledge.
  • Taxonomy: The taxonomy includes Requirement Conflicting, Code Inconsistency, and Knowledge hallucinations.The first two conflict with task requirements or contextual code, while Knowledge hallucinations conflict with real-world knowledge.
  • Scope boundary: The definition excludes indirect conflicts requiring logical reasoning or calculations because those are treated as inference or deduction errors.Hallucinations instead require a direct misalignment with provided or expected information.

C. Difference between Code Hallucinations and Errors

The paper separates hallucinations from general code errors by requiring direct semantic conflict for hallucinations. Syntax, reasoning, incompleteness, and structural invalidity are treated as distinct error types.

  • Distinction: Code containing hallucinations does not necessarily contain execution errors, and code without hallucinations may still contain general errors.Useless statements can be hallucinatory without usually causing functional errors.
  • Error types: Syntactic errors are nonsemantic problems usually detectable during compilation.They are distinguished from semantic conflicts between generated code and established facts.
  • Error types: Logical reasoning errors produce incorrect deductions or calculations without directly conflicting with requirements.The generated code may remain semantically consistent with the stated requirements while implementing incorrect logic.
  • Error types: Incomplete implementation partially fulfills requirements without directly conflicting with their semantics.This differs from invalid generation, which is formally or structurally incomplete.
  • Error types: Invalid generation involves structurally incomplete code, such as malformed function definitions or missing control structures.Unlike incomplete implementation, the code lacks formal completeness or coherent structure.

C. Code Hallucination

The study builds on prior code-generation benchmarks and hallucination research by constructing a taxonomy-focused empirical analysis. It uses diverse models and tasks to examine hallucinations in standalone and repository-level function generation.

  • Related work: Prior hallucination studies examined repository-level conflicts and package hallucinations, including erroneous references to nonexistent libraries.These studies motivate broader analysis of hallucination categories in code generation.
  • Related work: Existing code-generation benchmarks cover standalone functions, multi-dependency tasks, repository-level tasks, competitions, and dynamically updated settings.Researchers have evaluated generated code for properties including security, usability, and correctness.
  • Study design: The study constructs its taxonomy by collecting LLM-generated samples, applying open coding, and iteratively establishing hallucination categories.This procedure is designed to analyze hallucinations according to their semantic conflict sources.
  • Study design: The evaluation uses HumanEval and CoderEval, covering standalone and repository-level function generation in Python and Java.HumanEval contains 164 Python tasks, while CoderEval includes Python and Java functions and methods from open-source projects.
  • Study design: The study evaluates CodeLlama, GPT-4, DeepSeek-Coder, and DeepSeek-R1, generating one code sample per problem with greedy decoding.Greedy decoding is used to reduce nondeterministic variability in the results.

2) Manual Analysis:

The manual analysis used thematic coding of generated code to build and refine a taxonomy of hallucinations. It applied expert review, independent annotation, and quality-control procedures to identify hallucinatory snippets.

  • The analysis began with a pilot study of 500 randomly sampled code snippets to enrich categories, establish a codebook, and develop an initial taxonomy.
  • The remaining 80% of code snippets were independently labeled by ten annotators with Java or Python programming experience to refine and expand the taxonomy.
  • Two experts independently evaluated the pilot samples using problem descriptions, reference code, and execution results to identify hallucinations.
  • Each later sample was annotated by two different participants, while new uncovered categories, causes, or impacts were documented for codebook refinement.
  • 1,212 hallucinatory snippets were found among 3,120 generated samples, producing a taxonomy with 3 primary categories and 12 leaf categories.

B. Taxonomy of Code Hallucinations

The taxonomy organizes code hallucinations into requirement conflicts, code inconsistencies, and knowledge conflicts, with distinct subcategories and concrete examples. Requirement conflicts are the most prevalent and consequential category.

  • Requirement Conflicting: Requirement Conflicting hallucinations account for 39.60% and occur when generated code semantics directly conflict with the stated requirements.
  • Requirement Conflicting: Behavior Conflicting represents 35.40% and includes deviations in functional logic or execution flow, such as using a value where the requirement specifies a dictionary key.
  • Code Inconsistency: Code Inconsistency accounts for 25.50% and includes Undefined Variables, Useless Statements, Inconsistent Libraries, and Fragmented Logics.
  • Code Inconsistency: Useless Statements comprise unexecuted statements at 0.58% and executed statements without output impact at 6.02%.
  • Knowledge Hallucinations: Knowledge Hallucinations conflict with real-world knowledge, including Computer Science knowledge conflicting at 33.25%, Mathematics and Natural Science at 1.40%, and Common Sense at 0.25%.
  • Knowledge Hallucinations: Computer Science knowledge conflicts include Library/Project knowledge at 25.99%, Algorithm knowledge at 4.95%, and Computer Theory knowledge at 2.31%.

V. CODE HALLUCINATION ANALYSIS

The analysis compares hallucination distributions across LLMs and benchmarks, finding recurring category differences and model-related patterns. It also highlights subtle inconsistencies and benchmark-specific knowledge demands.

  • Overall Distribution of Hallucinations: Behavior Conflicting is the most frequent hallucination type across LLMs except DeepSeek-R1, where it ranks third.
  • Overall Distribution of Hallucinations: Library/Project and Undefined Variables hallucinations occur frequently, particularly in CoderEval, where generated functions depend on project and library context.
  • Overall Distribution of Hallucinations: DeepSeek-R1 contains only a small fraction of executed Useless Statements hallucinations, while such issues appear more often in other models.
  • Overall Distribution of Hallucinations: Although Inconsistent Libraries are least frequent, they are subtle, difficult to detect, and may increase debugging costs and potential risks.
  • Overall Distribution of Hallucinations: Hallucination frequency may decrease as model parameter size increases across DeepSeek-Coder-1.3B, DeepSeek-Coder-7B, and DeepSeek-R1-671B.

2) Hallucination Distribution across Benchmarks:

Across HumanEval and CoderEval, Behavior Conflicting hallucinations are most prevalent, while other hallucination types vary with task scenario, language, and available context. Different models show largely consistent hallucination-type distributions.

  • HumanEval contains more Mathematics & Natural Science knowledge conflicts and Useless Statements because its tasks emphasize mathematics, algorithms, and logical reasoning.
  • CoderEval contains more Library/Project hallucinations because repository-level tasks depend on contextual code and third-party libraries.
  • Hallucination-type distributions remain largely consistent across models, supporting the robustness of the taxonomy.
  • Behavior Conflicting hallucinations are the most prevalent type across all evaluated LLMs and benchmarks.
  • Larger models generally tend to hallucinate less, while hallucination distributions vary across programming scenarios and languages.

B. RQ2. Hallucination Cause Analysis.

The analysis attributes code hallucinations to model-related limitations and prompt deficiencies, especially missing domain knowledge and unclear requirements. Model-related causes dominate overall and contribute to nearly every hallucination category.

  • Cause proportions are calculated over samples containing hallucinations, and multi-cause or multi-impact samples retain one cause and impact for visualization.
  • 1.90% of hallucinations involve Ambiguous or Incomplete requirements that leave task behavior or constraints open to interpretation.For example, “merging” integer lists may mean concatenation or elementwise addition, prompting the model to guess.
  • 26.24% of hallucinations involve Lack of Domain-Specific Knowledge when necessary codebase or specialized-domain information is absent from the prompt.
  • 80.86% of hallucinations are attributed to Model-related Causes, substantially exceeding other cause categories.These causes include poor training data, small parameter size, optimization objectives, and architecture deficiencies.
  • Model-related Causes are primary contributors to almost all hallucination categories, although requirement factors also matter for Undefined Variables and Library/Project hallucinations.

2) Impact of Prompt Length and Complexity:

Prompt complexity is associated with hallucination rates more clearly than prompt length, whose effect is largely explained by its relationship with complexity. Hallucinations most often produce incorrect functionality, while also affecting readability, efficiency, and maintainability.

  • 2) Impact of Prompt Length and Complexity:: Higher prompt complexity consistently correlates with increased hallucination rates, whereas prompt length alone shows no significant effect when complexity is fixed.
  • 2) Impact of Prompt Length and Complexity:: The 95% Bootstrap confidence interval for prompt-length mediation through complexity is (0.0113, 0.0725), with both bounds above zero.
  • C. RQ3. Impact of Hallucination.: 95.30% of hallucinations result in Incorrect Functionality, making it the most frequent impact.
  • C. RQ3. Impact of Hallucination.: 14.27% of hallucinations cause Poor Readability, potentially hindering collaboration and extending debugging and maintenance work.
  • C. RQ3. Impact of Hallucination.: 5.20% of hallucinations cause Low Efficiency, although compiler optimizations may mitigate these effects during execution.
  • C. RQ3. Impact of Hallucination.: 3.88% of hallucinations cause Poor Maintainability and Scalability through problematic coding style, structure, or interfaces.
  • C. RQ3. Impact of Hallucination.: Eight of ten hallucination categories result in Incorrect Functionality, with Behavior Conflicting the most frequent hallucination category.
  • C. RQ3. Impact of Hallucination.: Hallucinatory code is more prone to errors, although non-hallucinatory code can also contain unrelated errors.

VI. DISCUSSION

The discussion frames prompt enhancement as a lightweight mitigation direction guided by the hallucination taxonomy. Preliminary experiments with Self-Refine, CoT, and RAG reduced hallucinations in their targeted settings, with different strategies addressing different causes.

  • Self-Refine and chain-of-thought prompting are proposed for Requirement Conflicting hallucinations, while RAG targets knowledge conflicts.
  • The three prompting strategies substantially reduce hallucinations in their respective target datasets, with particularly effective mitigation of Requirement Conflicting hallucinations.
  • RAG notably reduces Knowledge Hallucinations on CoderEval by supplying relevant project-specific context.
  • Self-Refine can make a Behavior Conflicting hallucination milder, but the illustrated refined code remains partially conflicting with the requirement.
  • The taxonomy is presented as guidance for selecting mitigation methods according to developer priorities, such as time complexity.

B. Implications

The study argues that hallucination-aware evaluation and mitigation are needed because functional correctness alone does not fully assess generated-code quality and reliability. It also identifies task scope as an important direction for future research.

  • Functional correctness metrics and benchmarks should be supplemented with measures that detect hallucinations in generated code.The authors link this broader evaluation to more comprehensive assessment of code quality and reliability.
  • Most hallucinations are attributed to model-related causes, motivating stronger reasoning capabilities through advanced training or fine-tuning.
  • Integrating retrieval-augmented generation and domain-specific knowledge is proposed as a promising direction for hallucination mitigation.These techniques could provide external information to support and refine the model’s reasoning process.
  • Because the study focuses on NL2Code, hallucination patterns may differ in code translation, unit-test generation, program repair, and code review.The authors identify task-specific investigation as an opportunity for future work.

2) Implications for Developers:

The study emphasizes clear prompting, result checking, and testing as practical safeguards for reducing and detecting code hallucinations. Its conclusions are bounded by the studied languages, benchmarks, and code-generation setting.

  • Implications for Developers:: Models with larger parameter sizes tend to hallucinate less, making robust model selection relevant for developers.
  • Implications for Developers:: Clear, concise, and comprehensive prompts can improve the quality of generated code and support hallucination mitigation.Prompt engineering is presented as a promising training-free mitigation direction.
  • Implications for Developers:: Rigorous result checking, iterative prompt refinement, code review, and testing help validate logical correctness and consistency with requirements.
  • Threats to Validity: The empirical study covers Python and Java tasks from two benchmarks, including standalone and repository-level function generation.The authors note that other programming languages and code-generation scenarios remain open areas for investigation.
  • Threats to Validity: Manual labeling of hallucination categories, causes, and impacts is somewhat subjective, creating a threat to internal validity.
  • Conclusion: The study reports that code LLMs are frequently affected by hallucinations, which can lead to incorrect functionality, poor readability, poor maintainability, and low efficiency.
Loading 2404.00971v3…