Source-linked AI summary
LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation
Ziyao Zhang, Yanlin Wang, Chong Wang, Jiachi Chen, Zibin Zheng
TL;DR
Practical repository-level code generation exposes hallucinations that are insufficiently characterized by standalone-function studies. This paper manually studies six LLMs to build a taxonomy, analyze distributions and causes, and test RAG mitigation, which consistently improves all studied models.
Problem
Prior code-generation hallucination research focused on standalone functions, leaving practical repository-level hallucinations involving contextual dependencies and software-engineering factors insufficiently studied.
Method
The authors manually analyze six LLMs on CoderEval to construct a hallucination taxonomy, investigate distributions and four potential factors, and evaluate repository-based RAG prompts.
Results
The study identifies three major hallucination categories with eight subcategories, finds Task Requirement Conflicts most prevalent, and reports consistent improvement from lightweight RAG across studied LLMs.
Takeaways & Limitations
The findings connect practical code-generation hallucinations to task requirements, factual knowledge, project context, and repository information used for mitigation.
Takeaways & Limitations
The study focuses on Python and a limited CoderEval dataset of 230 tasks, and its manual annotation lacks a formal inter-rater reliability measure.
Abstract
from arXiv · showhide
Code generation aims to automatically generate code from input requirements, significantly enhancing development efficiency. Recent large language models (LLMs) based approaches have shown promising results and revolutionized code generation task. Despite the promising performance, LLMs often generate contents with hallucinations, especially for the code generation scenario requiring the handling of complex contextual dependencies in practical development process. Although previous study has analyzed hallucinations in LLM-powered code generation, the study is limited to standalone function generation. In this paper, we conduct an empirical study to study the phenomena, mechanism, and mitigation of LLM hallucinations within more practical and complex development contexts in repository-level generation scenario. First, we manually examine the code generation results from six mainstream LLMs to establish a hallucination taxonomy of LLM-generated code. Next, we elaborate on the phenomenon of hallucinations, analyze their distribution across different models. We then analyze causes of hallucinations and identify four potential factors contributing to hallucinations. Finally, we propose an RAG-based mitigation method, which demonstrates consistent effectiveness in all studied LLMs. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/LLMCodingHallucination
I. INTRODUCTION
Practical repository-level code generation introduces contextual dependencies that standalone-function benchmarks do not fully capture. This study investigates hallucination manifestations, distributions, causes, and RAG-based mitigation across six LLMs.
- Motivation: Repository-level generation involves contextual dependencies beyond standalone-function generation, including user-defined functions and project-defined data protocols.Existing benchmarks primarily measure functional correctness through test-case pass rates and provide limited analysis of failure causes.
- Study Scope: The study examines hallucinations in practical code generation through three research questions covering taxonomy, LLM comparisons, and root causes.The authors also evaluate a mitigation approach based on retrieval-augmented generation.
- Study Scope: Six mainstream LLMs are evaluated on CoderEval, while manual open coding iteratively develops a taxonomy from generated code.The process samples 10% of tasks initially, refines the taxonomy, and labels the remaining 90%.
- Findings: Hallucinations are organized into three major categories and eight subcategories, with Task Requirement Conflicts most prevalent across models.The categories are Task Requirement Conflicts, Factual Knowledge Conflicts, and Project Context Conflicts.
- Findings: The study identifies four potential hallucination factors and proposes a lightweight RAG mitigation method that consistently improves all studied LLMs.The retrieval library is built from repository code, and similarity matching supplies relevant snippets as prompts.
II. BACKGROUND & RELATED WORK
Hallucinations are a recognized problem in generative LLMs, but code-generation research has insufficient coverage of practical software-engineering contexts. Earlier work focused mainly on standalone functions, whereas this study targets repository-level generation.
- Code Generation Context: Code generation automates production of code from natural-language specifications, reducing manual coding workload and supporting developers’ higher-level tasks.The paper motivates repository use because private APIs and project resources may be unavailable to pretrained models.
- Hallucinations: Natural-language hallucinations include content conflicting with inputs, facts, or contextual information, and code-generation hallucinations can similarly undermine reliability.The background links fact conflicts to limitations in training data, knowledge updates, or reasoning capabilities.
- Related Work: Prior code-generation hallucination research studied standalone function or script benchmarks and categorized fine-grained semantic issues such as dead code and repetition.The present work instead examines repository-level generation from a holistic perspective spanning phenomena, mechanisms, and mitigation.
- Research Gap: The paper identifies a research gap around hallucinations involving software-engineering factors such as development environments, system resources, external constraints, and code repositories.These factors are presented as relevant to failures in actual development scenarios.
III. EVALUATION SETUP
The evaluation uses real-world Python repository tasks and compares mainstream open- and closed-source LLMs under repeated sampling. Each task is evaluated in its associated project context.
- Dataset: CoderEval provides 230 Python code-generation tasks extracted from diverse repositories, each containing a description, ground-truth snippet, test cases, and project environment context.The dataset is used to simulate practical development scenarios.
- Studied LLMs: The study evaluates mainstream LLMs spanning open- and closed-source models and different parameter sizes.The model set includes ChatGPT, CodeGen, PanGu-α, DeepSeekCoder, CodeLlama, and StarCoder2.
- Studied LLMs: CodeLlama is represented by CodeLlama-7b-Python-hf, while StarCoder2 is represented by StarCoder2-7B.Both are included among the six evaluated models.
- Generation Procedure: Each task produces 10 code snippets per LLM using nuclear sampling with temperature 0.6, following CoderEval’s settings.Repeated generation supports analysis of hallucination patterns across outputs.
C. Taxonomy Annotation
The taxonomy is constructed through manual open coding of generated code, beginning with an initial task sample and progressing through discussion, refinement, and broader annotation. It distinguishes requirement, knowledge, and project-context conflicts.
- Taxonomy Annotation: Manual open coding is used to identify hallucination types in LLM-generated code.The taxonomy records hallucination categories and their meanings in generated code.
- Initial Open Coding: The initial stage analyzes 23 of CoderEval’s 230 tasks, yielding 1,380 generated snippets from six models for preliminary taxonomy development.Each snippet is tested in its corresponding development environment to assess correctness.
- Preliminary Taxonomy Construction: Annotators document hallucinations and their locations, discuss similar cases, and group them into a preliminary taxonomy.A single code snippet may contain multiple hallucinations.
- Full Taxonomy Construction: Three experienced Python programmers independently annotate the remaining snippets and document newly observed hallucination types for further discussion.The taxonomy is refined when cases fall outside existing categories.
- Taxonomy Categories: Task Requirement Conflicts include functional and non-functional requirement violations, such as wrong or missing functionality, security, performance, style, and code smells.These examples illustrate the taxonomy’s requirement-conflict branch.
IV. EVALUATION RESULTS
The evaluation establishes a taxonomy of hallucinations in practical code generation and illustrates its categories through requirement, knowledge, and project-context conflicts.
- Hallucination taxonomy: The taxonomy contains three categories: Task Requirement Conflicts, Factual Knowledge Conflicts, and Project Context Conflicts, divided into eight subtypes.The categories cover requirements, factual knowledge, and project-specific contexts.
- Task Requirement Conflicts: 43.53% of hallucinations are Task Requirement Conflicts, including functional and non-functional requirement violations.Functional Requirement Violation accounts for 36.66%, while Non-functional Requirement Violation accounts for 6.86%.
- Examples: The examples show generated code overlooking timezone handling, using unsafe YAML loading, conflicting with OCFL requirements, misusing library APIs, and invoking a nonexistent parameter.The API example uses datetime.timedelta() with a non-existing year parameter.
- Factual Knowledge Conflicts: 31.91% of hallucinations are Factual Knowledge Conflicts involving background, library, or API knowledge.The subtypes include Background Knowledge Conflicts, Library Knowledge Conflicts, and API Knowledge Conflicts.
- Project Context Conflicts: 24.56% of hallucinations are Project Context Conflicts involving environments, dependencies, and non-code resources specific to a project.These conflicts differ from factual knowledge conflicts because they concern project-specific rather than publicly accessible facts.
B. RQ2: LLM Comparison
Across the evaluated models, Task Requirement Conflicts are the most common hallucination type. Their frequency varies substantially, with CodeGen and StarCoder2 highest and DeepSeekCoder and CodeLlama lowest.
- Cross-model distribution: Task Requirement Conflicts are the most common hallucination type across all evaluated models.Factual Knowledge Conflicts and Project Context Conflicts remain at approximately the same frequency.
- Model comparison: CodeGen and StarCoder2 show notably higher frequencies of Task Requirement Conflicts than the other models.The comparison is based on the distribution of hallucination counts across the three taxonomy categories.
- Model comparison: DeepSeekCoder and CodeLlama show the lowest occurrence of Task Requirement Conflicts among the compared models.The passage reports this specifically for the most prevalent hallucination category.
C. RQ3: Root Cause Analysis
The root-cause analysis identifies four possible contributors to hallucinations in practical code generation: training data quality, intention understanding, knowledge acquisition, and repository-level context awareness.
- Training data quality: Low-quality training repositories may contain mismatched documentation, insecure implementations, misused APIs, outdated usage, and limited domain diversity.These flaws can be incorporated into models’ knowledge bases and contribute to generated hallucinations.
- Intention understanding capacity: Limited intention understanding can produce functionally or non-functionally inaccurate code because models generate from recognized patterns rather than fully capturing specific user needs.The passage links this limitation to the effectiveness and trustworthiness of code generation.
- Knowledge acquisition capacity: Limited knowledge acquisition prevents models from reliably incorporating incorrect, missing, or post-training domain and library knowledge.The analysis contrasts this limitation with human developers’ ability to continuously learn during development.
- Repository-level context awareness: Repository-level context awareness is constrained by token limits, quadratic computation growth, and irrelevant project information.Feeding all code, documents, and non-code resources into an LLM is described as challenging and impractical.
- Identified factors: Four possible contributing factors are training data quality, intention understanding capacity, knowledge acquisition capacity, and repository-level context awareness.The study reports that deficiencies in any of these factors can lead to hallucinations.
A. Motivation
At inference time, hallucinations trace to insufficient task understanding, missing factual knowledge, and inability to access necessary repository resources, challenging practical code generation.
- Inference-stage limitations: Three inference-stage limitations underlie hallucinations: insufficient task understanding, missing factual knowledge, and inaccessible repository code and non-code resources.These limitations are identified as challenges for LLM code generation in practical development settings.
B. RAG-based Mitigation
The paper implements a lightweight RAG method that retrieves repository code snippets as prompts for practical code generation. Similarity-based retrieval supplies context to the LLM during generation.
- Repository source files are scanned with a sliding window to construct retrieval corpora for each CoderEval repository.The window extracts consecutive code lines and advances by a fixed slicing step for complete coverage.
- The mitigation method is evaluated under Pass@1, with results reported in Table I.
- A sparse bag-of-words model compares task queries and code snippets using Jaccard similarity.The ten highest-scoring snippets are returned as prompts for the LLM.
C. Evaluation
The evaluation compares raw generation with repository-informed RAG across six LLMs and uses case studies to illustrate mitigation. The discussion also identifies difficult-to-detect hallucinations and directions for future detection and mitigation.
- Evaluation setup: The study compares Raw and RAG-based generation across six LLMs using Pass@1 on the CoderEval dataset.Raw prompting uses docstrings and signatures, whereas RAG adds ten retrieved code snippets.
- Case studies: RAG case studies show CodeGen replacing an incorrect replace-based solution with splitlines to satisfy a one-line conversion requirement.
- Case studies: A second case study reports ChatGPT correctly using self.pop() to mitigate a Project Context Conflict.
- Future directions: Some hallucinations, including incomplete functionality and security issues, may evade static checks and test cases and reach production systems.The paper therefore calls for more effective techniques to identify and localize hallucinations.
- Future directions: The paper suggests expanding mitigation beyond repository-only RAG with sources such as online search, API documents, and StackOverflow discussions.It also discusses query refinement and multi-agent workflows involving testing and mitigation.
VII. THREATS TO VALIDITY
The paper discusses threats to external, internal, and construct validity. These concern limited language and dataset coverage, manual annotation reliability, model bias, and evaluation design.
- External Validity: The taxonomy and root-cause analysis focus on Python and a CoderEval dataset containing 230 coding tasks.The authors identify extension to other programming languages and larger datasets as future directions.
- Internal Validity: Manual taxonomy annotation lacks a formal inter-rater reliability measure.Annotators resolved discrepancies through discussion, and one author reviewed all labeled data.
- Internal Validity: The authors mixed outputs from six models before annotation to mitigate model bias during labeling.
- Construct Validity: Mitigation evaluation uses six models and the test cases available in CoderEval.The paper presents this as a standard approach for evaluating generated-code correctness.
- Conclusion: The study combines empirical taxonomy construction, hallucination-cause analysis, distribution analysis, and RAG-based mitigation in practical development scenarios.