PaperlayerView exact revision on arXiv

Source-linked AI summary

We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs

Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala

arXiv:2406.10279v3cs.SEcs.AIcs.CRcs.LG

TL;DR

Package hallucinations can expose software users to package-confusion attacks, yet their prevalence and possible mitigations had not been thoroughly investigated. The paper systematically studies the phenomenon across code-generating models and evaluates mitigation strategies; it reports that 19.7% of generated packages are fictitious.

  • Problem

    Package hallucinations—fictitious package recommendations in generated code—can be exploited through malicious packages, while their prevalence and mitigation effectiveness had not been thoroughly investigated.

    Source: §1 · pp. 1–2

  • Method

    The paper systematically studies package hallucinations across code-generating LLMs and model settings, then evaluates mitigation strategies.

    Source: §1 · p. 2

  • Results

    19.7% of generated packages were fictitious; fine-tuning reduced DeepSeek hallucinations by 83%, to a 2.66% rate.

    Source: §7 · p. 13, §6.2 · p. 13

  • Takeaways & Limitations

    Repeated hallucinated names can be exploited by publishing matching malicious packages, and simple package-name cross-referencing may not detect the attack after publication.

    Source: §1 · p. 2

    Newer code-generation models may have different performance and hallucination tendencies, and fewer commercial models were included because of funding constraints.

    Source: §7 · p. 13

Abstract

from arXiv · show

The reliance of popular programming languages such as Python and JavaScript on centralized package repositories and open-source software, combined with the emergence of code-generating Large Language Models (LLMs), has created a new type of threat to the software supply chain: package hallucinations. These hallucinations, which arise from fact-conflicting errors when generating code using LLMs, represent a novel form of package confusion attack that poses a critical threat to the integrity of the software supply chain. This paper conducts a rigorous and comprehensive evaluation of package hallucinations across different programming languages, settings, and parameters, exploring how a diverse set of models and configurations affect the likelihood of generating erroneous package recommendations and identifying the root causes of this phenomenon. Using 16 popular LLMs for code generation and two unique prompt datasets, we generate 576,000 code samples in two programming languages that we analyze for package hallucinations. Our findings reveal that that the average percentage of hallucinated packages is at least 5.2% for commercial models and 21.7% for open-source models, including a staggering 205,474 unique examples of hallucinated package names, further underscoring the severity and pervasiveness of this threat. To overcome this problem, we implement several hallucination mitigation strategies and show that they are able to significantly reduce the number of package hallucinations while maintaining code quality. Our experiments and findings highlight package hallucinations as a persistent and systemic phenomenon while using state-of-the-art LLMs for code generation, and a significant challenge which deserves the research community's urgent attention.

1 Introduction

Package hallucinations can expose software users to package-confusion attacks, yet their prevalence, characteristics, and possible mitigations had not been thoroughly investigated. This study examines them across code-generating LLMs and settings, characterizes related behaviors and packages, and evaluates mitigation strategies.

1 Introduction

  • An attacker can publish a malicious package under a hallucinated name, which trusting users may install and thereby spread through codebases and dependency chains.The paper describes this as a package-confusion attack enabled by code-generating LLMs.Source: §1 · p. 2
  • Earlier work had not thoroughly established package hallucination prevalence in state-of-the-art commercial and open-source models, their nature, or the effectiveness of mitigations.Simple cross-referencing can fail when an adversary has already published the hallucinated name.Source: §1 · p. 2
  • The study examines hallucination prevalence in Python and JavaScript and related behaviors including repetition, output verbosity, and models’ ability to detect their own hallucinations.These behaviors are among the paper’s stated characterization goals.Source: §1 · p. 2
  • The analysis examines how training-data recency, model temperature, and decoding strategies affect package hallucinations.Source: §1 · p. 2
  • The authors characterize hallucinated packages by semantic similarity to popular packages, recurrence across models, and effects of recently removed packages.Source: §1 · p. 2
  • The paper evaluates mitigation techniques intended to reduce package hallucinations while retaining effective code generation, and releases Python and JavaScript prompt and generated-sample datasets.The released datasets comprise 19,500 coding prompts and 586,000 generated samples.Source: §1 · p. 2

2 Background and Related Work

The background links open package repositories and LLM-generated code to software supply-chain risks, including package confusion and hallucinated package names. Related work had not comprehensively measured package hallucinations across models or established their characteristics and mitigation effectiveness.

2.1 Background

  • Package confusion attacks use deceptive names, including typosquatting, combosquatting, brandjacking, and similarity attacks, to trick users into downloading malicious packages.The text distinguishes these attacks from corrupting legitimate packages or creating unique malicious packages.Source: §2.1 · p. 2
  • Code-generating LLMs have improved on coding prompts, while prior work also reports insecure generated code and a false sense of security among developers.Source: §2.1 · p. 3
  • Hallucinations include fact-conflicting errors, and package hallucinations can be exploited by publishing malicious packages under fictitious names that users trust and install.The paper describes possible cascading vulnerabilities through package and code dependencies.Source: §2.1 · p. 3

2.2 Related Work

  • Prior preliminary tests found package hallucinations in commercial LLMs, but did not cover popular open-source models or systematically characterize hallucinations and model properties.The paper contrasts this with its broader two-language evaluation and detailed analysis.Source: §2.2 · pp. 3–4

3 Research Questions

The paper organizes its investigation into five questions, spanning package-hallucination prevalence, model settings, model behaviors, package attributes, and mitigation. Its assumed attacker targets users who trust LLM-recommended package names and install them without sufficient verification.

3 Research Questions

  • The assumed attacker cannot manipulate the models, but can identify hallucinated names and publish matching packages for users who install recommendations without sufficient verification.Source: §3 · p. 4
  • RQ1 asks how prevalent package hallucinations are in Python and JavaScript code generated by commercial and open-source LLMs across coding tasks.Source: §3 · p. 4
  • RQ2 examines how training data and decoding strategies affect package hallucinations.Source: §3 · p. 4
  • RQ3 examines persistence within and across models, output verbosity, and whether models detect their own hallucinations after generation.Source: §3 · p. 4
  • RQ4 examines semantic similarity to popular packages, cross-language hallucinations, and generated packages recently removed from repositories.Source: §3 · p. 4
  • RQ5 asks whether retrieval augmentation, self-detected feedback, decoding strategies, and supervised fine-tuning can reduce hallucinations with minimal compromise to code quality.Source: §3 · pp. 4–5

4 Experiment Design

The experiment design builds diverse coding prompts, generates code with selected models, and detects package hallucinations in a three-phase pipeline.

4 Experiment Design

  • The pipeline has three phases: prompt-dataset generation, code generation, and hallucination detection.Source: §4 · p. 5

4.1 Prompt Dataset

  • The prompt datasets combine Stack Overflow questions with prompts generated from popular package descriptions, with temporal splits supporting recency analysis.The Stack Overflow questions were selected from 240 relevant tags; both datasets were divided into recent and all-time segments.Source: §4.1 · p. 5

4.2 Code Generation

  • The study tested off-the-shelf models in Python and JavaScript, generating 576,000 code samples across its tests.Fourteen models were tested in both languages, while two Python-specific models were tested only in Python.Source: §4.2 · pp. 5–6

4.3 Detection Methodology and Heuristics

  • Package names were identified through install commands and by asking the generating model to name packages for either the code or its original task prompt.Install commands occurred naturally in 7% of outputs; the other two heuristics simulate users asking which packages are required.Source: §4.3 · pp. 6–7
  • Names absent from the January 10, 2024 PyPI and npm master lists counted as hallucinations, making rates a lower bound if those lists already contained malicious packages.Source: §4.3 · p. 7

5 Evaluation Results

The evaluation measures package hallucinations across models and languages, then examines settings, model behaviors, and hallucination characteristics. RQ1 covers Python and JavaScript; RQs 2–4 narrow the analysis to Python, selected models, and sampled prompts.

5.1 Prevalence of Package Hallucinations (RQ1)

  • 19.7% of 2.23 million recommended packages were hallucinations, including 205,474 unique names; commercial models averaged 5.2% versus 21.7% for open-source models.Python averaged 15.8% hallucinations versus 21.3% for JavaScript.Source: §5.1 · p. 7, Figure 2 · p. 7

5.2 Impact of Model Settings (RQ2)

  • All models hallucinated more as temperature rose, while decoding changes raised rates 1.16% on average; at maximum temperatures, open-source models produced more hallucinated than valid packages.The effect of increased temperature was especially sharp above 1 for the OpenAI models.Source: §5.2 · p. 8, Figure 3 · p. 8
  • Recent-topic prompts produced higher hallucination rates across all 16 evaluated Python models; the RQ2 summary reports a 10% increase for more recent topics.Source: §5.2 · pp. 8–9, Figure 4 · p. 8

5.3 Model Behaviors (RQ3)

  • 43% of hallucinated packages reappeared in all ten repeated queries, while 39% never reappeared; 58% were repeated more than once.Hallucination rate correlated with the number of unique packages recommended; the lowest-rate models used smaller package subsets and scored highest on EvalPlus, suggesting both outcomes may improve together.Source: §5.3 · p. 9, Figure 5 · p. 9
  • GPT-4 Turbo, GPT-3.5, and DeepSeek each detected their own hallucinations with accuracy above 75%.Source: §5.3 · p. 10, Figure 7 · p. 10

5.4 Characteristics of Hallucinations (RQ4)

  • 81% of distinct hallucinated package names were generated by only one model, even though hallucinations often repeat within a model; their exact forms are generally model-specific.Source: §5.4 · p. 10, Figure 8 · p. 10
  • 13.4% of hallucinations had Levenshtein distance 1–2 and 48.6% had distance 6 or higher; deleted packages accounted for 0.17%, while JavaScript packages accounted for 8.7% of hallucinated Python packages.Source: §5.4 · p. 11, Figure 9 · p. 11

6 Mitigation

The section contrasts reactive filtering with proactive pre-generation strategies and evaluates RAG, self-refinement, and fine-tuning on two models. All tested strategies reduce package hallucinations, while fine-tuning also lowers code quality.

6.1 Mitigation Strategies

  • Pre-generation techniques aim to prevent hallucinations at their source, unlike post-generation filters, which remove content after it is generated.The paper characterizes curated-package filtering as blunt and reactive, requiring constant verification and updating.Source: §6.1 · p. 12

6.2 Mitigation Implementation and Results

  • RAG retrieves semantically similar valid-package statements for prompts, self-refinement iteratively rechecks generated packages, and fine-tuning trains on 560,000 valid responses.Fine-tuned models were evaluated for code quality with HumanEval.Source: §6.2 · pp. 12–13
  • The techniques were tested individually and as an ensemble on DeepSeek Coder 6.7B and CodeLlama 7B, selected as better- and worse-performing models in initial experiments.The paper evaluated RAG, self-refinement, and fine-tuning.Source: §6.2 · pp. 12–13
  • 83% fewer hallucinations, to 2.66%, followed DeepSeek fine-tuning; the full ensemble reduced hallucinations 85% for DeepSeek and 64% for CodeLlama.All tested strategies reduced hallucinations; Table 4 presents their performance.Source: §6.2 · p. 13, Table 4 · p. 13
  • Fine-tuning decreased HumanEval code quality by 26.1% for DeepSeek and 3.1% for CodeLlama, despite reducing package hallucinations.The resulting scores remained comparable to several other high-performing models.Source: §6.2 · p. 13, Table 5 · p. 13

7 Discussion and Conclusion

The conclusion reports fictitious packages in generated output and describes behavioral patterns and mitigation strategies. The authors caution that newer models and limited commercial-model coverage constrain how fully the findings represent current commercial systems.

7 Discussion and Conclusion

  • Newer code-generation models may have different hallucination tendencies, and funding constraints limited the commercial models studied.The authors say their findings may not fully represent newer commercial LLMs.Source: §7 · p. 13
  • 19.7% of generated packages were fictitious, presenting a threat through package-confusion attacks.The study included commercial and open-source code-generating models.Source: §7 · p. 13
  • The study characterized package-hallucination behaviors and proposed mitigation strategies to support reliability and security in AI-assisted software development.Source: §7 · p. 14

Ethics Considerations

The authors frame the research around package-confusion threats and disclose it to model providers. They describe ethical choices in package testing, data handling, and the controlled nature of the experiments.

Ethics Considerations

  • The researchers disclosed their findings to OpenAI, Meta, DeepSeek, and Mistral AI, and report that each provider would consider them for future models.Source: Ethics Considerations · p. 14
  • For ethical reasons, the researchers did not publish actual packages using hallucinated names to PyPI, citing risks to repository integrity and potential terms-of-service concerns.Source: Ethics Considerations · p. 14
  • The Stack Overflow dataset contained scraped questions only, without user information or answers, and collection was designed to follow the site's terms and data-use policies.Source: Ethics Considerations · p. 14
  • The study involved no human subjects and was conducted in controlled settings without impact on external persons or entities.Source: Ethics Considerations · p. 14

Open Science

The appendix describes which research assets will be released and which will be withheld, then provides examples of prompts, model instructions, and testing resources.

Open Science

  • The authors will withhold the master list of hallucinated package names and detailed results of individual tests, sharing them by request with verified researchers.They cite potential misuse for package-hallucination attacks.Source: Open Science · p. 14
  • Release materials include code and data to reproduce the primary experiments, figures, and mitigation tests, plus the LLM and Stack Overflow datasets.A script can generate code and packages, evaluate hallucinations, and save results.Source: Open Science · p. 14
  • The planned release also includes two fine-tuned models and the RAG database with its creation code and data.Source: Open Science · p. 14

A Truncated List of LLM Generated Coding Prompts

B System Messages and Prompts

  • Figures 10–12 show the system messages and prompts sent to models; the displayed instruction requests code-only responses unless explanation is necessary.Source: B System Messages and Prompts · p. 18

C Model Parameters and Testing Environment

D Python vs. JavaScript Hallucination

The section presents a linear relationship in models’ propensity to hallucinate across Python and JavaScript, alongside figures labeled as two package-generation heuristics.

D Python vs. JavaScript Hallucination

  • A model’s propensity to hallucinate across Python and JavaScript has a linear relationship, shown in Figure 14.The supplied passage does not specify the direction or strength of the relationship.Source: D Python vs. JavaScript Hallucination · p. 18
  • Figure 11 is labeled as showing package generation phase—Heuristic 1.Source: Figure 11 · p. 18
  • Figure 12 is labeled as showing package generation phase—Heuristic 2.Source: Figure 12 · p. 18

E Complete Results for Python and JavaScript

Tables 7 and 8 provide complete package-hallucination experiment results for all tested models in Python and JavaScript.

E Complete Results for Python and JavaScript

F.1 Hallucinations

This section distinguishes intrinsic from extrinsic hallucinations and identifies several forms of code-generation errors, while its figures and table are labeled as covering prompt generation, language-specific rates, and model parameters.

F.1 Hallucinations

  • Intrinsic hallucinations are directly verifiable from source content, whereas extrinsic hallucinations are not.The passage associates intrinsic code hallucinations with traceable syntax errors and extrinsic ones with complex interactions or gaps in training data.Source: §F.1 · p. 18
  • The coding-prompt example asks an assistant to create simple prompts for Python or JavaScript code generation and return no code.Source: §F.1 · p. 19
  • Figure 13 is labeled as covering coding-prompt generation, Figure 14 as comparing Python and JavaScript hallucination rates, and Table 6 as outlining model parameters.Source: Figure 13 · p. 19, Figure 14 · p. 19, Table 6 · p. 19
  • Code-generation hallucinations can include functional bugs, wrong-task code, dead code, and exploitable security vulnerabilities.Source: §F.1 · p. 19
  • Prior mitigation strategies for standard natural-language generation include prompt engineering and model-architecture enhancements.Examples include retrieval-augmented generation, self-refinement, supervised fine-tuning, inference-time intervention, and knowledge graphs.Source: §F.1 · p. 19
  • Higher temperatures increase sampling of low-frequency tokens and raise hallucination risk, while lower temperatures produce more predictable outputs.Source: §F.1 · p. 19

G Packages and Modules Explanation

Packages collect modules, which developers download from repositories and import into code; module names do not map uniquely to package names.

G Packages and Modules Explanation

  • A package is a collection of related modules; using a module may require downloading its package from a repository and importing the module into code.The passage gives Python `import` and JavaScript `require` as examples.Source: G Packages and Modules Explanation · p. 19
  • Module names need not match package names, and different packages may contain modules with the same name, so imports do not uniquely identify package dependencies.Source: G Packages and Modules Explanation · p. 19
Loading 2406.10279v3…