Source-linked AI summary
Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities
Arjun Krishna, Erick Galinkin, Leon Derczynski, Jeffrey Martin
TL;DR
Package hallucinations can expose software supply chains to attacks when plausible nonexistent dependencies are registered with malicious code. This paper measures the behavior across programming languages and LLMs, finding that hallucination rates vary by language and model characteristics, with all tested models exhibiting hallucinations.
Problem
LLM-generated code can recommend nonexistent packages that attackers may register with malicious implementations, creating a software supply-chain risk.
Source: §1 · p. 1, §2.1 · p. 2Method
The paper studies package hallucination across languages and models, using varied code-generation prompts and measuring the share that produce at least one hallucinated package.
Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 4Results
All tested models had package hallucination rates from 0.22% to 46.15%; JavaScript averaged 14.73%, versus 23.14% for Python and 24.74% for Rust.
Source: §6 · p. 8Takeaways & Limitations
Model and language choice matter: larger models tend to have lower package-hallucination propensity, while coding-specific models tend to have higher propensity.
Source: §6 · p. 8The 11-model sample may have been too small to determine whether coding and general-purpose models differ significantly in package hallucination rates.
Source: §4.3 · p. 6
Abstract
from arXiv · showhide
Large Language Models (LLMs) have become an essential tool in the programmer's toolkit, but their tendency to hallucinate code can be used by malicious actors to introduce vulnerabilities to broad swathes of the software supply chain. In this work, we analyze package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies. By analyzing this package hallucination behaviour we find potential attacks and suggest defensive strategies to defend against these attacks. We discover that package hallucination rate is predicated not only on model choice, but also programming language, model size, and specificity of the coding task request. The Pareto optimality boundary between code generation performance and package hallucination is sparsely populated, suggesting that coding models are not being optimized for secure code. Additionally, we find an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages. Our metrics, findings and analyses provide a base for future models, securing AI-assisted software development workflows against package supply chain attacks.
1. Introduction
Package hallucination can turn plausible but nonexistent dependencies into a software supply-chain risk: attackers may register those names with malicious implementations. The paper studies this behavior across languages and LLMs and evaluates defensive strategies for AI-assisted development.
1. Introduction
- LLMs’ unpredictable or incorrect outputs can include package references, introducing security vulnerabilities into software supply chains as their use in code generation grows.The paper notes that AI assistance completes as many code characters as developers manually type, increasing the importance of securing these systems.Source: §1 · p. 1
- Attackers can register hallucinated package names with malicious implementations, allowing generated code to introduce malware into dependency chains.The paper describes this as a potential route from an LLM recommendation to broader supply-chain impact.Source: §1 · p. 1
- A plausible-sounding nonexistent package such as securehashlib can appear in code requested for password hashing, creating an attack vector if developers trust the recommendation.The example prompt asks for Python code to make a password safe to store.Source: §1 · p. 1
1.1. Contributions
- The study examines legitimate and fictional package references across programming languages and LLMs, then develops and evaluates defensive strategies for AI-assisted development.Source: §1 · p. 2, §1.1 · p. 2
1. Introduction
- The introduction frames four questions: how often package hallucination occurs, and how it varies with language, model size, and coding-specific versus general-purpose models.Source: §1 · p. 2
2. Background
Package hallucinations can be exploited by registering nonexistent names, while existing squatted packages create a related risk. The paper defines the terms and categories used to distinguish these behaviors and model types.
2.1. Security impact
- Public package repositories let users claim names, enabling attackers to register malicious code under names that LLM output tends to leave unregistered.Source: §2.1 · p. 2
- Attackers can probe accessible models for nonexistent package names, register them with malicious payloads, and thereby exploit package hallucination in a way compared to typosquatting.The paper notes that this risk can remain even for users of smaller local private models intended to reduce code leakage.Source: §2.1 · p. 2
- Package hallucination is not the only repository risk: existing squatted packages may use plausible names, and the paper notes that some packages have been pre-emptively registered to block squatters.The example arangodb RubyGem is described as a benign placeholder.Source: §2.1 · p. 2
2.2. Definitions
- Package hallucination is defined as generated code importing external dependencies the model could not have been trained on, with registration timing used to identify such packages.The criteria include absence from the appropriate repository and registration after the model’s knowledge cutoff, or within 90 days before first publication when that cutoff is unavailable.Source: §2.2 · p. 3
- The paper distinguishes induced hallucination, prompted by an explicit nonexistent package or fictional API, from natural hallucination, generated without such a specific request.Source: §2.2 · p. 3
- Coding models are optimized for code generation and understanding and trained on a greater proportion of code; models such as Llama 3.1 variants remain categorized as general-purpose.Source: §2.2 · p. 3
3. Method
The study measures whether LLM-generated code imports packages absent from the relevant repository, using varied models, prompt combinations, and repository snapshots. It reports package hallucination rate across repeated requests.
3. Method
- The experiment uses garak to prompt target LLMs, identify imported packages, and compare them with the language’s primary repository to flag likely unobserved packages.Source: §3 · p. 3
3.1. Selection of models
- Models were selected to vary in parameter count, provider, and purpose, with a preference for open-weight models and GPT-4o included for completeness.The selection considers model resource consumption and inference speed alongside benchmark performance.Source: §3.1 · pp. 3–4
3.2. Building package hallucination prompts
- Code-generation prompts combine a request stub and coding-task description, with a language placeholder; the tested languages are JavaScript, Python, and Rust.The prompt set spans all combinations of request stubs and task descriptions.Source: §3.2 · p. 4
3.3. Generating lists of known-good packages
- The study scraped package names and release dates to reconstruct known-good package lists for PyPI, NPM, and crates.io.These are identified as the most popular repositories for their respective languages; NPM has nearly an order of magnitude more packages than PyPI, which is about three times the size of crates.io.Source: §3.3 · p. 4
3.4. Metrics
- Package Hallucination Rate (PHR) is the proportion of prompts producing at least one hallucinated package, with each request repeated five times to reduce spurious-output effects.Source: §3.4 · p. 4
4. Analysis
Package hallucination appeared in every tested model and language, with rates varying across languages and model characteristics. The analysis finds lower rates associated with larger models and stronger HumanEval performance, and proposes package-name verification as a targeted mitigation.
4. Analysis
- Every tested model hallucinated packages across all tested languages, though rates varied with model size and other parameters.Source: §4 · p. 4
4.1. Programming Languages
- Rust had the highest mean PHR at 24.74%, while Python had the highest overall PHR at 46.15%; Python also showed the greatest across-model variance (σ = 16.03).Source: §4.1 · pp. 4–5
4.3. Code vs. Non-Code
- Code-specialized models had higher mean PHRs across the three languages, but differences were significant only for JavaScript and were not significant after GPT-4o was removed.Source: §4.3 · p. 6
4.4. Model Size
- PHR was inversely correlated with model size (ρ = −0.542, p = 0.00114), more strongly with log-transformed size (ρ = −0.593, p = 0.00028); Mamba-Codestral was an exception.Source: §4.4 · pp. 6–7, Figure 3 · p. 7
4.5. Coding Benchmarks
- HumanEval scores had a strong inverse correlation with average PHR (ρ = −0.7887), while MBPP had a weaker inverse correlation (ρ = −0.2919); the Pareto frontier was sparsely populated.Source: §4.5 · pp. 7–8
4.6. Mitigation
- A package list reflecting each model’s training cutoff can flag suggestions for packages not yet in the repository, but this does not detect malicious packages already indexed.Source: §4.6 · p. 8
5. Related Work
Prior work spans LLM code hallucinations and vulnerabilities, package-confusion attacks, and supply-chain defenses; this study connects those areas through package hallucination as an attack vector. It extends earlier package-hallucination research across languages and adversarial prompts while distinguishing hallucination-driven introduction of new malicious packages from detection of existing malware.
6. Conclusion
- The study bridges research on LLM code hallucinations and vulnerabilities, package-confusion attacks, and supply-chain defenses by examining package hallucinations as an attack vector.It frames package hallucinations as a new attack vector for LLMs.Source: §5 · p. 8
- Compared with Spracklen et al. (2024), the work surveys more languages, examines explicitly adversarial prompts, and reports correlations involving pre-training, model size, benchmarks, and hallucination.Source: §5 · p. 8
- Unlike Zahan et al. (2024), which detects existing NPM malware, this study focuses on hallucinations that could enable new malicious packages to enter the supply chain.Source: §5 · p. 8
6. Conclusion
The study examines package-hallucination vulnerabilities across programming languages and model architectures, finding that hallucinations occur across tested models and vary by language and model characteristics. It presents these findings as a foundation for addressing the risk in AI-assisted software development.
6. Conclusion
- All tested models hallucinated packages, with rates from 0.22% to 46.15%; JavaScript averaged 14.73%, versus 23.14% for Python and 24.74% for Rust.JavaScript had the most consistent performance (σ = 8.4), while Python and Rust had higher variance (σ = 16.03 and σ = 14.37, respectively).Source: §6 · p. 8
- Package-hallucination rates varied with model choice; larger models tended to hallucinate less, while coding-specific models tended to hallucinate more.The authors note that larger models may be impractical in some deployment scenarios.Source: §6 · p. 8
- The authors identify reducing hallucinations in smaller models, studying code optimization and security, and developing cross-language, cross-architecture defenses as future work.They also call for investigating whether non-transformer models are more robust.Source: §6 · pp. 8–9
- The study presents a foundation for understanding and addressing package-hallucination vulnerabilities in AI-assisted software development.Source: §6 · p. 9
Software and Data
The research used open-source software and openly available models, citing them where used.
Software and Data
- The research used open-source software and openly available models, citing them where used.Source: Software and Data · p. 10
Impact Statement
The paper highlights package hallucination as a security risk in code-generation tools, reporting that the failure mode appeared in every tested model. It also identifies coding-benchmark scores as a possible heuristic while noting deployment trade-offs involving privacy, cost, and code security.
Impact Statement
- Package hallucination is a subtle failure mode with significant security risk, and appeared at some level in every model tested.Source: Impact Statement · p. 10
- Package-hallucination rates were negatively correlated with coding-benchmark scores, offering a commonly available heuristic for evaluating coding-assistant models.Source: Impact Statement · p. 10
- Choosing larger models may improve security against package hallucinations, but third-party hosting can create tensions with data privacy, cost, and generated-code security.Source: Impact Statement · p. 10
A. Prompt composition
Prompt construction uses request stubs and coding tasks to form varied programming prompts. The examples range from general code requests to library-focused requests with different framings.
A. Prompt composition
- The examples vary from general code-generation requests to requests for libraries or modules that support a task.Some examples also vary the framing, including demographic descriptors.Source: A. Prompt composition · p. 12
- Prompt construction combines request stubs with coding tasks to create prompts.The supplied material identifies these components in Tables 6 and 7 but does not expose their contents.Source: Table 6 · p. 12, Table 7 · p. 12
B. Extracting package names from generated code
The paper extracts package references from generated code using language-specific regular expressions designed for idiomatic import patterns. It covers Python, JavaScript, and Rust reference forms.
B. Extracting package names from generated code
- Language-specific regular-expression extractors capture idiomatic package import patterns to identify references in generated code.The method first extracts package references as a step toward identifying package hallucinations.Source: B. Extracting package names from… · p. 12
- Python extraction covers direct, qualified, and aliased imports.The listed patterns include direct imports, imports from a package, and aliased imports.Source: B. Extracting package names from… · p. 12
- JavaScript extraction covers ES6 module imports and CommonJS require patterns.Both modern ECMAScript modules and CommonJS are included.Source: B. Extracting package names from… · p. 12
- Rust extraction covers use declarations, external crate declarations, and direct path references.These are the Rust reference forms specified for extraction.Source: B. Extracting package names from… · p. 12