HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing

I. INTRODUCTION

• Package hallucination—recommending nonexistent packages [9], [10]—creates serious supply-chain risks, which tool-using agents can amplify by downloading malicious packages without developers’ awareness,.The introduction illustrates the risk with a nonexistent PyPI package recommendation that could be registered with malicious code.Source: §I · p. 1
• Existing package-hallucination studies cover limited datasets, while testing needs broad, diverse coding tasks that remain code-relevant.Prior task-generation approaches struggle to produce coding tasks with multiple valid implementations or may generate non-code-related prompts.Source: §I · pp. 1–2
• HFUZZER uses LLM-guided phrase-based fuzzing to generate diverse coding tasks, extracting code-relevant Object–Predicate–Complement phrases from package descriptions or tasks.Phrases let the model infer a wider range of reasonable information; the paper presents this as the first use of fuzzing to test LLMs for package hallucinations.Source: §I · p. 2
• HFUZZER found an average of 2.60x more unique hallucinated packages than GPTFUZZER-A across tester models, triggered hallucinations in all target models, and found 46 unique packages with GPT-4o.GPT-4o also hallucinated packages while assisting with environment configuration, not only during code generation.Source: §I · p. 2

II. METHODOLOGY

• HFUZZER uses a tester model to extract phrases from package information and tests a target model through input and output access.The workflow has Phrase Extraction and a Fuzzing Loop, and supports closed-source target models.Source: §II · p. 2, Figure 2 · p. 3

B. Fuzzing Loop

• Each round selects a three-phrase seed according to phrase power, then recombines its phrases to generate a coding task rather than mutating the seed.Phrase power represents a phrase’s potential and determines its selection probability.Source: Figure 2 · p. 3
• HFUZZER asks the target to generate code and recommend required packages, evaluates those packages to calculate a Hallucination Score, and adjusts related phrase power.Package indices such as PyPI are queried to verify recommendations and avoid false positives from tester-model errors.Source: Figure 2 · p. 3
• For tasks that trigger package hallucinations, HFUZZER extracts new phrase compositions and adds them to the seed pool; rounds continue until the query budget is exhausted.The process returns generated coding tasks and corresponding model outputs.Source: Figure 2 · p. 3, B. Fuzzing Loop · p. 3

II. METHODOLOGY

• HFUZZER discards intermediate results with invalid output formats, permits “None” when the tester cannot generate reasonable output, and allows the target to refuse unrelated tasks.The framework also uses LLMs’ semantic understanding and natural-language processing to handle complex inputs.Source: Figure 2 · p. 3

A. Phrase Extraction

• Phrase extraction represents package information as ⟨Object, Predicate, Complement⟩ and places each component type in a separate phrase pool with power.Object identifies what the package processes, Predicate its method, and Complement contextual details beyond the other two components.Source: A. Phrase Extraction · p. 3
• HFUZZER prompts an LLM to extract phrases from package information, assigns them initial power, and permits “None” when information is insufficient.System and user prompts specify the extraction target, package information, and expected response format.Source: A. Phrase Extraction · p. 3

1) Seed Selection:

• HFUZZER selects one phrase from each of three pools using probabilities proportional to phrase power, which is adjusted to reduce repeated selections.Each seed combines three phrases to provide richer context for the model.Source: 1) Seed Selection · pp. 3–4

2) Task Generation:

• HFUZZER prompts an LLM with the seed phrases to generate a solvable coding task, then extracts the task from a predefined output tag.The prompt includes a formatting example; the illustrative task combines HTTP/2 requests, handling, and a styled HTML page.Source: 2) Task Generation · p. 4

3) Hallucination Triggering:

• HFUZZER triggers hallucinations through code generation followed by package recommendation, asking the model for installation commands for packages used in its code.The model is instructed to return “None” if it cannot produce code, reducing false positives from unintended tasks; package recommendation also tests environment configuration and helps assess module-to-package mismatches.Source: 3) Hallucination Triggering · p. 4

4) Hallucination Evaluation:

• HFUZZER treats packages as hallucinated if they were registered after the model’s knowledge cutoff or absent from the appropriate repository, excluding standard libraries from that definition [9].The evaluation targets hallucinations within existing model knowledge rather than whether models can infer unknown packages.Source: 4) Hallucination Evaluation · p. 4
• The evaluator classifies recommendations as standard, existing, other-language, or nonexistent packages using repository membership and the model’s knowledge cutoff.Other-language packages appear in Libraries.io but are neither standard libraries nor registered in the appropriate repository before the cutoff; nonexistent packages appear in none of those sets.Source: 4) Hallucination Evaluation · p. 4
• HFUZZER uses Libraries.io, a cross-language package index, to check whether a recommended package originates from another language [10], [9].The index aggregates package repositories from multiple programming ecosystems.Source: 4) Hallucination Evaluation · p. 4
• HFUZZER calculates a hallucination score from recommended-package counts, weighting nonexistent packages more heavily than other-language packages.The weights are α = 1 and β = 0.51 because nonexistent packages indicate constructed packages, while other-language recommendations may reflect language confusion; the example scores one nonexistent package among two recommendations as 0.5.Source: 4) Hallucination Evaluation · p. 4

5) Power Adjustment:

• HFUZZER lowers phrase power by 0.8 for tasks yielding no hallucinated packages, by 0.6 for tasks yielding no package recommendations, and otherwise adjusts it using HS and the new-hallucination share.The share is Nnew/(Nold + Nnew); with HS 0.5, old power 1, and a first-seen hallucinated package, updated power is 0.875.Source: 5) Power Adjustment · pp. 4–5

6) Seed Pool Expansion:

• HFUZZER extracts phrase compositions from coding tasks to reduce repeated compositions and declining task diversity across rounds.It assigns initial power to newly extracted phrases, with Powernew emphasizing new hallucinated packages more than Formula 2; k is set to 0.6.Source: 6) Seed Pool Expansion · p. 5
• With k = 0.6, Example B6 reports that HFUZZER extracts new phrase compositions from the task in Figure 1, and assigns those extracted compositions power 1.0.Source: 6) Seed Pool Expansion · p. 5
• At each round’s end, HFUZZER records coding tasks, target-model outputs, and hallucination evaluation results to support reproducing discovered package hallucinations.Source: 6) Seed Pool Expansion · p. 5

III. EVALUATION

• The evaluation asks whether HFUZZER triggers package hallucinations more effectively than GPTFUZZER-A, generates more diverse tasks, and benefits from its modules.These are the study’s three research questions: effectiveness, task diversity, and module contributions.Source: §III · p. 5

A. Experimental Setup

• HFUZZER is tested through the OpenAI library API, with package information obtained from repository APIs and libraries.io.The study tests for hallucinations rather than detecting whether model outputs contain them; it compares against GPTFUZZER-A, adapted from GPTFuzzer [20], because Drowzee assumes a unique answer and does not apply to package hallucinations.Source: A. Experimental Setup · p. 5
• Python is the target language, and the study selects popular open- and closed-source models to cover different training data and architectures.The passage describes Python as popular, with a well-developed ecosystem and widespread use in related studies [9], [10], [5].Source: A. Experimental Setup · p. 5
• Package Hallucination Rate (PHR) measures the proportion of responses containing hallucinated packages, while unique hallucinated package count assesses testing ability.The study uses PHR and unique hallucinated package count for RQ1 and RQ3; PHR indicates whether generated coding tasks trigger hallucinations.Source: A. Experimental Setup · pp. 5–6
• The Diversity Index, defined as the total number of clusters and noise points, measures task diversity for RQ2 and RQ3.The study uses this metric to assess task diversity for both HFUZZER and GPTFUZZER-A.Source: A. Experimental Setup · p. 6
• Experiments run on a server with four NVIDIA A800 GPUs, a 32-core Intel Xeon Platinum 8358P CPU, 2TB of memory, and vLLM v0.6.1.Source: A. Experimental Setup · p. 6

• The evaluation uses nine tester and target models in 81 combinations, with top-100 SourceRank Python packages and 1,000 rounds per run.Tester temperature was fixed at 0, target temperature at 0.7, and the maximum token limit at 3,000.Source: B. RQ1: The Capability in Testing LLMs · p. 6
• HFUZZER finds an average 2.60× more unique hallucinated packages than GPTFUZZER-A, outperforming it across all target models and tester models.The largest improvement is 4.89x for target DeepSeek-V3 and 5.13× when DeepSeek-Coder is the tester.Source: B. RQ1: The Capability in Testing LLMs · pp. 6–7
• HFUZZER triggers hallucinations across all target models, with average PHR ranging from 0.26–0.66% for four models to 16.47% for Mistral-v0.3.GPT-4o, GPT-4o mini, DeepSeek-V3, and Qwen2.5-Coder have PHRs of 0.26–0.66%; DeepSeek-Coder, Meta-Llama-3, and Meta-Llama-3.1 have PHRs of 5.68%, 7.10%, and 9.82%.Source: B. RQ1: The Capability in Testing LLMs · p. 6, Figure 3 · p. 6
• HFUZZER works across tester models without a consistently strongest tester, and remains effective with smaller-scale LLMs.It triggers hallucinations in most model combinations; its unique-package advantage over GPTFUZZER-A is largest with DeepSeek-Coder as tester (5.13×).Source: B. RQ1: The Capability in Testing LLMs · pp. 6–7, Figure 4 · p. 6
• HFUZZER yields an average 3.02× improvement over GPTFUZZER-A in stability tests, with statistically significant gains for most model combinations and consistently lower CV.Some combinations are not significant because low target-model hallucination rates limit differences in means relative to variances, producing smaller effect sizes and larger CV.Source: B. RQ1: The Capability in Testing LLMs · p. 7
• Across models, hallucinated packages often combine technical terms from tasks, and 190 non-existent plus 176 other-language packages were recommended by multiple models.pkg_resources was recommended by all models, while 17 packages were recommended by more than half; the authors suggest many shared hallucinations may originate in training data.Source: B. RQ1: The Capability in Testing LLMs · p. 7
• Qwen2.5-Coder’s low PHR is partly associated with refusals, which limit both fuzzers, while related Qwen2.5-code models also refuse tasks that newer Qwen models answer.The authors describe this as similar to prior over-refusal findings and leave its further investigation beyond this paper’s scope.Source: B. RQ1: The Capability in Testing LLMs · p. 7
• Despite generally stronger code-generation performance, Llama 3.1 recommends more hallucinated packages than Llama 3, even after specific packages are removed.The authors note that this unexpected observation requires broader experiments; they also suspect training data may relate to widely recommended google-cloud packages, especially with a google.protobuf import.Source: B. RQ1: The Capability in Testing LLMs · p. 7

C. RQ2: The Diversity of Generated Tasks

• HFUZZER’s tasks have a 2.36× higher average Diversity Index than GPTFUZZER-A across parameter settings, with greater diversity across all model combinations.The comparison uses DBSCAN clustering of generated tasks and is reported in Figure 5.Source: C. RQ2: The Diversity of Generated Tasks · p. 7
• Larger DBSCAN ε decreases Diversity Index, whereas increasing minS raises it by splitting small clusters and treating more tasks as noise.The evaluation varies ε across 0.1, 0.2, and 0.3 and minS across 1, 3, and 5.Source: C. RQ2: The Diversity of Generated Tasks · p. 7
• Across tester models, the average CV of Diversity Index is 0.15 for HFUZZER and 0.29 for GPTFUZZER-A; across target models, CVs are around 0.1.For ε = 0.1, 0.2, and 0.3, the corresponding CV pairs are (0.03/0.16), (0.11/0.24), and (0.32/0.47).Source: C. RQ2: The Diversity of Generated Tasks · pp. 7–8

D. RQ3: The Impact of Different Modules

• The Phrase module has the largest ablation impact: it improves unique hallucinated packages by 2.06× and Diversity Index by 4.40×.The study tests Mate-Llama-3.1 and Mistral-v0.3 as target models with all tester models.Source: Table IV · p. 8, D. RQ3: The Impact of Different Modules · p. 8
• The Power module improves unique hallucinated packages by 1.39× and Diversity Index by 1.25×; Seed Pool Expansion improves them by 1.27× and 1.26×, respectively.Removing each module reduces performance relative to HFUZZER.Source: Table IV · p. 8, D. RQ3: The Impact of Different Modules · p. 8

C. RQ2: The Diversity of Generated Tasks

• Across multiple RQ1 runs, HFUZZER increases average Diversity Index by 2.46× across parameter settings (p < 0.05; CV ≈0.01).The increase is reported as statistically significant.Source: C. RQ2: The Diversity of Generated Tasks · p. 8

D. RQ3: The Impact of Different Modules

• The RQ3 ablations remove Seed Pool Expansion, replace Power-based selection with random selection, remove Phrase in favor of package descriptions, or remove all modules.The w/o All variant relies only on the LLM to generate coding tasks.Source: D. RQ3: The Impact of Different Modules · p. 8
• The w/o All variant performs poorly because relying solely on the LLM often produces repetitive coding tasks, underscoring the value of guiding generation.The comparison uses two target models that recommended the largest number of unique hallucinated packages in RQ1 and all tester models.Source: D. RQ3: The Impact of Different Modules · p. 8

IV. CASE STUDY

• HFUZZER found 46 unique hallucinated packages in GPT-4o, including 11 packages for other languages.The large-scale evaluation used information from the top 1,000 packages for 10,000 rounds, with GPT-4o mini as the tester model.Source: §IV · p. 8
• The case study distinguishes code errors, where imports are incorrect, from package errors, where imports are correct but installation commands name hallucinated packages.Examples include the incorrect `flask_livereload` import for the intended `livereload` package and `pip install pywt` despite the correct `pywt` import for PyWavelets.Source: §IV · p. 8
• 34 hallucinated packages were package errors and 12 were code errors, suggesting environment configuration may be a more prone setting for package hallucinations.Two authors classified packages independently using whether the code could parse the correct package after installation; the authors note that the hypothesis requires investigation across models.Source: §IV · p. 9
• The analysis identified five placeholders; all hallucinated packages with comments were marked as placeholders or assumptions, while 12 hallucinated packages contained such comments.The authors also note that the model may ignore comments when analyzing required packages in code.Source: §IV · p. 9

A. The Effect of Potential Data Contamination

• The contamination study used the first 100 SourceRank Python packages released after model training cutoffs, running 1,000 rounds with all tester models and Meta-Llama-3.1 and Mistral-v0.3.Its results were compared with Section III-B, which used the top 100 pre-cutoff packages.Source: A. The Effect of Potential Data… · p. 9
• Post-cutoff packages yielded slightly higher Diversity Indices than pre-cutoff packages: 0.3%, 7.6%, and 24.8% at DBSCAN ε = 0.1, 0.2, and 0.3, respectively.HFUZZER also found more hallucinated packages with post-cutoff packages.Source: A. The Effect of Potential Data… · p. 9
• Post-cutoff tasks included unfamiliar elements such as model context protocol and expanded the seed phrase pool, which may explain the increased hallucinations and diversity.The passage attributes increased hallucinations to LLMs’ difficulty answering accurately about information beyond their training data [43], and greater response randomness to increased task diversity.Source: A. The Effect of Potential Data… · p. 9

B. The Effect of Parameter Settings

• The parameter study tested score, power, and temperature variants on the two model combinations with the most unique hallucinated packages in RQ1.Score settings were α/β = 1.25/0.25 or 0.75/0.75; power settings were k1/k2/k3 = 0.05/0.8/0.6, 0.15/0.7/0.6, or 0.15/0.8/0.7; five temperatures were tested. Other settings matched RQ1, and Default was the Section III setting.Source: B. The Effect of Parameter Settings · pp. 9–10
• Parameter variants found fewer unique hallucinated packages than the Default, but their overall effects were limited and results consistently outperformed GPTFUZZER-A.Temperature had no significant effect on the number of unique hallucinated packages, with no clear trend observed.Source: Figure 8 · p. 10, B. The Effect of Parameter Settings · p. 10
• Parameter variants produced Diversity Indices between 85% and 96% of the Default, averaged over all clustering parameters.The result is shown in Figure 8.Source: Figure 8 · p. 10, B. The Effect of Parameter Settings · p. 10
• Temperature reduced the Diversity Index to 89% at 0.2, but increased it slightly by 1.02%–1.26% at 0.4–1.0.The temperature effect on unique hallucinated-package counts was not significant, with no clear trend.Source: B. The Effect of Parameter Settings · p. 10

C. The Impact of Package Hallucinations

• Hallucinated packages have appeared in GitHub repositories, and one case study found a hallucinated package downloaded over 30,000 more times than a randomly named package in three months.The authors warn that developers and end-to-end software agents such as Devin may unknowingly install hallucinated packages, increasing potential exposure to malicious attacks.Source: C. The Impact of Package Hallucinations · p. 10

D. Potential downstream tasks

• Fine-tuning, Retrieval Augmented Generation, and Self-Detected Feedback reduce package hallucination rates, with fine-tuning reducing PHR by 83% on DeepSeek Coder 6B.HFUZZER can provide data for fine-tuning and future mitigation evaluation.Source: D. Potential downstream tasks · p. 10
• Research reports a negative correlation between PHR and code-benchmark performance, making package-hallucination reduction a potential path to improving code-model performance [9].The authors say HFUZZER provides data and evaluation methods for follow-up studies of techniques such as fine-tuning and model editing.Source: D. Potential downstream tasks · p. 10

E. Testing LLM VS. Detecting LLM Hallucinations

• HFUZZER generates diverse, logical inputs to test LLMs, complementing studies that analyze outputs to detect hallucinations,,,,,,,,,,,.The cited approaches detect hallucinations, with some detecting package hallucinations as a side effect; HFUZZER instead tests the model.Source: E. Testing LLM VS. Detecting LLM… · p. 10

F. Threats to Validity

• The evaluation focuses on Python despite HFUZZER being language-agnostic, and regex extraction may be unstable despite one-shot prompting and validation checks.The authors examine tester-model temperature on two model combinations for cost reasons and plan broader sampling-strategy evaluation.Source: F. Threats to Validity · p. 10

A. Fuzzing

• Prior fuzzing work spans seed grouping, in-memory mutation, API-level and dynamic techniques, semantic alignment, and LLM-based input generation or mutation, whereas HFUZZER tests the LLM itself [59],,,,,,,.Examples of LLM-based fuzzing include coverage feedback with an LLM mutator and generating anomalous programs for deep-learning libraries.Source: A. Fuzzing · p. 11

B. LLM Jailbreak

• Jailbreak research tests vulnerabilities through template-based red-team attacks, [20] and proposes representation- and inference-based defenses, [70].Prior work uses templates to preserve prompt structure or mutate initial templates; defense approaches adjust hidden representations or re-prompt based on inferred intent.Source: B. LLM Jailbreak · p. 11

C. LLM Hallucination

• Hallucination research includes prefix-tuning for mitigation and detection using internal states, uncertainty, classifiers, and metamorphic testing.The metamorphic approach assumes a unique answer.Source: C. LLM Hallucination · p. 11
• Package hallucinations reference unregistered or post-cutoff packages and can expose developers to attacker-registered malicious packages [10], [9],,.The cited definition includes packages first registered after the model’s knowledge cutoff date.Source: C. LLM Hallucination · p. 11
• Code-hallucination research classifies code-generation hallucinations [5], mitigates low-frequency API hallucinations [73], and detects them through execution validation.Other studies investigate package hallucination specifically [10], [9], which HFUZZER targets.Source: C. LLM Hallucination · p. 11
• Unlike hallucination detection and mitigation methods, HFUZZER generates tasks to trigger model hallucinations, similarly to red-team attacks.The paper presents this approach as a response to the security risks of package hallucinations.Source: C. LLM Hallucination · p. 11

VII. CONCLUSION AND FUTURE WORK

• HFUZZER triggered package hallucinations across all selected models, found 2.60x more unique hallucinated packages than GPTFUZZER-A, and identified 46 for GPT-4o.It also generated more diverse tasks; GPT-4o hallucinated packages during both code generation and environment configuration.Source: §VII · p. 11
• Future work plans broader testing across more LLMs and languages, open-sourcing results to promote mitigation research, and expanding HFUZZER to code hallucinations.Source: §VII · p. 11