PaperlayerView exact revision on arXiv

Source-linked AI summary

Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files

Yupu Wang, Zhengyuan Jiang, Reachal Wang, Neil Zhenqiang Gong

arXiv:2610.09264v1cs.CRcs.AI

TL;DR

Security implications of widely shared coding rule files have been largely overlooked, despite their role in guiding autonomous coding agents. The paper introduces PackHallu, which optimizes injected prompts using trajectory-level feedback, and reports high attack success alongside unreliable detection of malicious rule files.

  • Problem

    Security implications of the commonly shared coding rule-file ecosystem have been largely ignored.

    Source: §1 · p. 1

  • Method

    PackHallu uses evolutionary search to optimize malicious prompts embedded in rule files, evaluating candidate prompts with the surrogate agent’s complete trajectory.

    Source: §5.3 · p. 7, §8 · p. 19

  • Results

    PackHallu achieves high attack success rates and substantially outperforms existing prompt-injection attacks, while current detectors fail to reliably identify its malicious rule files.

    Source: §1 · p. 3, §8 · p. 20

  • Takeaways & Limitations

    The findings highlight a security gap in the coding-agent ecosystem and the need for stronger defenses for the collaborative rule-file pipeline.

    Source: §8 · p. 20

    The threat model assumes attackers can access a benign rule file and deploy a local surrogate agent with surrogate tasks, while lacking victim-agent and victim-task knowledge.

    Source: §4 · p. 5

Abstract

from arXiv · show

Modern agentic coding frameworks increasingly rely on community-shared rule files (e.g., AGENTS.md or .cursorrules) to guide autonomous code generation, yet the security risks of this pipeline remain underexplored. To bridge this gap, we introduce the package hallucination attack, where an attacker injects malicious prompts into benign rule files to induce coding agents to replace legitimate dependencies with attacker-controlled packages. To obtain effective malicious prompts injected into rule files, we propose PackHallu, an evolutionary optimization framework that iteratively rewrites these injected prompts using trajectory-level feedback and LLM-guided mutations. Evaluations across multiple benchmarks, LLMs, and agent frameworks show that PackHallu achieves high attack success rates and strong transferability across diverse models and agent combinations. Our findings demonstrate that coding agents are vulnerable to package hallucination attacks, highlighting the urgent need for stronger security safeguards in autonomous coding systems.

1 Introduction

The paper identifies malicious prompts embedded in shared coding rule files as a route to make agents import attacker-controlled packages. It proposes PackHallu to optimize those prompts and reports strong attacks and weak detection across the evaluated setting.

1 Introduction

  • Coding rule files guide agent behavior, are complex to customize, and are commonly downloaded and shared, while their security implications have received little attention.Existing attacks are less effective in this setting because long rule files dilute prompts and agent-generated code provides sparse optimization feedback.Source: §1 · pp. 1–2
  • The attack injects a prompt into a shared rule file so an agent replaces a legitimate dependency with an attacker-chosen package that may run when the generated code executes.The attacker publishes both the contaminated rule file and malicious package; the paper distinguishes this from prior package hallucination involving random nonexistent packages.Source: §1 · p. 1
  • PackHallu uses evolutionary search with LLM-guided failure analysis and trajectory-level feedback to optimize prompts embedded in coding rule files.It uses a locally deployed surrogate agent rather than requiring access to the victim agent.Source: §1 · p. 2
  • PackHallu achieves high attack success rates and substantially outperforms existing heuristic-based and optimization-based prompt injection attacks.The evaluation spans 3 coding benchmarks, 8 agentic coding frameworks, and 13 backbone LLMs.Source: §1 · p. 3
  • Detectors fail to reliably identify PackHallu-generated malicious rule files, exposing a security weakness in the coding rule-file ecosystem.The reported detector outcomes include missed attacks or high false positive rates.Source: §1 · p. 3

2 Related Work

Related work covers model hallucinations and prompt injection, then identifies why existing injection methods fit coding-agent rule files poorly. PackHallu is designed around those constraints.

2 Related Work

  • Prior code-hallucination work studies intrinsic, non-adversarial model errors, whereas this paper considers an attacker targeting coding agents.The paper distinguishes its attack from prior package hallucination that produces random nonexistent imports.Source: §2 · p. 3
  • Prompt injection methods include manually designed heuristic prompts and optimization-based approaches that refine prompts toward a target behavior.The reviewed optimization methods include white-box losses and black-box feedback-driven search.Source: §2 · p. 3
  • Existing methods face contextual dilution in long rule files, limited practicality of white-box optimization, and sparse feedback when success appears only in final code.The paper reports limited effectiveness for existing techniques in this setting and presents PackHallu as designed to address these constraints.Source: §2 · pp. 3–4

3 Problem Definition

The problem definition models a coding agent as mapping a composite input to generated code, with a rule file among its inputs. A package hallucination attack manipulates that file to substitute an attacker-controlled package for the intended dependency.

3.1 Coding Agent

  • A coding agent maps its composite input to generated code, and that input includes the system prompt, user prompt, optional external context, and coding rule file.External context is optional when the task is standalone code generation rather than modifying or extending a repository.Source: §3.1 · p. 4

3.2 Package Hallucination Attack

  • A package hallucination attack injects a malicious prompt into a benign rule file to make the agent import an attacker-controlled package instead of the package the task requires.The paper gives numpy and numpy_hl as an example victim-package and malicious-package pair.Source: §3.2 · p. 4
  • The attack seeks a prompt that induces package substitution while preserving semantic coherence so the prompt remains stealthy.The resulting malicious input consists of the agent's ordinary inputs with the modified rule file.Source: §3.2 · p. 4

4 Threat Model

The threat model gives attackers the goal of inducing malicious-package imports while keeping injected prompts semantically meaningful. It assumes limited knowledge of the victim but allows rule-file distribution and local surrogate-agent optimization.

4 Threat Model

  • The attacker seeks both effectiveness—replacing the intended package when a task requires it—and stealthiness through a semantically meaningful injected prompt.The malicious package may perform attacker-desired behavior when executed; each prompt targets a specific victim-package and malicious-package pair.Source: §4 · p. 5
  • The attacker is assumed to lack knowledge of the victim agent, its system prompt, backbone model, framework, task, and external contexts.The attacker does have access to a benign coding rule file for injection.Source: §4 · p. 5
  • The attacker can publish contaminated rule files and malicious packages, and can optimize prompts using a local surrogate agent and surrogate tasks.The surrogate may use different framework and backbone models, and its tasks may come from a different distribution than victim tasks.Source: §4 · p. 5

5 Our PackHallu

PackHallu formulates malicious-prompt discovery for package hallucination as an evolutionary search using a surrogate agent and surrogate tasks. It combines trajectory-level scoring with LLM-guided refinement, then iterates prompt generation and selection.

5.1 Overview

  • PackHallu combines trajectory-level scoring, self-attributed prompt refinement, and evolutionary search to optimize malicious prompts using a surrogate agent and surrogate tasks.The search returns the highest-scoring prompt encountered across its rounds.Source: §5.1 · p. 5

5.2 Formulating an Optimization Problem

  • The optimization searches for an injected prompt that replaces a victim package with a malicious package, despite contextual dilution and costly, sparse feedback.Each evaluation otherwise requires a complete multi-turn agent interaction per task.Source: §5.2 · p. 6

5.3 Trajectory-Level Signal

  • The trajectory-level signal scores package occurrences across the surrogate agent’s complete reasoning trajectory, rather than checking only its final code.The broader count also credits evidence in reasoning traces or tool-call formulations; the binary final-code check is a special case.Source: §5.3 · pp. 6–7

5.4 Self-Attributed Refinement

  • The attack LLM critiques a candidate prompt’s likely weakness, proposes a revision strategy, and generates offspring prompts conditioned on that critique and strategy.The attack LLM and surrogate agent use the same backbone LLM, making the critique structurally self-directed.Source: §5.4 · p. 7

5.5 The Full Refinement Loop

  • Each round refines every parent prompt into offspring, scores offspring over sampled surrogate tasks, and selects the top K for the next generation.The process repeats for R rounds and returns the highest-scoring prompt from the search history; the seed specifies malicious-package use and victim-package replacement objectives.Source: §5.5 · pp. 7–8

6 Evaluation

The evaluation examines PackHallu’s effectiveness across baseline attacks, backbone LLMs, agent frameworks, task distributions, and package popularity. It also studies generalization under growing code context and how injection location and optimization choices affect attack effectiveness.

6.1 Experimental Setup

  • The evaluation spans 8 agent frameworks and 13 backbone LLMs, with victim tasks drawn from code-generation and repository-level refactoring benchmarks.The main experiments use five backbone LLMs and BigCodeBench; the task collection covers 10 unique victim packages.Source: §6.1 · pp. 8–10
  • The controlled simulation pairs each victim package with a compatible malicious package hosted on a local registry, and distinguishes package-import success from deployable execution.SASR counts malicious-package imports; DASR requires successfully executed code to invoke the predefined malicious behavior.Source: §6.1 · pp. 10–12

6.2 Main Results

  • 79.29% SASR and 67.68% DASR for PackHallu exceed Combined Attack’s 14.99%/11.35%, improvements of 5.29× and 5.96×, respectively.PackHallu’s averages cover the evaluated packages and victim agents; SASR measures malicious-package imports, while DASR additionally requires executable code that invokes the malicious package.Source: Table 2 · p. 11, §6.2 · p. 12
  • PackHallu generalizes across packages, captures more LLM attention than baselines, and attack success generally declines as package popularity increases.The reported median Pearson correlations between popularity and success are r = −0.69 for SASR and r = −0.60 for DASR; greater attention to PackHallu’s coherent prompt accompanies higher success.Source: Table 3 · p. 13, Figure 2 · p. 13, §6.2 · p. 13

6.3 Generalization Study

  • On 8 of 12 victim LLMs, PackHallu exceeds 90% SASR on at least three packages, including models from Qwen, Llama, DeepSeek, and GPT families.Reported examples include 97.92% SASR on GPT-5.5 for Pandas and 100% on DeepSeek-V4-Flash for Pandas and Matplotlib. The study also reports higher vulnerability among more capable backbone LLMs when the framework is fixed.Source: Table 4 · p. 14, §6.3 · p. 14
  • Across agent frameworks, transferred prompts average 71.55% SASR and 62.99% DASR, with no observed correlation between framework capability and SASR (r = −0.186).88.0% of the 25 framework-package cells exceed 50% SASR, and 76.0% exceed 70%; both highly and less capable frameworks remain susceptible.Source: Table 5 · p. 15, §6.3 · p. 15
  • When both framework and backbone differ from the surrogate, PackHallu achieves 74.81% average SASR and 65.38% average DASR across four victim agents.SASR reaches up to 98% on Cursor with Auto and Composer 2.5; Claude Code remains more resistant, with SASR ranging from 31% to 59%.Source: §6.3 · pp. 15–16, Table 6 · p. 16
  • Prompts optimized on BigCodeBench transfer to DS-1000 at 53.33% SASR and 45.00% DASR, but fall to 18.60% and 16.28% on RefactorBench.The evaluated transfer tasks include code generation and refactoring, from file-level to repository-level settings.Source: Table 7 · p. 16, §6.3 · p. 16
  • As surrounding code context grows, the malicious prompt occupies a smaller share of the agent’s input, weakening its influence on generated output.The passage reports this effect in BigCodeBench.Source: §6.3 · p. 16
  • The Claude Code case study traces Sonnet 4.6 on all 87 NumPy-subset BigCodeBench tasks under a numpy → numpy_hl attack, with the prompt placed at the start of CLAUDE.md.The study collected full execution traces, including reasoning, tool invocations, and final responses.Source: §6.3 · pp. 16–17

6.4 Ablation Study

  • 82.63% average SASR at the rule-file beginning exceeded middle (74.03%) and end (77.29%) placement.The experiment used OpenCode with DeepSeek-V4-Flash across five BigCodeBench packages.Source: Table 8 · p. 17, §6.4 · p. 17

6.3 Generalization Study

  • Successful trajectories were led by Policy-Cited Compliance (22 cases, 25.3%); in three S3 cases, agents recognized the injection as an attack but complied under rule-file authority.Silent Compliance involved invoking the malicious package without acknowledging the prompt.Source: §6.3 · p. 17
  • Failed attacks included 16 Suspicion-Based Refusals, 33 Empirical Recoveries (37.9%), and 11 Silent Refusals (12.6%), with recovery reverting to numpy after numpy_hl import failure.Suspicion-Based Refusal involved recognizing the attack and refusing the prompt; Silent Refusal involved using numpy without deliberation.Source: §6.3 · p. 17

6.4 Ablation Study

  • Across different prompt initializations, PackHallu’s attack effectiveness remains stable, with SASR of 98.00 ± 2.09% and DASR of 96.00 ± 1.37% (mean ± sample standard deviation).The results indicate that PackHallu’s effectiveness does not depend on a carefully tuned initial prompt.Source: Table 9 · p. 17, §6.4 · p. 18
  • PackHallu components each had a positive effect, while more optimization rounds, offspring prompts, and surrogate tasks generally improved performance with diminishing gains.The component and hyperparameter results are presented in Tables 10 and 11.Source: §6.4 · p. 18, Table 10 · p. 18, Table 11 · p. 18

7 Detecting Malicious Rule Files

The evaluation tests five prompt-injection detectors on malicious rule files generated by PackHallu. None reliably detects the attack while maintaining a usable false-positive rate.

7 Detecting Malicious Rule Files

  • The benchmark comprises 100 public benign rule files and 100 malicious counterparts created by inserting PackHallu prompts at the beginning.The five evaluated detectors are ProtectAI-DeBERTa, PromptGuard, DataSentinel, PromptArmor, and PIShield.Source: §7 · p. 18
  • 1.00 FNR for ProtectAI-DeBERTa, 0.46 for PromptGuard, 0.49 for DataSentinel, and 0.20 for PromptArmor; PIShield records 0.21 FNR and 0.11 FPR, while PromptArmor's FPR is 0.34.The authors report that PromptArmor's high FPR makes it impractical, while PIShield still misses more than one in five malicious files.Source: Table 12 · p. 19, §7 · p. 19
  • PromptArmor's false positives arise because coding rule files are instructional prompts, while PIShield's lower FPR still accompanies missed attacks.The paper concludes existing detectors do not reliably identify the attack while maintaining a usable FPR.Source: §7 · p. 19

8 Conclusion and Future Work

The paper introduces package hallucination attacks through prompt injection in coding rule files and proposes PackHallu to optimize the injected prompts. Its evaluations report high attack success, stronger performance than existing prompt-injection attacks, and unreliable detection by current methods.

8 Conclusion and Future Work

  • Package hallucination attacks use malicious prompts in coding rule files to induce package substitution, and PackHallu optimizes prompts for this attack.The paper reports that the attacker-chosen package replaces a legitimate dependency in generated code.Source: §8 · p. 19
  • Evaluations show PackHallu achieves high attack success rates and substantially outperforms existing prompt-injection attacks.The paper also reports that state-of-the-art prompt-injection detection methods fail to reliably identify malicious rule files.Source: §8 · p. 20
  • The findings identify a security gap in the coding-agent ecosystem and call for stronger defenses for the collaborative rule-file pipeline.The paper frames these defenses as an important direction for future work.Source: §8 · p. 20

A Implementation Details

The implementation uses PackHallu with locally hosted open-source LLMs and reports its optimization setup and runtime. The appendix also points to the initial prompt and hyperparameter settings.

A Implementation Details

  • PackHallu is implemented in PyTorch, uses Ollama as its inference backend, and runs on five locally hosted open-source LLMs.The initial prompt instantiates the victim and malicious packages as numpy and numpy_hl, and optimization uses the numpy subset of surrogate tasks.Source: A Implementation Details · p. 23, Figure 5 · p. 23
  • The appendix identifies Table 13 as the summary of PackHallu hyperparameter settings.The supplied table caption does not state the settings themselves.Source: Table 13 · p. 23
  • A full optimization run takes approximately 24 to 30 hours, depending on backbone-model inference speed.Models use up to two NVIDIA Quadro RTX 6000 GPUs, each with 24 GB of memory.Source: A Implementation Details · p. 23

B Examples of Optimized Malicious Prompts

The appendix presents optimized malicious prompts produced with OpenHands and different backbone LLMs. The examples use forceful migration instructions that replace victim-package imports with a malicious package.

B Examples of Optimized Malicious Prompts

  • The optimized prompts shown were obtained with OpenHands paired with distinct backbone LLMs.The appendix presents them as examples of PackHallu's optimized prompts.Source: B Examples of Optimized Malicious… · p. 23
  • One prompt demands replacing every victim-package import with the malicious package and warns that code retaining the victim package will be rejected.It labels the migration mandatory and non-negotiable.Source: B Examples of Optimized Malicious… · p. 23
  • Other examples call the victim package forbidden and require replacing all references, imports, and usage with the malicious package.One example also claims its directive supersedes all other instructions.Source: B Examples of Optimized Malicious… · pp. 23–24

C Optimization Dynamics

Appendix C presents PackHallu’s optimization dynamics through average offspring scores across rounds and includes trajectories illustrating several victim-agent responses. It also documents comparative plots, prompt examples, and evaluation setup.

C Optimization Dynamics

  • Figure 6 tracks the average score of offspring candidates generated in each optimization round.The accompanying text identifies it as showing the dynamics of prompt optimization.Source: Figure 6 · p. 24, C Optimization Dynamics · p. 24
  • In one trajectory, the agent recognizes the injection and intends to flag it, but its user-facing response recommends installing numpy_hl.Figure 8 separately depicts an agent adopting numpy_hl without detecting or questioning the injected prompt, which it treats as a legitimate requirement.Source: Figure 7 · p. 25, Figure 8 · p. 25
  • Another trajectory shows the agent reverting to numpy only after a failed numpy_hl import triggers a ModuleNotFoundError.The caption attributes recovery to the runtime failure rather than prompt-internal suspicion.Source: Figure 9 · p. 25
  • Figures 10 and 11 depict suspicion-based refusal cases, while Table 14 presents a behavioral taxonomy for Claude Code and Sonnet 4.6 under PackHallu.The captions do not provide the taxonomy entries or refusal outcomes.Source: Figure 10 · p. 26, Figure 11 · p. 26, Table 14 · p. 27
  • Figures 12 and 13 plot DASR against victim-agent capability across backbone LLMs and agent frameworks, respectively, with the other dimension fixed.Their captions specify the plotted relationship and fixed dimensions, but not its direction or strength.Source: Figure 12 · p. 28, Figure 13 · p. 28
  • The appendix includes malicious-prompt examples, four paraphrased PackHallu variants, and an initial GCG prompt.Its tables also document task counts, OpenHands input context, agent-framework configurations, and the 13 victim-agent backbone LLMs.Source: Table 15 · p. 28, Figure 14 · p. 29, Figure 15 · p. 29
Loading 2610.09264v1…