PaperlayerView exact revision on arXiv

Source-linked AI summary

Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systems

Hongyan Chang, Ergute Bao, Xinjian Luo, Ting Yu

arXiv:2601.07072v1cs.CRcs.AI

TL;DR

IPI research has often left unresolved whether malicious text is retrieved under realistic corpora and natural queries. This paper decomposes malicious content into retrieval and attack components, develops a black-box trigger-construction method, and finds reliable end-to-end attacks across retrieval and agentic systems.

  • Problem

    Prior evaluations often assume malicious text is already in context, leaving retrieval under realistic corpora and natural queries unresolved.

  • Method

    The paper decomposes IPI into a trigger fragment and an attack fragment, then uses a practical black-box algorithm to construct triggers that surface arbitrary attack objectives.

  • Results

    The evaluation across 11 information-retrieval benchmarks and 8 embedding models demonstrates a practical end-to-end IPI threat spanning realistic retrieval and downstream attacks.

  • Takeaways & Limitations

    IPI defenses must secure both retrieval and system-level components because retrieval is the decisive bottleneck and malicious text can drive harmful agent behavior.

  • Takeaways & Limitations

    The evaluation is limited to embedding-based retrieval and does not assess hybrid pipelines or reranking mechanisms, which could potentially mitigate the attack.

Abstract

from arXiv · show

Large language models (LLMs) increasingly rely on retrieving information from external corpora. This creates a new attack surface: indirect prompt injection (IPI), where hidden instructions are planted in the corpora and hijack model behavior once retrieved. Previous studies have highlighted this risk but often avoid the hardest step: ensuring that malicious content is actually retrieved. In practice, unoptimized IPI is rarely retrieved under natural queries, which leaves its real-world impact unclear. We address this challenge by decomposing the malicious content into a trigger fragment that guarantees retrieval and an attack fragment that encodes arbitrary attack objectives. Based on this idea, we design an efficient and effective black-box attack algorithm that constructs a compact trigger fragment to guarantee retrieval for any attack fragment. Our attack requires only API access to embedding models, is cost-efficient (as little as $0.21 per target user query on OpenAI's embedding models), and achieves near-100% retrieval across 11 benchmarks and 8 embedding models (including both open-source models and proprietary services). Based on this attack, we present the first end-to-end IPI exploits under natural queries and realistic external corpora, spanning both RAG and agentic systems with diverse attack objectives. These results establish IPI as a practical and severe threat: when a user issued a natural query to summarize emails on frequently asked topics, a single poisoned email was sufficient to coerce GPT-4o into exfiltrating SSH keys with over 80% success in a multi-agent workflow. We further evaluate several defenses and find that they are insufficient to prevent the retrieval of malicious text, highlighting retrieval as a critical open vulnerability.

1 Introduction

Retrieval is the critical barrier determining whether indirect prompt injection can become an end-to-end attack. This work overcomes that barrier with trigger fragments and demonstrates successful exploits under natural queries and realistic corpora.

  • The retrieval barrier: Prior evaluations often assume malicious text is already in context, leaving IPI risk across arbitrary queries and realistic corpora unresolved.This limits assessment in settings such as enterprise knowledge bases, clinical repositories, and financial email systems.
  • The retrieval barrier: Unoptimized malicious text is never retrieved on natural queries across 11 BEIR corpora, making retrieval the bottleneck of IPI.The result holds regardless of corpus size and query length.
  • End-to-end impact: A single poisoned email coerces GPT-4o to execute malicious Python that exfiltrates SSH keys in up to 80% of trials with zero user interaction.The exploit operates on general queries about common subjects rather than contrived retrieval triggers.
  • End-to-end impact: The study evaluates attacks across both RAG and agentic systems and finds existing defenses can be reliably bypassed by adaptive attack variants.These results establish successful IPI under realistic retrieval pipelines.
  • Trigger-based attack: A trigger fragment guarantees retrieval while an attack fragment carries arbitrary malicious instructions, separating retrieval from payload construction.The trigger is a compact token sequence designed to surface the attack fragment under natural queries.
  • Trigger-based attack: The black-box attack requires only embedding-model API access and is evaluated across 11 information-retrieval benchmarks and 8 open- and closed-source embedding models.The approach avoids requiring gradients or access to embedding-model parameters.

2 Problem Formulation

The paper models IPI as constructing a prefix that makes an attacker-provided payload rank among the retrieved items for any natural query. The threat model assumes black-box embedding access and focuses on retrieval, not payload construction or downstream behavior.

  • Retrieval-based LLM Systems: A retrieval-based LLM embeds queries and corpus items, ranks them by cosine similarity, and returns the top-K items to the base model.Cosine similarity maps two embedding vectors to [-1,1].
  • Attack objective: The adversary aims to coerce the system into executing an arbitrary attack fragment encoding objectives such as misinformation, phishing, or Python-command execution.The user query may be any natural query, while the attack fragment is attacker-specified.
  • Black-box threat model: The adversary has no access to retriever or LLM parameters and can query only the embedding model through standard APIs.This reflects proprietary embedding deployments accessible through restricted interfaces.
  • Overall attack framework: The formal objective is to construct a prefix x such that x concatenated with the attack fragment ranks in the top-K retrieved items.Retrieval of the concatenated item is intended to ensure execution of the attack fragment.
  • Scope and assumptions: The work studies retrieval of an attacker-provided payload, not construction of the payload or its downstream effect on the LLM.Those issues are treated as the focus of direct prompt-injection literature.
  • Scope and assumptions: The black-box setting excludes gradient-based white-box attacks, while directly prepending the query is the prior black-box baseline and does not always ensure retrieval.The baseline is x = q.

3 Prefix Construction Attack

The attack formulates trigger construction as black-box optimization of prefix similarity to a target query. A CEM-inspired factorized search concentrates limited API queries on high-scoring token sequences while respecting a token budget.

  • Similarity Search for Prefix: The attack maximizes cosine similarity between a prefixed attack fragment and the target query embedding.The attack fragment is assumed to have already been crafted; optimization focuses on the trigger prefix.
  • Similarity Search for Prefix: Top-K retrieval is characterized by comparing the candidate’s score with the external corpus items’ scores, with experiments using K = 5.For K = 1, the threshold is the highest corpus-item similarity score.
  • Optimization formulation: Because the external dataset is unobserved, finding a prefix satisfying the exact retrieval inequality is NP-hard, motivating bounded approximate optimization.The practical goal is an ε-optimal prefix in a bounded search space.
  • Optimization formulation: The token budget n bounds prefix length because unlimited sequences would make the search space impractical; increasing n can improve the optimal and ε-optimal scores.The paper also reports empirical verification of this relationship.
  • Black-box search challenge: Naive greedy search requires n|V| computations because every position must test every vocabulary token.This combinatorial cost motivates a more query-efficient method.
  • CEM Attack for Prefix Search: The method adapts the Cross-Entropy Method to sample candidate prefixes, select elite sequences, and update a distribution toward higher black-box scores.Each iteration uses a fixed batch of target-function queries, matching limited API access.
  • CEM Attack for Prefix Search: A fully factorized distribution represents token choices across positions with an n-by-|V| matrix, avoiding the |V|^n cost of a joint representation.The factorized distribution is repeatedly refined during optimization.
  • CEM Attack for Prefix Search: The algorithm initializes uniform per-position distributions, samples N length-n sequences per iteration, scores them, and updates the distribution under NT ≤ B.Its inputs include the attack fragment, embedding model, target query, token length, batch size, elite fraction, smoothing, and iteration count.

9 Output the best sequence as the trigger fragment

The algorithm iteratively samples candidate sequences, retains the highest-scoring fraction, and updates token-position distributions toward those elite samples. Under the stated linear-score condition, it provides an efficient near-optimality guarantee while using fewer black-box evaluations than naive or brute-force search.

  • Cross-Entropy Method: The algorithm samples N sequences independently from the current distribution, evaluates each sequence, and selects the top-λ fraction by score.The selected elite set S contains the highest-scoring samples used for subsequent updates.
  • Cross-Entropy Method: It updates each token-position distribution using the fraction of each token among the selected high-scoring sequences, with smoothing controlled by α.This update concentrates probability on tokens favored by the elite samples while retaining smoothing through α ∈ (0,1).
  • Utility Guarantee: Under a linear score across token positions, T = O(log|V|) iterations and N = O(log 1/δ) samples per iteration return an ε-optimal sequence with probability at least 1−δ.The guarantee achieves f(x) ≥ f(x*) − ε for any δ ∈ (0,1).
  • Cost: The method uses O(log|V| log 1/δ) black-box accesses to f, compared with n|V| accesses for greedy search and O(|V|^n) for brute-force sampling.This scaling addresses the combinatorial explosion caused by the token sequence search space.

4 Evaluation on Trigger Fragment

The evaluation tests whether CEM trigger fragments can surface arbitrary attack fragments under natural queries across diverse retrieval corpora and embedding models. Results show near-perfect retrieval in many settings, with attack difficulty driven more by corpus competition than corpus size and with mixed transferability across models and positions.

  • Setup: The evaluation uses 11 BEIR datasets, subsampling 100 target queries per dataset to test retrieval under diverse scenarios.Each dataset contains a document corpus, queries, and relevance labels.
  • Effectiveness in Retrieval: A single malicious insertion reaches the top five with near-perfect recall on seven datasets using only 5–10 trigger tokens.These datasets are NFCorpus, Natural Questions, SciFact, HotpotQA, DBPedia, SciDocs, and FEVER.
  • Effectiveness in Retrieval: On harder corpora, increasing the trigger length to around 15 tokens typically raises retrieval above 80% and sometimes above 90%, outperforming the baselines at matched lengths.The attack also uses shorter triggers to achieve comparable performance.
  • Effectiveness in Retrieval: Across eight FiQA embedding models, the attack consistently achieves near-perfect performance regardless of model size, architecture, or access type.The evaluated models span open-source and proprietary systems, including small and large models.
  • Efficiency: The default optimization uses at most 150,000 black-box embedding queries and costs $0.21 with voyage-3.5-lite or OpenAI’s text-embedding-3-small.Open-source models complete in 1.6–7.6 minutes on a single H100 GPU.
  • Transferability: Transferability is mixed: prefixes can transfer across positions and attack fragments, while transfer across embedding models is stronger within some model families and weaker across others.A Q3-0.6B prefix achieves only 10% retrieval on the OpenAI model, whereas Qwen-family transfer is stronger.

5 End-to-end Evaluations

The paper evaluates optimized indirect prompt injection end to end in RAG and agentic systems, showing that retrieved malicious text can steer outputs, tool use, and code execution.

  • 5 End-to-end Evaluations: Once retrieved, optimized malicious text can hijack system behavior across denial of service, phishing propagation, tool misuse, and code execution.The paper evaluates retrieval and downstream impact as distinct stages of successful indirect prompt injection.
  • 5.1 Case Study: RAG: A single malicious document reliably coerces most tested LLMs into producing a fixed target answer across nearly all datasets.The evaluation covers 11 datasets and 11 LLMs; attack success rates are often close to 1.
  • 5.1 Case Study: RAG: The RAG evaluation defines attack success as poisoned-corpus queries producing the target answer when the clean corpus does not.The target response is “Yes,” and cases where the clean system already outputs it are excluded.
  • 5.1 Case Study: RAG: Knowledge poisoning reaches ASR 0.58 with a two-token trigger and 0.50 with a single-token trigger on NQ using LLaMA-2-7B.These results match or approach a query-prepending method while using shorter triggers.
  • 5.2 Case Study: Agentic Systems: The agentic evaluation covers RAG-driven single-agent and multi-agent workflows involving downstream tool use and inter-agent coordination.Table 2 separately records phishing propagation, contact broadcasting, and phishing-link outcomes.
  • 5.2 Case Study: Agentic Systems: A 10-token prefix achieves near-perfect retrieval and outperforms query concatenation and repetition baselines despite using fewer tokens.The fusion variant further improves semantic alignment and stabilizes retrieval across tasks.
  • 5.2 Case Study: Agentic Systems: In code execution, end-to-end attack success reaches 26% on GPT-4o-mini, compared with 4% for the idealized IPI baseline.This comparison shows that evaluating attack fragments as already present in context can underestimate risk in this setting.

6 Evaluation on Defense

The evaluation finds that query paraphrasing, perplexity filtering, and token masking do not provide durable protection against adaptive attacks.

  • Defense overview: None of the three evaluated defenses provides durable protection once the attacker adapts.Small initial gains collapse under adaptive attack variants.
  • Query Paraphrasing: Joint optimization over multiple query paraphrases restores attack performance and can surpass the baseline.This makes paraphrasing ineffective against the adaptive attack.
  • Perplexity Filtering: Repeating malicious text lowers its perplexity below clean documents while preserving attack effectiveness, defeating perplexity filtering.The adaptive repetition strategy can make malicious text appear more natural than benign content.
  • Token Masking: Token masking is also ineffective in the evaluated setting.The paper reports this conclusion with results deferred to Figure 14 and the appendix.

7 Related Work

Prior indirect prompt injection evaluations commonly assume poisoned text is already retrieved, while this work targets the retrieval stage itself.

  • Indirect Prompt Injection: Many prior prompt-injection setups guarantee that the poisoned item appears in context by fixing tool outputs or using specially constrained queries.These settings focus on ranking manipulation rather than end-to-end retrieval under natural queries.
  • Indirect Prompt Injection: Query+ concatenates the user query to malicious text and is treated as the strongest practical black-box heuristic baseline.Prior work reports only a slight similarity boost from this approach.
  • Indirect Prompt Injection: Suffix-level defenses operate after malicious content enters context, whereas this work studies whether the content is retrieved in the first place.Because the optimized prefix can surface arbitrary suffixes, the effectiveness of suffix-level defenses is outside scope.

8 Limitations

The evaluation is limited to embedding-based retrieval and retrieval-stage defenses, does not test hybrid or reranking pipelines, and does not guarantee transfer across different embedding architectures.

  • Scope of work: The study does not evaluate hybrid pipelines or reranking mechanisms, which could potentially mitigate the attack.Both the attack evaluation and defense evaluation are restricted to embedding-based retrieval systems.
  • Transferability: The attack does not ensure transferability when reference and target embedding models use different architectures.The authors identify closing this transferability gap as future work.

9 Conclusion

The paper identifies retrieval as the decisive bottleneck in indirect prompt injection and shows that a black-box trigger-construction algorithm can reliably surface malicious objectives for arbitrary queries. Evaluations across benchmarks, embedding models, and downstream attacks establish IPI as a practical end-to-end threat.

  • Conclusion: IPI decomposes into a trigger fragment and an attack fragment, enabling a black-box algorithm to surface arbitrary attack objectives for arbitrary queries.The trigger fragment is constructed to make the malicious text retrievable, while the attack fragment encodes the objective.
  • Conclusion: Retrieval is the decisive bottleneck that determines whether indirect prompt injection succeeds in realistic systems.
  • Conclusion: The evaluation spans benchmarks, embedding models, and downstream attacks, supporting the conclusion that IPI is a practical end-to-end threat.

Ethical Considerations

The paper frames its work as a controlled security evaluation intended to improve defenses against retrieval-based LLM risks. It also acknowledges potential misuse beyond IPI and cautions that addressing this narrow threat does not resolve broader harms associated with LLM deployment.

  • Ethical principles: The research aims to improve security awareness and defensive measures for retrieval-based LLM systems using public benchmarks and synthetic corpora rather than real-world deployments.
  • Ethical principles: The authors restrict dissemination of exploit content and frame the analysis around system hardening and defense.
  • Potential misuse: CEM could potentially be adapted beyond IPI and LLM agents to manipulate embedding-based search or recommendation systems.The experiments themselves remain confined to controlled benchmarks and security evaluation.
  • Second-order effects: The paper cautions that progress against indirect prompt injection should not be interpreted as solving broader safety, environmental, cognitive, or intellectual-property harms.

Open Science

The paper provides source code and detailed evaluation instructions through a Zenodo repository. The release is intended to support scientific reproducibility and controlled defensive testing.

  • Open Science: The Zenodo repository contains the attack algorithm for constructing trigger fragments and scripts for end-to-end evaluations.
  • Open Science: The paper identifies the repository as the location for its source code and detailed instructions.

A.1.1 Different Metrics

Across retrieval and downstream exploit evaluations, the attack remains effective as trigger fragments grow, while adaptive defenses can be bypassed. Exact token masking weakens retrieval, but perplexity-based detection and paraphrasing provide limited protection.

  • Retrieval metrics: 29.5% average Recall@5 at n = 3 rises to 95.6% at n = 10, with MRR@5 of 0.79 and 100% Recall@5 on several datasets.The reported perfect datasets include NFCorpus, NQ, HotpotQA, DBPedia, SCIDOCS, FEVER, and SciFact.
  • Downstream effects: A single poisoned retrieval can drive downstream behavior, including affirmative responses, email sending, survey creation, and execution of malicious Python code.The case studies show these effects across targeted-answer and agentic workflows.
  • Query paraphrasing: Paraphrasing reduces Recall@5 by less than 10% on most datasets, while adaptive attacks restore or exceed original effectiveness.SciDocs increases from 95.8% to 97.6%, and MSMARCO rises to 79.1% versus a 69.8% no-defense baseline.
  • Perplexity filtering: Duplicating malicious content lowers its average perplexity from 154.1 to 14.4, allowing poisoned text to resemble fluent clean content.Further repetition lowers perplexity even more, undermining perplexity-based filtering.

D.2 Detailed Analysis of CEM Attack

The CEM attack analysis shows that selecting top-scoring samples amplifies the probability of sampling ε-optimal tokens, yielding a high-quality sequence after logarithmically many iterations and samples.

  • Per-token amplification: Each iteration increases the expected score of sampled sequences and amplifies the probability of sampling good tokens.The proof uses the positive score gap, selection conditioning, and Markov’s inequality.
  • CEM analysis: CEM repeatedly updates a factorized token distribution using the top-λN highest-scoring sampled sequences.The analysis assumes an additive objective and updates token probabilities in parallel across positions.
  • Utility guarantee: With T = O(log 1/min_i µ_i) iterations and N = O(log 1/δ) samples per iteration, Algorithm 1 returns an ε-accurate sequence with probability at least 1−δ.The guarantee is length-independent in its stated iteration and sample-count form, while the overall query complexity is O(log 1/min_i µ_i log 1/δ).
  • Utility guarantee: Under uniform initialization, the iteration count becomes T = O(log|V|), while ε and n affect the logarithm’s base through the initial token probabilities and gaps.The analysis notes that these dependencies are omitted from the displayed asymptotic expression but influence its constants.
  • Per-token amplification: Conditioning on selection into the top-λN set strictly increases the probability that a sequence contains an ε-optimal token.The positive token gap creates a higher conditional selection probability for sequences containing elite tokens.
Loading 2601.07072v1…