Source-linked AI summary
Prompt Injection attack against LLM-integrated Applications
Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, Leo Yu Zhang, Yang Liu
TL;DR
Prompt injection poses a poorly understood security risk to LLM-integrated applications, where malicious inputs can override predefined instructions. The paper develops HOUYI, a black-box attack methodology inspired by traditional injection attacks, and finds broad susceptibility across real-world services. Its results motivate stronger defenses while acknowledging that some vulnerabilities may not remain reproducible as applications evolve.
Problem
Prompt injection mechanisms and exploitability in real-world LLM-integrated applications remain insufficiently understood despite existing heuristic attacks.
Method
HOUYI combines a pre-constructed prompt, context-separating injection prompt, and malicious question within a black-box, iterative attack methodology.
Results
31 of 36 real-world LLM-integrated applications were susceptible to HOUYI, with 10 vendors acknowledging the findings.
Takeaways & Limitations
HOUYI can expose prompt-leak and prompt-abuse scenarios, including original-prompt theft and use of providers’ LLM computation.
Takeaways & Limitations
Some detected vulnerabilities may become non-reproducible as protection systems and backend LLMs evolve.
Abstract
from arXiv · showhide
Large Language Models (LLMs), renowned for their superior proficiency in language comprehension and generation, stimulate a vibrant ecosystem of applications around them. However, their extensive assimilation into various services introduces significant security risks. This study deconstructs the complexities and implications of prompt injection attacks on actual LLM-integrated applications. Initially, we conduct an exploratory analysis on ten commercial applications, highlighting the constraints of current attack strategies in practice. Prompted by these limitations, we subsequently formulate HouYi, a novel black-box prompt injection attack technique, which draws inspiration from traditional web injection attacks. HouYi is compartmentalized into three crucial elements: a seamlessly-incorporated pre-constructed prompt, an injection prompt inducing context partition, and a malicious payload designed to fulfill the attack objectives. Leveraging HouYi, we unveil previously unknown and severe attack outcomes, such as unrestricted arbitrary LLM usage and uncomplicated application prompt theft. We deploy HouYi on 36 actual LLM-integrated applications and discern 31 applications susceptible to prompt injection. 10 vendors have validated our discoveries, including Notion, which has the potential to impact millions of users. Our investigation illuminates both the possible risks of prompt injection attacks and the possible tactics for mitigation.
1 Introduction
Prompt injection threatens LLM-integrated applications, while existing attacks provide limited practical coverage. The paper introduces HOUYI, a systematic black-box method using context separation and reports substantial real-world attack success.
- Prompt injection overrides LLM instructions through malicious prompts, but its initiating patterns and underlying mechanisms remain insufficiently understood.
- Existing techniques achieved only partial exploitation in two of ten commercial applications during the pilot study.The study attributes failures to differing prompt usage across applications, including prompts treated as questions versus analytical data.
- HOUYI constructs attacks from Framework, Separator, and Disruptor components to separate malicious commands from established application context.The components integrate a pre-constructed prompt, trigger context separation, and provide a malicious question targeting the adversary’s objective.
- HOUYI uses context inference, payload generation, and feedback phases to iteratively adapt black-box attacks across service providers.The method infers application semantics from interactions, generates prompts using guidelines, and refines payloads based on responses.
- 86.1% success across 36 real-world services enabled original-prompt theft and exploitation of providers’ LLM computation.The authors report potential impacts involving millions of users and millions of US dollars in financial losses.
- Existing defenses mitigate traditional prompt injection but remain vulnerable to malicious payloads generated by HOUYI.
2 Background
LLM-integrated applications combine predefined prompts with user inputs to generate task-specific responses. This architecture enables useful services but also exposes applications to prompt injection through public interfaces and externally supplied content.
- Applications typically merge user input into provider-designed prompts before sending the combined prompt to an LLM.A provider may place a user question into a placeholder such as “Answer the following question as a kind assistant: <PLACE_HOLDER>”.
- Prompt injection manipulates application output by overriding or nullifying predefined prompts in the merged input.A malicious user can redirect an application from its intended task to a distinct query, such as printing attacker-chosen text.
- Internet-connected applications may also receive deceptive payloads through requested or supplied online resources.
- The threat model assumes public endpoint access and arbitrary input manipulation without access to application internals.Attackers may infer information from service responses while targeting outputs that deviate substantially from intended functionality.
3 A Pilot Study
The pilot study evaluates existing prompt injection strategies on ten commercial applications and finds that application design often limits their effectiveness. Its analysis identifies prompt interpretation, formatting, and workflow constraints as key barriers to successful exploitation.
- The pilot study surveys existing attack patterns and tests their exploitability on ten real-world LLM-integrated applications.The applications span chatbot, writing assistant, code assistant, business analysis, and creative generation categories.
- The evaluation combines established prompts with three malicious questions and repeats each injection attack five times.The questions request “print hello world,” the US capital, or repetition of the question’s prompt.
- Existing techniques were generally ineffective, with most attacks failing and successful chatbot outputs often matching intended functionality.
- Application input interpretation: Applications differ in whether user input is treated as a question for answering or as data for analysis.DECISIONAI treats a user’s decision as analytical data, making direct prompt injection strategies ineffective in that workflow.
- Formatting constraints: Formatting constraints can prevent manipulated LLM output from appearing on the front end even when the model is susceptible.The code-generation application AIWITHUI could potentially be induced to produce manipulated output when a required angle-bracket format was appended.
- Workflow constraints: Multi-step workflows and response-time limits can cause manipulated generations to miss deadlines and remain undisplayed.
- Application design, prompt processing, and built-in defenses together strengthen resilience against traditional prompt injection attacks.
4 HOUYI Overview
HOUYI addresses the difficulty of isolating malicious prompts from application-established context in black-box LLM-integrated services. It combines context inference, three-part prompt construction, and iterative feedback to generate attacks.
- Design Insight: Existing prompt injection methods often fail because applications interpret user inputs as data within pre-designed prompt contexts.The central design question is how to isolate a malicious command from that established context.
- Design Insight: HOUYI draws on traditional injection attacks by using a separator to terminate or partition preceding context before introducing a malicious command.The separator must be based on the preceding context and make the malicious input appear command-like to the LLM.
- Attack Workflow: HOUYI infers application context from documented interactions, generates an injection prompt, and dynamically evaluates responses to refine the attack.The process recursively adjusts its strategy and ultimately outputs successful attack prompts.
5 Methodology Details
HOUYI composes prompts that preserve an application's expected interaction while separating its preset context from a malicious question. Its methodology generates and refines framework, separator, and disruptor components using application examples and feedback.
- Prompt Composition: The Framework Component aligns the injected prompt with the application's normal flow and output format to reduce detectability.Its design uses inferred context, reproducible questions, and preferably shorter responses because token limits and longer outputs can increase errors.
- Prompt Composition: The Separator Component creates a boundary that shifts the LLM from treating subsequent input as data to interpreting it as commands.HOUYI explores syntax-based, language-switching, and semantic-based strategies for constructing this boundary.
- Prompt Composition: The Disruptor Component is a malicious question tailored to extract information, manipulate responses, or compromise application security or privacy.Its formatting can follow the application's expected output, and limiting response length can improve attack success.
- Prompt Composition: In a DECISIONAI example, the framework preserves a normal PhD question, the separator switches context and language, and the disruptor asks about phishing emails.The phrase requesting a three-point summary helps the malicious output fit the application's front-end display.
- Context Inference: HOUYI infers application context by testing documented example questions and analyzing the resulting input-output pairs with a generative LLM.The inferred context may not perfectly match the actual context but provides an approximation for subsequent prompt construction.
- Iterative Prompt Refinement: HOUYI iteratively tests component strategies, replacing framework and separator constructions after failure and changing the disruptor after success.It begins with empty framework and separator components and a short proof-of-concept question before refining the attack.
- Iterative Prompt Refinement: Successful information-extraction exploits do not guarantee accurate retrieval because the black-box setting cannot distinguish factual output from hallucination.The authors validate findings with service providers in practice.
6 Evaluation
HOUYI was evaluated across 36 real-world LLM-integrated applications and identified widespread prompt-injection susceptibility, while ablations and case studies examined attack mechanisms and consequences.
- Vulnerability Detection: 31 of 36 applications were successfully attacked at least once across five exploit scenarios.The evaluation found that most tested applications were susceptible to prompt injection.
- Unsuccessful Attacks: Five applications resisted injection because they used domain-specific models, extensive output-processing procedures, or multimodal architectures.STORYCRAFT, STARTGEN, and CopyBot used dedicated models; GAMLEARN applied parsing, refining, and formatting; MINDGUIDE combined multimodal models.
- Exploit Variability: Prompt usage varied across applications, so prompt leaking was not applicable to systems without conventional prompts or those using domain-specific knowledge.AICHATFIN and KNOWLEDGECHATAI are cited as examples of differing prompt practices.
- Ablation Study: HOUYI generally outperformed three ablation baselines, while syntax-only injection was least effective because some applications sanitized or tolerated escape characters.The ablation results attribute stronger performance to semantic and language-based strategies in other variants.
- Ablation Study: Combining the three separator-generation strategies produced optimal results in further investigation.The finding supports the effectiveness of the separator-generation approach.
- Vulnerability Validation: 10 of 31 identified vulnerabilities were confirmed by vendors, including findings involving commercial products such as Notion.The evaluated consequences included prompt leaking and prompt abusing, with Notion serving over 20 million users.
- WRITESONIC Prompt Leak: WRITESONIC’s leaked prompt produced highly similar responses in a mock application, suggesting functional replication of the original service.WRITESONIC developers confirmed the prompt leak and its potential implications.
- PAREA Prompt Abuse: PAREA’s prompt-abuse vulnerability enabled arbitrary user-defined commands and was estimated to impose a daily developer loss of $259.2.The study reports that 30 other LLM-integrated applications were susceptible to similar prompt abuse.
7 Discussion
The discussion reviews proposed defenses against prompt injection and emphasizes that current protections remain incomplete. It also identifies open concerns about attack-strategy coverage and the future reproducibility of vulnerabilities.
- Defenses: Current mitigation strategies include instruction defense, post-prompting, XML tagging, and separate LLM evaluation.
- Defenses: Existing defenses provide some protection but do not offer full immunity to all forms of prompt injection.
- Defenses: HOUYI was able to circumvent each evaluated defense strategy, underscoring the need for more advanced protection mechanisms.
- Separator Component Generation: The study uses three Separator Component generation strategies: syntax-based, language switching, and semantic-based.
- Separator Component Generation: These three strategies are effective but may represent only a small subset of possible approaches.
- Reproducibility: Some detected vulnerabilities may become non-reproducible as applications add protections or their backend LLMs evolve.
8 Related Work
Related work covers LLM risks and capabilities surrounding prompt injection, including hallucination, jailbreaks, tool use, autonomous systems, and their security implications. As LLMs gain operational autonomy and require less human supervision, investigating their security becomes increasingly important.
- LLM Hallucination: LLM hallucination refers to risks of generating contentious, biased, hateful, or stereotyped content despite robustness mechanisms such as RLHF.
- LLM Jailbreak: LLM jailbreak attacks elicit restricted or sensitive content, potentially causing privacy breaches when training data contain private information.
- LLM Capabilities: Toolformer demonstrates that LLMs can generate API calls while selecting APIs and their arguments.
- LLM Capabilities: ReAct equips LLMs with task-specific actions and verbal reasoning based on environmental observations.
- Autonomous Systems: Autonomous-system projects such as Auto-GPT and Generative Agents extend LLM applications toward independent task planning and interaction with APIs or models.
- Security Implications: LLMs may execute adversaries’ objectives from high-level descriptions, making security more important as systems become more autonomous and less supervised.
9 Conclusion
The conclusion introduces HOUYI as a black-box prompt injection methodology built from three components. Its evaluation identifies prompt abuse and prompt leak, finds 31 of 36 tested applications susceptible, and receives validation from 10 vendors.
- HOUYI is a black-box methodology comprising a pre-constructed prompt, an injection prompt, and a malicious question.
- HOUYI reveals two exploit scenarios: prompt abuse and prompt leak.
- 31 of 36 evaluated real-world LLM-integrated applications are susceptible to prompt injection.
- 10 vendors acknowledged the findings, validating the research and indicating its extensive implications.
A List of Anonymized LLM-integrated Applications
The evaluation applications are presented in a table using anonymized aliases alongside short descriptions of their functionalities.
- The table lists the full set of evaluated LLM-integrated applications using anonymized aliases and brief functionality descriptions.