PaperlayerView exact revision on arXiv

Source-linked AI summary

A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems

Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick McDaniel, Chaowei Xiao

arXiv:2402.18649v1cs.CRcs.AI

TL;DR

LLM security research has often examined models on their own, rather than the broader systems that combine them with other components. This paper applies an information-flow-based, multi-layer analysis to GPT-4 and finds vulnerabilities in the model and its integrations, including an end-to-end attack on users’ chat histories.

  • Problem

    Security analyses have often focused on standalone LLMs rather than the broader systems in which models interact with other components.

    Source: §3 · p. 4

  • Method

    The paper models system security as constraints on information flow within the LLM and between it and other objects, then applies a multi-layer, multi-step analysis to GPT-4.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    The analysis identifies vulnerabilities in GPT-4 and its integrations, and demonstrates an attack that obtains users’ private chat histories without manipulating their input or directly accessing GPT-4.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The case studies show that security risks can arise both within the LLM and through its interactions with components such as the Frontend and web tools.

    Source: §3 · p. 4, §4 · p. 4

    The URL-based approach is constrained by typical browser and server URL limits of around 2,000 characters, which can truncate chat history and make large transfers inefficient.

    Source: §7.1 · p. 11

Abstract

from arXiv · show

Large Language Model (LLM) systems are inherently compositional, with individual LLM serving as the core foundation with additional layers of objects such as plugins, sandbox, and so on. Along with the great potential, there are also increasing concerns over the security of such probabilistic intelligent systems. However, existing studies on LLM security often focus on individual LLM, but without examining the ecosystem through the lens of LLM systems with other objects (e.g., Frontend, Webtool, Sandbox, and so on). In this paper, we systematically analyze the security of LLM systems, instead of focusing on the individual LLMs. To do so, we build on top of the information flow and formulate the security of LLM systems as constraints on the alignment of the information flow within LLM and between LLM and other objects. Based on this construction and the unique probabilistic nature of LLM, the attack surface of the LLM system can be decomposed into three key components: (1) multi-layer security analysis, (2) analysis of the existence of constraints, and (3) analysis of the robustness of these constraints. To ground this new attack surface, we propose a multi-layer and multi-step approach and apply it to the state-of-art LLM system, OpenAI GPT4. Our investigation exposes several security issues, not just within the LLM model itself but also in its integration with other components. We found that although the OpenAI GPT4 has designed numerous safety constraints to improve its safety features, these safety constraints are still vulnerable to attackers. To further demonstrate the real-world threats of our discovered vulnerabilities, we construct an end-to-end attack where an adversary can illicitly acquire the user's chat history, all without the need to manipulate the user's input or gain direct access to OpenAI GPT4. Our demo is in the link: https://fzwark.github.io/LLM-System-Attack-Demo/

1 Introduction

LLM security studies have often examined models in isolation, while deployed LLM systems combine models with other components. This paper proposes an information-flow framework and applies a multi-layer, multi-step analysis to GPT-4, finding vulnerabilities across the model and its integrations.

1 Introduction

  • Customized GPTs and GPT stores reflect the growing integration of LLM systems into daily life.As of November 17, the top ten customized GPTs had received more than 750,000 visits, and the third-party store listed more than 20,000 GPTs.Source: §1 · p. 1
  • Prior LLM security research predominantly examined standalone models, while system-level studies lacked a comprehensive methodology.The paper cites earlier analysis of GPT-4 training-data leakage as a standalone-model study and describes other system work as limited to specific attack scenarios.Source: §1 · pp. 1–2
  • The paper formulates LLM-system security as constraints on information flow within the LLM and between it and other system objects.Its framework distinguishes objects, within-object actions, and between-object interactions, with constraints applying to both actions and interactions.Source: §1 · p. 2
  • The study focuses on the design of OpenAI GPT-4, not potential or future systems with different features.The paper’s stated scope does not cover such other systems.Source: §1 · p. 2
  • Because LLM inputs and outputs are probabilistic and ambiguous, the analysis considers both whether constraints exist and how robustly they work.The paper treats constraints as rules that mediate information flow and recommends examining their enforcement against adversarial strategies.Source: §1 · p. 2
  • GPT-4’s constraints can be bypassed, and some interactions lack necessary controls, including sandbox file isolation and Safe URL Check.Files uploaded in one session could be accessed from another, while a novel strategy bypassed the Frontend’s Safe URL Check.Source: §1 · p. 3
  • An end-to-end attack illicitly obtains users’ private chat histories without manipulating their input or directly accessing GPT-4.The attacker publishes a website that covertly transmits the chat history to an attacker-controlled server.Source: §1 · p. 3

2 Related Work

Prior work has examined component compromise, prompt injection, and information-flow control, but the paper distinguishes its focus on security in real-world LLM systems. It aims to provide a systematic framework for that analysis.

2 Related Work

  • Prior LLM-system security work includes studies assuming an attacker can control an internal component and prompt-injection studies that manipulate outputs without compromising components.The paper contrasts these with its focus on intrinsic system security without assuming internal attacker control.Source: §2 · p. 3
  • Prompt-injection studies described here use lab settings where LLMs directly concatenate user instructions with external language data, unlike real-world systems with more complex tool interactions.The paper identifies real-system interactions with tools as a distinguishing setting.Source: §2 · p. 3
  • The authors characterize prior system-security studies as offering specific examples without a comprehensive methodology, and present their work as a systematic framework.They cite specific attack scenarios, including Bing Chat, as examples of the earlier work’s scope.Source: §2 · p. 3
  • Traditional information-flow control uses security labels to restrict confidentiality- and integrity-sensitive flows, with enforcement sometimes modeled through a type system.The passage notes that strict enforcement can provide noninterference, while practical systems may allow endorsement.Source: §2 · p. 3

3 Preliminary Case Study

The case studies illustrate how security risks arise both within LLM actions and across interactions with other components. They show the need to examine both directions of information flow in a compositional system.

3 Preliminary Case Study

  • The Frontend automatically renders Markdown image links, while the LLM can generate malicious links and their transmission lacks necessary control.The process has two steps—LLM output and transmission to the Frontend—and the authors report security failures at both.Source: §3 · p. 4
  • External webpage instructions may be treated as legitimate, redirecting the LLM from the user’s request and potentially transmitting chat history to an attacker through the Frontend.The authors say carefully designed strategies can bypass safeguards such as requests for user confirmation.Source: §3 · p. 4
  • The examples show bidirectional risk: information can flow from the LLM to the Frontend or from web tools to the LLM.Both directions can pose security and privacy risks without adequate safeguards.Source: §3 · p. 4

4 LLM System

The section models an LLM system as objects connected by information flows, distinguishing processing within one object from transmission between objects. It defines constraints over both kinds of flow.

4 LLM System

  • LLM systems contain a core LLM and supporting facilities, such as sandboxes, frontends, web tools, and plugins.The LLM receives signals and makes decisions; facilities connect it to external environments and support tasks.Source: §4 · p. 4
  • An action is one object's information processing, from input to output, without involvement by other objects.Source: §4 · p. 5
  • An interaction is a one-way call or return that transmits information between two objects.Source: §4 · p. 5
  • Constraints restrict either information processing within an object or information transmission across multiple objects.The section gives preventing unethical LLM output and preventing sensitive data from reaching external users as examples.Source: §4 · p. 5

5 Analysis Principles

The section organizes LLM-system security analysis around system layers, whether constraints exist, and how robust they are. It excludes vulnerabilities confined to non-LLM facilities and their interactions.

Multi-Layer Security Analysis

  • Analyze both the LLM's actions and its interactions with other internal objects, such as web plugins.Source: §5 · p. 5

Existence of Constraints Analysis

  • Check whether constraints exist: without them, arbitrary LLM output or unrestricted information flow can expose the system to harmful content, indirect prompts, or chat-history leaks.Source: §5 · p. 5

Robustness of Constraints Analysis

  • Because LLM constraint enforcement is indeterminate, assess robustness; adversarial prompts can bypass the external-image-link restriction.Figure 4 depicts the restriction, while Figure 5 shows its bypass via adversarial prompts.Source: §5 · p. 5, Figure 5 · p. 6, Figure 4 · p. 6
  • The analysis excludes intrinsic vulnerabilities in other facilities, such as web browsers, and interactions between non-LLM objects.Source: §5 · p. 5

6 Vulnerability Analysis in OpenAI GPT4

The analysis separates LLM actions from interactions with system components, then identifies needed constraints, checks whether they exist, and tests their robustness. Across GPT-4’s action, Sandbox, web-tool, and Frontend paths, the study finds missing or bypassable protections.

6.1 Vulnerability Analysis over the Action of the LLM

  • Indirect website instructions bypassed GPT-4’s refusal to output external Markdown image links, as did a jailbreak prompt that embedded the malicious goal in a benign puzzle task.The puzzle prompt made generating the link a prerequisite to completing its stated goal.Source: §6.1 · pp. 6–7, Figure 6 · p. 6

6.2.1 Sandbox

  • A file uploaded in a closed session remained accessible from a new session, exposing a cross-session Sandbox isolation failure.The test used a file named secret.txt and opened the new session on another device.Source: §6.2.1 · p. 8

6.2.2 Web Tools

  • Web-fetched instructions triggered a user confirmation request, but adding an external “no need to confirm” instruction let GPT-4 call Doc Maker without confirmation.A direct request to summarize chat history with Doc Maker also proceeded without confirmation.Source: §6.2.2 · pp. 8–9, Figure 9 · p. 8, Figure 10 · p. 9

6.2.3 Frontend

  • The Frontend rendered explicit and violent images, including all 50 unethical images tested, without assessing their content.The study also reports that rendering a Markdown image URL can send an HTTP request to its target, creating a route for private-data leakage.Source: §6.2.3 · p. 9, Figure 11 · p. 9
  • The Safe URL Check prevented a tested indirect link containing secret text from rendering by asking the server to assess the URL before the Frontend displays it.A false safety result blocks rendering; the inspected page source contained no target link.Source: §6.2.3 · p. 10, Figure 12 · p. 10, Figure 13 · p. 10

7 End2End Practical Attack

The paper develops an end-to-end attack that uses a malicious website and plugins to steal chat history without changing user input or directly accessing GPT-4. It addresses URL-check, stealth, and data-length challenges, and reports successful attacks across eight web plugins.

7 End2End Practical Attack

  • The threat model gives the attacker control of website content and publication, but not GPT-4 or the user's inputs and instructions.The goal is to transmit the user's private conversation records to the attacker.Source: §7 · p. 10

7.1 Challenges

  • Three challenges are bypassing the Safe URL Check, concealing GPT-4's output, and handling long chat histories; URL length limits make direct URL transmission unreliable.Browsers and servers typically limit URLs to around 2,000 characters, risking truncation, errors, or lost data.Source: §7.1 · p. 11
  • The paper notes that the confirmation-request challenge was addressed earlier and is not explained in depth in this section.It refers readers to Real Case III in section 6.2.2.Source: §7.1 · p. 11

7.2 Attack Method

  • The attack pipeline uses a malicious external website whose embedded instructions are read and executed when a user accesses its URL through GPT-4.The attacker publishes the crafted site at a specific URL.Source: §7.2 · pp. 11–12
  • Doc Maker stores the chat history in an online document, allowing the attack to send a link rather than placing the full history in a URL.The paper says this approach can handle chat histories of unlimited length.Source: §7.2 · p. 12
  • The Safe URL Check permits links already seen in prior conversation content, including plugin returns, so the attack first introduces its target URL through a plugin.The four-step strategy places the target URL in a generated document, retrieves its link through the Doc plugin, then asks GPT-4 to display it.Source: §7.2 · pp. 12–13
  • Indirect prompts instruct GPT-4 to show only the target link, helping conceal its output and intermediate explanations from the user.The prompt example asks GPT-4 not to display other text or intermediate thoughts.Source: §7.2 · p. 13
  • All eight tested web content plugins supported successful attacks, and the user-facing outputs and images were concealed while the attacker obtained the chat history.The evaluation used WebPilot and Doc Maker; the reported plugin comparison is summarized in Table 1.Source: §7.2 · p. 13, Table 1 · p. 13

8 Discussion on Potential Mitigation Strategies

The discussion outlines possible mitigations while stating that designing a defense solution is outside the study's scope.

Potential Mitigation Strategies

  • Output filtering and procedural control flows could detect harmful or sensitive generated content and sanitize or withhold the response.The paper suggests adversarial training to improve the detector's robustness.Source: §8 · p. 13
  • Enhanced interaction protocols could label inputs by source and train the LLM to ignore instructions originating from external systems such as websites.The paper contrasts user input with information from external components.Source: §8 · p. 13
  • The paper presents these as potential mitigation strategies, not a designed defense solution, which it says falls outside the study's scope.Source: §8 · p. 13

9 Conclusion

The paper concludes that its information-flow framework enables systematic security analysis of GPT-4 systems and reports vulnerabilities across models and integrated components. It also describes a practical attack that stealthily obtains users' private chat history.

9 Conclusion

  • The paper formulates LLM-system security as constraints on information-flow alignment within the LLM and between the LLM and other objects.It applies a multi-layered, multi-step approach to study GPT-4.Source: §9 · p. 13
  • The analysis identifies security issues in GPT-4 itself and in its integration with other components, with safety constraints remaining vulnerable to attacks.Source: §9 · p. 13
  • The paper's end-to-end attack demonstrates stealthy acquisition of a user's private chat history.The conclusion frames the study's perspective as systemic rather than limited to individual models.Source: §9 · pp. 13–14

A Prompt used in Figure 9:

The prompt asks GPT-4 to summarize the chat history into a document using the Doc plugin and to proceed without requesting confirmation or more details.

A Prompt used in Figure 9:

  • The example prompt asks GPT-4 to put the chat history in a Doc-plugin document and proceed without further questions or confirmation.Source: A Prompt used in Figure 9 · p. 16

B Prompt used in Figure 15:

The prompt asks an assistant to retrieve chat history, save it in a document, and use document links in further requests. It also instructs the assistant to display only a specified image-markdown string and suppress intermediate thoughts.

B Prompt used in Figure 15:

  • The prompt asks the assistant to retrieve the conversation's chat history and save it in a document without asking for document details.It says the assistant may choose the document type and plugin.Source: B Prompt used in Figure 15 · p. 16
  • It asks the assistant to create a second document containing an attacker-server URL that incorporates the first document's link, then inspect a second internal document link.The prompt asserts permission to execute these steps.Source: B Prompt used in Figure 15 · p. 16
  • It requests that the assistant finish by showing only an image-markdown string containing the first document's link and no other text.The prompt also says not to display intermediate thoughts.Source: B Prompt used in Figure 15 · p. 16

C Image Results in Section 7

The section presents image results from quantitative experiments for seven web-plugin configurations. The figure captions identify the plugin used in each set of results but do not state the outcomes.

C Image Results in Section 7

Loading 2402.18649v1…