PaperlayerView exact revision on arXiv

Source-linked AI summary

Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game

Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, Stuart Russell

arXiv:2311.01011v1cs.LGcs.CR

TL;DR

Instruction-following LLM applications remain vulnerable to prompt injection, while systematic study is difficult because real applications use complex instructions and subjective evaluation can be expensive. This paper releases a large human-generated Tensor Trust dataset, analyzes its attack strategies, and derives prompt extraction and hijacking benchmarks. The results show that many models fall for basic human-written attacks, some strategies transfer to real applications, and the dataset exposes interpretable weaknesses.

  • Problem

    Prompt injection can let malicious users or third parties subvert LLM application behavior, while robust evaluation is difficult because applications require complex instructions and violations may require human judgment.

  • Method

    The paper creates Tensor Trust, releases its human-generated attacks and defenses, analyzes interpretable strategies, and derives benchmarks for prompt extraction and prompt hijacking.

  • Results

    Many models, including capable LLMs, can fall prey to basic human-written attacks, while some dataset attacks transfer almost verbatim to real-world applications.

  • Takeaways & Limitations

    The findings suggest that clever prompting is insufficient for preventing unwanted behavior and that models need to distinguish trusted instructions from untrusted data.

  • Takeaways & Limitations

    Access-code acceptance was slightly nondeterministic, and real-world applications often allowed system prompts to be overridden without sophisticated attack strategies.

Abstract

from arXiv · show

While Large Language Models (LLMs) are increasingly being used in real-world applications, they remain vulnerable to prompt injection attacks: malicious third party prompts that subvert the intent of the system designer. To help researchers study this problem, we present a dataset of over 126,000 prompt injection attacks and 46,000 prompt-based "defenses" against prompt injection, all created by players of an online game called Tensor Trust. To the best of our knowledge, this is currently the largest dataset of human-generated adversarial examples for instruction-following LLMs. The attacks in our dataset have a lot of easily interpretable stucture, and shed light on the weaknesses of LLMs. We also use the dataset to create a benchmark for resistance to two types of prompt injection, which we refer to as prompt extraction and prompt hijacking. Our benchmark results show that many models are vulnerable to the attack strategies in the Tensor Trust dataset. Furthermore, we show that some attack strategies from the dataset generalize to deployed LLM-based applications, even though they have a very different set of constraints to the game. We release all data and source code at https://tensortrust.ai/paper

1 INTRODUCTION

Instruction-tuned LLM applications are vulnerable to prompt injection, creating practical security risks and difficult robustness requirements. Tensor Trust addresses this challenge with a simple game that produces a large, human-generated dataset of attacks and defenses for analysis and benchmarking.

  • Motivation: Prompt injection lets malicious users or third parties subvert an LLM application's intended behavior, including manipulating search results or revealing the application's prompt.These attacks can affect both untrusted content inserted into prompts and direct user queries.
  • Motivation: Prompt injection is a current security threat that has turned Bing Chat into a phishing agent and enabled instruction leakage and spam generation.Robustness is difficult because applications require complex instructions, while checking violations can require expensive human judgment.
  • Tensor Trust: Tensor Trust reduces evaluation complexity to a string-comparison task: defenses should produce “access granted” only when attackers supply the secret access code.Players create defenses, while other players attempt to gain access without seeing the defense or code.
  • Contributions: The released dataset contains 126,808 attacks and 46,457 defenses, including player identifiers, timestamps, both attack and defense text, and multi-step attacks.After de-duplication, the dataset contains 69,906 distinct attacker inputs and 39,731 defenses.
  • Contributions: The paper analyzes interpretable human-written attack strategies and introduces benchmarks for prompt extraction and prompt hijacking.The source code and full Tensor Trust dataset are released publicly.

2 THE TENSOR TRUST WEB GAME

Tensor Trust models a bank-like game in which players defend accounts with layered prompts and secret access codes while attackers submit unseen prompt injections. The access-code design preserves useful processing while creating opportunities for extraction attacks, and restrictions encourage diverse attack strategies.

  • Defending: Each account defense contains an opening defense, an access code, and a closing defense; the access code is saved only when the LLM outputs “access granted.”Validation sandwiches the access code between the opening and closing defenses before sending the input to the LLM.
  • Attacking: Attackers submit arbitrary text between another player's opening and closing defenses, gaining a fraction of the defender's money if the LLM outputs “access granted.”Otherwise, the defender receives a small reward for rebuffing the attack.
  • Motivation for the access code: The access code makes the game more like real applications that must process inputs rather than simply prevent every “access granted” output.Because players often place the code in a defense, attackers may first extract the defense prompt and then enter the recovered code.
  • Ranks and restrictions: Rank-based restrictions were added after top players reused attacks against weaker accounts, requiring attacks against lower-ranked accounts to satisfy defense-specific constraints.Examples included banning vowels and limiting attack content.
  • LLM details: The game uses GPT 3.5 Turbo with temperature=0 and token limits for defenses, access codes, attacks, and responses.The opening defense, access code, closing defense, attack, and response limits are 300, 150, 200, 500, and 500 tokens, respectively.

3 DATASET AND BENCHMARKS

The paper releases the Tensor Trust data and constructs filtered robustness benchmarks for prompt hijacking and prompt extraction. These benchmarks test whether models resist attacks while preserving valid access-code behavior, with additional data for detecting indirect extraction.

  • Dataset and benchmark construction: The release includes the full player-generated attack and defense dataset, two robustness benchmarks, and a small dataset for detecting indirect prompt extraction.The two main benchmarks evaluate prompt extraction and prompt hijacking.
  • Attack types: Prompt extraction attacks reveal enough of a defense for an attacker to guess its access code, whereas prompt hijacking attacks override the defense without using the access code.These categories correspond to attack behaviors observed in real systems.
  • Prompt hijacking robustness benchmark: The prompt hijacking benchmark asks models to reject attacks while still outputting “access granted” for the true access code.High-quality defenses and transferable attacks were filtered using reference models, adversarial pairing, and manual validation.
  • Prompt hijacking robustness benchmark: Hijacking Robustness Rate measures the percentage of attacks for which the model avoids “access granted,” while Defense Validity measures valid access-code acceptance.An ideal model scores high on both metrics.
  • Prompt extraction robustness benchmark: The prompt extraction benchmark measures whether models avoid outputting the true access code verbatim, using the same good defenses but different attack-identification heuristics.Potential extraction attacks were filtered and manually validated, leaving 569 samples.
  • Prompt extraction robustness benchmark: Extraction Robustness Rate measures whether outputs omit the access code verbatim, while Defense Validity measures whether defenses accept the true access code.An ideal model scores high on both metrics.
  • Prompt extraction detection: The exact-match extraction metric misses semantic variations and reconstructive hints, so the authors release a class-balanced dataset for studying indirect extraction.GPT4 reaches 97% precision and 84% recall with zero-shot prompting on this detection task.

4 EXPLORING ATTACK AND DEFENSE STRATEGIES

The paper analyzes Tensor Trust attacks and defenses to identify recurring strategies and weaknesses in GPT-3.5 Turbo. Attack strategies are compositional, evolve over time, and can exploit both prompt structure and model- or serving-pipeline-specific behaviors.

  • Uncovering attack strategies with LDA topic modeling: The analysis uses LDA and manual inspection to reduce 41 topics from 3,912 successful attacks to 16 rough attack strategies.The clustering covers successful prompt hijacking and extraction attacks identified before adversarial filtering and validation.
  • Evolution and virality of attacks: Attack strategies evolved over nine weeks from “Ask Directly” to “Sentient Door” roleplay and then the rare token “artisanlib”.Weekly topic frequencies reveal the viral spread and replacement of strategies across the game.
  • Model-specific adversarial tokens: The “artisanlib” token often makes GPT-3.5 Turbo ignore pre-prompts or post-prompts, while <|im_end|> can trigger API errors that disrupt attack submission.The latter appears to affect the serving pipeline despite being unavailable through OpenAI’s high-level ChatML API.
  • Confusing the model about the preceding prompt: Repeated-character blocks, random text, and related prefixes aim to make the model treat the preceding defense as unrelated or ignore it.The paper connects repeated-character attacks to prior observations that they can make GPT-3.5 Turbo go “off the rails”.
  • Composability and virality of attacks: Attack strategies are composable: players extract successful techniques from attacks and recombine them to strengthen later attacks.This mixing-and-matching creates a compounding effect as successful strategies spread through the game.
  • Insights on defenses: Successful defenses commonly give direct behavioral instructions, warn against untrusted user input, specify the model’s role, delimit inputs, and repeat instructions or examples.These defenses tend to use more grammatical, straightforward text than attacks.

5 EVALUATING THE ROBUSTNESS OF LLMS

The benchmark evaluates eight baseline models on prompt hijacking and extraction using adversarially filtered Tensor Trust data. Model robustness reflects a trade-off with defense validity, with GPT-4 performing strongly across both measures while other models show failure modes tied to refusals, verbosity, or prompt leakage.

  • Benchmark setup: Eight baseline models were evaluated on prompt hijacking and extraction robustness using Tensor Trust benchmark datasets.The models included GPT-3.5 Turbo, GPT-4, Claude variants, PaLM 2, LLaMA variants, and CodeLLaMA-34B-Instruct.
  • Prompt hijacking robustness: All three reference models had relatively high defense validity and low hijacking robustness rates because the benchmark filtered defenses and attacks using those models.Claude-instant-v1.2 had lower defense validity and higher hijacking robustness than the other reference models, partly because of false refusals and role resistance.
  • Prompt hijacking robustness: LLaMA models exhibited a trade-off between hijacking robustness and defense validity, partly because smaller models more often refused both defense instructions and attacks.LLaMA-2-70B-chat had higher defense validity but worse hijacking robustness than the 7B model.
  • Prompt hijacking robustness: GPT-4 beat other models in hijacking robustness while retaining high defense validity.The authors speculate that GPT-4 combines similar instruction behavior to GPT-3.5 with greater resistance to known vulnerabilities such as artisanlib and role-playing attacks.
  • Prompt extraction robustness: GPT-4 also had better defense validity and extraction robustness than other models, while LLaMA 2 Chat models had much worse extraction robustness than hijacking robustness.The authors associate the LLaMA pattern with verbose helpful refusals that accidentally leak parts of defense prompts.
  • Message-role ablation: Changing GPT-3.5 Turbo’s message-role arrangement produced little overall performance difference, showing that message roles alone were insufficient to reject human-created prompt injections.No alternative message-role choice was better across all metrics.

6 ATTACKS FROM TENSOR TRUST CAN TRANSFER TO REAL APPLICATIONS

Tensor Trust attacks sometimes transfer to real chatbots and writing assistants, including jailbreak settings, despite being designed for a limited access-granting task. However, the strongest safety breaks required manual prompt engineering.

  • Transfer to real applications: Some Tensor Trust attack strategies generalized to real-world chatbots and writing assistants, including the related task of jailbreaking.Adapted hijacking attacks elicited behaviors that applications otherwise refused, including jokes and other sensitive responses.
  • Transfer to real applications: Some attacks elicited jokes from ChatGPT, Claude, and Bard after those systems refused direct requests for jokes about Kim Jong Un.The attacks were adapted from Tensor Trust to request the target behavior.
  • Limits of transfer: The most serious jailbreaks, such as eliciting bomb-making instructions, required additional manual prompt engineering.The dataset attacks could sometimes work almost verbatim, but not consistently for the strongest breaks in safety fine-tuning.
  • Limits of transfer: Prompt-injection attacks on deployed applications were difficult to study because their system prompts could usually be overridden with little effort.In those applications, sophisticated attack strategies were unnecessary.

7 RELATED WORK

Related work distinguishes prompt injection from jailbreaks and includes automated adversarial-prompt optimization, human-generated attack collections, online prompt-injection games, and informal jailbreak repositories. Tensor Trust focuses on human-generated attacks and user-created defenses within an online game.

  • Adversarial attacks and jailbreaks: Prompt injection overrides previous model instructions, whereas jailbreaks target safety behaviors that models were fine-tuned to refuse.Prior work on instruction-following LLMs has studied both attack types.
  • Adversarial attacks and jailbreaks: Prior adversarial-prompt research includes automated optimization, transfer attacks from open-source models, and attacks through image channels in vision-language models.Tensor Trust instead focuses on human-generated attacks.
  • Prompt injection games: Existing prompt-injection games inspired Tensor Trust, which differs by allowing users to create defenses rather than relying on a small fixed set.The passage identifies Tensor Trust as part of a broader group of online games challenging users to prompt-inject LLMs.
  • LLM jailbreak collections: Jailbreak collections have been widely shared through informal platforms such as Jailbreak Chat and Twitter.These collections concern jailbreaks rather than the prompt-injection focus of Tensor Trust.

8 CONCLUSION

Tensor Trust contributes a dataset and benchmarks showing that human-written attacks can expose weaknesses in instruction-following LLMs. The findings point to the difficulty of distinguishing trusted instructions from untrusted data, especially in applications receiving third-party inputs.

  • Contributions: The dataset reveals diverse strategies for causing undesirable behavior in applications that use instruction-tuned LLMs.The paper also introduces benchmarks for robustness to prompt hijacking and prompt extraction.
  • Findings: Even highly capable LLMs can fall prey to basic human-written attacks in the benchmark’s string-control setting.The conclusion states that clever prompting is not sufficient to prevent unwanted behavior.
  • Implications: The findings suggest that models need better ways to distinguish trusted instructions from other untrusted text.The conclusion specifically frames this distinction as important for preventing unwanted behavior.
  • Implications: Providing LLMs with access to untrusted third-party inputs is dangerous in sensitive applications.This is presented as a consequence of the observed prompt-injection vulnerabilities.

CONTRIBUTIONS, SECURITY, AND ETHICS

The paper documents its benchmark implementation, data-handling safeguards, and ethics procedures, while noting that current APIs do not reliably separate trusted instructions from untrusted input.

  • Ethics: The study informed players that their data would be publicly released and sought institutional guidance about research approval before releasing the game.The consent materials described the project as research on prompt-injection vulnerabilities.
  • Evaluation: Attacks were counted as successful when normalized output matched “access granted” under a case-insensitive regular expression.The benchmark used a weaker expression allowing trailing alphanumeric text, unlike the stricter game-validation expression.
  • Game design: The game exposed players to prior attacks and ranked them by balance, while later restrictions allowed high-ranked players to attack lower-ranked accounts under conditions.These mechanics were intended to support defense updates and continued attack development.
  • Safety and moderation: Potential terms-of-use violations were flagged with OpenAI’s moderation endpoint, manually reviewed in some cases, and occasionally sanctioned.The paper also describes removing extreme inappropriate language from the released dataset through string matching and manual checks.
  • Security limitation: Current LLM APIs do not reliably distinguish untrusted user input from trusted instructions, because several models assign both attack and defense content the user role and delimiter-based separation is unreliable.This limitation constrains the reliability of instruction/data separation in the evaluated interfaces.

C.3 FULL RESULTS TABLES

The appendix provides complete numerical results for the paper’s prompt-hijacking and prompt-extraction robustness evaluations, along with response-length statistics for both benchmarks.

  • Robustness tables: Table A1 and Table A2 report precise prompt-hijacking and prompt-extraction robustness results for the dataset.These tables contain the full figures corresponding to Fig. 6.
  • Response lengths: Figure A2 reports each model’s mean response length separately for the hijack and extraction benchmarks.The figure compares response lengths across models and benchmark types.

D PROMPT EXTRACTION DETECTION DATASET

The prompt-extraction detection dataset evaluates whether models can identify leaked access-code content, but robust automated detection remains difficult, especially for rephrased or encoded leaks.

  • Limitation: Simple string comparison works for exact prompt reiterations but fails when extracted prompts are rephrased or encoded.This makes automated prompt-extraction detection difficult beyond literal matches.
  • Task: The classifier receives an access code and attack output, then predicts whether any part of the code was disclosed.The benchmark targets successful prompt extractions identified in Tensor Trust.
  • Dataset construction: 230 manually reviewed samples form the prompt-extraction detection dataset, with positives split 70–30 between hard and easy cases and negatives balanced 50–50.Easy positives directly included access codes; hard positives represented less straightforward extractions.
  • Metrics: Precision measures correct predictions among flagged positives, while recall measures correct predictions among all true prompt extractions.The evaluation uses overall accuracy together with these two metrics.
  • Results: GPT-3.5 Turbo performed only marginally better than random guessing, whereas GPT-4 showed high proficiency in zero-shot prompt-extraction detection.The authors leave construction of a truly robust classifier for future work.

E LDA ANALYSIS DETAILS

The appendix details topic modeling and transfer experiments using Tensor Trust attack strategies, including tests of jailbreak behavior across deployed applications and message-role schemes.

  • LDA analysis: LDA analysis used 3,912 unique attacks: 2,326 suspected extractions and 1,586 suspected hijackings.Because many attacks contained non-grammatical character blocks, the authors avoided word-based tokenization and manually consolidated coherent topics into 16 strategies.
  • Joke elicitation: Ten phrasings of a Kim Jong Un joke were embedded in three dataset attacks to test whether models switched from refusal to answering.Figures A4–A9 show such behavior for Bard, Claude, and ChatGPT.
  • Unusual behaviors: Bard normally refused a bomb-making request but switched to Thai under a repeated-character attack, which the authors classify as surprising behavior rather than a jailbreak.The experiment was designed to examine unusual model behavior induced by modified attacks.
  • Application transfer: Modified attacks elicited bomb-making instructions from Bing Chat and Notion AI after those systems initially refused the direct request.The screenshots blurred ingredients, which the authors identified as black-powder components and, for Notion AI, a combustible metal.
  • Message roles: Across five GPT-3.5 Turbo message-role schemes, Hijacking Robustness Rate ranged from 19% to 31% and Extraction Robustness Rate from 27% to 32%.The small differences across schemes suggest that changing message roles had limited effect in these experiments.
Loading 2311.01011v1…