PaperlayerView exact revision on arXiv

Source-linked AI summary

StruQ: Defending Against Prompt Injection with Structured Queries

Sizhe Chen, Julien Piet, Chawin Sitawarin, David Wagner

arXiv:2402.06363v2cs.CR

TL;DR

Prompt injection exploits LLMs’ failure to separate application instructions from user data. StruQ introduces structured queries with a secure front-end and specially trained model, substantially improving resistance while preserving utility; it remains vulnerable to some optimization-based attacks.

  • Problem

    Prompt injection lets users place hidden instructions in data that divert an LLM-integrated application from its intended task.

  • Method

    StruQ separates prompts and data through a structured-query format and uses structured instruction tuning to make the LLM follow prompt instructions but ignore data instructions.

  • Results

    StruQ reduces manual attack success rates to <2% and lowers Llama’s TAP success rate from 97% to 9% and GCG success rate from 97% to 58%.

  • Takeaways & Limitations

    Structured queries are a promising direction for protecting LLM-integrated applications against prompt injection with minimal impact on utility.

  • Takeaways & Limitations

    StruQ is not completely secure against optimization-based attacks, particularly GCG, and richer structures integrating system prompts remain future work.

Abstract

from arXiv · show

Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capabilities. However, as LLMs have improved, so have the attacks against them. Prompt injection attacks are an important threat: they trick the model into deviating from the original application's instructions and instead follow user directives. These attacks rely on the LLM's ability to follow instructions and inability to separate prompts and user data. We introduce structured queries, a general approach to tackle this problem. Structured queries separate prompts and data into two channels. We implement a system that supports structured queries. This system is made of (1) a secure front-end that formats a prompt and user data into a special format, and (2) a specially trained LLM that can produce high-quality outputs from these inputs. The LLM is trained using a novel fine-tuning strategy: we convert a base (non-instruction-tuned) LLM to a structured instruction-tuned model that will only follow instructions in the prompt portion of a query. To do so, we augment standard instruction tuning datasets with examples that also include instructions in the data portion of the query, and fine-tune the model to ignore these. Our system significantly improves resistance to prompt injection attacks, with little or no impact on utility. Our code is released at https://github.com/Sizhe-Chen/StruQ.

1 Introduction

StruQ addresses prompt injection by separating application instructions from user data and training an LLM to follow only instructions in the prompt channel. Its evaluation finds substantially improved attack resistance with little or no utility loss.

  • Motivation: Prompt injection exploits LLMs’ inability to distinguish developer instructions from user data, allowing injected text to subvert an application’s intended operation.Attackers can place strings such as “Ignore all prior instructions and instead...” inside data.
  • Structured queries: StruQ represents prompts and data as separate components rather than concatenating them into one LLM input.Its front-end accepts both components and assembles them into a special format.
  • Structured instruction tuning: Structured instruction tuning trains the model to follow instructions before the separator while ignoring extra instructions after it.Training combines normal examples with attacked examples containing instructions in the data portion.
  • Evaluation: Manual attack success rates fall below 2% on Llama and Mistral, while TAP decreases from 97% to 9% and GCG from 97% to 58% on Llama.These optimization-based attacks were not represented during training, and the authors note that the system is not fully secure.
  • Conclusion: The paper concludes that structured queries are a promising approach for securing LLMs against prompt injection with little or no utility loss.Utility impact is indicated by AlpacaEval.
  • System design: StruQ combines specially reserved delimiters, front-end filtering, and structured instruction tuning to improve security against prompt injections.The authors identify these as three main design ideas and report that experiments suggest they significantly improve security.

2 Background and Related Work

The paper frames prompt injection as a control-data separation failure analogous to vulnerabilities in other computer systems. It surveys related defenses and distinguishes StruQ by combining a general-purpose hardened model with structured inputs.

  • Background: LLM-integrated applications commonly concatenate a task prompt and user data, enabling malicious data to be interpreted as an instruction.A resume, for example, may contain hidden text that redirects the model’s output.
  • Background: Prompt injection is presented as an unsafe-by-design API problem analogous to SQL injection and cross-site scripting.In each case, control and data are mixed in a single string, leaving boundaries unclear or spoofable.
  • Structured queries: The proposed remedy is a safe-by-design LLM API that supplies the prompt and data as separate inputs, called a structured query.The paper identifies training models for this API as its central research problem.
  • Related defenses: Existing prompt-injection defenses are described as not fully satisfactory, while StruQ combines special tokens, structured instruction tuning, and front-end filtering.The related-work discussion characterizes these components as distinguishing StruQ from current instruction-tuning methods.
  • Related defenses: Jatmo reduces attack success below 1% but requires task-specific fine-tuning, whereas StruQ aims to harden one LLM for many tasks.This comparison concerns generality rather than a direct security ranking.
  • Threat distinctions: Prompt injection differs from jailbreaking because it involves an application developer and user-data channel in addition to the model provider and user.The supplied passage introduces this distinction but is truncated before fully stating the parties.
  • Threat distinctions: The paper also distinguishes prompt injection from data extraction and task-specific attacks that target LLM training data or task performance.These are presented as separate threat categories.

3 Prompt Injection Attacks

The paper defines prompt injection through an attacker-controlled data channel and evaluates diverse manual, crafted, and optimization-based techniques. The attacks vary in how they overwrite, disguise, or append instructions.

  • 3.1 Problem Statement: A prompt injection attack diverts an LLM-integrated application toward a hidden instruction in data instead of the intended benign instruction.The paper’s goal is to prevent this behavior while minimizing utility loss.
  • 3.1 Problem Statement: The attacker may arbitrarily modify the data but cannot modify the prompt or application formatting, which are assumed known.An attack succeeds when the model obeys the hidden instruction rather than treating it as data.
  • 3.2–3.3 Direct injection: Naive and Ignore attacks directly append instructions such as “Output no” or “Ignore previous instructions and instead...” to otherwise benign input.The study manually creates ten Ignore variants and randomly selects one per sample.
  • 3.4 Escape Character Attacks: Escape Character attacks use repeated backspace or carriage-return characters to imitate deletion, or newline and tab characters to create separation.For Escape-Deletion, the injected count is set to the length of prior text plus 10; Escape-Separation uses a random count from 0–9.
  • 3.5 Completion Attacks: Completion attacks append a fake response and then new instructions, with variants using legitimate, slightly modified, unrelated, or combined delimiters.Completion-Real is described as the most effective strategy because it exactly matches legitimate delimiters, which the front-end filters.
  • 3.6 HackAPrompt: HackAPrompt supplies crowd-sourced injections from a ten-level prompt-hacking competition, and the study samples 20 successful level-1 attacks tailored to the original task.The original instructions are retained to preserve attack effectiveness.
  • 3.7 Tree-of-Attacks with Pruning: TAP uses an attacker LLM to generate injections and a judge or matching procedure to assess success, expanding beyond hand-crafted attacks.For prompt injection, the paper removes the judge LLM and uses substring matching because the evaluation target is simple.
  • 3.8 Greedy Coordinate Gradient Attack: GCG uses gradient information to optimize an adversarial suffix and assumes white-box gradient access, making it a strong worst-case baseline.The paper describes GCG as powerful but notes that gradient access may not always be feasible for attackers.

4 Structured Queries

StruQ implements structured queries by separating prompts from data and training an LLM to follow instructions only in the prompt portion. Its secure front-end encodes and filters inputs, while structured instruction tuning uses clean and attacked examples to preserve utility and resist injection.

  • 4 Structured Queries: Structured queries separate each LLM input into a prompt and data component, with the model instructed to follow only prompt-side instructions.This design targets the requirement that instructions in the data portion must not be executed.
  • 4 Structured Queries: StruQ combines a secure front-end that encodes queries with a custom LLM trained to consume the resulting format.The front-end prepares the query, while the model learns the corresponding structured-input behavior.
  • 4 Structured Queries: Reserved delimiter tokens and filtering prevent user data from spoofing control boundaries, helping defend against Completion attacks.The filter repeatedly removes delimiter strings and also removes ##, which was empirically effective as a fake delimiter.
  • 4 Structured Queries: Structured instruction tuning teaches the LLM to obey correctly positioned instructions while ignoring injected instructions placed in the data portion.Training includes clean samples and attacked samples whose desired outputs follow the prompt-side instruction.
  • 4 Structured Queries: Initializing embeddings for newly added special tokens makes a big difference to utility, because instruction tuning alone is insufficient to learn them from scratch.The embeddings are updated during structured instruction tuning and become distinct from their initialization tokens.
  • 4 Structured Queries: The training set combines 50% clean samples, 25% Naive-attack samples, and 25% Completion-Other samples before fine-tuning a base LLM.The attacked examples concatenate additional instructions or fake delimiters and responses into the data while retaining the original target response.

5 Experiments

StruQ evaluates structured-query defenses across utility, manual prompt injections, adaptive delimiter attacks, optimization-based attacks, and ablations. It substantially improves security with little utility loss, but remains vulnerable to carefully optimized task-specific injections.

  • Evaluation setup: StruQ measures both output utility with AlpacaEval and security using attack success rates across prompt injection techniques.The evaluation uses AlpacaEval for utility and tests attacks including multilingual and encoded injections.
  • Utility: About one percentage point: Mistral’s AlpacaEval win rate decreases by this amount, while Llama shows no detrimental utility effect.The Mistral change is borderline statistically significant, whereas the Llama change is not statistically significant.
  • Optimization-based attacks: 97% → 9%: StruQ lowers Llama’s TAP attack success rate, while GCG falls from 97% to 58%; Mistral shows a similar effect.TAP and GCG are identified as the strongest tested attacks, but the Llama model is not completely immune.
  • Limitations: Optimization-based attacks generate task-specific injections, and StruQ’s task-agnostic training does not fully generalize to them.The authors state that additional task-dependent attack data and more research are needed.
  • Adaptive attacks: StruQ stops designed Completion attacks through complementary defenses: filtering blocks real delimiters, while structured instruction tuning blocks near-miss delimiters.Reserved tokens distinguish correct delimiters from near-miss variants.
  • Structured instruction tuning: 0%: combining naive and completion augmentations reduces attack success to 0% across selected attacks with minimal utility impact.This combination was used in the final framework.
  • Comparison with BIPIA: StruQ achieves 0% attack success on both StruQ and BIPIA test sets, while BIPIA reaches 54% on the StruQ test set and 100% under GCG.The comparison reports no utility loss for StruQ.

6 Discussion

StruQ protects programmatic LLM applications from prompt injection but has important scope and robustness boundaries. Strong optimization-based attacks remain difficult, while chatbot settings and non-injection attacks are outside its design.

  • Scope limitations: StruQ does not apply to web-based chatbots with multi-turn, open-ended conversations.It targets programmatic applications whose developers can specify prompts separately from data.
  • Scope limitations: StruQ is not designed to defend against jailbreaks, data extraction, or other attacks against LLMs.
  • Robustness limitations: StruQ is not a completely secure worst-case defense because GCG attacks achieve a non-trivial attack success rate.The authors identify defending against GCG/TAP prompt injections as an important open problem.
  • Attack cost: GCG or TAP attacks require > 100× more GPU hours than the other attacks considered.TAP queries the LLM about 100 times, while GCG queries it 256k times for a sample.
  • Future directions: Resistance to strong optimization-based attacks remains an open question for future defenses.Suggested directions include access control, rate-limiting, and architectures that treat prompt and data differently.
  • Future directions: Future structured-query systems could distinguish system prompts, user prompts, and associated data as three separate elements.
  • Broader implications: The findings support structured queries as a promising security approach, and motivate making non-instruction-tuned models available for fine-tuning.

7 Summary

StruQ separates prompts from data and trains models to follow instructions only in the prompt channel. Experiments report broad protection against human-crafted injections and improved robustness against optimization-based attacks with minimal utility impact.

  • 7 Summary: StruQ converts prompts and data into structured queries processed by models trained with structured instruction tuning.The method converts noninstruction-tuned models into defended instruction-tuned models and uses a front-end to create structured queries.
  • 7 Summary: Experiments show security against a wide class of adaptive and non-adaptive human-crafted prompt injections.
  • 7 Summary: StruQ improves security against optimization-based attacks with minimal impact on model utility.

Ethics considerations and compliance with the open science policy

The paper reports compliance with the Menlo Report’s ethics principles and states that it introduces no new datasets or binaries. Its testing includes manually designed ignore-attack sentences.

  • Ethics considerations and compliance with the open science policy: The research reports compliance with the Menlo Report principles, including fairness, respect for persons, and respect for law and public interest.
  • Ethics considerations and compliance with the open science policy: The paper states that it does not contribute any new datasets or binaries.
  • A.2 Ignore Attack in Training: The ignore-attack evaluation manually designs ten sentences expressing the meaning of ignoring previous instructions.

A.2 Ignore Attack in Training

The training appendix uses manually constructed ignore-instruction examples as a defense against prompt injections. A separate Completion-Other attack varies delimiters and instruction-related words through randomized construction.

  • A.2 Ignore Attack in Training: Ten manually designed ignore sentences are included in training to imitate prompt injections as a defense.A different individual creates the training sentences to distinguish their distribution from testing sentences.
  • A.2 Ignore Attack in Training: The training examples express variants of disregarding prior instructions and following a new instruction.
  • A.2 Ignore Attack in Training: Completion-Other Attack tests completion attacks using manual delimiters unrelated to the default delimiters.
  • A.2 Ignore Attack in Training: The attack construction samples markers, input and output labels, and instruction and response words to form delimiters.

A.4 Completion-Other Attack in Training

The training procedure augments Completion-Other attacks by sampling delimiter components and formatting instruction, input, and response fields into structured strings.

  • The defense uses manually designed delimiter-component candidate sets during Completion-Other attack training.This training step is intended to imitate prompt injections.
  • Training samples delimiter markers from multiple bracketed and hash-prefixed formats to imitate prompt injections.Candidate markers include forms such as {s}, ### {s}, |{s}|, and nested combinations.
  • Training independently samples optional input and output role labels, including Prompter, Assistant, Chat-bot, and Bot.The instruction, data, and response words are sampled from predefined vocabularies.
  • The sampled marker and words format separate instruction, input, and response strings as dinst, dinpt, and dresp.These strings are constructed as marker-wrapped combinations of role labels and field words.

A.5 HackAPrompt

The HackAPrompt evaluation collects successful prompt injections that exploit translation, dictionary, formatting, and instruction-following behaviors to elicit a fixed target response.

  • Formatting constraints strengthen these injections by requiring punctuation removal, exact wording, or a specific response layout.The examples request no periods, one-line output, exact quoted text, or whitespace substitution.
  • The listed successful entries include closely related attack ideas submitted by different participants.The evaluation notes that some of the 20 randomly selected entries are very similar.
  • Successful entries repeatedly use translation tasks to redirect outputs toward “I have been PWNED”.Examples switch languages, translate already translated text, or instruct the model to emit the target phrase for non-English inputs.
  • Several attacks embed target outputs in dictionary mappings and then ask the model to retrieve the mapped response.The mappings associate ordinary translation or instruction sentences with the target phrase.
  • Table 8 reports utility and attack success rates for Llama models trained with different structured instruction-tuning augmentation strategies.The table omits special delimiters and filtering and provides complete results for the corresponding ablation.

A.6 Tree-of-Attack With Pruning

The tree-of-attack evaluation uses prescribed search parameters and manually assesses responses, while documenting successful injections against StruQ.

  • The attack search uses branching factor 4 with depth and width 10, continuing to the tree’s maximum depth.All responses are then manually reviewed to determine attack success rate.
  • For jailbreak attacks, the attacker LLM receives a goal and a target string that the target LLM should be forced to produce.The stated examples pair a harmful tutorial goal with a target response beginning “Sure, here is a tutorial”.
  • The TAP examples include successful prompt injections that instruct the model to print exactly “Hacked!”.One example embeds the target in a chef scenario followed by “### Command”.
  • Other successful injections demand the target regardless of previous instructions or condition its output on a magic command.These examples explicitly override task context or insert a conditional print expression.
  • Table 9 compares utility and attack success rates for Completion-Real and Completion-Close attacks across delimiter combinations on Llama 7B.The experiments use the naivecompletion-augmented Llama training set and include default delimiters as a reference.

B Additional Ablation Study Results

The additional ablation study examines structured instruction-tuning augmentation strategies and special-delimiter choices, with complete results provided in supplementary tables.

  • The paper provides detailed ablation results for these additional experiments.The results cover both training augmentation and delimiter design choices.
  • The study presents the highest attack success rate for different structured instruction-tuning augmentation strategies in Table 5.Complete results for this analysis appear in Table 8.
  • A separate ablation studies the choice of special delimiters in Table 6, with complete results reported in Table 9.The analysis compares delimiter configurations used by the system.
Loading 2402.06363v2…