PaperlayerView exact revision on arXiv

Source-linked AI summary

Effective Prompt Extraction from Language Models

Yiming Zhang, Nicholas Carlini, Daphne Ippolito

arXiv:2307.06865v3cs.CLcs.AI

TL;DR

Anecdotal prompt-extraction reports have been difficult to verify because attackers rarely have access to the ground truth. This paper systematically evaluates extraction attacks using prompt sources with known ground truth and confidence estimates for candidate leaks. It finds widespread extraction across models and high-precision predictions, including attacks on real services.

  • Problem

    Earlier reports of extracting hidden prompts were difficult to assess because attackers rarely had the ground-truth prompts.

    Source: §1 · p. 1

  • Method

    The paper evaluates prompt extraction using known prompt sources and estimates candidate success from consistency among extractions of the same prompt.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    More than 50% of prompts were approximately extracted for each of 11 models across three held-out sources, and the estimator achieved over 90% precision for five tested models.

    Source: §5 · p. 5

  • Takeaways & Limitations

    The authors conclude that prompts should not be treated as secrets and that output-filtering and prompt-based defenses can be insufficient to prevent extraction.

    Source: §9 · p. 10

    The threat model assumes queries are appended to the conversation; inserting them within a user instruction may make prompts more difficult to extract.

    Source: Limitations and Ethics Statement · p. 10

Abstract

from arXiv · show

The text generated by large language models is commonly controlled by prompting, where a prompt prepended to a user's query guides the model's output. The prompts used by companies to guide their models are often treated as secrets, to be hidden from the user making the query. They have even been treated as commodities to be bought and sold on marketplaces. However, anecdotal reports have shown adversarial users employing prompt extraction attacks to recover these prompts. In this paper, we present a framework for systematically measuring the effectiveness of these attacks. In experiments with 3 different sources of prompts and 11 underlying large language models, we find that simple text-based attacks can in fact reveal prompts with high probability. Our framework determines with high precision whether an extracted prompt is the actual secret prompt, rather than a model hallucination. Prompt extraction from real systems such as Claude 3 and ChatGPT further suggest that system prompts can be revealed by an adversary despite existing defenses in place.

1 Introduction

Prompts can be central to an LLM product’s behavior, yet anecdotal reports of extracting hidden prompts have been difficult to verify. The paper systematically evaluates these attacks and reports successful extraction from both held-out prompt sources and real services.

1 Introduction

  • A product’s prompt can be its most significant component because someone who obtains it can essentially replicate the behavior of the prompted LLM.Source: §1 · p. 1
  • The evaluation finds prompt extraction attacks effective across 11 LLMs, with high-precision estimates and successful attacks against real services.The paper reports translation-based extraction from systems including Bard, Bing Chat, Claude, and ChatGPT.Source: §1 · p. 1
  • Figure 2 shows a Japanese attack query recovering Bing Chat’s system prompt by back-translation, up to translation errors.Source: Figure 2 · p. 2
  • The paper also discusses text-based defenses intended to prevent prompt extraction and how they can be circumvented.Source: §1 · p. 2

2 Threat Model

The threat model considers an adversary who queries a conversational service to reconstruct its secret prompt. Success is measured by exact sentence containment or by an approximate token-coverage metric.

2 Threat Model

  • The service passes both the secret prompt and the user query to an LLM, whose generation is returned through the API.Source: §2 · p. 2
  • The adversary’s goal is to reconstruct the secret prompt from the responses to a set of attack queries.Source: §2 · p. 2
  • Exact match requires every sentence of the true prompt to appear as a substring in the guess, rather than requiring the full prompt as one exact string.Source: §2 · p. 2
  • Exact match can miss guesses differing only in trivial formatting, so the paper also defines approximate match using Rouge-L recall and a 90% threshold.The approximate score represents the fraction of prompt tokens covered by the longest common subsequence.Source: §2 · pp. 2–3
  • The adversary can make about 100 queries, with the API reset between queries, and has no assumed access to model internals or token likelihoods.Source: §2 · p. 3

3 Simple Prompt Extraction Attack

The attack gathers candidate prompt extractions and assigns them confidence estimates, then returns the candidate with the highest estimate. Its confidence mechanism uses consistency across extractions to distinguish likely leaks from plausible hallucinations.

3 Simple Prompt Extraction Attack

  • The attack gathers candidate prompts from multiple queries, scores their likelihood of matching the ground truth, and returns the highest-confidence candidate.Source: §3 · p. 3, §3.2 · p. 4

3.1 Attack queries

  • Five handwritten seed queries, such as asking the model to repeat the conversation, are already sufficient to extract most prompts in the development set.Source: §3.1 · p. 3
  • GPT-4 paraphrases of the seed queries expand the attack to 105 queries per prompt; running it for one prompt costs less than $1 on GPT-4.Source: §3.1 · p. 3

3.2 Guessing the prompt

  • Because generated extractions can be plausible hallucinations, the attacker needs a way to estimate whether each candidate is actually successful.Source: §3.2 · p. 3
  • A DeBERTa model fine-tuned on 16,000 development-set extractions estimates the leaked-token ratio in one extraction conditioned on the others.Source: §3.2 · p. 3
  • For GPT-4 extraction experiments, the authors use the 15 most effective queries identified on the development set.Source: §3.2 · p. 3
  • The estimate P(e_i) marginalizes the model’s prediction over permutations of the other extractions.Source: §3.2 · p. 4
  • The final guess is the extraction with the highest estimated success probability, accompanied by its confidence score.Source: §3.2 · p. 4

4 Controlled Experimental Setup

The controlled evaluation uses three prompt sources, separating attack-development DEV data from final test sets, and evaluates 11 models from four families.

4.1 Datasets

  • Unnatural Instructions contributes 500 test prompts and 200 DEV prompts from instruction-tuning data sampled using human-written instruction-output pairs.Source: §4.1 · p. 4
  • ShareGPT test prompts are initial user messages from user-shared ChatGPT conversations; incomplete or over-256-token conversations are filtered, with 200 test and 200 DEV prompts sampled.Source: §4.1 · p. 4
  • Awesome-ChatGPT-Prompts is a curated test set of 153 role-adaptation prompts resembling system messages for real LLM-based services.Source: §4.1 · p. 4

4.2 Models

  • The main experiments evaluate 11 models from four families: GPT-3.5-turbo/GPT-4, Alpaca, Vicuna, and Llama-2-chat.Source: §4.2 · p. 4
  • The table's headline is that a majority of prompts are extractable across models and held-out datasets; each cell reports the percentage of guesses matching ground truth.Source: Table 1 · p. 5

5 Extraction Attack Results

Across 11 LLMs and three held-out prompt sources, extraction attacks recover most prompt content from every model tested. The section also evaluates confidence-based verification, capability-related variation, and identifying the model behind a service.

LLMs are prone to prompt extraction

  • More than 50% of prompts were approximately extracted from each of 11 LLMs; exact extraction averaged 69.0% for Vicuna1.3-33B, while approximate recovery averaged 68.0% for Alpaca-7B.Approximate extraction means that over 90% of prompt tokens were recovered.Source: §5 · p. 5

Prompt-query separation does not safeguard models

  • Despite prompt-query separation, extraction reached 87.0% for GPT-3.5 and 86.0% for GPT-4, and substantial proportions of prompts were extracted from all three Llama-2-chat models.The passage gives no extraction percentages for the Llama-2-chat models here.Source: §5 · p. 5

Implications for prompt extraction attacks

  • The attack uses only generation-API access and makes no model- or service-specific assumptions, so the authors characterize its results as a lower bound on what dedicated attackers could achieve.Dedicated attackers could issue more queries and choose them strategically.Source: §5 · p. 5

Prompt extraction attack is high-precision

  • For all five evaluated models except Alpaca-7B, confidence-estimator precision exceeded 90% across datasets; Alpaca-7B reached 80%, with precision maintained across a wide recall range.Figure 3 plots precision and recall; its circles mark the decision boundary P > 90%.Source: §5 · p. 5, Figure 3 · p. 6

Model capability correlates with extractability

  • Extractability had a weak positive correlation with MMLU score (Pearson’s r = 0.28), but capability did not fully explain vulnerability: GPT-4 extraction was 83.5%, versus 89.4% for GPT-3.5.MMLU is used as a capability proxy, not a comprehensive measure.Source: §5 · p. 6, Figure 4 · p. 7

Can the LLM behind a service be identified?

  • Comparing service generations with candidate models using Rouge-L identified the model with 71.7% accuracy across 60 prompted APIs.The heatmap compares actual and predicted models.Source: Figure 5 · p. 7, §5 · p. 7

6 Prompt Extraction from Production Models

The authors adapt prompt extraction for production services by asking models to translate outputs into other languages and back-translating the results. They report consistent extractions from several services despite defenses, while Claude 3 Sonnet’s recovered prompt differs from the available reference.

Translation-based prompt extraction

  • The attack requests translations into target languages, then back-translates them to English; consistency across languages provides some confidence that the recovered text matches the prompt.Languages with little shared vocabulary with English may be less likely to trigger an English-only output filter.Source: §6 · p. 7

System prompt extraction from production services

  • Japanese translation enabled extraction of Bing Chat’s system prompt despite instructions to refuse disclosure, while the same attack query in English failed.The authors suggest an English-only post-generation filter may explain the difference.Source: §6 · p. 8
  • Claude 3 Sonnet extractions matched the available reference closely but consistently omitted three sentences and added one; the authors suggest the prompt may have changed since publication.Multiple language-based extractions produced the same prompt, which the authors consider plausibly correct.Source: §6 · p. 8, Table 2 · p. 8
  • Consistent prompts were also extracted from Bard and ChatGPT, supporting the authors’ conclusion that the attack can work on state-of-the-art industry LLMs despite explicit instructions against extraction.The reported extractions are provided in Appendix D.2.Source: §6 · p. 8

7 Output Filtering Does Not Prevent Prompt Extraction

Output filtering can sharply reduce verbatim prompt extraction, but attackers can evade a 5-gram filter by manipulating generated text. The defense’s effectiveness falls as models become more capable of following such instructions.

7 Output Filtering Does Not Prevent Prompt Extraction

  • 0% extraction success was reported for Vicuna1.5-13B, GPT-3.5, and GPT-4 when the 5-gram defense returned an empty string for generations overlapping the secret prompt.The attack in this setting relies on models generating the prompt verbatim.Source: §7 · p. 8
  • Interleaving prompt words with a removable symbol and Caesar-ciphering generated text are two strategies for bypassing output filters.The authors present these as proof-of-concept strategies for manipulating output so the prompt can be recovered.Source: §7 · p. 9
  • 35.2% of prompts remained extractable on Vicuna1.3-33B under the 5-gram defense, while it was mostly ineffective on Llama-2-chat-70B and nearly blocked extraction on Alpaca-7B.The reported success pattern varies with model capability; successful evasions mostly used interleaving, not Caesar encryption.Source: §7 · p. 9, Table 3 · p. 9

8 Related Work

Related work connects prompt extraction to both the value of prompts and broader adversarial-prompting research. This paper distinguishes its generated-text threat model from recovery using next-token probabilities.

8 Related Work

  • Prompting, instruction-tuning, and preference-tuning made prompts valuable intellectual property that designers often keep secret.Prior work also studied prompt engineering and techniques for adapting model behavior.Source: §8 · p. 9
  • Adversarial prompting includes attacks that induce degenerate behavior, toxic text, jailbreaks, and prompt injection into applications built on prompted LLMs.The passage situates prompt extraction as a special case of prompt injection, with the objective of making a model leak its prompt.Source: §8 · pp. 9–10
  • Morris et al. (2023) recover prompts from next-token probabilities with an inversion model, whereas this work assumes access only to generated text.Source: §8 · p. 10

9 Conclusion

The conclusion argues that simple extraction attacks expose prompts despite prompt-based and output-filtering defenses. The accompanying limitations and appendix material describe the threat model, model setups, attack queries, and extraction-confidence procedure.

9 Conclusion

  • Prompt-based defenses and output filtering are insufficient to prevent prompt extraction in general, and the authors report extracting secret system messages from Claude and Bing Chat.They call for future work on mitigating prompt-extraction risks in real-world applications.Source: §9 · p. 10

Limitations and Ethics Statement

  • The evaluation omits adaptive query selection and attacker knowledge of the underlying model; its threat model also assumes queries are appended, while other defenses may make extraction harder but could be circumvented.The authors also note possible malicious use against real systems and potential harm.Source: Limitations and Ethics Statement · p. 10

A Attack Queries

  • The attack-query appendix includes five author-written queries and ten sampled queries from a set of 100 generated by GPT-4 using the handwritten queries.The appendix also lists models used to measure attack efficacy.Source: A Attack Queries · p. 14, Table 4 · p. 14, Table 5 · p. 14

B.1 OpenAI’s GPT-3.5 and GPT-4

  • For GPT-3.5 and GPT-4, the API uses a secret prompt as the system message and treats the incoming query as the conversation’s first utterance.Source: §B.1 · p. 15

B.2 LLaMA; B.3 Alpaca

  • The LLaMA setup conditions generated text on the secret prompt and query; the authors instead evaluate instruction-following variants because LLaMA 1 has limited ability to adapt to arbitrary prompts.Alpaca is described as fine-tuned on 52k instruction-completion pairs generated by GPT-3.Source: §B.2 · p. 15, §B.3 · p. 15

B.4 Vicuna

  • The evaluated Vicuna variants span 7B, 13B, and 33B parameters for version 1.3 and 7B and 13B for version 1.5.Vicuna is described as an open-source dialog model; version 1.5 has no 33B variant.Source: §B.4 · p. 15

B.5 Llama-2-chat

  • The Llama-2-chat experiments cover 7B, 13B, and 70B models; these models are optimized for dialog with instruction-tuning and reinforcement learning from human feedback.Source: §B.5 · p. 16

C DeBERTa Model Details

  • The DeBERTa confidence model predicts token overlap between the true prompt and a candidate extraction using the other extractions from that prompt as context.The authors report 16,000 permuted training examples, average eight permutations during evaluation, and provide hyperparameters.Source: C DeBERTa Model Details · p. 17, Table 7 · p. 17

D Prompt Extraction Examples

The appendix provides sampled extraction examples and prompt excerpts from multiple services, then examines extraction under random sampling and the effect of prompt length.

D.1 Additional Qualitative Examples

  • Table 8 reports attack-generated GPT-4 guesses for a random sample of 4 UNNATURAL prompts, with percentage of leaked tokens and exact-match and approx-match successes; Table 9 samples 5 guesses around the 90% leaked-token approx-match threshold.Source: Table 8 · p. 18, Table 9 · p. 19

D.2 Extracted System Prompts

  • Tables report extracted system prompts for Bard, Bing Chat, ChatGPT, and Claude 3 Sonnet; only one Bing Chat extraction is printed due to length, with Arabic, Chinese, and Japanese extractions in the released dataset.Source: §D.2 · p. 20, Table 10 · p. 20, Table 11 · p. 21
  • The Bing prompt also specifies a self-introduction and multilingual communication, alongside stylistic expectations for answers and imaginative-content abilities.Source: §D.2 · p. 21
  • The extracted Bing prompt includes instructions about markdown formatting, security and copyright responses, chat settings, and predefined tools.Its listed rules include refusing to discuss guidelines or rules except chat settings.Source: §D.2 · p. 21

E.1 Sampling temperature has a small impact on extraction success

  • With temperature set to 1, random sampling had a negligible-to-small effect on attack efficacy depending on the model, and most prompts remained extractable.The passage reports experiments on Alpaca-7B, Vicuna1.3-33B, and Llama-2-chat-70B.Source: §E.1 · p. 24, Table 14 · p. 24

E.2 Longer prompts are slightly harder to recover

  • Prompt length had a significant but weak negative correlation with the ratio of leaked tokens across three datasets and 11 models (Pearson’s r = −0.07).The authors interpret this as longer prompts being only marginally harder to extract.Source: §E.2 · p. 24

E.3 Extraction Results Against the 5-gram Defense

The appendix reports that the 5-gram defense can be evaded, describes precision-recall validation for prompt extraction, and gives the experiments’ computational cost.

E.3 Extraction Results Against the 5-gram Defense

E.4 Precision and Recall Results for Prompt Extraction

  • Heuristic P verifies successful extractions with high precision; Figure 6 shows precision-recall curves for the remaining five models, with circles marking the P > 90% decision boundary.Source: Figure 6 · p. 26, §E.4 · p. 26

F Computational Infrastructure and Cost

Loading 2307.06865v3…