PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt Stealing Attacks Against Large Language Models

Zeyang Sha, Yang Zhang

arXiv:2402.12959v1cs.CRcs.CL

TL;DR

Because prompt quality depends on prompt design, recovering prompts from generated answers is a relevant security question. The paper proposes an attack that predicts prompt type and structure, then reconstructs a similar prompt. The authors report better performance than a baseline, while noting a limitation for in-context prompt similarity.

  • Problem

    The paper asks whether well-designed prompts can be inferred from LLM-generated answers, given the value placed on prompt engineering.

    Source: §1 · p. 1

  • Method

    The attack classifies prompts as direct, role-based, or in-context, predicts role or context count, and uses these features with answers to reconstruct prompts.

    Source: §1.1 · p. 2

  • Results

    The authors report that the parameter extractor and prompt reconstructor perform much better than the baseline.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    The authors conclude that prompts can be stolen from generated responses and call for more effective defenses against prompt stealing.

    Source: §8 · p. 12

    Prompt similarity is very low for in-context prompts because the exact randomly selected contexts cannot be recovered from the answer, although answer similarity is considered good enough for a successful attack.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

The increasing reliance on large language models (LLMs) such as ChatGPT in various fields emphasizes the importance of ``prompt engineering,'' a technology to improve the quality of model outputs. With companies investing significantly in expert prompt engineers and educational resources rising to meet market demand, designing high-quality prompts has become an intriguing challenge. In this paper, we propose a novel attack against LLMs, named prompt stealing attacks. Our proposed prompt stealing attack aims to steal these well-designed prompts based on the generated answers. The prompt stealing attack contains two primary modules: the parameter extractor and the prompt reconstruction. The goal of the parameter extractor is to figure out the properties of the original prompts. We first observe that most prompts fall into one of three categories: direct prompt, role-based prompt, and in-context prompt. Our parameter extractor first tries to distinguish the type of prompts based on the generated answers. Then, it can further predict which role or how many contexts are used based on the types of prompts. Following the parameter extractor, the prompt reconstructor can be used to reconstruct the original prompts based on the generated answers and the extracted features. The final goal of the prompt reconstructor is to generate the reversed prompts, which are similar to the original prompts. Our experimental results show the remarkable performance of our proposed attacks. Our proposed attacks add a new dimension to the study of prompt engineering and call for more attention to the security issues on LLMs.

1 Introduction

Prompt quality depends on how prompts are designed, creating demand for prompt-engineering expertise. The paper proposes an answer-based attack that infers prompt properties and reconstructs prompts, with reported gains over a baseline.

1 Introduction

  • Prompt output quality depends heavily on the input, while companies invest in experts to craft prompts for desired outputs.The introduction also describes paid tutorials and courses teaching prompt design.Source: §1 · p. 1

1.1 Our Contribution

  • The attack classifies prompts as direct, role-based, or in-context, then predicts the role or context count from answers.The role classifier covers 15 popular roles; the in-context classifier predicts the number of contexts.Source: §1.1 · p. 2
  • ChatGPT first generates a direct reversed prompt from the answer; the attack then adds the predicted role or generated contexts when needed.For in-context prompts, ChatGPT generates similar question-answer examples according to the predicted context count.Source: §1.1 · p. 2
  • The evaluation uses RetrievalQA and Alpaca-GPT4 with ChatGPT and LLaMA.The paper measures prompt similarity and answer similarity for reconstruction.Source: §1.1 · p. 2
  • On RetrievalQA generated by ChatGPT, the primary classifier reaches 0.833 accuracy, while role-based answer similarity rises from 0.616 to 0.703 with extracted parameters.The answer-similarity comparison is for role-based prompts on RetrievalQA generated by ChatGPT.Source: §1.1 · pp. 2–3

2 Preliminary

The preliminary section situates LLMs as text-generation systems and describes prompt engineering as a way to adapt their outputs to tasks. It distinguishes direct, role-based, and in-context prompts.

2.1 Large Language Models

  • LLMs generate text conditioned on prompts; the section traces their development through transformer-based GPT models and ChatGPT.It describes ChatGPT as based on GPT-3.5 and optimized with reinforcement learning from human feedback.Source: §2.1 · p. 3

2.2 Prompt Engineering

  • Prompt engineering develops and optimizes prompts for users’ specific tasks, with direct, role-based, and in-context forms.The paper limits its treatment of in-context prompts to text used directly as context.Source: §2.2 · p. 3
  • Prior work cited in the section connects role-based prompts with uses including toxic-content generation, game design, and jailbreak prompts.The section also notes few-shot learning and newer in-context approaches such as chain-of-thought and tree-of-thought contexts.Source: §2.2 · p. 3

3 Threat Model

The threat model considers an adversary who uses generated answers to infer prompt structure and produce similar reversed prompts. The adversary has answer access but no access to the original prompts.

3.1 Adversary’s Goal

  • The adversary seeks both to infer prompt parameters and to produce a reversed prompt with similar semantics, structure, and answer behavior.The parameter information is intended to support the prompt-stealing process.Source: §3.1 · p. 3

3.2 Adversary’s Background Knowledge

  • The adversary can access answers generated from the original prompts but is assumed to have no knowledge of those prompts.Source: §3.2 · p. 3
  • The method overview shows parameter extraction predicting prompt type and further information before reconstruction uses those parameters and the answers.Source: Figure 2 · p. 4
  • The adversary is assumed to know which LLM generated the answers.The paper motivates this assumption by noting that answers shared online are often labeled with the model used.Source: §3.2 · p. 4

4 Prompt Stealing Attack

The attack first estimates prompt type and structure from generated answers, then uses those estimates with the answers to construct reversed prompts. Its extractor uses a primary classifier and two type-specific sub-classifiers.

4.1 Parameter Extraction

  • A questionnaire of 10 frequent ChatGPT users informed the three prompt categories used to build the classifiers’ training data.Source: §4.1 · p. 4
  • The role-based data covers 15 commonly used roles, assigned either by ChatGPT-rated relevance or at random.Source: §4.1 · p. 4
  • In-context training examples add randomly selected existing prompts and their answers as context to original prompts.The authors use this construction to examine whether varying contexts affect generated answers.Source: §4.1 · p. 4
  • The primary classifier predicts direct, role-based, or in-context prompt type from generated answers; sub-classifiers estimate role or context count.The primary classifier uses a pretrained BERT backbone.Source: §4.1 · p. 5, §4.3 · pp. 5–6

4.2 Prompt Reconstruction

  • The reconstructor asks ChatGPT to infer a direct prompt from the given answer, then assembles the reversed prompt using extracted parameters.Source: §4.2 · p. 5
  • For role-based prompts, reconstruction adds an “Assume you are” instruction with the predicted role.Source: §4.2 · p. 5
  • For in-context prompts, ChatGPT generates the predicted number of similar question-answer pairs, which are added to the direct reversed prompt.Source: §4.2 · p. 5

4.3 Summary

  • The paper presents the extractor and reconstructor as a hierarchical attack and reports better performance than a baseline.The conclusion describes the reversed prompts as intended to be similar to the originals.Source: §4.3 · pp. 5–6

5 Experiments

The experiments evaluate prompt stealing with ChatGPT and LLaMA, training the attacks on ChatGPT data and testing generalization to LLaMA. The hierarchical parameter extractor outperforms a direct-classification baseline, while reconstruction quality varies by prompt type and training conditions.

5.1 Experimental Setup

  • The study used ChatGPT and LLaMA, training on ChatGPT data and testing LLaMA as a different target model; 500 RetrievalQA prompts were reserved for attack training.The researchers randomly selected 600 prompts from the two datasets, using the remaining RetrievalQA prompts and selected Alpaca-GPT4 prompts for testing.Source: §5.1 · p. 6

5.2 Performance of Parameter Extraction

  • 0.833 accuracy on RetrievalQA and 0.811 performance on Alpaca-GPT4 for ChatGPT prompt-type classification; Alpaca-GPT4 results support generalization across prompt distributions.The LLaMA primary classifier is a two-class classifier covering direct and role-based prompts because long-context input is limited.Source: §5.2 · p. 7, Table 2 · p. 7
  • 0.732 accuracy for role prediction on ChatGPT-generated RetrievalQA answers exceeded the context-number classifier’s 0.614 in a 4-class task.The role classifier’s result is for a 15-class task; LLaMA’s input limit also excluded in-context prompts from its sub-classifier evaluation.Source: §5.2 · p. 7
  • 0.693 accuracy versus 0.234 for the direct 20-class baseline on RetrievalQA answers generated by ChatGPT; the hierarchical extractor combines primary and sub-classifiers.The reported comparison is the overall performance of the parameter extractor.Source: §5.2 · pp. 7–8, Figure 3 · p. 7

5.3 Performance of Prompt Reconstruction

  • 0.832 PS and 0.768 AS for ChatGPT’s direct-prompt reconstruction, while in-context reconstruction had low PS but adequate AS because the original context could not be recovered.For role-based prompts on RetrievalQA generated by ChatGPT, reconstruction reached 0.803 PS and 0.703 AS; the authors identify AS as the better reconstruction metric.Source: §5.3 · p. 8, Table 4 · p. 9

5.4 Ablation Study

  • More training prompts improved primary-classifier performance, with RetrievalQA/ChatGPT accuracy rising from 0.507 at 100 prompts to 0.833 at 500; gains diminished at larger training sizes.For Alpaca-GPT4 answers generated by ChatGPT, performance rose from 0.412 at 100 prompts to 0.793 at 300, then improved by 0.08 from 300 to 500.Source: Figure 4 · p. 8, §5.4 · p. 9
  • 0.087 performance on RetrievalQA/ChatGPT answers when roles were randomly assigned; the authors report better classifier performance when roles align with prompts.The random-role result is from the role-based sub-classifier.Source: §5.4 · p. 9, Table 5 · p. 9
  • Different reconstruction prompts had limited influence in the direct-prompt experiment: the direct version scored 0.832 similarity and the role-based version 0.865.This comparison used direct prompts as the original prompts for simplicity.Source: §5.4 · p. 10

6 Defense

The defense section defines goals and user-level capabilities, then tests prompt- and answer-perturbation strategies. The defenses reduce prompt-stealing performance, but their benefits trade off against utility and operational effort.

6.1 Defender’s Goals and Capabilities

  • The defender aims to reduce prompt-stealing performance, measured by prompt and answer similarity, while preserving answer utility measured against the original answer.Higher similarity between defended and original answers indicates higher utility.Source: §6.1 · p. 10
  • Defenders are assumed to access original prompts and answers, but not modify deployed models or access attack-model classifiers or their gradients.The proposed defenses therefore perturb prompts or answers without fine-tuning the deployed LLM.Source: §6.1 · p. 10

6.2 Method

  • Prompt-based defense appends a predefined instruction asking the LLM not to reveal information from the original prompt.The intended effect is to make the original prompt harder to reverse.Source: §6.2 · pp. 10–11, Figure 7 · p. 11

6.3 Results

  • Both defense families reduce attack performance; prompt-based defense mitigates more than answer-based defense but causes a larger utility decline.The authors characterize the current defenses as achieving only a subpar trade-off.Source: Table 7 · p. 11, §6.3 · pp. 11–12

6.2 Method

  • Answer-based defense perturbs generated answers through either summarizing and rewriting them or explicitly removing critical content.The authors call these variants inconspicuous and conspicuous defense, respectively.Source: §6.2 · p. 11

6.3 Results

  • On RetrievalQA prompts generated by ChatGPT, conspicuous defense lowers AS from 0.768 to 0.486, while inconspicuous defense lowers it by 0.077.Utility is 0.813 for conspicuous defense and 0.980 for inconspicuous defense.Source: §6.3 · p. 11
  • The defenses require time-consuming LLM modifications for users, motivating more automated deployment that avoids substantially increasing operational difficulty.The authors call for better defenses because the proposed strategies achieve only a subpar trade-off between attack similarity and utility.Source: §6.3 · p. 12

7 Related Work

Prior LLM security work covered several vulnerabilities, but the paper identifies prompt security as an underexplored area. It situates its attack alongside prompt-stealing work on text-to-image systems and earlier LLM prompt-reversal findings.

7.1 Security Issues of LLMs

  • Earlier LLM security research examined toxicity, stereotypes, adversarial robustness, and privacy, while this paper identifies prompt security as an unaddressed perspective.The authors present prompt stealing as the first attack in this framing of LLM security.Source: §7.1 · p. 12

7.2 Prompt Stealing Attacks

  • Text-to-image prompt stealing used image captioning, including ClipCap by Mokady et al. [22], and specialized datasets proposed by Shen et al. [35].The cited ClipCap approach uses CLIP [27] to translate an image into a prompt.Source: §7.2 · p. 12
  • Although Zhang et al. found that an LLM system prompt could be reversed, the authors note no prior attack based on LLM-generated answers.The paper positions its work as addressing that gap.Source: §7.2 · p. 12

8 Conclusion

The conclusion presents prompt stealing as a two-stage attack that infers prompt properties and uses them to reconstruct similar prompts from generated answers. It reports successful parameter prediction and a trade-off between defense performance and utility.

8 Conclusion

  • Parameter extraction successfully predicts prompt type and further information, enabling the reconstructor to generate more similar reversed prompts.The authors also report that two proposed defenses involve a fair trade-off between defense performance and utility.Source: §8 · p. 12
  • The attack classifies prompts as direct, role-based, or in-context, then predicts the role or context count before reconstructing prompts with ChatGPT.For role-based prompts, the system adds the predicted role; for in-context prompts, it generates and adds a certain number of contexts.Source: §8 · p. 12
  • The authors urge more effective defenses and further LLM security research because prompts can be stolen from generated responses.They emphasize studying LLM security from the perspective of prompts.Source: §8 · p. 12
Loading 2402.12959v1…