PaperlayerView exact revision on arXiv

Source-linked AI summary

PLeak: Prompt Leaking Attacks against Large Language Model Applications

Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, Yinzhi Cao

arXiv:2405.06823v3cs.CRcs.AIcs.LG

TL;DR

Prompt leaking threatens confidential system prompts, while prior manually crafted attacks have limited effectiveness. PLeak optimizes adversarial queries incrementally on shadow prompts and a shadow LLM, then transfers them to target applications. It substantially outperforms prior baselines and exactly reconstructs prompts for 68% of 50 Poe applications.

  • Problem

    System prompts are confidential intellectual property, but existing prompt-leaking attacks rely mainly on manually crafted queries and have limited effectiveness.

  • Method

    PLeak uses closed-box optimization with shadow system prompts and a shadow LLM, incrementally optimizing adversarial queries and applying adversarial transformations before target reconstruction.

  • Results

    68% of 50 real-world Poe applications had their system prompts exactly reconstructed, and PLeak outperformed manually crafted and adapted jailbreaking baselines.

  • Takeaways & Limitations

    Automated adversarial-query optimization can effectively leak system prompts in offline and real-world LLM applications.

  • Takeaways & Limitations

    The evaluation used publicly available system prompts, and further defenses and gradual multi-query bypasses remain future work.

Abstract

from arXiv · show

Large Language Models (LLMs) enable a new ecosystem with many downstream applications, called LLM applications, with different natural language processing tasks. The functionality and performance of an LLM application highly depend on its system prompt, which instructs the backend LLM on what task to perform. Therefore, an LLM application developer often keeps a system prompt confidential to protect its intellectual property. As a result, a natural attack, called prompt leaking, is to steal the system prompt from an LLM application, which compromises the developer's intellectual property. Existing prompt leaking attacks primarily rely on manually crafted queries, and thus achieve limited effectiveness. In this paper, we design a novel, closed-box prompt leaking attack framework, called PLeak, to optimize an adversarial query such that when the attacker sends it to a target LLM application, its response reveals its own system prompt. We formulate finding such an adversarial query as an optimization problem and solve it with a gradient-based method approximately. Our key idea is to break down the optimization goal by optimizing adversary queries for system prompts incrementally, i.e., starting from the first few tokens of each system prompt step by step until the entire length of the system prompt. We evaluate PLeak in both offline settings and for real-world LLM applications, e.g., those on Poe, a popular platform hosting such applications. Our results show that PLeak can effectively leak system prompts and significantly outperforms not only baselines that manually curate queries but also baselines with optimized queries that are modified and adapted from existing jailbreaking attacks. We responsibly reported the issues to Poe and are still waiting for their response. Our implementation is available at this repository: https://github.com/BHui97/PLeak.

1 INTRODUCTION

PLeak introduces an automated, closed-box attack that optimizes adversarial queries to reconstruct confidential system prompts. It uses incremental optimization and adversarial transformation, outperforming manually crafted and adapted jailbreaking baselines in offline and Poe evaluations.

  • System prompts are confidential intellectual property because they determine an LLM application's functionality and performance.Poe allows developers to keep system prompts confidential, and 55% of surveyed Poe applications did so.
  • Prior prompt-leaking attacks manually craft queries, creating scalability limitations and limited effectiveness.
  • PLeak formulates prompt leaking as a closed-box optimization problem using shadow system prompts and a shadow LLM.The optimized adversarial query is intended to make shadow applications output their system prompts.
  • Incremental search optimizes adversarial queries gradually, beginning with a few system-prompt tokens and increasing the target length step by step.This decomposition addresses the large search space and exact word-by-word matching objective.
  • Adversarial transformation modifies queries and outputs so PLeak can reconstruct system prompts through an inverse transformation despite response filtering defenses.Examples include adding prefixes and reversing word order.
  • Across five datasets and five LLMs, PLeak outperformed manually crafted and adapted jailbreaking baselines on Exact Match, Semantic Similarity, and related metrics.The offline evaluation reports higher Exact Match and Semantic Similarity, while the detailed results state gains across all four metrics.
  • 68% of 50 real-world Poe applications had their system prompts exactly reconstructed, versus 20% for manually curated queries and 18% for adapted jailbreaking attacks.

2 OVERVIEW

PLeak treats an LLM application as a backend LLM combined with a system prompt and user query, then reconstructs the target prompt through a two-phase pipeline. It optimizes adversarial queries offline and applies them to the target application, reversing transformations and aggregating responses.

  • Definitions: An LLM predicts the next token from preceding tokens, and decoding repeatedly generates tokens until an END token is produced.
  • Definitions: Top-k and Top-p sampling select next tokens from restricted probability sets, while beam-sample combines beam search with sampling.
  • Definitions: An LLM application concatenates a user query with its system prompt, sends the resulting prompt to a backend LLM, and relays the response.
  • Problem Formulation: The attack optimizes multiple adversarial queries and a post-processing function so aggregated application responses reconstruct the target system prompt.The reconstructed prompt is denoted p_r, while P aggregates the target application's responses.
  • Threat Model: The target application keeps its system prompt confidential, while the adversary can query the application and process its responses.
  • PLeak Pipeline: Phase 1 optimizes adversarial queries against shadow system prompts and a shadow LLM, then applies an adversarial transformation.The default transformation is identity unless defenses are being evaluated.
  • PLeak Pipeline: Phase 2 sends transformed adversarial queries to the target application and reconstructs the prompt by inverting the transformation and computing common response parts.

3 PLEAK

PLeak is a closed-box attack that optimizes adversarial queries to make shadow LLM applications reproduce system prompts. It uses incremental search and gradient-based token optimization, then reconstructs the target prompt from multiple responses.

  • 3.1 Phase 1: Offline AQ Optimization: PLeak formulates adversarial-query generation as optimizing shadow queries so shadow LLMs output their system prompts.The optimization uses a shadow dataset of system prompts and a shadow LLM.
  • 3.1 Phase 1: Offline AQ Optimization: Incremental search progressively optimizes queries for longer prefixes of each shadow system prompt, beginning with the first t tokens.Prompts shorter than the current target length are excluded from that optimization step.
  • 3.1 Phase 1: Offline AQ Optimization: PLeak approximately minimizes the discrete loss by estimating each adversarial token's effect with gradients, retaining top-k candidates, and evaluating their true losses.The procedure repeats token updates until the adversarial query no longer changes.
  • 3.2 Phase 2: Target System Prompt Reconstruction: PLeak reconstructs the target prompt by inverse-transforming responses, finding text shared by response pairs, and selecting the longest common text.Multiple adversarial queries are combined because their performances may differ.

4 IMPLEMENTATION AND EXPERIMENTAL SETUP

The evaluation combines offline experiments across five datasets and five LLMs with a real-world Poe sample, using multiple attack baselines and prompt-reconstruction metrics. PLeak is implemented with PyTorch and evaluated under defined initialization and scoring procedures.

  • Experimental Setup: Offline experiments use five datasets as system prompts and five LLMs, including GPT-J-6B, OPT-6.7B, Falcon-7B, LLaMA-2-7B, and Vicuna.Adversarial-query generation takes two hours for seven-billion-parameter LLMs on an A100.
  • Experimental Setup: The real-world evaluation randomly selects 50 Poe applications with open system prompts, whose prompts average 96.55 tokens with a 61.25-token standard deviation.Open prompts are selected for evaluation convenience.
  • Baselines and Metrics: Attack success is evaluated with substring-match and exact-match accuracy, while real-world attacks require additional validation because the attacker does not know when leakage succeeds.Exact Match requires the reconstructed prompt to equal the target prompt after punctuation exclusion.
  • Baselines and Metrics: PLeak is compared with two manually crafted prompts and two optimized adaptations, GCG-leak and AutoDAN-leak.When a baseline has multiple adversarial queries, evaluation uses its best query.

5 EVALUATION

The evaluation examines PLeak’s attack performance, real-world effectiveness, parameter sensitivity, transferability, and resilience to defenses. Across these settings, PLeak generally outperforms manually crafted and adapted optimized-query baselines, while performance varies with prompt, query, model, and defense conditions.

  • RQ1: Prompt Leaking Attack Performance: PLeak outperforms manually crafted and adapted optimized-query baselines across all four metrics on five datasets and five LLMs.The RQ1 evaluation compares PLeak with Perez et al., Zhang et al., GCG-leak, and AutoDAN-leak.
  • RQ1: Prompt Leaking Attack Performance: PLeak achieves SM accuracy above 0.9 in 15 of 20 cases, whereas prior manually crafted methods usually obtain substantially lower scores.Zhang et al. reaches a highest reported SM score of 0.363 on SQuAD2 with LLaMA-2, while Perez et al. is usually near zero.
  • RQ1: Prompt Leaking Attack Performance: PLeak’s EM accuracy ranges from 0.327 to 1.00 and averages 0.823 across five datasets and five LLMs.The reported average substantially exceeds either baseline, while EED and SS comparisons also favor PLeak across LLMs.
  • RQ2: Attacks against Real-world LLM Applications on Poe: On Poe, PLeak reconstructs 42% of target prompts with one AQ and 68% with multiple AQs, compared with 2% for Perez et al.Adversarial transformation improves robustness to possible keyword filtering and raises SM by around 0.50.
  • RQ3: Parameter Analysis: Increasing shadow-dataset size stabilizes and improves attack performance, while increasing AQ length from 8 to 18 tokens raises EM, SM, and SS and lowers EED.For AQ lengths from 8 to 10 tokens, EM rises from 0.429 to 0.902 and SM from 0.440 to 0.903; gains flatten at longer lengths.
  • RQ4: Transferability and RQ5: PLeak against Defenses: PLeak transfers across LLMs and datasets, but transfer quality depends on prompt composition and target conditions.The same AQ reaches SM and EM above 0.9 across tested alternative LLMs, while dataset transfer is strongest when prompt compositions match; keyword filtering reduces baselines to near zero but PLeak retains EM 0.30 and SM 0.34.

6 DISCUSSION

The paper addresses privacy and disclosure concerns by using publicly available system prompts and notifying Poe about discovered leaks. It also identifies real-world deployment and stronger defenses as future work.

  • Ethics: All experiments use publicly available system prompts, so no private information is used in the evaluation.The authors state that performance on public and private prompts was the same in their own Poe application, differing only in availability.
  • Ethics: The authors responsibly notified Poe about system prompt leaks and its open-source PLeak implementation in December 2023.They planned to allow Poe 45 days before formal publication.
  • Real-world Attack Deployment: Real-world deployment remains future work because attackers lack the original system prompt needed to evaluate reconstruction success.The paper suggests comparing output distributions between the target application and one using the reconstructed prompt, potentially across multiple queries.

7 RELATED WORK

The related work spans LLM applications, in-context learning, attacks and jailbreaks, prompt injection, and model-stealing research. PLeak differs by optimizing adversarial queries rather than manually crafting injected instructions or data.

  • LLMs and Their Applications: LLMs support zero-shot and few-shot learning, while in-context learning uses demonstrations whose order can affect few-shot accuracy.Prior work also formalizes in-context learning as Bayesian inference and studies recovery of latent concepts.
  • LLMs and Their Applications: Question-answering applications evaluate whether LLMs understand questions and produce accurate answers using system prompts or knowledge.Related work includes extractive tasks that retrieve answers from system prompts and methods intended to improve human preference alignment.
  • Attacks against LLMs or Their Applications: Prior attacks include adversarial triggers, training-data extraction, indirect prompts, and jailbreaking prompts that mislead an LLM’s original goal.PLeak draws on token-replacement ideas associated with GCG, Wallace et al., and HotFlip.
  • Prompt Injection Attacks: PLeak’s adversarial query can be viewed as injected instruction or data that redirects an LLM application to leak its system prompt.Unlike existing prompt injection attacks based on heuristics, PLeak crafts the query by solving an optimization problem.
  • Prompt Leaking and Model Stealing: The closest prompt-leaking approaches manually craft adversarial queries, which limits scalability and effectiveness for real-world applications.The paper also distinguishes PRSA by its different attack scenario and threat model.
  • Prompt Leaking and Model Stealing: Earlier model-stealing work reconstructs model parameters, training hyperparameters, or graph-training links through queries to model APIs.These studies concern conventional machine-learning models rather than system prompts in LLM applications.

8 CONCLUSION

PLeak is presented as an automated prompt-leaking attack that optimizes adversarial queries on shadow data and a shadow LLM, then transfers them to target applications. Evaluations on offline and real-world applications show that it outperforms manually curated and jailbreak-adapted query baselines.

  • Conclusion: PLeak optimizes adversarial queries on a shadow dataset and shadow LLM to maximize exact or partial system-prompt output, then transfers them to target LLMs.Its adversarial transformation enables later reconstruction of original outputs through an inverse function.
  • Conclusion: PLeak significantly outperforms manually curated adversarial queries and optimized queries adapted from prior jailbreaking attacks in offline and real-world evaluations.The conclusion summarizes the comparison across both evaluation settings without reporting a specific metric.

A A SELECTIVE LIST OF POE APPLICATIONS

The appendix provides a selective list of Poe applications by presenting adversarial queries generated by PLeak in Table 14.

  • A A SELECTIVE LIST OF POE APPLICATIONS: Table 14 presents adversarial queries generated by PLeak for real-world Poe applications.The table is included in the appendix as a selective list of Poe applications.
Loading 2405.06823v3…