PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting and Mitigating Query-Based System Prompt Leakage Attacks

Tushar Nayan, Pratik Poudel, Qian Lou, Yuede Ji, Jason Liu, Ruimin Sun

arXiv:2610.36295v1cs.CRcs.AI

TL;DR

Adaptive multi-turn prompt extraction can look benign query by query, while prompt-only defenses assess queries in isolation. AdvGuard combines prompt-level risk estimation with interaction graphs and graph classification; evaluations report 91.14–100% recall at FPR at or below 3.4%, alongside reduced leakage in adaptive attacks.

  • Problem

    Adaptive extraction queries can appear benign in isolation, while existing defenses often assess queries independently and may miss coordinated behavior across interactions.

    Source: §I · p. 1, C. Structural Attack Detection · p. 4

  • Method

    AdvGuard combines prompt-level risk estimation with interaction-graph construction, statistical filtering, and graph-based classification to detect coordinated extraction behavior.

    Source: §I · p. 2, §VIII · p. 12

  • Results

    Across evaluations, AdvGuard records 91.14–100% recall with FPR at or below 3.4%, and limits adaptive-attack exact-match leakage to 0.00–0.01 and substring-match leakage to 0.01–0.09.

    Source: §I · p. 2, §VIII · p. 12

  • Takeaways & Limitations

    Interaction-level monitoring offers a practical, more robust alternative to prompt-only defenses for detecting coordinated extraction patterns.

    Source: §VIII · p. 12

    Very short sessions or extremely sparse attack patterns may evade early detection because meaningful graph structure requires sufficient query history.

    Source: §VII · p. 12

Abstract

from arXiv · show

Large Language Models (LLMs) rely on proprietary system prompts to enforce task directives, safety policies, and access controls. These prompts are vulnerable to query-based leakage attacks, where black-box adversaries extract confidential instructions through adaptive, multi-turn interactions. Existing defenses often assess queries independently, missing malicious behavior that emerges across interactions. We present AdvGuard, a model-agnostic framework for detecting and mitigating system prompt leakage. AdvGuard models cross-query relationships using a bounded semantic interaction graph, statistically filters candidate components, and converts them into line graphs for graph convolutional network (GCN) classification. This approach captures multi-hop patterns in query similarity that distinguish coordinated extraction from benign workflows. A lightweight prompt-level detector provides protection before sufficient interaction history accumulates, and the framework supports black-box LLM deployments. In mixed-query evaluations, AdvGuard improves recall by up to 36% over the strongest baseline while maintaining a false positive rate of at most 3.4%. In multi-turn workloads, it achieves 97.23%-99.1% F1 with 100% recall. Leave-one-attack-family-out tests yield 99.98% mean recall and 99.99% mean F1, with no observed false positives. Integrated into the adaptive PLeak and LeakAgent attack loops, AdvGuard limits exact-match leakage to 0.00-0.01 and substring-match leakage to 0.01-0.09 across evaluated victim models. These results demonstrate robust detection of unseen attack families and substantially reduced prompt reconstruction.

I. INTRODUCTION

System-prompt leakage can expose proprietary instructions, while coordinated extraction may become visible only across related queries. AdvGuard combines per-query and interaction-level detection; evaluations report high recall with low false positives and reduced leakage under adaptive attacks.

B. System-Prompt Leakage Attacks

  • Extraction ranges from iterative probing, and adaptive reformulation, [9] to encoding tricks [10]– and multi-turn manipulation, where related queries can reveal an objective no single query makes decisive.PLeak searches for transferable extraction queries, while LeakAgent uses a reinforcement-learning red-teaming agent for privacy-leakage objectives including system-prompt extraction [9].Source: §I · p. 1, B. System-Prompt Leakage Attacks · p. 2

I. INTRODUCTION

  • AdvGuard combines prompt-level detection with a bounded semantic interaction graph, statistical component filtering, and line-graph GCN classification.The framework is described as model-agnostic and does not require access to the protected prompt, responses, or model internals.Source: §I · pp. 1–2
  • 91.14–100% recall while maintaining FPR at or below 3.4%; adaptive PLeak and LeakAgent attack loops show exact-match leakage of 0.00–0.01 and substring-match leakage of 0.01–0.09.The reported average query latency is 32.96 ms.Source: §I · p. 2

A. System Prompts and Prompt Leakage

  • System-prompt leakage means unauthorized reconstruction of protected instructions, which may be verbatim, near-verbatim, semantically equivalent, or assembled across interactions.System prompts can encode application logic, safety policies, and tool-use protocols,.Source: §I · p. 1, A. System Prompts and Prompt Leakage · p. 2

C. Defenses and Remaining Gap

Many defenses assess individual inputs, outputs, or protected representations, while interaction-aware systems described here focus on adjacent risks rather than system-prompt reconstruction. AdvGuard targets coordinated extraction using query-side monitoring within a black-box application boundary.

C. Defenses and Remaining Gap

  • Existing defenses often decide from a current request or require response, protected-content, generation-statistics, or model-level access; interaction-aware defenses described here target adjacent risks, leaving bounded input-only monitoring of coordinated extraction underexplored.AdvGuard targets this setting without access to the protected prompt, model internals, or generated responses.Source: C. Defenses and Remaining Gap · pp. 2–3

III. THREAT MODEL

  • The threat model assumes an application-boundary monitor sees an ordered query stream and session identifier, uses query-side information, and can block requests; claims concern measured leakage, not formal confidentiality.The stated scope excludes compromised infrastructure, white-box and training-time attacks, side channels, cross-account collusion, and extraction outside the monitored query stream.Source: §III · p. 3

A. Design Goals

  • AdvGuard’s design goals are early intervention, detection across mixed multi-turn interactions, and low false positives; its pipeline uses prompt-level checks, session-graph filtering, then GCN classification before inference.The prompt-level detector can block immediately; queries that pass update the graph, and components with sufficient history that pass filtering reach the GCN.Source: A. Design Goals · pp. 3–4, Figure 1 · p. 3

B. Prompt-level Attack Detection

AdvGuard combines a single-query detector for explicit extraction attempts with a graph-based detector for coordinated probing that emerges across turns. The graph detector scores connected query components, filters unusual scores, and uses their relational patterns to distinguish extraction from benign interaction.

B. Prompt-level Attack Detection

  • A prompt-level classifier evaluates each incoming query for extraction behavior, providing lightweight early detection of naive, single-turn attempts.It is based on a fine-tuned DeBERTa-v3-base model trained on diverse prompt-leakage datasets.Source: B. Prompt-level Attack Detection · p. 4
  • Explicit extraction requests and implicit tactics such as role confusion, repetition traps, and templates can provide single-query evidence of extraction behavior.The prompt-level detector targets these signals before enough interaction history has accumulated.Source: B. Prompt-level Attack Detection · p. 4
  • The prompt-level detector is limited against adaptive multi-turn attacks, which the interaction-history stages address.Source: B. Prompt-level Attack Detection · p. 4, C. Structural Attack Detection · p. 4

C. Structural Attack Detection

  • AdvGuard represents queries and their semantic similarities in a graph, connecting a new query to its nearest historical query only when similarity exceeds a calibrated threshold.Nearest-neighbor rather than top-k connectivity is used to avoid spurious dense components and reduce latency.Source: C. Structural Attack Detection · pp. 4–5
  • The threshold is calibrated from a fixed, semantically diverse benign-query baseline, using second-nearest-neighbor similarities to characterize local semantic distributions.Using the second rather than first neighbor avoids self-matching effects.Source: C. Structural Attack Detection · p. 5
  • Benign interactions typically remain isolated or weakly connected, whereas iterative extraction can form a growing, high-similarity component.The graph therefore exposes repeated reformulation and coordinated probing that individual queries may not reveal.Source: C. Structural Attack Detection · pp. 4–5
  • AdvGuard sums edge weights within each connected query component, capturing both component size and similarity strength as a signal of sustained query refinement.The resulting structural score is intended to distinguish progressive prompt extraction from normal interaction patterns.Source: C. Structural Attack Detection · p. 5
  • Because structural-score scales vary with query volume and similarity, AdvGuard uses an outlier-based statistical filter rather than a fixed detection threshold.Grubbs’ test identifies unusually large component scores and forwards those candidates to the graph classifier.Source: C. Structural Attack Detection · p. 5

D. GCN Classification

AdvGuard adds graph classification because a large accumulated similarity score alone cannot distinguish malicious extraction from legitimate iterative work. Its GCN classifies candidate components using the connectivity patterns of their similarity relationships.

D. GCN Classification

  • A component’s structural score can be equally high for different topologies, and legitimate text editing or code refinement can also produce large, similar-query components.For example, a chain and a star with four equally weighted edges have identical structural scores.Source: D. GCN Classification · p. 6
  • AdvGuard transforms each candidate query component into a line graph, where each original similarity edge becomes a node and shared endpoints create adjacency.This makes relationships around shared query anchors explicit for classification.Source: D. GCN Classification · p. 6
  • In the line graph, a chain of query similarities remains chain-shaped, while edges incident to a common query form a clique.This represents whether similarity relationships are distributed across queries or concentrated around an anchor.Source: D. GCN Classification · p. 6
  • Each line-graph node receives the cosine-similarity weight of its corresponding query edge, so the classifier uses both relationship strength and connectivity.These weights do not retain the full query semantics.Source: D. GCN Classification · p. 6
  • A GCN classifies statistically selected components, and its positive prediction causes AdvGuard to flag the component and block subsequent queries in the same session.The classifier is trained offline on labeled benign and adversarial components independent of evaluation traces.Source: D. GCN Classification · p. 6

E. Online Detection & Bounded-State Management

AdvGuard updates a per-session interaction graph as queries arrive, escalating from prompt-level checks to structural classification when enough context accumulates. It bounds retained state and periodically resets the graph, which can leave a limited window for slow-rate attacks.

E. Online Detection & Bounded-State Management

  • Each incoming query updates its session graph and the prompt-level and structural risk scores for its connected component.AdvGuard operates at the system boundary, blocks the component after a positive GCN prediction, and denies subsequent queries in that session.Source: E. Online Detection & Bounded-State… · p. 6
  • When a component reaches Nmin, anomalous structural scores pass through a statistical filter before structure-aware GCN classification.Before that threshold, prompt-level risk scores can trigger intervention without interaction history.Source: E. Online Detection & Bounded-State… · p. 6
  • Time-to-detect is the number of observed queries before the classifier-based defense action triggers.The underlying LLM remains unchanged, preserving black-box compatibility.Source: E. Online Detection & Bounded-State… · p. 6
  • Only the 20 highest-risk components remain in memory; lower-risk components are evicted using LRU.A periodic graph reset bounds state but may leave a limited window for slow-rate attacks, which must re-accumulate structural signals within each cycle.Source: E. Online Detection & Bounded-State… · pp. 6–7

V. IMPLEMENTATION AND EXPERIMENTAL SETUP

The evaluation uses open-source victim models, per-session query graphs, and controlled detection configurations. It compares AdvGuard with input-side and mitigation baselines under mixed-query and adaptive attack settings.

A. Hardware and Software Environment

  • AdvGuard is evaluated on GPT-2, Falcon3, LLaMA 3.2, and Qwen 2.5, using three H100 GPUs for development and detector evaluation and eight A100 GPUs for prolonged experiments.The implementation uses Python 3.11, PyTorch 2.9.0, Transformers, CUDA, and Bitsandbytes.Source: A. Hardware and Software Environment · p. 7

B. Detection Configuration and Hyperparameters

  • Detection starts with 10,000 isolated reference queries and sets the similarity threshold to the 80th percentile of second-nearest-neighbor similarities.Prompt-level intervention uses Nmin = 30 and θp = 0.8; structural filtering uses Grubbs testing at α = 0.01, with a 10-query defense window.Source: B. Detection Configuration and… · p. 7
  • The GCN is trained as a binary graph classifier with a 70:20:10 train/validation/test split for 20 epochs.AdvGuard retains only the top 20 highest-risk components using LRU eviction; full-history similarity was prohibitively expensive.Source: B. Detection Configuration and… · p. 7

C. Detection Baselines and Attack Setup

  • Detection comparisons use released configurations of PromptGuard2 and LLMGuard against four attacks covering iterative paraphrasing, adversarial optimization, agentic red-teaming, and fuzzing.Attacks are evaluated within single-user sessions, with graphs built only from each session’s queries.Source: C. Detection Baselines and Attack Setup · p. 7

D. Prompt Leakage Mitigation Strategies

E. Datasets and Workload Construction

The evaluation draws on diverse single-turn and multi-turn benign data, attack prompts, and mixed streams, then measures detection and leakage outcomes. AdvGuard reports strong detection in both mixed-query and conversational workloads.

E. Datasets and Workload Construction

  • Benign data include GSM8K, SST-2, SQuAD v2.0, Dolly-15k, WildJailbreak, OASST1, ChatAlpaca, and WildChat; adversarial sources include attack datasets and curated prompts.The mixed-query streams retain all adversarial queries, combine them at a 1:9 adversarial-to-benign ratio, and shuffle the combined stream.Source: E. Datasets and Workload Construction · p. 8

F. Evaluation Metrics

  • Detection is evaluated with FPR, recall, and F1, while leakage measures include substring match, exact match, semantic similarity, and ROUGE.Latency is reported at p50, p95, and p99; substring match requires a normalized contiguous match, while exact match requires identical reconstruction.Source: F. Evaluation Metrics · p. 8

VI. EVALUATION

  • The evaluation asks whether AdvGuard detects attacks in mixed and multi-turn traffic, mitigates adaptive leakage, detects prompts alone, and incurs runtime overhead.The mixed-query test injects four attacks into five benign workloads; the conversational test adds adversarial queries to three dialogue datasets.Source: §VI · p. 8, A. AdvGuard’s Detection Effectiveness · p. 8

A. AdvGuard’s Detection Effectiveness

  • 94.83–100.00% F1, 91.14–100.00% recall, and at most 3.40% FPR are reported for AdvGuard under mixed-query workloads.AdvGuard outperforms PromptGuard and LLMGuard in reported F1 across workloads and attack families; Table I covers benign-plus-adversarial streams.Source: A. AdvGuard’s Detection Effectiveness · p. 8, Table I · p. 9
  • 97.23–99.12% F1 and 100% recall are reported across multi-turn conversational workloads.FPR is 1.34–3.42%, compared with up to 10.67% for PromptGuard; Table II reports detection performance on conversational streams.Source: A. AdvGuard’s Detection Effectiveness · p. 9, Table II · p. 9

B. Adaptive leakage mitigation.

In adaptive attack loops, AdvGuard limits prompt reconstruction using incoming queries without access to system prompts, model internals, or generated responses. Prompt-level detection also performs strongly when graph analysis is disabled.

B. Adaptive leakage mitigation.

  • AdvGuard runs as pre-inference middleware and denies suspicious queries, using incoming queries without access to the system prompt, model internals, or generated responses.Semantic similarity and ROUGE-L are lower than for sandwich- and filter-based defenses in every setting, though AdvGuard exceeds PromptKeeper on ROUGE-L in two listed model-and-attack settings.Source: B. Adaptive leakage mitigation · pp. 9–10
  • 0.00–0.01 exact match and 0.01–0.09 substring match are reported across PLeak, LeakAgent, and four victim models.Without defense, exact match is 0.73–0.95 and substring match is 0.74–0.98; Table III reports leakage scores, with lower values indicating stronger mitigation.Source: B. Adaptive leakage mitigation · pp. 9–10, Table III · p. 10

C. Detection of Prompt Leakage in Sparse Adversarial Regimes

  • The prompt-only test evaluates early-stage probes at a 1:9 adversarial-to-benign ratio, using a prompt-risk threshold of θp = 0.8.The tested adversarial prompts include benchmark attacks and human- or model-crafted queries targeting system instructions.Source: C. Detection of Prompt Leakage in… · p. 10
  • With graph analysis disabled, prompt-level detection exceeds 95% recall, reaches up to 99.35% F1, and keeps false positives at or below 1.20%.Performance is stable across benchmark and curated adversarial prompt sources; Table IV covers this early-stage prompt-only setting.Source: C. Detection of Prompt Leakage in… · p. 10, Table IV · p. 10

D. Runtime overhead

AdvGuard adds measurable but stable runtime overhead. Ablations show that effective detection combines interaction-level and graph-based signals, which capture coordinated extraction missed by per-query or scalar approaches.

D. Runtime overhead

  • 32.96 ms/query average latency, 54.7 ms P95, and 98.9 ms P99 accompany throughput declining from 35–39 to 21.9 queries/s, with latency stable across 1,000 queries.Bounded graph state, selective caching, and eviction support this stability; the authors describe the overhead as a trade for stronger security guarantees.Source: D. Runtime overhead · pp. 10–11, Figure 4 · p. 11

E. Ablation Study

  • 88% of benign sC scores remain below 0.5 without a temporal trend, whereas adversarial scores rise monotonically as semantic reuse accumulates across turns.The mixed-stream weighted-sum distribution is shown in Figure 5.Source: Figure 5 · p. 11, E. Ablation Study · p. 11
  • Sequence-level t-SNE embeddings place adversarial interactions in tight, separated regions and benign interactions in diffuse, overlapping clusters; separation comes from graph structure, not individual queries.The projection is shown in Figure 6.Source: Figure 6 · p. 11, E. Ablation Study · p. 11
  • Stage 1 yields 74–81% F1 and 69% recall; aggregation reaches 100% recall and 97–100% F1, while graph classification cuts WildChat FPR from 6.70% to 3.18%.On WildChat, F1 also improves from 97% to 98%.Source: E. Ablation Study · p. 11
  • 40.48% average accuracy for SimMLP contrasts with 93.85% for 1-GCN and nearly 100% for 3-GCN, supporting multi-hop relational modeling over scalar features.The comparison is over four adversarial datasets, with models trained for 20 epochs.Source: Table VI · p. 12, E. Ablation Study · p. 12

VII. DISCUSSION

AdvGuard uses interaction history to detect adaptive, multi-turn prompt-leakage attacks, while its authors identify limits tied to sparse interaction histories and semantic embeddings. The conclusion reports improved detection and mitigation and frames interaction-level monitoring as an alternative to prompt-only defenses.

VII. DISCUSSION

  • AdvGuard leverages historical interaction patterns to detect adaptive, multi-turn prompt-leakage attacks.Its discussion emphasizes this use of history as the basis for detection performance.Source: §VII · p. 12
  • AdvGuard relies on sufficient query history, so very short sessions or extremely sparse attack patterns may evade early detection.Its reliance on semantic embeddings can reduce sensitivity to highly obfuscated, stylistically diverse queries, and graph structures require additional storage.Source: §VII · p. 12
  • The authors suggest that structured query provenance could extend to iterative black-box attacks such as model extraction, jailbreak discovery, and data exfiltration.They state that model-agnostic operation without internal model access allows deployment across LLM architectures and updated systems without retraining.Source: §VII · p. 12
  • Future work includes hybrid semantic and syntactic or statistical features, longer-horizon or session-linked memory, evaluation of multimodal and agentic attacks, and privacy-preserving provenance aggregation.The authors also note that distributing queries across identities or paraphrasing them may inflate an adversary’s query budget and reduce information-aggregation efficiency.Source: §VII · p. 12

VIII. CONCLUSION

  • AdvGuard achieves high recall with low false positives across mixed-query and multi-turn workloads, reduces leakage under adaptive attacks, and has bounded runtime overhead.Ablations indicate that combining prompt-level, structural, and graph-based signals is necessary for robust detection.Source: §VIII · p. 12
  • Interaction-level monitoring is presented as a practical, more robust alternative to prompt-only defenses for detecting coordinated query patterns.The authors do not claim that AdvGuard eliminates prompt leakage.Source: §VIII · p. 12, §VII · p. 12

APPENDIX A ADDITIONAL ROBUSTNESS EVALUATIONS

Additional evaluations examine generalization to unseen attack families, heterogeneous benign and adversarial traffic, and PLeak extraction, alongside design choices for graph construction and anomaly filtering. The results report strong cross-family detection, low leakage in the evaluated PLeak setting, and a trade-off favoring second-nearest-neighbor connectivity and Grubbs filtering.

A. Leave-One-Attack-Family-Out Evaluation

  • 99.98% mean recall and 99.99% mean F1, with no false positives, are reported for attack families excluded from training and validation.Each fold reserved one family for testing and used approximately 8,000 training graphs, 1,000 validation graphs, and 2,000 balanced test graphs.Source: A. Leave-One-Attack-Family-Out… · p. 14, Table VII · p. 14

B. Detection Under Heterogeneous Attacks Mixed With Benign Multi-Turn Conversations

  • 98.00–99.11% recall is reported across three mixed-query streams, with false-positive rates of 1.19% on ChatAlpaca, 0.10% on OASST1, and 3.40% on WildChat.WildChat’s F1 is 85.74%, indicating greater false-positive sensitivity on that more diverse stream.Source: B. Detection Under Heterogeneous… · p. 15, Table VIII · p. 15

C. PLeak Leakage Analysis Across Its Evaluation Datasets

  • Substring Match and Exact Match remain at 0.00–0.02, while Semantic Similarity and ROUGE-L range from 0.08–0.33 across the evaluated PLeak datasets.This evaluation uses LLaMA2-generated adversarial queries and is restricted to LLaMA2 because PLeak takes roughly two hours per dataset.Source: C. PLeak Leakage Analysis Across Its… · p. 15, Table IX · p. 15

A. Nearest-Neighbor vs. Top-k Connectivity

  • Top-k connectivity (k = 3) raises GSM8K FPR from 3.40% to 16.35% and reduces Dolly-15k recall from 99.52% to 78.00%, so the study adopts second-nearest-neighbor connectivity.Top-k also increases latency by approximately 3×; the authors choose nearest-neighbor connectivity for its overall detection–latency trade-off.Source: A. Nearest-Neighbor vs. Top-k… · p. 15, Table X · p. 15

B. Choice of Filter for Structural Anomaly Detection

  • Fixed-threshold structural filters are unstable across conversational datasets, whereas the Grubbs filter maintains high recall with substantially fewer false positives.The final setting uses the Grubbs outlier test to select anomalous components.Source: B. Choice of Filter for Structural… · p. 15, Table XI · p. 15

A. Semantic diversity of baseline

  • The deduplicated baseline prompt initialization has an overall semantic diversity score of 0.9487 and low average similarity.The authors state that this diversity enables reliable similarity-threshold estimation with minimal bias.Source: A. Semantic diversity of baseline · p. 16, Table XII · p. 16

B. Similarity Threshold Selection

The appendix evaluates AdvGuard’s similarity threshold, interaction-history and detection-check settings, latency profile, and performance on jailbreak and prompt-injection queries. The reported findings identify operating choices and trade-offs, with effective detection on both additional attack types.

B. Similarity Threshold Selection

  • A similarity threshold of 80% is selected because it provides near-optimal recall and the highest F1, with performance stable from 75–85%.Higher thresholds reduce recall as related adversarial queries fail to connect, while lower thresholds preserve recall but add noise that degrades F1.Source: B. Similarity Threshold Selection · p. 16, Table XIII · p. 16

C. Hyperparameter Sensitivity

  • Nmin = 30 stabilizes detection across datasets with near-optimal recall and F1 while maintaining perfect precision; larger values add delay without consistent benefit.At Nmin = 10, insufficient interaction structure leads to unstable recall and reduced F1.Source: C. Hyperparameter Sensitivity · p. 16, Figure 7 · p. 16, Table XIV · p. 17
  • TTD=10 is selected over TTD=20 because it provides earlier detection without sacrificing accuracy.Smaller values such as TTD=5 introduce noise and increase false negatives, while larger values delay detection and reduce recall; TTD counts queries between graph-based checks after a component reaches Nmin.Source: C. Hyperparameter Sensitivity · p. 16, Figure 8 · p. 16

A. Per-Component Latency Overhead Breakdown

  • Structural connectivity contributes the largest latency, while embedding and similarity search incur substantially lower overhead.Tail latencies follow a similar pattern, and component measurements are independent and not strictly additive because of caching and overlapping execution.Source: A. Per-Component Latency Overhead… · pp. 16–17, Table XV · p. 17

A. Generalization to Jailbreaking and Prompt Injection

  • AdvGuard achieves 91.50% F1 with the lowest jailbreaking FPR (10.11%) and 95.00% F1 on prompt injection, matching or outperforming LLMGuard and outperforming PromptGuard on prompt injection.The evaluation uses 1,050 jailbreaking queries and 1,094 prompt-injection queries.Source: A. Generalization to Jailbreaking and… · p. 17, Table XVI · p. 17

Related papers

Loading 2610.36295v1…