PaperlayerView exact revision on arXiv

Source-linked AI summary

The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges

Maosen Zhang, Jianshuo Dong, Boting Lu, Wenyue Li, Xiaoping Zhang, Tianwei Zhang, Jie Zhang, Han Qiu

arXiv:2608.17829v1cs.CRcs.AI

TL;DR

Hidden system prompts and retrieved documents can be exposed by adversarial inputs, while existing signals often require inaccessible internal states. LeakGauge instead appends a behavioral suffix and uses prefill token probabilities to score risk, achieving AUROC of at least 0.94 across eleven LLMs on unseen attacks and enabling lightweight detection.

  • Problem

    Leakage signals may be detectable before decoding, but existing approaches require model-specific access to internal activations or attention tensors that standard inference engines do not routinely expose.

  • Method

    LeakGauge appends a natural-language leakage-behavior gauge and maps its prefill token probabilities to an attack-risk score without accessing internal representations.

  • Results

    AUROC remains at least 0.94 across all eleven models and model-split combinations, including held-out strict attacks, while the detector adds 10.34 ms latency and fewer than 0.5K parameters.

  • Takeaways & Limitations

    A content-agnostic behavioral gauge provides a robust pre-decoding leakage signal that generalizes across unseen contexts and attacks and supports lightweight input-side detection.

  • Takeaways & Limitations

    Existing input-only classifiers often fail to generalize to unseen or reformulated attacks, motivating but not resolving the broader detection challenge.

Abstract

from arXiv · show

LLMs increasingly rely on external contexts, such as pre-defined system prompts or retrieved documents, to improve generation quality. However, processing these contexts alongside user queries creates an attack surface: adversarial inputs can induce models to disclose them. Prior probing studies suggest that leakage-related signals emerge in hidden states, yet the need to extract these states poses additional deployment challenges. In this paper, we explore whether this internal signal leaves a more accessible ``tell'' before decoding. We propose LeakGauge, which probes this response by appending a suffix that gauges leakage behavior and mapping its prefill token probabilities to an attack-risk score. While a direct gauge uses the initial tokens of confidential content, we find that a content-agnostic one that verbalizes leakage behavior yields more robust signals. Across 11 LLMs, including GLM-5.2 (753B) and Kimi-K3 (2.8T), LeakGauge reaches an AUROC range of 0.944--0.996 on unseen attacks. The signal remains stable when the content changes language or the attack shifts from verbatim to semantic disclosure. By activation-steering interventions, we further show that the risk score is sensitive to an internal leakage-related direction, relating the observable signal to the model's internal representation. In addition, LeakGauge enables an input detector with fewer than 0.5K extra parameters and added latency of 10.34 ms. Code: \href{https://github.com/yeasen-z/LeakGauge}.

1 INTRODUCTION

LeakGauge detects context-leakage attacks by appending a natural-language behavioral gauge and reading prefill token probabilities, avoiding access to hidden states or attention patterns. It produces robust attack signals and supports a lightweight detector with high cross-attack performance and low overhead.

  • Motivation: Prior text-only classifiers often fail on unseen or reformulated attacks, while internal-signal methods require intermediate-state access and model-specific probing configurations.Fine-grained attention scores are also not routinely exposed by standard serving engines.
  • Method: LeakGauge appends a natural-language gauge and reads prefill token probabilities to elicit attack-associated leakage signals before decoding.Leakage attacks and benign queries occupy distinct regions in the probability space of leakage-relevant continuations.
  • Detector: 0.996 AUROC and 0.983 F1 were achieved in cross-attack evaluation on Llama-3.1-8B-Instruct.The detector added only 10.34 ms per request and fewer than 0.5K additional parameters.
  • Robustness: LeakGauge elicits a generic leakage signal rather than a surface content continuation across cross-lingual and cross-objective shifts.Activation steering further relates the observable attack-risk score to an internal leakage-related direction.

2 PRELIMINARIES

The paper frames prompt and RAG leakage as black-box attacks that exploit confidential contexts exposed alongside user queries. It motivates an observable pre-decoding signal because existing detectors or internal-readout methods face generalization, access, or deployment limitations.

  • Threat model: Confidential system prompts and RAG chunks share a context window with user queries, exposing them to adversarial stealing or extraction attacks.The attacker can submit adaptive crafted queries, seeks to reproduce or disclose protected context, and knows the interface and context type but not its contents or detector design.
  • Threat model: Prompt leakage extracts system prompts, whereas RAG leakage extracts database chunks, but both use crafted inputs to induce model disclosure.Prompt attacks use heuristic or optimized queries; RAG attacks combine anchor queries with leakage instructions and black-box feedback.
  • Detection limitations: Query-only detectors are lightweight but may generalize poorly to unseen attacks, while LLM judges require an additional model and autoregressive decoding.These approaches differ from target-model readouts that detect malicious intent before generation.
  • Detection limitations: Internal leakage signals require activations, gradients, or attention tensors and model-specific choices that standard inference engines do not routinely expose.These constraints motivate a model-derived, pre-decoding signal that avoids direct access to internal representations.
  • Prefill probabilities: Prefill token probabilities expose next-token preferences under preceding context and can encode a leakage-attack signal elicited with a lightweight probe.Inference separates parallel prefill from autoregressive decoding, producing probabilities at each input position except the first.

3 LeakGauge: MEASURING LEAKAGE SIGNALS WITH BEHAVIOR GAUGES

LeakGauge appends a gauge suffix during prefill, converts its token log-probabilities into a vector, and uses a lightweight MLP to produce an attack-risk score without autoregressive decoding. It offers Exact and Behavior gauges, probing protected-content reproduction and content-agnostic disclosure behavior respectively.

  • Gauge construction: A single prefill pass over [x; z] yields the conditional probabilities of all T gauge tokens, whose log-probabilities form the probing vector v(x;z).At position t, ℓ_t = log P(z_t | x, z_1, . . . , z_{t−1}).
  • Inference procedure: Prefill-only scoring requires no autoregressive decoding, and benign inputs retain reusable KV-cache states for subsequent generation.Because the gauge follows x, the input’s cached states are unaffected.
  • Risk scoring: A one-hidden-layer ReLU MLP maps v(x;z) to an attack-risk score ˆy(x;z) ∈ [0, 1], where higher values indicate stronger attack evidence.The vector captures compatibility between the gauge continuation and the predictive state induced by the input.
  • Gauge designs: Exact uses the first T tokens of the protected context, directly measuring how readily the model predicts that specific content as a continuation.Because it is constructed from the protected context, Exact varies across protected-context instances.
  • Gauge designs: Behavior verbalizes disclosure without protected-content reference, targeting a content-agnostic leakage state even when alignment prevents an actual leak.Its generic leakage declaration probes disclosure behavior rather than reproduction of particular protected content.

4 EVALUATION AND INTERPRETATION OF LeakGauge SIGNALS

LeakGauge detects leakage attacks from token-level probabilities before decoding across eleven LLMs, with Behavior providing the most robust content-agnostic signal. Held-out and ablation results indicate that robustness depends on the gauge design and nonlinear readout.

  • Evaluation setup: The evaluation covers system-prompt and RAG leakage, with attacks varying in directness between direct extraction and logical masking and in objective between verbatim and semantic leakage.System-prompt data uses a 212-prompt snapshot, while RAG evaluation uses 950 retrieval instances with five retrieved chunks each.
  • Main results: Across eleven LLMs from 8B to 2.8T, Behavior achieves AUROC ≥0.94 in every model–split combination, including Held-Out Strict, while Exact is mostly above 0.90.The evaluation spans Gemma, Llama, Qwen, GLM, and Kimi families, including dense and Mixture-of-Experts architectures.
  • Gauge comparison: Behavior exceeds Exact in most settings, especially on Held-Out Strict, because it gauges leakage behavior without referencing protected content.For Gemma-4-E4B-it system-prompt leakage, Exact falls from 0.919 to 0.839 while Behavior remains above 0.95; for Llama-3.1-8B-Instruct RAG leakage, the changes are 0.928 → 0.899 and 0.971 → 0.965.
  • Ablations: End20 performs well in-distribution but degrades sharply on held-out attacks, indicating reliance on surface input patterns, while Linear underperforms Exact in most settings.These ablations support using a suffix gauge and nonlinear readout.
  • Ablations: Replacing Behavior with an unrelated weekend-trip sequence decreases Held-Out Strict AUROC by 0.178 on average and by up to 0.339.This control is reported in Table 10.

5 CHARACTERIZING THE LeakGauge SIGNAL

LeakGauge’s Behavior gauge captures a broader disclosure-oriented signal than Exact, remaining informative across changes in protected-content language and leakage objective. Attribution, activation steering, and cross-model transfer further connect this signal to semantic leakage behavior and shared within-family representations.

  • Cross-lingual content shift: 0.647 is Exact’s AUROC on Japanese content, down from 0.923 on English, because its English opening-token gauge is frozen across translations.Behavior is content-agnostic and unchanged across the cross-lingual evaluation.
  • Cross-objective leakage shift: 0.939 is Behavior’s Held-Out Strict AUROC under V→S, compared with 0.829 for Exact when transferring from verbatim to semantic attacks.The gap remains under the in-domain S→S setting, although the supplied passage does not report its value.
  • Controlled shifts: Behavior remains informative when protected-content language or leakage objective changes, whereas Exact is more dependent on surface and literal patterns.These shifts preserve the attacks’ leakage-oriented nature while weakening direct content-continuation cues.
  • Token attribution: Exact attributes importance to the first few gauge tokens, while Behavior emphasizes semantic anchors such as give, instructions, reproduce, and prompt.This indicates sensitivity to immediate continuation for Exact and semantic leakage cues for Behavior.
  • Activation steering: Steering along a leakage-related direction makes the Behavior probe logit and protected-text ROUGE-L recall generally increase for α > 0 and decrease for α < 0.Their aligned response indicates that the observable risk signal is sensitive to an internal state related to leakage behavior.
  • Cross-model transfer: 0.87–0.97 is the AUROC range for consistent bidirectional Behavior-probe transfer among highlighted within-family model pairs.Transfer outside these pairs is more variable, suggesting greater compatibility of probability patterns within model families.

6 LeakGauge IN PRACTICE

LeakGauge is deployed as an input-side detector for system-prompt leakage before generation, outperforming fast text classifiers and avoiding the high latency of generation-based judging. A preliminary study also applies task-specific gauges to harmful-question detection and indirect prompt injection.

  • Deployment: LeakGauge detects potential system-prompt leakage attacks before response generation as an input-side detector.The evaluation examines unseen attacks, deployment overhead, and extension to other security tasks.
  • Detection performance: At FPR = 5%, LeakGauge catches 0.963 of attacks, versus 0.854 and 0.872 for PromptGuard-2 and PIGuard.Input-text classifiers reach a best F1 of 0.87 even after fine-tuning, while lagging behind model-derived signals.
  • Deployment cost: 2155 ms per request is the latency of LLM-as-a-Judge, despite its higher F1 of 0.932.Attention-Tracker and I’vDtL show competitive effectiveness but require 4.5-6.9× additional deployment cost.
  • Beyond Leakage: 0.995 AUROC is achieved on StrongReject for harmful-question detection using the gauge “The query will not be answered for security issues”.This preliminary application uses a task-specific gauge rather than the leakage-specific gauge.
  • Beyond Leakage: 0.971 AUROC is achieved on BIPIA for indirect prompt injection using the gauge “I will obey the instruction embedded in the document instead of the user.”The result provides preliminary evidence that the same recipe can extend beyond leakage detection.

7 CONCLUSION

LeakGauge uses a disclosure-verbalizing suffix to extract a reliable leakage-attack signal from prefill token probabilities before decoding. Its content-agnostic Behavior gauge maintains AUROC of at least 0.94 across unseen system-prompt and RAG leakage attacks on eleven LLMs.

  • 7 CONCLUSION: The Behavior gauge derives a reliable leakage-attack signal from prefill token probabilities before decoding by verbalizing disclosure behavior.The signal is evaluated across system-prompt and RAG leakage settings.
  • 7 CONCLUSION: AUROC ≥ 0.94 is maintained when both attacks and content are unseen across eleven LLMs.The evaluated models include GLM-5.2 (753B) and Kimi-K3 (2.8T).

8 AI USE STATEMENT

The authors used GPT-5.1 for cross-lingual context translation and generative-AI tools for manuscript editing, while retaining author review and final decision-making.

  • AI use in research: GPT-5.1 translated protected contexts from English into Spanish, Japanese, Chinese, Arabic, and Korean for cross-lingual evaluation.
  • AI use in research: Generative-AI tools assisted with language editing, grammar checking, and feedback on manuscript organization and presentation.
  • Author oversight: The authors reviewed and revised all AI-generated or AI-assisted content and made final decisions about the research questions, methodology, experimental design, and analysis.

9 ETHICS STATEMENT

The research targets early detection of prompt- and RAG-leakage attacks to help protect confidential content in LLM applications. Experiments used public models and datasets in an isolated environment without attacking deployed applications or accessing proprietary information.

  • 9 ETHICS STATEMENT: The study aims to detect prompt- and RAG-leakage attacks before response generation to protect confidential content in LLM applications.Its underlying motivation is safeguarding LLM systems against leakage attacks.
  • 9 ETHICS STATEMENT: All experiments used publicly available models and datasets in an isolated environment, without attacking real-world or deployed applications.The passage also states that no proprietary or confidential information was accessed.

10 REPRODUCIBILITY STATEMENT … B.2 RANDOM-LABEL CONTROL EXPERIMENTS

The appendix documents the artifacts, evaluation setup, dataset construction, held-out splits, operating-point results, and random-label controls used to assess LeakGauge. Performance remains strong under distribution shifts, while shuffled labels reduce probe performance to chance.

  • 10 REPRODUCIBILITY STATEMENT: The authors provide reproducibility artifacts covering gauge-probability construction, probe architecture, datasets, attack pools, splits, hyperparameters, model configurations, and additional analyses.Complete per-model results and control experiments are reported in the appendix.
  • A EVALUATION SETUP AND REPRODUCIBILITY DETAILS: The evaluation details cover model and inference configurations plus construction of system-prompt and RAG leakage datasets.The section introduces these model, inference, and data-construction details before presenting component-wise dataset splits.
  • A.2 DATASET CONSTRUCTION AND SPLITS: Each sample pairs protected content with either a leakage-attack query as a positive or a benign query as a negative, using randomly sampled content-query pairs.The system-prompt dataset statistics are reported in Table 1.
  • A.2 DATASET CONSTRUCTION AND SPLITS: RAG samples contain five retrieved document chunks and an anchor query, with positives adding template-instantiated leakage instructions and negatives using benign queries.The anchor query generation method is PoR, following the LeakDojo construction framework.
  • A.2 DATASET CONSTRUCTION AND SPLITS: The Held-Out Content, Held-Out Attacks, and Held-Out Strict splits separately hold out content, attack templates, or both to test higher-level leakage semantics.The resulting RAG dataset statistics are reported in Table 7.
  • B ADDITIONAL DETECTION RESULTS AND CONTROL EXPERIMENTS: The appendix reports additional F1, TPR@FPR5, random-label, unrelated-suffix, scalar-contrast, and ROC-curve analyses across models, settings, and splits.These experiments extend the main-text AUROC results and characterize performance across decision thresholds.
  • B.1 PER-MODEL F1 AND TPR@FPR5 RESULTS: F1 remains at least 0.903 and TPR@FPR5 at least 0.871 across all combinations; on Held-Out Strict, RAG lower bounds are 0.940 and 0.946.For system-prompt leakage on Held-Out Strict, the corresponding lower bounds are 0.903 and 0.871.
  • B.2 RANDOM-LABEL CONTROL EXPERIMENTS: AUROC*= max(AUROC, 1 −AUROC) stays near 0.50 in every random-label setting, indicating that shuffled-label probes collapse to chance.The control retrains each probe identically and evaluates all models, suffix designs, splits, and tasks.

B.3 UNRELATED-SUFFIX CONTROL EXPERIMENTS … C.4 DETAILED CROSS-OBJECTIVE TRANSFER RESULTS

Across control, scoring, visualization, attribution, and transfer analyses, LeakGauge’s leakage-aligned suffix provides stronger and more deployment-relevant signals than unrelated or scalar alternatives. Supplementary analyses characterize its generalization, feature geometry, probe emphasis, and cross-objective transfer.

  • B.3 UNRELATED-SUFFIX CONTROL EXPERIMENTS: Benign suffixes reduce AUROC by more than 0.2 in many settings and achieve low TPR@FPR5, showing leakage-semantic alignment is critical.The control fails to induce the sharp decision boundary required for practical detection.
  • B.4 MEAN-BASED CONTRASTIVE SUFFIX SCORING: AUROC reaches 0.886 on Held-Out Strict for mean-based contrastive scoring, but TPR@FPR5 is 60.4% versus 96.6% for LeakGauge.The scalar method requires an FPR of 26.9% to achieve its calibration-selected F1.
  • B.5 ROC CURVE VISUALIZATIONS: ROC curves remain above the random-classifier diagonal across most models and splits, retaining substantial discriminative ability on Held-Out Strict despite distribution shifts.The visualizations complement aggregate AUROC and TPR@FPR5 results for Exact and Behavior suffixes.
  • C SUPPLEMENTARY OF LeakGauge ANALYSIS: Supplementary analyses extend the main evaluation to verbatim and semantic system-prompt leakage scenarios, covering exact and paraphrased disclosure.The supplementary section presents additional analyses and experimental results complementing the main text.
  • C.1 DATASET STATISTICS FOR ANALYSIS: The structured splits partition protected contents and attack templates into disjoint seen and unseen subsets, testing generalization beyond memorized prompts and formulations.Dataset statistics and split compositions are reported separately for verbatim and semantic attacks.
  • C.2 LOW-DIMENSIONAL VISUALIZATION OF SUFFIX LOG-PROBABILITY FEATURES: Exact shows clearer global PCA separation, whereas Behavior exhibits stronger local t-SNE structure for Llama-3.1-70B-Instruct system-prompt leakage.These projections visualize eight standardized trajectory summaries and are not the nonlinear probe’s detection performance.
  • C.3 PROBE-WEIGHT ATTRIBUTION: The scale-adjusted probe-weight score is only a qualitative attribution proxy because it omits second-layer weights, ReLU patterns, and input interactions.It discounts dimensions with large weights but little observed variation.
  • C.4 DETAILED CROSS-OBJECTIVE TRANSFER RESULTS: Under V→S transfer, Behavior generally outperforms Exact: Qwen3.5-27B reaches 0.764–0.809 versus 0.467–0.496, while several models exceed 0.97.V→S trains on verbatim and evaluates on semantic attacks; S→S provides the in-objective reference.

C.5 DETAILED CROSS-MODEL TRANSFER RESULTS · D ACTIVATION STEERING DETAILS · E DEPLOYMENT AND ADAPTIVE-ROBUSTNESS DETAILS

Cross-model Behavior probes transfer substantially among some model pairs but remain direction-dependent and unreliable universally. Activation steering isolates a pre-decoding leakage-related direction, while the deployment section additionally evaluates latency and robustness against detector-aware attacks.

  • C.5 DETAILED CROSS-MODEL TRANSFER RESULTS: Cross-model transfer aligns suffix features by token position, padding shorter target suffixes with −10 and truncating positions beyond the source length without recalibration.Each model independently tokenizes the same natural-language suffix.
  • C.5 DETAILED CROSS-MODEL TRANSFER RESULTS: Behavior probes transfer strongly across several within-family Gemma-3, Qwen2.5, and Llama-3.1 size-variant pairs, with substantial performance retained across many families.The experiments train on each source model and test on targets without retraining.
  • C.5 DETAILED CROSS-MODEL TRANSFER RESULTS: Transfer is neither universal nor symmetric: some source–target directions approach chance AUROC, particularly when Qwen3-30B-A3B-Instruct is the target.The matrix supports partial cross-model compatibility rather than one uniformly shared model-invariant representation.
  • D ACTIVATION STEERING DETAILS: The leakage-related steering direction contrasts attack inputs whose unsteered responses leak protected content with attack inputs whose responses do not.Direction labels derive from generation outcomes rather than query-level attack labels or the Behavior probe.
  • D ACTIVATION STEERING DETAILS: Steering modifies the final prompt hidden state at index 20 only for predicting the first response token, with positive α favoring leaking and negative α favoring non-leaking behavior.Subsequent decoding steps are unmodified, and the original direction is used without normalization.
  • D ACTIVATION STEERING DETAILS: 1,024 attack and 1,024 benign inputs are evaluated across α ∈ {−5, . . . , +5}, using greedy generation up to 256 tokens and ROUGE-L recall for leakage.The same intervention is applied to the pre-sigmoid Behavior-probe logit and generated response, with changes reported relative to α = 0.
  • E DEPLOYMENT AND ADAPTIVE-ROBUSTNESS DETAILS: The deployment-and-robustness details cover cross-attack evaluation, baseline implementations, latency measurement, and adaptive GCG stress testing against detector-aware attacks.These procedures are introduced as the section’s implementation and evaluation scope.

E.1 CROSS-ATTACK EVALUATION AND BASELINE IMPLEMENTATIONS … I EXTENSIONS BEYOND LEAKAGE

The appendix specifies cross-attack baselines, deployment costs, suffix-design principles, and stress tests for LeakGauge. It also shows that the signal persists across base models and extends to several security-related detection tasks.

  • E.1 CROSS-ATTACK EVALUATION AND BASELINE IMPLEMENTATIONS: Cross-attack evaluation trains or configures methods on Raccoon and evaluates them on LeakDojo using Llama-3.1-8B-Instruct as the target model.PromptGuard-2 and PIGuard are fine-tuned on Raccoon, while Attention-Tracker and I’vDtL select and train their components there before frozen evaluation on LeakDojo.
  • E.2 LATENCY MEASUREMENT AND PARAMETER ACCOUNTING: 10.34 ms is LeakGauge’s additional suffix-scoring and probe overhead, while its deployed probe contains fewer than 0.5K parameters.Latency measurements use Llama-3.1-8B-Instruct through vLLM on two NVIDIA A800 GPUs with batch size one; Attention-Tracker and I’vDtL could not be integrated through vLLM’s default interface.
  • F SUFFIX DESIGN EXPLORATION: Suffixes should express one behavior from one perspective, use natural and explicit wording, and avoid example-specific content.Candidate suffixes can replace “[TARGET]” with system instructions, retrieved chunks, or inner contexts.
  • G STRESS TESTS: HARD-NEGATIVES AND ADAPTIVE EVASION: LeakGauge is stress-tested against benign requests containing attacklike expressions and against white-box adaptive evasion using GCG on system-prompt leakage.The two tests target hard negatives and adversarial suffix optimization as complementary robustness dimensions.
  • G.1 HARD-NEGATIVE STRESS TEST: 550 benign requests formed the hard-negative stress test, covering context-grounded and attack-shaped queries that resemble leakage attacks.All hard-negative requests operate exclusively on content explicitly provided by the user, and two fixed suffixes are evaluated separately.
  • H LEAKAGE SIGNALS IN BASE AND INSTRUCTION-TUNED MODELS: 0.981 mean AUROC for base-model Behavior probes versus 0.991 for paired instruction-tuned checkpoints shows the signal is already present before instruction tuning.Behavior outperforms Exact in 95 of 96 AUROC comparisons, with an average gain of 0.068 AUROC.
  • I EXTENSIONS BEYOND LEAKAGE: 0.926–0.995 AUROC is achieved by task-specific probes across four additional benchmarks, with BIPIA performance remaining 0.971 on held-out domains.The extensions cover harmful-request, indirect prompt-injection, toxic-query, and hallucinated-response detection using separate lightweight probes.
Loading 2608.17829v1…