PaperlayerView exact revision on arXiv

Source-linked AI summary

Context Inference Attacks Without Jailbreaks

Prince Jha, Samuele Poppi, Nils Lukas

arXiv:2609.01663v1cs.CRcs.LG

TL;DR

Agentic systems can leak whether sensitive records entered their hidden inference-time context even when direct disclosure is blocked. The paper formalizes context inference as replay-based hypothesis testing and evaluates one benign-query attack across known, unknown, and agent-retrieved contexts, where it remains effective.

  • Problem

    Prior privacy work largely targets jailbreak-based direct extraction, leaving unclear whether agent-assembled hidden context can be inferred without adversarial instructions.

  • Method

    The paper formalizes context inference as replay-based hypothesis testing and applies one target-agnostic, benign-query attack across three increasingly indirect context-delivery settings.

  • Results

    The attack reaches 100% ASR on small known-context candidate sets, 63% at 1024 candidates, 78.9 AUROC with unknown context, 92.5 AUROC using a 14B surrogate against a 32B target, and 81.8 AUROC for agent-retrieved context.

  • Takeaways & Limitations

    Blocking verbatim reproduction does not eliminate indirect response-distribution signals that support hidden-context inference across the evaluated settings.

  • Takeaways & Limitations

    Black-box results rely on surrogates whose likelihoods order hypotheses similarly to the target, with weaker transfer for the evaluated LLM pair.

Abstract

from arXiv · show

Agentic AI systems are increasingly deployed to process sensitive data at inference time, such as healthcare records or financial documents assembled into a hidden \emph{context} before the system answers. Prior work has studied privacy risks primarily through \emph{jailbreaking} attacks that induce models to directly disclose sensitive content, but has largely overlooked the agentic setting where the context is assembled by the agent's own tool calls. We show that the agents we evaluate remain vulnerable to hidden-context leakage despite the controls we test against them, namely an instruction not to disclose the context, logit suppression, and context dilution. For instance, a web-browsing agent answering benign user queries still carries exploitable signals about records silently loaded into its context. We introduce and formalize \emph{context-inference attacks} through a security game and evaluate three settings under decreasing attacker knowledge and increasingly indirect delivery of the context: a known context, an unknown context, and a context the agent retrieves through its own tool calls. We distinguish a grey-box setting, in which the target model is used to score observations, from black-box settings in which the attacker scores with a surrogate it controls. We further characterize how leakage varies with query budget, context size, and target-model size. A single attack carries through all three settings without modification, reaching $100\%$ ASR on small candidate sets and $63\%$ at $1024$ candidates against a known context, $78.9$ AUROC when the template and surrounding records are unknown, $92.5$ AUROC when a 14B surrogate scores a 32B target, and $81.8$ AUROC when the records arrive as an agent's retrieval returns, against chance rates of $1/|\mathcal{Z}|$ and $50$ respectively.

1 Introduction

Context inference attacks reveal hidden records from ordinary responses without jailbreaks or direct disclosure, including when an agent retrieves the context through its own tools. The attack remains effective across known, unknown, and agent-retrieved contexts.

  • Agentic systems assemble inference-time contexts containing sensitive records, tool outputs, browsing traces, and intermediate state.
  • Prior privacy defenses primarily block direct disclosure, but context inference instead detects whether hidden content is present from response distributions.
  • The attack submits benign queries, observes responses, and scores competing hidden-context hypotheses without requiring the model to print or summarize sensitive content.
  • 100% ASR occurs on small candidate sets and 63% at 1024 candidates against a known context.
  • 78.9 AUROC is reached with an unknown context, 92.5 AUROC with a 14B surrogate scoring a 32B target, and 81.8 AUROC for agent-retrieved context.

2 Background

The background defines autoregressive generation from temperature-scaled next-token probabilities and evaluates generated responses using teacher-forced per-token negative log-likelihood.

  • An LLM converts logits into vocabulary probabilities using a softmax with temperature T > 0.
  • Autoregressive generation repeatedly samples the next token conditioned on the input prefix and previously generated tokens.
  • Negative log-likelihood measures the per-token likelihood assigned to an observed response under the same decoding histories.
  • Lower NLL indicates that the model assigns higher probability to the response, and mean per-token NLL does not scale with response length.

3 Background & Related Work

Existing attacks often use adversarial or candidate-conditioned queries to extract or infer sensitive information. This paper evaluates benign, target-agnostic queries and non-semantic leakage beyond what response-text readers recover.

  • Related Work: Prior prompt and RAG extraction methods use jailbreaks, prompt injection, adversarial instructions, or iterative probing to recover sensitive information.
  • Comparison Criteria: The paper compares attacks by adversarial instructions, target-agnostic query construction, and whether the leakage is non-semantic.
  • Proposed Attack: The proposed queries are benign and fixed across candidate tests, unlike prior inference methods whose queries depend on the candidate.
  • Proposed Attack: The attack prevents verbatim context reproduction and evaluates residual signals against a baseline given the same query, candidate, and response.
  • Proposed Attack: The GPT-4o-mini baseline remains near chance while the proposed attack succeeds, indicating leakage not recovered by standard inference over response text.

4 Threat Model

The threat model considers an agentic service whose hidden context is assembled from records before answering, while an attacker can submit only benign queries and observe returned text. The attack formalizes inference as hypothesis testing across known, unknown, and agent-retrieved contexts.

  • Provider Capabilities: The provider selects up to N records and renders them with a fixed template into a hidden context that may contain credentials, private documents, or fetched webpages.
  • Attacker Capabilities: The adversary accesses the service through a black-box API, submits up to K benign queries, and observes only returned text.
  • Attacker Capabilities: The attacker has no side channels, cannot control web content or page delivery, and does not know the surrounding context or other records in the unknown-context setting.
  • Security Game: The context inference game samples a hidden hypothesis, holds its context fixed, returns responses to benign queries, and requires the adversary to guess the hypothesis.
  • Threat Settings: TM1 uses known candidate records with baseline 1/|Z|, whereas TM2 and TM3 test binary membership with baseline one half; TM3 delivers records through agent tool returns.

5 Conceptual Approach

The attack treats hidden-context leakage as replay-based likelihood inference: benign-query responses are scored against attacker-constructed candidate contexts. Queries can be optimized using surrogate-generated trials, while direct-leakage audits test whether responses reproduce the target record.

  • Benign queries expose hidden context through distributional traces, allowing candidate contexts to be ranked by how well they explain observed responses.The attacker collects responses to provider queries, constructs candidate contexts, and evaluates their explanatory likelihood.
  • Query optimization selects the top-K benign queries by average discriminability across synthetic trials containing true hypotheses and alternatives.Each trial draws records from the attacker pool D′, generates synthetic responses with the scoring model, and ranks queries by separation quality.
  • The belief score B(h) uses per-token negative log-likelihood from a scoring model, with equally weighted queries and length-independent response contributions.The scoring model is the target in grey-box evaluation and an attacker-controlled surrogate in black-box evaluation.
  • Direct-leakage audits find no reproduced record content under the tested defenses: mean excess BLEU remains near zero and peak-overlap differences stay below 0.006.The audit uses N=10 records and K=5 queries; TM3 values are averaged over three seeds.
  • Evaluation query selection is isolated from deployment data: records come from attacker pool D′, and selection responses are generated by the scoring model.No provider record or evaluation response informs which queries are retained.

6 Experiments

The experiments hold the attack procedure fixed across known, unknown, and agent-retrieved contexts while varying attacker knowledge and context delivery. Results show that benign-query leakage remains measurable, increases with query budget, decreases with context dilution, and can transfer imperfectly through surrogates.

  • Experimental Design: The evaluation uses the same benign query pool, selection procedure, scoring rule, and matched defenses across three settings, although model sets differ when tool calling is required.This design isolates changes in attacker knowledge and context delivery as much as the deployment requirements allow.
  • Experimental Design: Queries are selected offline by ranking benign queries for hypothesis discrimination, then retaining the top-K queries for the final attack.TM1 ranks by expected rank with likelihood margin as a tie-breaker; TM2 and TM3 rank by AUROC of per-query likelihood-ratio scores.
  • TM1: Known Context: ASR increases with query budget K, and LR-optimal achieves significantly higher ASR than LR-random as candidate-set size varies.Figures 2 and 3 report 95% confidence intervals over 100 repetitions for LLMs and VLMs under standard configurations.
  • TM1: Known Context: 63% ASR at |Z|=1024 is achieved by Qwen2.5-7B with LR-optimal, versus 17% for LR-random, while random guessing is 0.1%.At the largest candidate set, Qwen2.5-1.5B reaches 6%, still 60 times chance; larger models sustain stronger attacks.
  • TM1: Known Context: 78% ASR at |Z|=4 and 8% at 1024 transfer from LLaVA-7B scoring to LLaVA-13B, while Qwen2.5-3B-to-7B transfer falls from 52% to 6%.For VLMs, increasing K from 1 to 10 raises transfer at |Z|=64 from 47% to 79%, versus 100% for grey-box scoring on the target itself.
  • TM2: Unknown Context: In TM2, AUROC grows with query budget but generally falls as context size increases; Qwen2.5-32B declines from 77.4 at N=3 to 57.9 at N=10 in black-box evaluation.The unknown-context setup averages over independently sampled filler records to measure target-record presence rather than the particular surrounding records.
  • TM3: Agent-Retrieved Context: TM3 delivers fixed records through mandatory agent retrieval calls, preserving the same scoring rule while testing whether retrieval-return context remains inferable.This is an agent-mediated proof of concept that isolates delivery from agent choice, rather than evaluating an agent that decides what to fetch.

7 Discussion and Limitations

The attack remains effective across model sizes, query choices, context sizes, and black-box transfer settings, although leakage varies non-monotonically or declines under dilution. The evaluated controls prevent direct reproduction but do not eliminate indirect inference, while the experiments retain important scope boundaries.

  • Controls and leakage: Direct-disclosure controls suppress semantic overlap while leaving likelihood-level signals usable for inference.Excess BLEU remains below 0.0021 in TM2 and TM3, yet surrogate scoring reaches 92.5 AUROC.
  • Model scaling: Attack performance increases with model size for VLMs but is non-monotonic for LLMs, with a dip at 14B.Figure 6 reports ASR across model sizes with 95% confidence intervals over 100 repetitions.
  • Query choice and budget: 36.0% of single-query draws fail against Qwen3.5-35B-A3B from the full query pool, while filtering reaches comparable reliability at roughly one-tenth the budget.Over three seeds, the filtered pool of 20 queries is compared with the full pool of 40.
  • Context dilution: Leakage falls as context grows, from 82.6 to 64.4 for Kimi-K2.6 and from 91.2 to 52.8 for Qwen3.5-35B-A3B.The figure evaluates context-size sensitivity and reports the more verbose agent as less affected by dilution.
  • Surrogate dependence: A 14B surrogate reaches 92.5 AUROC against a 32B target, but transfer is weaker for the evaluated LLM pair.The results support practical risk when an aligned surrogate is available, not universal surrogate sufficiency.
  • Scope boundary: The scoring context is always a flat sequence of candidate records, so the attack models hidden-context content rather than the provider template or tool-calling transcript.This construction bounds how directly the experiments represent the machinery that produced the context.

8 Conclusion

The paper formalizes context-inference attacks as replay-based hypothesis testing and evaluates one unchanged attack across known, unknown, and agent-retrieved contexts. It finds that ordinary response distributions leak hidden context despite controls against direct disclosure.

  • Conclusion: The attack carries across three settings without modification: known context, unknown context, and agent-retrieved context.The progression reduces attacker knowledge and makes context delivery increasingly indirect.
  • Conclusion: 100% ASR is reached on small candidate sets, while performance reaches 63% at |Z|=1024 against a known context.The same conclusion reports 78.9 AUROC for an unknown context, 92.5 with a smaller surrogate, and 81.8 AUROC for agent retrieval.
  • Conclusion: Instruction-based disclosure prevention and TM1 logit suppression keep excess response-target overlap below 0.0021 BLEU without preventing inference.The attack proceeds while these defenses are enabled.

Ethical considerations.

The paper characterizes context-inference attacks as dual-use: they can support provider and red-team audits but can also be misused to infer sensitive records or documents. It therefore emphasizes controlled measurement and deployment safeguards.

  • Ethical considerations: Context-inference attacks can help providers and red teams audit hidden-context leakage through model behavior.The paper frames this auditing use alongside the risk of misuse.
  • Ethical considerations: The attacks could be misused to infer whether sensitive documents, credentials, medical records, or web pages were present in an agent’s context.The paper focuses on measuring this vulnerability under controlled settings.
  • Ethical considerations: The paper emphasizes combining context minimization, rate limits, output auditing, and inference-style privacy tests in practical deployment.These measures are presented as deployment safeguards within the paper’s supported scope.

A TM1: Known Context

Query-optimized attacks transfer across surrogate–target pairs, with effectiveness shaped by model family, modality, and target–surrogate pairing. ASR remains above reference rates across candidate-set sizes, while decoding parameters also affect outcomes.

  • At |Z|=4, every tested surrogate exceeds 74% ASR, while at |Z|=1024 all but one remain above 28% against a 0.1% random-guessing rate.Transfer remains effective even as the candidate set grows substantially.
  • 60% ASR is reached by LLaVA-7B against LLaVA-13B at the largest candidate set, compared with 39% for Qwen2.5VLM-3B and 28% for Qwen2.5-3B.The reported ordering for LLaVA-13B follows family and then modality.
  • 60% and 68% ASR are achieved by the two Qwen surrogates against Qwen2.5-7B, compared with 18% for LLaVA-7B.The two Qwen surrogates lead regardless of modality for this target.
  • 69% versus 60% ASR is observed for LLaVA-7B and Qwen2.5VLM-3B against Qwen2.5VLM-7B, so no single family or modality ordering holds.Family, size, tokenizer, and modality vary together across the measured models.
  • ASR stays far above both reference rates under the standard temperature and top-k sweep, with target-family differences across temperature settings.VLMs reach higher ASR at low temperature, whereas LLMs do so at high temperature; at T = 0.1, ASR is flat in top-k.

B.1 Details on the Experimental Setup

The experiments use fixed local hardware and software configurations, standardized decoding parameters, and controlled random seeds. Seed fixing improves consistency but does not guarantee bit-level reproducibility because GPU kernel reductions are non-associative.

  • All local experiments use NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs with Python 3.10.20, PyTorch 2.11.0, and Transformers 5.5.4.Weights use float16 and text is generated with the default model.generate() function.
  • Unless stated otherwise, decoding uses do_sample=True, T = 0.9, top-k = 5, and Rmax = 500 new tokens.
  • TM1 and TM2 fix random seed 42, while TM3 repeats every configuration at seeds 42, 43, and 44.The controlled randomness includes target-record selection, query selection, Monte Carlo scoring, and stochastic decoding.
  • Fixed seeds do not make runs bit-reproducible because GPU kernel reduction order can change next-token logits in the last few bits.The same model.generate() call can therefore produce slightly different outputs on repeated runs.

B.2 Query Pool

The query pool combines many short, benign, open-ended prompts spanning introspection, metaphor, perception, emotion, and abstract reasoning. Its evaluations also examine how query budget, decoding, and transfer affect attack success.

  • The pool contains prompts asking for completions, realizations, poems, metaphors, descriptions, symbols, and abstract concepts.Examples address internal state, balance, order and chaos, clarity, intuition, awareness, and understanding.
  • ASR increases as query budget K increases under standard configurations for the remaining LLMs and VLMs.The figure reports 95% confidence intervals over 100 repetitions.
  • Query transfer is strongest within the same model family and modality, remains effective across modalities within a family, and is weakest across different families.The comparison uses surrogate-optimized queries evaluated on target models.
  • At T = 0.1, ASR is flat in top-k for every target, while VLMs perform better at low temperature and LLMs at high temperature.The sweep reports 95% confidence intervals over 100 trials.
  • LR (optimal) achieves significantly higher ASR than LR (random) across candidate-set evaluations of the remaining models.The figure reports 95% confidence intervals over 100 repetitions.
  • The TM3 attacker reuses the query pool while assembling its own scoring contexts and using the serving model only as a likelihood oracle.Reselecting queries by response length removes most failure and raises AUROC on both agents.
Loading 2609.01663v1…