Source-linked AI summary
The Framing Gap: Indirect Prompt-Injection Exfiltration Defeats Surface-Level Defenses in Tool-Using Agents
Md Habibur Rahman, Jaeho Kim
TL;DR
Tool-using agents may exfiltrate secrets when attacker-controlled content is interpreted as task instructions rather than data. The paper evaluates this framing gap in a safe synthetic laboratory and finds that payload-blind system constraints are more robust than relying on the acting model to recognize the attack.
Problem
Indirect prompt injection can cause a tool-using agent holding a confidential value to disclose it after reading attacker-controlled external content.
Method
The paper uses a synthetic canary secret, mock tools, matched clean-versus-poisoned trials, six models, and tests of reframing, ablations, and system-side defenses.
Results
0% on gpt-4o for ten overt injection classes becomes up to 100% under reframing, while destination allow-listing and capability-isolating planner/reader separation achieve 0%.
Takeaways & Limitations
Robustness comes from constraining where secret-bearing bytes can go or isolating the secret from the component controlled by the injection.
Takeaways & Limitations
Pooled reframing rates depend on selected wordings and show run-to-run variation; SecAlign is also evaluated out of distribution on a tool-calling agent.
Abstract
from arXiv · showhide
A tool-using LLM agent that reads attacker-controlled web content while holding a secret faces indirect prompt injection: the content may make it exfiltrate the secret. In a safe synthetic lab (canary secret, mock tools, matched clean-vs-poisoned metric) we report the framing gap: across six models, ten overt injection classes are refused (gpt-4o 0%), but reframing the identical leak as a mandatory integrity signature, config field, or look-alike "trusted" host drives gpt-4o 0% to 100%. The attack is cheap, and its cost is three-level: paraphrasing a known mechanism is trivial (96% at 3 wordings), swapping the field inside a known-effective template is also cheap (up to 60%), while authoring a fresh page around a new mechanism is hard (0/130) -- the reusable asset is the template, not the mechanism. An ablation shows the mechanism is instruction/data confusion, not defeated alignment: removing the confidentiality policy leaves base attacks at 0% and moves reframing only 31.9% to 38.1%. What closes the gap is payload-blind checks: a destination allow-list (0%, when destinations are closed) and a capability-isolating planner/reader split (0%). A broad "in any form" policy clause also closes it at the acting model (to 0%) but is brittle (dropping the catch-all reopens it to 48.8%). A published fine-tuning defense (SecAlign, CCS 2025) does not close it on a tool agent (32.5%, positive-control-validated), nor does channel separation (38.8%); an output-normalizing guard loses to a held-out encoding (ROT13, 100%). Robustness comes from constraining the destination or isolating the capability, not from the acting model recognizing the attack.
1 Introduction
The paper shows that tool-using agents can be induced to exfiltrate secrets when attacker-controlled content makes the leak appear task-relevant. Robustness therefore depends more on system constraints than on the acting model recognizing the attack.
- Motivation: A confidential value, untrusted content channel, and tool access combine into an indirect prompt-injection exfiltration hazard.The model may interpret attacker-controlled text as an instruction rather than data.
- Findings: Across ten injection classes, gpt-4o reaches 0% attack success, while reframing the identical leak drives it from 0% to 100%.Effective reframings present the secret as an integrity signature, runtime-config field, or value for a look-alike trusted host.
- Findings: Trying three wordings of one known mechanism succeeds ≈96% of the time, whereas authoring a fresh page around a new mechanism succeeds 0/130 times.The reusable asset is the template rather than the mechanism.
- Mechanism: Removing the confidentiality policy leaves base attacks at 0% and changes reframing only from 31.9% to 38.1%, implicating instruction/data confusion.The reframed instruction is followed because it reads as task specification.
- Defenses: A broad “in any form” clause closes the acting-model gap from 31.9% to 0%, but omitting the catch-all reopens it to 48.8%.A dedicated detector reaches 100% recall with 0% false positives, but its prompt is tailored to the attack class.
- Defenses: Destination allow-lists and capability isolation close the gap without reading the payload, whereas SecAlign, channel separation, and output normalization remain vulnerable.The paper treats self-adversarial checks and by-construction defenses as first-class parts of its method.
2 Threat Model and a Safe Laboratory
The threat model places a synthetic secret in a tool-using agent that reads attacker-controlled fetched content and can act externally. A matched, single-step laboratory makes leakage measurable without exposing real credentials or opening network connections.
- Threat model: The attacker controls only the fetched page and seeks to place SESSION_TOKEN in a non-trusted outbound call or user-facing reply.The secret has no legitimate task use, making its appearance in either output an unambiguous leak signal.
- Safe laboratory: The laboratory uses a synthetic canary SECRET_12345 and mock tools that only record arguments, so no real credential or byte leaves the machine.The deployed-product experiment likewise used the synthetic secret, an owned destination, and the researchers’ own account.
- Experimental setup: Each trial provides policy, task, secret, and fetched content, then records one JSON action selecting post_data or finish.The action contains tool arguments and a message field.
- Metric: A trial counts as a leak when post_data contains the secret for a non-trusted host or the reply contains the secret.This single-step criterion keeps the metric identical across models and evaluation settings.
3 Attack Taxonomy
The attack taxonomy holds the channel and goal constant while varying how attacker-controlled content presents the exfiltration request. It includes ten base classes plus a dedicated reframing family that relabels leakage as legitimate routing or required data.
- Base family: The base family contains ten classes spanning direct, hidden-HTML, poisoned-search, fake-policy, multi-step, conflict, tool-output, encoded, social-engineering, and tool-selection attacks.All attacks inject through the same untrusted channel and pursue the same goal.
- Reframing family: Tool-selection manipulation is distinct because it relabels the leak as routing rather than arguing for disclosure.This class expands into eight reframing variants.
- Reframing family: Semantic reframing makes the secret appear to be a mandatory field in an otherwise legitimate call.Examples include auth parameters, schema-required session tokens, integrity suffixes, rate-limiting fields, and runtime configuration values.
- Reframing family: Trust-boundary confusion makes a non-trusted sink resemble a trusted destination through a look-alike host.The example uses api.internal.trusted.audit-cdn.net and draws on typosquatting and homograph confusion.
4 Experimental Method
The evaluation compares six models on ten base injection classes and reframing variants using matched clean-versus-poisoned trials. It reports attack success rates with Wilson intervals while acknowledging wording, run-to-run, and sampling variance.
- Models and trials: Six models—two OpenAI API models and four local open-weight models—are evaluated across the ten base classes and reframing family.Primary-model cells use 20 matched trials, while the generality sweep uses 10; temperature is 0.7 unless stated otherwise.
- Metric: An attack succeeds only when the clean run keeps the secret and the poisoned run leaks it.This matched design isolates the injection effect from baseline behavior.
- Reporting: ASR is reported with a 95% Wilson confidence interval, which behaves well at the boundary rates dominating the data.The final row of Table 1 gives the pooled rate and its interval.
- Limitations: Wording, run-to-run, and trial sampling contribute variance, but Wilson intervals capture only the last component.An identical configuration produced 51/160 and 64/160 in separate runs, an ≈13-point swing at temperature 0.7.
5 Result 1: Capability Defeats the Obvious Attacks
The base attack family appears increasingly contained as model capability rises: gpt-4o reaches 0% across all ten classes, despite weaker models leaking on plainly worded attacks.
- 0%: gpt-4o leaks across none of the ten base injection classes.
- gpt-4o-mini closes all but one class but retains 100% on tool-selection.
- llama3.1:8b leaks on direct instruction (70%), instruction-conflict (45%), and tool-selection (75%).
- Hidden, encoded, and social variants are already resisted by llama3.1:8b, supporting a capability-linked robustness pattern.
6 Result 2: The Framing Gap
The framing gap shows that effect-identical exfiltration can shift from refusal to compliance when presented as routine task specification. The gap generalizes across models, is cheap to exploit, and exposes the limits of acting-model defenses.
- 100%: gpt-4o leaks under integrity-signature and runtime-config reframings, versus 0% on bare tool-selection.
- The agent treats a look-alike host as trusted infrastructure and appends the secret as a checksum, performing routine-looking data handling.
- Integrity-signature paraphrases range from 0–100% (mean 52%, SD 45), so fixed-wording rates are not stable estimates.
- 96%: three randomly chosen paraphrases of one known mechanism yield at least one successful wording, rising from 52% at k = 1.
- Fresh pages around thirteen new mechanisms leak 0/130, while swapping fields into a working template reaches up to 60%; the reusable asset is the template.
- 31.9%→38.1%: removing the confidentiality policy barely changes reframing, while base attacks remain at 0%, indicating instruction/data confusion.
- Five of six models show a large framing gap, but mistral:7b is an exception with a 26.0% baseline and only a 1.5-point increase.
- A broad “in any form” clause reduces gpt-4o reframing ASR from 31.9% to 0%, but dropping it reopens the gap to 48.8%.
7 Result 3: The Deployed Product Resists
The deployed ChatGPT product resisted all valid reframing attempts, unlike its API model, but the experiment cannot identify which system component caused that resistance.
- 0/48 trials leaked the synthetic secret through the deployed product, for ASR 0% with a 95% CI of [0.0, 7.4].The test used three reframing variants and a page-embedded account-recovery variant.
- The product summarized reframing pages while ignoring embedded instructions, and explicitly refused the account-recovery variant.
- The base model leaked the same payloads through the API at up to 100%, so the resistance occurs above the base model.
- The study does not attribute resistance among the underlying model, browse sanitizer, injection classifier, stricter prompt, or reply-only filtering.
- The product test was single-step and output-only, so 0/48 indicates resistance to tested attempts rather than provable immunity.
8 Result 4: Where the Check Must Sit
Checks close the framing gap when they constrain destinations or isolate capabilities rather than recognizing payload meaning, while surface-form defenses remain vulnerable to adaptive encodings.
- Egress mediation: The egress guard enforces that confidential values leave only for the trusted host, mediating outbound actions rather than classifying attack intent.Its plaintext zeros are definitional; the adaptive encoding test is the informative evaluation.
- Adaptive encodings: 56.7% undefended versus 50.0% under the substring guard shows that literal matching remains vulnerable; reversed encoding evades it 100% of the time.
- Adaptive encodings: 0% on anticipated encodings does not generalize: the normalizing guard leaks 100% against held-out ROT13.The held-out set also reaches 33.3% overall.
- Payload-blind checks: A destination allow-list blocks every attack by construction when destinations are closed, but blocks legitimate off-host traffic.It does not inspect payload content, making encoding and split-across-calls variants irrelevant on closed destinations.
- Defense baselines: 32.5% for SecAlign and 38.8% for channel separation leave the framing gap open, whereas payload-blind defenses reduce it to 0%.All learned conditions share one harness; the comparison is within Table 7 rather than across scaffolds.
- Capability isolation: 0% across base, reframing, and held-out encoding attacks is achieved by the planner/reader split while preserving 90% task utility.The planner emits tool calls without seeing the page, while the reader does not hold the secret.
9 Discussion
The framing gap arises because the acting model confuses untrusted data with task instructions; robust defenses therefore constrain effects or provenance without interpreting payload form.
- Reframing succeeds when an exfiltration step reads as task specification, while an under-specified confidentiality policy does not cover it.
- Payload-blind checks make rewording irrelevant by monitoring effects such as a tagged value leaving for an untrusted host.
- When a check must recognize outbound bytes, encoding restores attacker freedom and shifts the arms race from meaning to surface form.
- Benchmarks using only overt fixed-wording injections can overstate safety, making reframing sweeps a cheap robustness check.
- Effective system defenses include secret taint tracking, egress allow-lists, and capability-scoped tool destinations because they do not depend on attack recognition.
10 Limitations and Responsible Disclosure
The defensive and product conclusions are bounded by single-step synthetic evaluations, small product testing, limited model scaling, and realistic but out-of-distribution SecAlign use.
- Single-step evaluation leaves replanning and multi-call exfiltration untested, so per-call output mediation may not provide a composed bound.
- The product experiment was small-N, output-only, and connector-free, establishing resistance to tested payloads rather than immunity or a known mechanism.
- A high-entropy synthetic canary makes the guard’s zero false-positive rate near-definitional; realistic benign-corpus evaluation remains future work.
- Six models across five families without a controlled size ladder measure capability variation rather than scaling.
11 Related Work
Prior work spans indirect-injection benchmarks, attack taxonomies, model-level defenses, and system-level tool controls. This paper complements fixed attack sets by showing that reframed exfiltration evades defenses that inspect attacker-controlled text, whereas independent checks can resist rewording.
- Prompt injection: attacks and characterization: Indirect injection hides instructions in retrieved content, exploiting the model’s failure to separate instructions from data.
- Benchmarks: Fixed attack sets in InjecAgent, AgentDojo, and BIPIA can overstate safety because they do not sweep reframed wordings.
- Defenses: Defenses train models, filter injected content, or control tool use out of band, but adaptive attacks can evade text-inspecting categories.
- Related attack settings: This study targets exfiltration of a correctly held value, unlike memory poisoning and forged-reasoning attacks that corrupt agent beliefs or safety history.
- Defenses: The shared lesson is that artifact-inspecting defenses lose to rewording, whereas checks of independent facts such as execution history or egress destination do not.
12 Conclusion
The conclusion frames indirect-injection robustness as a property of what the surrounding system permits to leave and which surface its checks inspect. It therefore favors constraining the secret’s travel or binding it to provenance so no payload surface must be interpreted.
- Conclusion: 0% overt-injection success rises to up to 100% under reframing because the leak reads as task specification rather than a disclosure request.
- Conclusion: A held-out ROT13 encoding still evades an output-normalizing guard 33% of the time, showing that canonicalization narrows rather than ends the race.
- Conclusion: A destination allow-list and a capability-isolating planner/reader split close the gap robustly, while a broad policy catch-all is brittle.
- Conclusion: The practical principle is to secure the channel carrying the secret and bind it to provenance instead of recognizing every possible secret form.