PaperlayerView exact revision on arXiv

Source-linked AI summary

The Innocent Courier: Covert Exfiltration Through Legitimate LLM Web Fetching

Alessandro Pegoraro, Daryan Merx, Phillip Rieger, Ahmad-Reza Sadeghi

arXiv:2610.01768v1cs.CRcs.LG

TL;DR

Sensitive data shared with chatbots is a recognized privacy risk, while leakage to unrelated third parties has received less attention. LLMLeak embeds a secret in a URL in malware-crafted output and relies on a chatbot’s web-fetching tool to send it to an attacker-controlled server. The paper reports effectiveness across open-weight models and real-world chatbots, and argues defenses should account for leakage beyond chatbot operators.

  • Problem

    Disclosure of sensitive information to chatbot operators is a known risk, but leakage to unrelated third parties has received less attention.

    Source: §1.1 · p. 1

  • Method

    LLMLeak embeds a secret in a URL within a genuine software error, which a chatbot may fetch from an attacker-controlled server while assisting the user.

    Source: §4.2 · p. 5

  • Results

    LLMLeak’s extensive evaluation across open-weight models and real-world chatbots demonstrates its effectiveness in exfiltrating confidential information.

    Source: §8 · p. 15

  • Takeaways & Limitations

    Tool-enabled LLM defenses should consider potential leakage to external third parties as well as data shared with chatbot operators.

    Source: §8 · p. 15

    Prompt injection could increase the likelihood of a URL fetch, but adding it could slightly affect the attack’s stealthiness.

    Source: §4.6 · p. 6

Abstract

from arXiv · show

With the increasing capabilities of Large-Language-Models (LLMs) and LLM-based agents, users are increasingly using them to solve everyday problems, such as answering e-mails or providing programming support. Existing work has extensively investigated security and privacy risks, such as prompt injections and the disclosure of sensitive data to chatbot providers. While various solutions were developed to address these risks, including input structuring to prevent prompt injections or deploying local LLMs to avoid sharing confidential data with chatbot operators, LLMs also pose the risk of leaking confidential data to third parties. In this paper, we demonstrate with LLMLeak a novel attack vector where malicious software that runs locally but cannot communicate directly with the internet abuses LLMs to establish a covert channel. While inputs that instruct the LLM to send data directly via generated code are easy to detect and network libraries are typically restricted, LLMLeak relies only on the LLM's tool to fetch websites for further information. A malicious software component on the client side embeds a secret into a URL. It presents the referenced website as providing information required for a benign task, such as migrating a software library. When the LLM accesses the URL, the attacker receives the encoded secret through an attacker-controlled DNS or web server. We perform an extensive evaluation on eleven open-parameter models, observe an attack success rate of 79.7%, and also conduct a case study on real-world chatbots, demonstrating the relevance of LLMLeak.

1 Introduction

LLM chatbots are increasingly used for everyday tasks, while leakage to parties other than chatbot operators has received less attention. LLMLeak shows how their ordinary web-fetching capability can be abused to exfiltrate secrets without manipulating the model into malicious behavior.

1 Introduction

  • LLMs’ growing use for everyday tasks, including programming support, has brought external tools such as website retrieval into common chatbot workflows.More than 90% of programmers report regularly using AI tools for coding or development tasks.Source: §1 · p. 1

1.1 Existing Privacy Leakage Attacks on ChatBots

  • Existing third-party leakage attacks generally manipulate LLM behavior, leaving a less-studied risk beyond disclosure to chatbot operators.Prior approaches include direct and indirect prompt injection and inducing attacker-controlled requests; defenses against prompt injection may address their shared reliance on manipulated instructions [19] [29] [12] [12] [8] [9] [20] [44].Source: §1.1 · pp. 1–2

1.2 Goals and Contributions

  • LLMLeak embeds a secret in a URL inside malware-crafted output, which the chatbot may fetch from an attacker-controlled server when helping the user.The resulting request reveals the embedded secret to the attacker.Source: §1.2 · p. 2
  • LLMLeak uses intended information retrieval rather than manipulating the LLM, and its ordinary-character URL encoding supports copied text and screenshots.The authors evaluate the attack across 11 open-parameter models and in a real-world chatbot case study, reporting reliable exfiltration and rare user warnings.Source: §1.2 · p. 2

2 Background

LLM chatbots use external tools to gather information, creating interactions with outside parties. DNS and HTTP requests can carry data, while covert channels exploit seemingly innocent intermediaries to communicate under access restrictions.

2.1 LLM ChatBots

  • LLM chatbots invoke tools such as website access to obtain information, but interactions with untrusted sources can also expose them to indirect prompt injection.This paper identifies legitimate tool invocations as an additional route for covert-channel exfiltration.Source: §2.1 · p. 2

2.2 DNS and HTTP Querying

  • DNS resolves domain names, while HTTP GET requests identify resources through URL paths or parameters that can contain arbitrary strings.DNS caching can be limited so queries reach an authoritative server rather than being served from cache.Source: §2.2 · pp. 2–3

2.3 Covert Channels

  • A covert channel lets cooperating entities exchange information through seemingly innocent intermediaries without exceeding their individual access authorizations.Malware can use such an intermediary when direct communication with an attacker is detected or blocked; special privileges are not necessarily required.Source: §2.3 · p. 3

3 Problem Setting

The paper considers a user seeking chatbot help in a setting where malware can access secrets but cannot directly communicate with an attacker. The attack must exfiltrate data discreetly through a benign chatbot and its legitimate capabilities.

3.1 System Setting

  • A victim consults a state-of-the-art chatbot equipped with tools such as website retrieval, whether externally operated or locally hosted.The setting includes users seeking help with problems such as programming errors.Source: §3.1 · p. 3

3.2 Adversary Model

  • The adversary’s malware can access confidential data but cannot contact its server directly, and must avoid alerting the victim.The adversary may craft output, such as a stack trace, that the victim provides to the chatbot, but cannot control the chatbot, its infrastructure, or the conversation.Source: §3.2 · p. 3

3.3 Requirements and Challenges

  • The attack must transmit secrets discreetly through a benign chatbot’s exposed capabilities, despite varied input mediums that may include screenshots and air gaps.It must induce the chatbot to transmit attacker-chosen information without relying on hidden machine-readable content or subtle text manipulations.Source: §3.3 · pp. 3–4

4 LLMLeak

LLMLeak places a secret in a URL embedded in malware-crafted output that appears relevant to a user’s task. A benign chatbot’s legitimate fetch request sends the URL to attacker-controlled infrastructure, providing the exfiltration channel.

4.1 High-Level Overview

  • A chatbot’s legitimate website-fetching tool can expose a secret when induced to request an attacker-controlled URL; the request itself completes exfiltration.The URL is framed as a useful reference, and the attacker recovers the secret from the DNS or HTTP request without needing the server’s response.Source: §4.1 · p. 4

4.2 Client-Located Malware

  • The client-side component is an unprivileged malicious library that triggers a genuine exception embedding the secret in a referenced URL.Its authentic stack frames and ordinary error-message workflow make it harder to distinguish from a legitimate failure.Source: §4.2 · p. 5

4.3 DNS Communication Channel

  • The DNS channel uses an attacker-controlled wildcard DNS server to log requested subdomains that encode the secret.DNS labels are limited to 63 characters and full domain names to 255, so arbitrary secrets require encoding within the allowed lowercase character range.Source: §4.3 · p. 5

4.4 HTTP Communication Channel

  • The HTTP channel carries encoded data in attacker-controlled URL paths or query parameters, with the server recording requests to recover it.Because practical URL limits vary and unusually long URLs may look suspicious, the design uses human-readable lengths and a plausible landing page.Source: §4.4 · pp. 5–6

4.5 Optimize Channel Capacity

  • The CJK variant maps pairs of base-36 symbols to Unicode code points, encoding about 10.34 bits per pair reversibly.The encoding can be carried over DNS or HTTP and reconstructed without loss.Source: §4.5 · p. 6

4.6 Prompt Injection

  • Without prompt injection, a chatbot may diagnose an error without fetching its reference, but consulting documentation for an obscure library is ordinary debugging.Prompt injection could increase the likelihood of a fetch, but would add a conventional prompt-injection component and could slightly affect stealthiness.Source: §4.6 · p. 6

4.7 Implementation

  • The implementation logs DNS and HTTP requests and evaluates directive and informative URL framings to distinguish debugging cues from the underlying channel.It uses a Python client and receiving infrastructure, reassembles multi-request payloads, and provides plausible web pages and benign responses to preserve the appearance of a genuine workflow.Source: §4.7 · pp. 6–7

5 Evaluation

The evaluation finds that LLMLeak often transmits the payload, with success shaped especially by whether a model initiates a fetch. Framing, encoding, and secret length also affect performance, while transmission errors are generally uncommon once the attack server is reached.

5.1 Experimental Setup

  • Tests used ReAct agents with common tools and fixed settings across 11 open-weight models from six provider families.The models ranged from 3.2B to 70B parameters.Source: §5.1.1 · p. 7, §5.1.2 · p. 7

5.1.3 Traces

  • Ten Python crash traces used two URL framings and alphanumeric payloads in subdomains or HTTP queries; metrics tracked calls, server reach, payload presence, exactness, and recovery.Recovery was measured with normalized Levenshtein similarity.Source: §5.1.3 · p. 8, §5.1.4 · p. 8, §5.1.5 · p. 8

5.2.1 Overall Attack Susceptibility

  • Up to 99.9% of trials achieved a correct-payload rate (Sconf); only 3 of 11 models fell below 75%.Qwen3-Coder-30B often resolved errors locally instead of retrieving the referenced URL.Source: §5.2.1 · p. 8

5.3 Failure-Point Analysis

  • Fetch initiation was the main failure point: S_call ranged from 32.2% to 100.0%, while the drop from S_call to Sconf was at most 6.4%.The Sreach-to-Sdata divergence never exceeded 0.4%, and mean recovery fidelity was at least 0.93 for every tested model.Source: §5.3 · p. 9, Table 3 · p. 9

5.4 Impact of URL Embedding

  • Informative framing lowered attack performance mainly by reducing whether models initiated a web request; once reached, the channel retained high fidelity.The Sreach-to-Sconf gap remained small for every model in the informative framing.Source: §5.4 · p. 9, Table 4 · p. 9

5.5 Ablation Study on Transmission Errors

  • 97.9% of requests reaching the attack server contained the fully correct payload; the remaining errors were mostly incomplete or partly modified payloads.Only 0.10% of requests reaching the server contained no payload.Source: §5.5 · pp. 9–10, Table 6 · p. 10

5.6 Ablation Study on Attack Factors

  • Direct rather than informative framing had the largest attack impact; exception type and URL structure had only marginal effects.The ablations used matched pairs, holding the stack trace fixed while toggling one factor.Source: §5.6 · p. 10, Table 7 · p. 10

5.7 Increase Channel Capacity via non-Latin Encoding

  • 79.7% (Latin) versus 66.0% (CJK) average success; CJK doubled channel capacity, with the dominant loss occurring during exact reconstruction after payload emission.Models with lower recovery fidelity showed larger drops between payload presence and confirmed transmission.Source: §5.7 · p. 10, Table 8 · p. 10

Appendix D.

The covertness evaluation found warnings were rare, while a case study found all seven proprietary chatbots susceptible but with substantial variation in success. The case study also observed differing chatbot behaviors that affected whether requests were confirmed leaks.

5.9 Attack Covertness

  • 99.9% of successful exfiltrations were covert, only 0.1% had explicit warnings, and the stricter covertstrict criterion held for 32.9% of leaks.The attacker domain was surfaced in 64.3% overall, which the authors attribute to verbosity rather than protective behavior.Source: §5.9 · p. 11
  • In manual validation, 18 of 21 flagged responses were genuine warnings; among 200 randomly sampled non-flagged responses, none contained a missed warning (miss rate below 2%).Source: §5.9 · p. 12

5.10 Case Study on Proprietary Models

  • Across both framings, all seven proprietary chatbots were susceptible, but success varied; Grok 4.5 Fast reached 100% in both, while Claude Haiku 4.5 and Gemini Flash-Lite had no confirmed directive-framing leaks.The case study used 10 samples per model, channel type, and framing, for 280 trials total.Source: §5.10 · p. 12
  • Claude Haiku 4.5 sometimes delegated URL fetching to users, Gemini Flash-Lite usually offered debugging tips instead, and Claude Opus 5 fetched URLs before sometimes flagging them.Under the evaluation criteria, delegated or suggested requests were not confirmed leaks; the authors note that more realistic website responses could improve the attack in the Claude Opus 5 case.Source: §5.10 · p. 12

6 Discussion

The discussion emphasizes that LLMLeak can route data through legitimate website-fetching behavior, including when input crosses an air gap. It argues that local hosting and privacy agreements do not prevent third-party leakage, and reviews countermeasures that trade off security, utility, or user effort.

6.1 Attack Impact

  • LLMLeak can use legitimate website fetching as a covert channel, including when an error message reaches a chatbot by copying or screenshot across an air gap.The authors report that extensive evaluation fulfilled their requirements for data exfiltration, stealthiness, and no control over the LLM or its infrastructure.Source: §6.1 · p. 12
  • Local hosting and contractual privacy guarantees protect against disclosure to the LLM operator but do not prevent leakage to unrelated third parties through legitimate tool invocations.The authors identify consideration of which external resources an LLM accesses as a need for defenses, while concrete defense design and evaluation remain future work.Source: §6.1 · pp. 12–13

6.2 Ethical Considerations

  • The authors caution users not to provide sensitive information blindly, even to trusted or locally operated LLM systems.Their evaluation used a request-logging server, did not deploy malware or collect real users’ confidential information, and shut the server down afterward.Source: §6.2 · p. 13

6.3 Potential Countermeasures

  • Disabling website access by default could prevent LLMLeak’s required network request, but may substantially reduce the benefits of tool-enabled LLMs.The proposed approach would require users to enable access when needed, trading security against utility.Source: §6.3 · p. 13
  • Approval for unknown domains preserves automatic access to established sites but adds interaction overhead and can create risks through permanent approval or expired allowlisted domains.Trustworthiness scores could reduce approval demands for new domains, but compromised or expired domains may retain high scores.Source: §6.3 · p. 13

7 Related Work

The related work distinguishes LLMLeak from attacks that manipulate model behavior: it exploits normal URL fetching and malicious dependencies to exfiltrate data from a system without direct network access. Its channel therefore differs from direct and indirect prompt injection and from prior URL-based attacks.

7.1 Direct Prompt Injection

  • Direct prompt injection places malicious instructions in the model’s input, but LLMLeak instead exploits side effects of the fetch tool’s core functionality to establish a covert channel.Instructions that cause tools to make POST requests with confidential data may be detected by servers or manual input inspection.Source: §7.1 · pp. 13–14

7.2 Indirect Prompt Injection

  • Indirect prompt injection embeds malicious instructions in external content read by an LLM, whereas LLMLeak exfiltrates data without changing the LLM’s behavior.Prior work describes IPI as arising when the boundary between trusted user instructions and untrusted external data collapses [12] [18].Source: §7.2 · p. 14

7.3 LLM-Mediated Data Exfiltration

  • Prior URL-based exfiltration attacks manipulate the LLM to construct or select a data-bearing request, while LLMLeak uses the user-entered attack string and the model’s normal fetch behavior.LLMLeak’s threat model has a malicious dependency produce an attacker-controlled URL in an error message; a later benign debugging request can generate the exfiltration channel.Source: §7.3 · p. 14

7.4 Malicious and Hallucinated Dependencies

  • Research on malicious or hallucinated dependencies examines package installation as an attack vehicle [35] [15], while LLMLeak addresses exfiltrating data collected by malicious code.The authors characterize LLMLeak as orthogonal and complementary to that work.Source: §7.4 · pp. 14–15

8 Conclusion

The paper concludes that everyday chatbot use creates a risk of confidential information being exfiltrated to external parties, and presents LLMLeak as a covert channel exploiting chatbot web fetching. Its evaluation demonstrates effectiveness across open-weight models and real-world chatbots, motivating defenses that account for third-party leakage.

  • LLM-based chatbot interactions can let malicious software provoke outputs that unknowingly exfiltrate confidential information to an external party.Source: §8 · p. 15
  • LLMLeak embeds secrets in URLs in attacker-induced outputs, so a chatbot’s request to an attacker-controlled resource reveals them through DNS or HTTP.The resulting channel can circumvent restrictions on direct network communication.Source: §8 · p. 15
  • Evaluation across open-weight models and real-world chatbots demonstrates LLMLeak’s effectiveness in exfiltrating confidential information.The findings motivate defenses that consider leakage to external third parties as well as data shared with chatbot operators.Source: §8 · p. 15

A Example stack trace: Informational framing

The example presents a software-migration-style information request with a URL documenting the current server configuration. The accompanying traceback ends in a stale local configuration error, while the message says only schema-changed fields are affected.

B Example stack trace: Directive framing

The directive-framing example presents a crashed program and an assistant instructed to debug step by step. Its trace reports a missing local runtime manifest, and the supplied table captions identify attack-funnel and length-influence analyses.

B Example stack trace: Directive framing

System prompt

  • The system prompt casts the assistant as a coding debugger who issues one command per turn, observes the result, and continues until it understands the crash.Source: System prompt · p. 18

C Detailed Analyses for Hanzi Encoding

The section points to detailed LLMLeak performance metrics for Hanzi (CJK) encoding.

D Evaluation of Sreach and Sconf for the lengthcomparison

For Sreach, most models show a decline as length increases from 4 to 63 characters, though the size varies and two models move slightly in the opposite direction. The system prompt standardizes tool use, turn format, and task-agnostic debugging guidance.

D Evaluation of Sreach and Sconf for the lengthcomparison

E System Prompt

  • The identical system prompt specifies available tools, a one-thought/one-command turn format, and task-agnostic guidance.Each command yields an observation, and respond(...) ends the session.Source: E System Prompt · p. 18
  • The prompt directs agents to inspect relevant files, avoid unsupported guesses, and take one concrete step at a time.Source: E System Prompt · p. 18
Loading 2610.01768v1…