PaperlayerView exact revision on arXiv

Source-linked AI summary

The Illusion of Local Privacy: Confidentiality Boundary Failures in Consumer LLM Serving Systems

Youssef Hamdi Zafan Ibrahim, Muhammad Ikram, Mohammed Khalaf Salama

arXiv:2609.18526v2cs.CR

TL;DR

The paper asks whether keeping inference local is enough to keep prompts confidential, given that serving software handles prompts across multiple stages. It develops LLAnalyzer to test four confidentiality boundaries separately; its retained observations show that local execution alone does not establish prompt confidentiality.

  • Problem

    It is unclear which components of a consumer local-LLM stack determine prompt confidentiality when computation remains on the user’s machine.

    Source: §I · p. 1, ORCID: 0009-0007-6508-1437 · p. 1

  • Method

    LLAnalyzer measures model admission, runtime lifetime, wrapper persistence, and serving-state isolation using boundary-specific evidence and criteria.

    Source: B. Confidentiality Boundaries · p. 2, F. Experimental Controls and… · p. 5

  • Results

    Retained records show runtime prompt canary recovery, configuration-specific wrapper persistence, and same-process saved-state transfer, while the latter does not establish cross-principal authorization.

    Source: D. RQ4. Isolation Boundary · p. 10, §VI · p. 11

  • Takeaways & Limitations

    Local execution is one condition in the trust model; prompt confidentiality also depends on admission, runtime lifetime, persistent storage, and client isolation.

    Source: §VI · p. 11

    The saved-state workloads did not test distinct principals, and timing records were not retained for independent validation.

    Source: D. RQ4. Isolation Boundary · p. 10

Abstract

from arXiv · show

Running large language models (LLMs) locally is often considered more private than cloud-hosted inference because user prompts remain on the device. We ask whether keeping inference local is, by itself, sufficient to keep those prompts confidential. Our results show that it is not: prompt confidentiality also depends on how the surrounding serving software handles prompt data before, during, and after inference. We examine four boundaries at which prompt confidentiality can fail in consumer local-LLM serving systems: model loading, runtime memory, wrapper-level persistence, and the serving interface. To study these boundaries, we develop LLAnalyzer, a measurement framework that tests each boundary separately and traces observed failures to the responsible software component. Applying LLAnalyzer to four open-weight model families and two consumer deployment platforms, we find markedly different behaviour across boundaries. In a 24-hour AFL++ campaign with more than 12 million executions, we observe no parser crashes or successful malformed GGUF loads within the explored state space. Runtime memory tells a different story: we recover prompts after inference because multiple plaintext representations survive in allocator-managed memory, and sanitisation reduces this residue without eliminating it. We also find that consumer wrappers can extend prompt lifetime through plaintext persistence. At the serving boundary, we uncover a previously undocumented authorization flaw in llama.cpp that allows one authenticated client to restore another tenant's saved conversation state; the attack succeeds in 200/200 controlled trials. Separately, shared prompt-prefix caching exposes a remote timing oracle that remains distinguishable under WAN conditions. We argue that local LLM systems need explicit guarantees for prompt lifetime, persistent storage, and tenant isolation.

I. INTRODUCTION

The paper asks which parts of a consumer local-LLM serving stack determine prompt confidentiality, and develops LLAnalyzer to test four boundaries separately. Its observations differ by boundary, showing that local execution alone does not establish confidentiality within the serving stack.

I. INTRODUCTION

  • Local inference raises the question of what happens to prompts inside serving software, beyond whether they are sent to a cloud provider.Prior work examined training-example memorisation [4],, local artifact recovery [6], [7],, and shared-cache leakage, [10], [11],,.Source: §I · p. 1, ORCID: 0009-0007-6508-1437 · p. 1

ORCID: 0009-0007-6508-1437

  • The study’s observations support treating local execution as one condition in the trust model, not as a guarantee about prompt handling across the stack.The paper identifies admission, runtime lifetime, persistent storage, and client isolation as relevant components.Source: ORCID: 0009-0007-6508-1437 · p. 2
  • LLAnalyzer separates prompt confidentiality into model admission, runtime lifetime, application persistence, and serving-state isolation, then traces observations to boundary-specific evidence.The paper frames this as a systems property rather than a single implementation-bug search.Source: ORCID: 0009-0007-6508-1437 · pp. 1–2
  • LLAnalyzer uses a reusable workflow to exercise the boundaries as separate logical surfaces and attribute observations to boundary-specific evidence sources.The paper also describes debugger and allocator evidence, descriptive sanitisation comparisons, and conservative provenance reporting.Source: ORCID: 0009-0007-6508-1437 · p. 2
  • Across four model families and two platforms, the study reports runtime prompt residue, wrapper persistence, and same-process saved-state transfer, alongside assertion aborts in retained malformed-input reports.The saved-state transfer findings are not two-principal authorization tests.Source: ORCID: 0009-0007-6508-1437 · p. 2

II. CONFIDENTIALITY BOUNDARY FRAMEWORK

  • The framework defines four distinct security objectives and ways of failing, rather than treating the deployment as one trusted environment.A prompt may be copied, transformed, retained in memory, persisted, or associated with serving-interface state.Source: §II · p. 2

A. Prompt Lifecycle in Consumer LLM Serving Systems

A local request passes through multiple components and representations, so a guarantee at one stage does not establish confidentiality at the others. The framework defines four boundaries and tests each with boundary-specific objectives, adversaries, and evidence.

A. Prompt Lifecycle in Consumer LLM Serving Systems

  • A request moves from a consumer application through model loading, parsing, prompt construction, tokenization, inference-state management, and potentially conversation recording or API exposure.The prompt can be copied, transformed, retained in memory, written to storage, or associated with serving-interface state.Source: A. Prompt Lifecycle in Consumer LLM… · p. 2

B. Confidentiality Boundaries

  • The four boundaries cover model admission, runtime prompt lifetime, wrapper-managed persistence, and separation of clients sharing a serving process.Each has a distinct security objective, and satisfying one does not imply satisfying another.Source: B. Confidentiality Boundaries · pp. 2–3
  • LLAnalyzer maps the request path to four confidentiality boundaries: model-file integrity, runtime lifetime, wrapper-managed persistence, and shared serving-state isolation.Figure 1 presents the architecture and evidence map.Source: Figure 1 · p. 3

C. Threat Model

  • The threat model distinguishes an unprivileged same-UID co-resident process, an authorized forensic collector, an authenticated network client, and a malicious model provider.The saved-state workloads did not instantiate two authenticated clients or distinct credentials, so they do not provide an A2 authorization result.Source: C. Threat Model · pp. 3–4
  • The framework separates residual-data existence, forensic recoverability under documented permissions, adversarial accessibility, and the resulting confidentiality consequence.For memory measurement, the collector F is distinct from the unauthorized co-resident adversary A1.Source: C. Threat Model · p. 4

III. LLANALYZER

  • LLAnalyzer varies the mechanism relevant to one boundary while holding the rest of the serving configuration fixed, using connected boundary-specific harnesses rather than one scanner.Its goals include attribution, methodological consistency across different instrumentation, and reproducibility using UUID canaries and controlled configurations.Source: §III · p. 4, A. Design Goals · p. 4

A. Design Goals

  • The retained implementation records source revision 721354f, but the executed June binaries were not hashed or retained, and equivalence to the later CPU rebuild is unknown.The GPU build revision is also unrecorded.Source: A. Design Goals · p. 4

B. Framework Architecture

LLAnalyzer uses a common experimental sequence to test each confidentiality boundary, while changing instrumentation and evidence sources to fit the boundary. Retained records, model scope, and comparison controls constrain how the results can be interpreted.

B. Framework Architecture

  • For every boundary, LLAnalyzer states a security objective, selects an adversary, exercises the boundary under controlled conditions, acquires evidence, and tests it against the objective.The instrumentation changes by boundary, but the experimental logic remains common.Source: B. Framework Architecture · p. 4

C. Boundary-Oriented Measurement

D. Measurement Pipeline

  • A fresh UUIDv4 canary is embedded in each prompt, recorded in a manifest, checked absent from the evidence source beforehand, and searched for after inference.The UUID contains 122 random bits, making accidental exact matches negligibly probable.Source: D. Measurement Pipeline · pp. 4–5

E. Experimental Configuration

  • The evaluated artifacts comprise four open-weight model families using GGUF Q4 K M quantisation, but retained records lack a run-local model SHA-256 manifest.GPU records use a CUDA build with an unrecorded revision; GPU scope is partial CUDA offload, with no full-offload or non-llama.cpp stacks tested.Source: E. Experimental Configuration · p. 5

F. Experimental Controls and Statistical Analysis

  • Cross-family comparisons are descriptive because runtime settings differ; twelve-tenant rows are Nemotron-only, multilayer diagnostics Phi-4-only, and GPU-v5 Nemotron CUDA.The experiments were run in June 2026 from retained records and were not rerun for this revision.Source: F. Experimental Controls and… · p. 5, E. Experimental Configuration · p. 5
  • The analysis reports descriptive proportions and denominators, uses Wilson intervals only when trial units support them, and claims no paired significance test.Success is recorded only when an exact UUID match meets the boundary-specific disclosure criterion.Source: F. Experimental Controls and… · p. 5

IV. EVALUATION

The evaluation tests prompt confidentiality in lifecycle order, from model admission through runtime memory and wrapper persistence to isolation between clients of a shared serving instance. Its integrity-boundary evidence is limited to retained malformed-input reports and does not establish universal parser robustness.

IV. EVALUATION

  • The evaluation asks whether local execution alone is sufficient by separately testing integrity, runtime lifetime, wrapper persistence, and shared-instance isolation.The questions follow prompt-lifecycle order and isolate the target boundary while holding unrelated factors constant wherever possible.Source: §IV · p. 5

validation or trigger memory-safety failures before inference begins?

  • Integrity testing used retained structured malformed GGUF artifacts; the unretained fuzzing campaign is not treated as a quantitative result.The artifacts targeted tensor metadata, architecture identifiers, dimensions, allocation sizes, offsets, RoPE parameters, and header fields.Source: validation or trigger memory-safety… · p. 6

A. RQ1. Integrity Boundary

  • Retained malformed-GGUF reports record SIGABRT assertion aborts and no SIGSEGV, but do not establish a unified crash rate or global parser robustness.The report families overlap, and the available evidence cannot establish the unretained historical fuzzing total.Source: validation or trigger memory-safety… · p. 6, Table I · p. 6

B. RQ2. Lifetime Boundary

The runtime-memory evaluation asks whether prompt plaintext remains recoverable after inference, how it persists, and whether sanitisation or GPU execution changes that exposure. It finds post-inference residue and traces it to prompt copies across request processing and allocator-managed memory.

1) RQ2.1: Does Prompt Plaintext Persist After Inference?

  • In two Nemotron twelve-tenant runs, canaries from 11/12 tenants were recovered; records contained 110 copies and 10 834 KB in one, and 78 copies and 3 352 KB in the other.These single-run measurements are byte occurrences and printable-cleartext volume, not counts of independent secrets or attacker sessions.Source: 1) RQ2.1: Does Prompt Plaintext… · p. 6, Table III · p. 8
  • Earlier tenants' canaries remain recoverable after subsequent requests, so prompt lifetime can outlast response generation and later inference within the same process.The retained observations attribute this residue to post-inference serving-process state, rather than content introduced by the acquisition procedure.Source: 1) RQ2.1: Does Prompt Plaintext… · pp. 6–7

2) RQ2.2: Why Does Prompt Residue Persist?

  • Ordinary request processing creates multiple plaintext representations, while allocator-managed reuse can retain released buffers; sanitisation must address the lifecycle, not allocator state alone.The traces associate duplication with stages such as JSON parsing, chat-template expansion, request construction, serialisation, and std::string reallocations.Source: 2) RQ2.2: Why Does Prompt Residue… · pp. 7–8
  • Across 200 model-switch trials, model A's canary remained in writable mappings after the swap, but the records do not establish cross-principal access.The process identity remained unchanged in all trials.Source: 2) RQ2.2: Why Does Prompt Residue… · p. 8

3) RQ2.3: Can Runtime Sanitisation Reduce Residual Recoverability at Acceptable Cost?:

The retained sanitisation configurations reduced some measures of residual plaintext, but both still allowed recovery from 11/12 tenants. The records do not establish a causal reduction or an equivalence result for latency.

3) RQ2.3: Can Runtime Sanitisation Reduce Residual Recoverability at Acceptable Cost?:

  • Both retained configurations exposed canaries from 11/12 tenants, despite differences in observed byte occurrences and printable cleartext.The rows are single-run observations under different protocols and do not identify a causal sanitisation effect.Source: 3) RQ2.3: Can Runtime Sanitisation… · p. 8
  • Table IV reports descriptive KV-cache zeroisation latency for 50 requests per model, without an equivalence margin or formal equivalence test.The retained latency differences are descriptive workload observations, not universal performance bounds.Source: Table IV · p. 8
  • Restoring the lifetime boundary requires handling prompt-bearing representations across their lifecycle; the evaluated sanitisation did not cover every surviving copy.Copy count and recovered byte volume measure residual magnitude, while tenant-level recovery indicates whether historical confidentiality was restored.Source: 3) RQ2.3: Can Runtime Sanitisation… · p. 8

4) RQ2.4: Does GPU Execution Fundamentally Change Runtime Confidentiality?:

Under partial CUDA offloading, retained records show host-memory observations, while the evidence does not establish direct device-memory recovery or a universal GPU result. Wrapper findings show that prompt persistence can depend on application configuration and remains bounded by the examined artifacts.

4) RQ2.4: Does GPU Execution Fundamentally Change Runtime Confidentiality?:

  • In 50 retained Nemotron CUDA trials, canary_hit_count was 9 in 47 trials and 10 in 3; extra_tenants[*].hits was 16 in every record.The shutdown field was (101, 8192), a mapping observation rather than proof of zero memory recovery or direct device-memory recovery.Source: 4) RQ2.4: Does GPU Execution… · p. 9, Table V · p. 9
  • Prompt construction and request processing remained host-side under partial offload, so relocating model computation did not remove those plaintext-retaining paths.The retained campaign directly evaluates Nemotron under partial CUDA offloading, not fully offloaded models or other GPU-serving architectures.Source: 4) RQ2.4: Does GPU Execution… · p. 9

C. RQ3. Persistence Boundary

  • With the recorded Nemotron LM Studio configuration, enabling sensitive-data logging yielded 10/10 canary recoveries, while disabling it yielded none in examined artifacts.This is a report-backed configuration comparison, not a fresh-install default claim.Source: C. RQ3. Persistence Boundary · pp. 9–10
  • Ollama records include 203 completed trigger files, two model-memory failures, and one incomplete metadata record; six LM Studio hook summaries showed external connections but no visible canary.TLS was not decrypted, so the hook observations do not establish absence of all external transmission.Source: C. RQ3. Persistence Boundary · p. 9

D. RQ4. Isolation Boundary

The retained isolation experiments show same-process state transfer across slot identifiers, with later model continuation as a separate observation; they do not establish a cross-principal authorization failure. The section situates these results as an evidence-bounded measurement case study and concludes that local execution alone does not guarantee prompt confidentiality.

D. RQ4. Isolation Boundary

  • The July-dated continuation records are model- and channel-specific, and the historical timing separation cannot be independently recomputed.The continuation results are not a hardware-equivalent factorial comparison, and neither workload tests distinct client credentials or resource ownership.Source: D. RQ4. Isolation Boundary · p. 10
  • 200/200 slot-restoration trials showed same-process state transfer by token-count equality, but no two-client or ownership control tested cross-principal authorization.The separate continuation workload recorded 126/200 KV-channel hits and 200/200 memory-channel hits; it was not a direct authorization test.Source: D. RQ4. Isolation Boundary · p. 10

V. RELATED WORK

  • LLAnalyzer complements prior local-serving and isolation work by keeping distinct evidence units separate, rather than claiming a new cache attack or general authorization vulnerability.The retained slot and KV-continuation records do not establish a mutually distrustful-tenant attack.Source: §V · p. 10

VI. CONCLUSION

  • Across four boundaries, retained GGUF reports contain assertion-abort observations and no recorded SIGSEGV, while runtime scans recover prompt canaries after inference.Wrapper logging produced 10/10 canary recoveries in the recorded mitigation test and none when disabled; slot restoration yielded 200/200 same-process token-count equality checks.Source: §VI · pp. 10–11
  • Local execution is not a confidentiality guarantee; prompt protection depends on controls over admission, copying, retention, and sharing, with evidence for any claimed principal boundary.The conclusion frames locality as a deployment property, not the endpoint of the trust model.Source: §VI · p. 11

OPEN SCIENCE

The retained records document experiment-specific conditions and provenance limits, including unavailable executed binaries and an unvalidated historical timing record. Experiments used controlled systems and synthetic prompts, while methods and configurations varied across workloads.

OPEN SCIENCE

ETHICAL CONSIDERATIONS

  • Experiments used researcher-controlled systems, accounts, tenants, and synthetic UUID-tagged prompts, without accessing human-subject data or production systems.Internet measurements were passive and non-destructive; no vendor report or vendor decision is established.Source: ETHICAL CONSIDERATIONS · p. 11

S1. EXPERIMENTAL METHODOLOGY

  • Trial counts and conditions differed by experiment, and hardware, artifacts, parameters, and revisions are not assumed identical across workload families.Slot, injection, continuation, and KV-overhead workloads used n=50-per-model files; two runtime rows were single-run observations.Source: S1. EXPERIMENTAL METHODOLOGY · p. 11

A. Hardware and Software Configuration

  • Headline CPU scans used parent-launched collectors reading writable process mappings and memory; the GPU campaign supports host-memory residuals, not direct device-memory access.CUDA debugger attach attempts were denied, and retained records lack complete permission metadata for retrospective reconstruction.Source: A. Hardware and Software Configuration · p. 12
  • Runtime mitigation comparisons included allocator-level controls and code-level sanitisation patches, reported separately to distinguish their effects.The paired KV-overhead workload used matched patched and unpatched request arrays, but the two twelve-tenant rows were not a matched causal comparison.Source: A. Hardware and Software Configuration · p. 12

B. Experimental Corpora, Models, and Parameters

The study constructs boundary-specific corpora with known ground truth rather than using a conventional model-accuracy dataset. It evaluates four model families and records the software components and provenance used in the evaluation.

  • The study uses malformed GGUF artifacts, synthetic UUID-tagged prompts, controlled conversations, persistence workloads, timing probes, and Internet deployment metadata as experimental corpora.The corpora are designed for boundary-specific confidentiality tests, not model-accuracy measurement.Source: B. Experimental Corpora, Models, and… · p. 12
  • The evaluation records software components and their retained provenance status, and the evaluated models use GGUF-compatible representations.Source: Table S2 · p. 13, B. Experimental Corpora, Models, and… · p. 13
  • Four model families were selected to vary architecture, parameter scale, chat-template behaviour, stopping behaviour, and reasoning style.This diversity supports examining which observations recur across model-labelled families and which remain model- or protocol-specific.Source: B. Experimental Corpora, Models, and… · p. 13

C. Confidentiality Boundaries and Adversary Models

The boundary framework maps evidence sources and measured properties to distinct confidentiality objectives. The retained Isolation workloads used one unauthenticated process and did not test relationships between authenticated clients.

D. Model Specifications

The study specifies the evaluated models and uses boundary-specific workloads to examine malformed model files, canary recovery, wrapper persistence, shared-process state transfer, continuation, and cross-model residue. The retained records constrain which results can be generalized or independently validated.

D. Model Specifications

  • Table S6 lists the evaluated models’ parameter counts, architectures, and artifact sizes; the artifacts use GGUF Q4 K M quantisation and come from lmstudio-community.Source: D. Model Specifications · p. 13, Table S6 · p. 13
  • The structured model-file tests use an 80-case malformed-input battery and a 21-file header corpus, while the historical fuzzing campaign is excluded because its artifacts are not retained.The corpus targets tensor dimensions, names, metadata lengths, and related inconsistencies; the 21 files are also processed through gguf-py.Source: D. Model Specifications · pp. 13–14
  • Fresh UUID canaries embedded in controlled prompts let the researchers attribute exact recoveries to originating requests across boundary-specific evidence sources.The canaries are checked against pre-experiment evidence sources before injection.Source: D. Model Specifications · p. 14
  • The wrapper workload checks application-managed files before and after repeated prompts, including configurations with sensitive-data logging disabled where supported.The retained Ollama tree has 203 successful trigger files, two model-memory failures, and one incomplete metadata-only record; LM Studio hook summaries show no visible canary, with TLS undecrypted.Source: D. Model Specifications · p. 14
  • The slot workload restores a saved slot-0 file into slot 1 on one unauthenticated loopback server, and all 200 retained trials show token-count equality.The workload does not test a second client, distinct API keys, wrong-key controls, or ownership checks.Source: D. Model Specifications · p. 14
  • July continuation records show 126/200 KV-channel hits and 200/200 memory-channel hits, but do not test distinct credentials and use different GPU offload settings by model.KV hits vary by model: Gemma 50/50, Qwen 50/50, Nemotron 25/50, and Phi-4 1/50.Source: D. Model Specifications · p. 14
  • All 200 D4 trials retain the same process identity and a positive post-swap observation of the prior model’s canary in writable mappings.The four model-swap directions have 50 trials each; this measures byte-occurrence residue, not output disclosure or two-principal authorization.Source: D. Model Specifications · p. 15
  • The retained timing vectors and analysis script are unavailable, so timing separation is historical context rather than an independently validated quantitative result.No Internet-prevalence count is used, and no vendor decision is inferred from policy text.Source: D. Model Specifications · p. 15

S2. STATISTICAL ANALYSIS

The analysis selects statistical treatment to match each boundary’s observable, reports uncertainty for binary outcomes, and preserves model-specific differences rather than pooling them. It also bounds deterministic findings and null results to their tested conditions.

S2. STATISTICAL ANALYSIS

  • The analysis chooses statistical treatment by the observable produced at each boundary and emphasizes repeated measurement, effect magnitude, and explicit positive and negative reporting.Source: S2. STATISTICAL ANALYSIS · p. 15
  • Binary outcomes are reported as x successful disclosures or anomalous trials out of n total trials, with two-sided 95% Wilson score intervals.Wilson intervals are used because some measurements lie near zero or one; zero successes in 50 trials does not establish a zero true probability.Source: S2. STATISTICAL ANALYSIS · p. 15
  • “Deterministic” means every repeated trial under a fixed condition has the same outcome, not that the result generalizes across versions, hardware, or deployments.Direct slot-state equality succeeding in 200 of 200 trials is one example.Source: S2. STATISTICAL ANALYSIS · p. 15
  • Runtime-memory results report residual cleartext copies, matching-region bytes, and direct values for each retained configuration.The two twelve-tenant Nemotron rows are single runs with different executable and runtime settings, so they do not support a causal relative-reduction estimate; copy counts are byte occurrences, not independent secrets, tenants, or trials.Source: S2. STATISTICAL ANALYSIS · p. 15
  • Model-specific results are reported separately because templates, stopping behaviour, reasoning channels, and allocation patterns affect confidentiality measurements.Measurements are not pooled into a single aggregate proportion.Source: S2. STATISTICAL ANALYSIS · p. 15
  • A null result is retained only when instrumentation and response fields are active, a positive control works, the process remains operational, and the outcome is reproducible.Nulls are reported with the instrumentation, controls, and deployment conditions that produced them; they do not establish universal security.Source: S2. STATISTICAL ANALYSIS · pp. 15–16
  • For each quantitative claim, the authors identify its retained script, configuration, raw log, or historical report, and flag claims that cannot be independently recomputed.Randomised identifiers and UUID canaries are recorded with their trials.Source: S2. STATISTICAL ANALYSIS · p. 16

Related papers

Loading 2609.18526v2…