PaperlayerView exact revision on arXiv

Source-linked AI summary

Answer-Only Disclosure: Private-Context LLM Assistance over Web APIs with a Leakage Ledger

Liang Yuan, Xiaohu Du, Ruikun Luo, Jinze Wang, Mashud Rana, Kaibin Wang, Jiangwen Zhou, Bo Li, Xiaoyu Xia

arXiv:2610.38126v1cs.CRcs.AI

TL;DR

Collaborative decoding can expose private content through the messages exchanged with a cloud API, even when the public view and final answer do not contain it. ADLER restricts cloud messages to functions of the public view and answer, while a client-side ledger bounds answer disclosure relative to a public reference. On GSM8K, unlimited-budget ADLER substantially improves small-client exact-match accuracy over the cloud-only setting, while budgets of 16 nats or less leave accuracy near that baseline.

  • Problem

    The cloud’s transcript can reveal private content beyond the public prompt and answer, so prompt-level privacy alone does not describe what a Web API learns.

    Source: §1 · pp. 1–2

  • Method

    ADLER sends answer prefixes through restart streaming and uses a client-side leakage ledger to limit answer disclosure relative to a public reference built from cloud replies.

    Source: §1 · p. 2

  • Results

    On GSM8K, unlimited-budget ADLER raises exact-match accuracy from 0.41 to 0.83 for a 0.5B client and from 0.67 to 0.85 for a 1.5B client.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Budgets of 16 nats or less leave GSM8K accuracy near restricted-cloud performance, making the disclosure cost of the gains explicit.

    Source: §1 · p. 2

    ADLER protects message content but does not address timing that can depend on local computation over X; the public view V is disclosed by design.

    Source: §8 · p. 9

Abstract

from arXiv · show

Web assistants increasingly answer questions about a user's private documents by pairing a small on-device model with a cloud LLM behind a Web API. In collaborative decoding, the document stays on the client, and the cloud receives only a public view that the user approves, such as the question alone, but the two models exchange messages while the answer is generated. We show that the message transcript can disclose private content that neither the public view nor the answer contains. In a planted-secret audit, speculative decoding with private drafts did not include the secret code in its answers, yet sent the code to the cloud in rejected drafts. We present ADLER, a decoding protocol for streaming LLM APIs in which every message to the cloud depends only on the public view and the answer the user sees, regardless of how the client combines the two models or how the cloud behaves. A client-side leakage ledger then bounds how much the answer itself reveals relative to a public reference process built from the cloud's replies. In the audit, no ADLER configuration exposed the secret beyond the answer. On GSM8K math problems with numerical values masked in the cloud's input, ADLER with an unlimited leakage budget raises the exact-match accuracy of a 0.5B client from 0.41 to 0.83 and of a 1.5B client from 0.67 to 0.85 under lenient answer parsing, close to the 0.92 achieved by uploading the full problem. It also exceeds a zero-leakage program baseline (0.69) by 0.14 and 0.16, respectively. In this baseline, the cloud writes a program that the client runs offline using the private numbers. The ledger records the disclosure charge associated with this gain; budgets of 16 nats or less leave accuracy near the cloud-only baseline on GSM8K.

1 Introduction

Collaborative decoding keeps private documents on the client but can expose them through messages exchanged with a cloud API. ADLER makes those messages depend only on the public view and answer, then uses a leakage ledger to bound what the answer reveals.

1 Introduction

  • Private-context drafts can reveal information absent from the answer, because rejected drafts are still sent to the cloud.The paper also reports that logit uploads expose functions of private context and that coupling public drafts to private-context decisions defeats per-token divergence accounting.Source: §1 · pp. 1–2
  • ADLER streams cloud continuations and sends answer prefixes at disagreements, so every cloud message depends only on the public view and answer.Its client-side ledger charges how far each answer token moves from a public reference built from cloud replies, stopping the tilt when the budget is spent.Source: §1 · p. 2
  • On GSM8K, unlimited-budget ADLER raises exact-match accuracy from 0.41 to 0.83 for a 0.5B client and from 0.67 to 0.85 for a 1.5B client, versus 0.92 with full-problem upload.Budgets of 16 nats or less remain near restricted-cloud performance; the zero-leakage program baseline reaches 0.69.Source: §1 · p. 2
  • Restart streaming cuts modelled communication time per GSM8K answer from 29.6 s to 5.4 s compared with token-by-token fusion.The planted-secret audit found no ADLER configuration exposed the code beyond the answer.Source: §1 · pp. 2–3

2 Setting and Threat Model

The setting separates a client’s private context and user-approved public view from a cloud model that receives the view. The cloud observes the transcript, while the answer itself remains the residual disclosure of interest.

2 Setting and Threat Model

  • The client holds private context X and a question; a user-approved mechanism M produces public view V, which may be the question alone, a redacted document, or retrieved passages.A small model q runs locally, while the cloud model p receives V and returns its top-K token probabilities.Source: §2 · p. 3
  • The cloud’s view is transcript T, including V, its randomness, and every client message; the user sees the complete emitted answer Y.The cloud is honest-but-curious by default, but the guarantees also hold for a malicious cloud returning arbitrary distributions and lookahead because they are enforced client-side.Source: §2 · p. 3
  • An answer-only protocol makes T a function of V, cloud randomness, and Y, leaving the answer’s own disclosure as the residual question.The disclosure guarantee compares the answer with a public reference process that samples with λ=0 and bounds additional disclosure by β nats.Source: §2 · p. 3

3 How Collaborative Decoding Leaks

Collaborative decoding can leak through private drafts or through the joint view of public drafts and emitted tokens. These channels expose information beyond what per-token answer accounting captures.

3 How Collaborative Decoding Leaks

  • Private drafts are part of the transcript even when residual correction preserves exact sampling from the restricted cloud distribution.Thus answer correctness does not prevent rejected draft content from reaching the server.Source: §3 · p. 3
  • Exact speculative decoding can emit an answer independent of private context while the transcript reveals all of that context.A local model can draft a token encoding X that the cloud rejects; in the planted-secret audit, all six code digits reached the cloud in rejected drafts in 5 of 48 sessions, although the answer never contained the code.Source: §3 · p. 3
  • With public drafts, the joint view of draft and emitted token can have infinite Rényi divergence from the reference even when token-level KL divergence is finite.Only an additive guarantee bounded by the sum of per-step total variation distances survives.Source: §3 · p. 3

4 ADLER: Answer-Only Decoding

ADLER streams cloud replies while keeping private context and local sampling on the client, then uses exponential tilting and a leakage ledger to control answer disclosure.

4.1 Restart Streaming

  • The cloud streams its greedy continuation and top-K probabilities; the client samples using private evidence and sends the answer prefix at disagreement or after L positions.The cloud resumes from the supplied prefix; lookahead positions after a disagreement may be computed but discarded.Source: §4.1 · p. 3
  • The transcript is a deterministic function of the public view, randomness, and answer for every fusion rule and cloud strategy, including malicious ones.The theorem concerns message content; timing is outside the model.Source: §4.1 · p. 3

4.2 Fusion by exponential tilting

  • Exponential tilting reweights a public reference distribution using a local score, increasing mass for tokens supported by private context while preserving the cloud distribution otherwise.The evidence and local terms let the method incorporate private-context shifts or pool the cloud and full-context local model; parameters are selected per model pair and task family on public validation data.Source: §4.2 · p. 4

4.3 Leakage Ledger

  • The ledger spends the per-answer budget on divergence from the public reference, bounding answer disclosure relative to that reference when cloud randomness is independent of private context given the public view.The bound transfers to the transcript; charges bound dependence on private context, not the information conveyed by any particular answer, and budgets add across answers.Source: §4.3 · p. 5

5 Experimental Design

The evaluation compares ADLER and controlled baselines across private-document QA and private-number math, using held-out test sets and prespecified validation choices.

Evaluation questions

  • The evaluation asks whether unlimited-budget ADLER outperforms restricted-cloud and local-only endpoints, whether tilting beats a linear mixture at matched budgets, and how richer public views affect the gap to full upload.The three research questions cover private documents and values, fusion choice, and public-view informativeness.Source: §5 · p. 5

Models and Serving

  • The study uses Qwen2.5-Instruct 7B as cloud model and 0.5B or 1.5B local models, with top-64 replies and lookahead 8; results are averaged over three runs unless stated otherwise.Public views include question-only, BM25 passage selection, evidence suffixes, and heuristic entity/number masking; GSM8K uses number masking.Source: §5 · p. 5

Tasks and Cohorts

  • Tests cover multi-hop QA on 2WikiMQA and MuSiQue private passages and GSM8K problems whose private numbers are replaced by placeholders in the cloud view.Document QA uses F1; GSM8K uses strict exact match and a lenient re-score, with 50 validation and 150 document-test questions per task and 50 validation and 300 test problems for GSM8K.Source: §5 · p. 5

Methods

  • ADLER is compared with restricted-cloud, local-only, and full-upload endpoints plus controlled adaptations of five published designs, including private-draft speculative decoding and a zero-leakage program baseline.The program baseline assumes offline execution without network access; design choices are isolated with a one-factor ablation.Source: §5 · p. 5

Score Selection and Statistics

  • Model-pair and task-family score parameters are chosen from declared candidates on validation data, while uncertainty is assessed with question-clustered paired bootstrap intervals across three runs.Test outcomes are not used for tuning; intervals use 10,000 resamples without multiplicity adjustment.Source: §5 · p. 5

6 Results

ADLER’s private-value gains bring both local models close to full upload, while private-document gains are narrower. The audit finds no secret exposure beyond answers in ADLER, and the results show that larger GSM8K gains require substantial ledger spend.

6.1 Answers to RQs

  • On multi-hop QA, the 0.5B tilt beats local-only on three of four panels but is significantly better than both endpoints on none; with BM25 on 2WikiMQA it falls 0.049 F1 below restricted.For the 1.5B model, the tilt beats restricted but not local-only; the 0.5B tilt beats both on HotpotQA, which was inspected during development.Source: §6.1 · pp. 5–6, Table 1 · p. 6
  • 0.5B ADLER reaches 0.769 strict and 0.834 lenient accuracy; 1.5B reaches 0.789 and 0.852, within 0.07 to 0.10 of full upload (0.870 / 0.918).On GSM8K with private numbers, the lenient gains over local-only are 0.42 for 0.5B and 0.18 for 1.5B.Source: §6.1 · p. 6, Table 2 · p. 6
  • Without a budget, the 0.5B tilt exceeds the linear mixture on every QA panel by 0.015 to 0.063 F1 and on GSM8K by 0.67.At 𝛽=16, QA estimates favor the tilt by 0.009 to 0.045 while it spends fewer nats; tuning the mixture’s weight cap does not close the gap.Source: §6.1 · p. 7
  • A more informative public view narrows the gap to full upload mostly by lifting the restricted cloud; under suffix and entity-masked views, the tilt stays within ±0.01 F1 of restricted.The tilt helps most when the public view carries the least evidence.Source: §6.1 · p. 7

6.2 Leakage Audit and Budget

  • Private-draft speculative decoding never put the planted code in its answers, but its drafts exposed the code in 5/48 extraction sessions; ADLER configurations did not expose it beyond answers.At 𝛽=16, the audit’s guessing bound is vacuous, so the observed zero exposure there is empirical rather than guaranteed.Source: §6.2 · p. 7, Table 3 · p. 7
  • Every GSM8K answer exceeded 16 nats of spend, while two-thirds of QA answers spent at most 16 nats; accordingly, budgets of 16 nats or less remove the GSM8K benefit but change QA F1 little.On MuSiQue with the question-only view, the tilt at 𝛽=16 still beats restricted by 0.061 for 0.5B and 0.078 for 1.5B.Source: §6.2 · p. 7

6.3 Comparison with Zero-Leakage Designs

  • The offline cloud-written program scores 0.691 on GSM8K and discloses nothing beyond the public view; ADLER exceeds it by 0.14 and 0.16 under lenient scoring, spending 72 and 52 nats.The authors recommend the program when that spend is unacceptable for this kind of arithmetic.Source: §6.3 · pp. 7–8

6.4 Ablation Study

  • GSM8K accuracy is an inverted U in evidence weight: the frozen 𝑎=2 scores 0.75, while 𝑎=1 loses a significant 0.09 and 𝑎=3 and 4 spend more but score lower.Increasing the budget yields little at 16 nats (0.013) or 32 nats (0.060); 64 nats buys about two-thirds of the unbudgeted gain (0.490 against 0.753). Restricting the reference to top-64 tokens changes accuracy by under 0.01, while a tighter clip costs 0.13; changing lookahead leaves all 300 answers token-identical while requests fall from 292 to 37 at 𝐿=16.Source: §6.4 · p. 8

7 Related Work

Prior work spans collaborative decoding and private inference, but the cited approaches differ in what they analyze or bound. ADLER is positioned as extending divergence guarantees from a single party’s output to a two-party Web protocol.

Speculative and Collaborative Decoding

  • Speculative sampling is exact for its target for any proposal [3] [16], while distributed, cascaded, and lossy variants optimize placement, deferral, or speed [2] [14] [22] [23] [34] [35].The section also notes that acceptance patterns can form a side channel [31].Source: §7 · p. 8
  • SpecSteer [20] and AsymSpec [18] let a full-context small model steer a restricted large model, but do not analyze transcript disclosure.Propositions 3.1–3.2 address that disclosure question.Source: §7 · p. 8

Private LLM Inference

  • Prompt sanitizers [10] and HaS [4] restore masked answers on the client, while CoGenesis [36] uploads local signals.The paper adapts HaS’s approach in its post-edit design.Source: §7 · p. 8
  • CoVeil [37] reduces exposure empirically, whereas DP-Fusion [28] and PMixED [11] bound a single party’s mixed-output divergence; ADLER extends the bound to a two-party Web protocol.CoVeil’s edge-side message pattern matches restart streaming at L=1; cryptographic and DP-in-forward methods are described as complementary [8] [17].Source: §7 · p. 8

8 Conclusion

ADLER addresses disclosure through decoding messages by making each cloud message depend on the public view and the answer, while bounding what the answer itself reveals. Its utility on private values comes with ledger spend, and the paper identifies content and evaluation boundaries.

8 Conclusion

  • ADLER makes every Web API message a function of the public view and answer, and bounds what the answer itself reveals.The conclusion frames this as protection against decoding messages revealing more than the answer.Source: §8 · p. 8
  • With an unlimited budget, ADLER brings a small client close to full upload on private values and ahead of a zero-leakage program; strong budgets forfeit most of that gain.The paper notes that this accuracy comes with ledger spend.Source: §8 · p. 9
  • ADLER protects message content, but timing may depend on local computation over X; public views are disclosed by design and the bounds take M as given.The KL guarantee is average-case and weaker than pure DP; public views are rule-based and network times are modelled.Source: §8 · p. 9

Future Work

  • Future work includes zero-leakage baselines for document tasks, learned public views, and a live Web API deployment.The paper reports that no personal data was used.Source: §8 · p. 9

A Proofs

The proofs establish that collaborative transcripts can reveal private context beyond the answer, then formalize how restart streaming and ledgered answer generation constrain disclosure. They also prove properties of the ledger’s charge and fusion rules.

Proof of Proposition 3.1

  • A private draft can make the transcript determine X even when the generated answer is constant and independent of X.The construction has an answer law identical to the cloud’s and zero conditional uncertainty about X given the transcript, view, and answer.Source: A Proofs · p. 10

Proof of Proposition 3.2

  • Under coupling, the joint view of a public draft and answer can have infinite divergence from the reference, while rejected drafts are never emitted.The proof also states that acceptance equals the event that the answer matches the draft.Source: A Proofs · p. 10

Proof of Theorem 4.1

  • Restart-streaming messages are answer prefixes ending at a disagreement or exhausted lookahead, and the recursion makes every message determined by the public view, cloud randomness, and answer.This establishes the transcript as a function of those inputs.Source: A Proofs · p. 10

Proof of Lemma A.1

  • The divergence charge is non-decreasing in the tilt parameter, supporting the ledger’s bisection procedure.For KL, the charge is strictly increasing when the score is nonconstant on the reference support; the mixture charge is non-strictly increasing.Source: A Proofs · p. 10

Proof of Proposition 4.3

  • For a fixed KL budget, the tilted distribution uniquely attains the maximum expected score, and the charge varies continuously and strictly with its tilt parameter.The proof also shows how special parameter choices recover particular distributions and how the tilt differs from a linear mixture.Source: A Proofs · p. 10

Proof of Theorem 4.4

  • The ledger’s per-token charges sum to at most β on every path, yielding a KL or Rényi divergence bound of β for the answer relative to the reference.The reference depends on the public view and cloud strategy, not on X; data processing transfers the bound to the transcript.Source: A Proofs · p. 10
  • The information bound gives I(X; T | V) ≤ β, while Pinsker’s inequality and related bounds constrain total variation and secret-recovery probability.For β=4 and m=10^6, the stated secret-recovery bound is 0.34.Source: A Proofs · p. 10

Implementation

  • The implementation passes exact-enumeration tests of the sequence-level KL and Rényi-2 bounds, Theorem 4.1, and Proposition 4.3.Source: A Proofs · p. 10

B Reproducibility

The reproducibility appendix specifies implementation and audit settings, the scope of a multi-hop QA results table, and prompt, scoring, and offline-execution procedures.

B.1 Settings

  • Runs use BF16 weights and deterministic PyTorch kernels, with answer caps of 32 tokens for QA and 512 for GSM8K, and shared per-question randomness across methods.The audit and ablations reuse the first test run’s random state; three test runs differ only in random state.Source: §B.1 · p. 10
  • Table 5 reports multi-hop QA test F1 and mean nats per answer for a 1.5B local and 7B cloud pair across three seeds.Its caption identifies paired differences and marks ADLER as best among methods keeping X local when bolded.Source: Table 5 · p. 11
  • The audit inserts a private account-code passage at a random position; the ledger uses KL with λ≤1 and 40 bisection steps, while private-draft decoding uses blocks of γ=5.Even-indexed audit documents ask for the code.Source: §B.1 · p. 11

B.2 Prompts and Scoring

  • QA uses official LongBench prompts and F1, while GSM8K uses step-by-step prompts and a lenient re-score that checks several answer formats.The appendix also specifies masking, post-edit and program-baseline prompts, a five-second program limit, and re-execution of 3,600 programs without network access; answers were unchanged.Source: §B.2 · p. 11

C Additional Results

Results cover a second model pair and HotpotQA, trace-based network costs, and protocol costs. The tilt improves on restricted-cloud results in some QA settings, while restart streaming incurs unused-position overhead.

C.1 Second Model Pair and HotpotQA

  • 0.054 on 2WikiMQA and 0.098 on MuSiQue: the 1.5B tilt beats restricted on question-only views and does not differ significantly from local-only.On HotpotQA, the 0.5B question-only tilt scores 0.382 versus 0.300 for restricted and 0.266 for local-only, with both differences significant; with BM25, it does not differ significantly from restricted (0.432 vs. 0.452).Source: §C.1 · p. 11

C.2 Trace-based Network Cost Model

  • At 100 ms round-trip time, GSM8K communication time falls from 29.2 to 5.2 seconds at 100 Mbit/s and from 33.7 to 7.5 seconds at 1 Mbit/s when lookahead increases from L=1 to L=8.The trace-based model assumes stateless requests that resend the prompt and answer prefix, excludes compute, and assumes transfer does not overlap streaming; round trips dominate, and at 10 Mbit/s bytes add under half a second per row.Source: §C.2 · p. 11

C.3 Protocol Cost

  • 44% (restricted) and 62% (tilt) of streamed QA positions are wasted, mostly because answers end before the 8-token lookahead is used; on GSM8K, waste is 19% and 30%.Full upload needs 37 requests per GSM8K answer versus 59 for restricted cloud, while prefix caching leaves per-answer cloud processing at 536 tokens for tilt and 594 for restricted on GSM8K.Source: §C.3 · p. 11
  • 0.06 above restricted on question-only MuSiQue: the tilt gains most when the public view carries least evidence; evidence-only is the only method significantly helping under entity masking, by 0.046.Figure 7 compares fusion methods with restricted cloud under four public views.Source: §C.3 · p. 11

D Score Selection on Validation Data

Validation scores guide the selection of score combinations and mixture weight caps. The selected score term differs by task family, and the 1.5B model favors a smaller value of a.

D Score Selection on Validation Data

  • The evidence term wins on GSM8K, geometric pooling on QA, and the 1.5B model prefers a=1 to a=2, consistent with Proposition 4.3(b).Figure 6 shows validation scores for five declared (a,b) candidates and endpoints; the highest score per model pair and task family is selected, with no ties.Source: D Score Selection on Validation Data · p. 11, Figure 6 · p. 12
  • The mixture’s weight cap was selected using the same validation-score procedure, with ties resolved in favor of the smaller cap.The passage reports that Figure 6 shows the candidates considered and that no ties occurred in the score selection.Source: D Score Selection on Validation Data · p. 11, Figure 6 · p. 12

Related papers

Loading 2610.38126v1…