PaperlayerView exact revision on arXiv

Source-linked AI summary

Anytime-valid detection of LLM weight exfiltration

Ines Ortega-Fernandez, Mateusz Kowalczyk, Keri Warr

arXiv:2610.11843v1cs.CRstat.ME

TL;DR

A compromised inference server can encode data in plausible token choices, but benign numerical nondeterminism also creates replay mismatches. The paper uses a calibrated prompt-level e-process to combine evidence across responses; it detects every resampled seed-blind stream, while seed-aware detection varies with attack rate and benign margin profile.

  • Problem

    Replay can expose exfiltration through token choices, but benign numerical nondeterminism also causes mismatches, and patient attackers may leave too little evidence in any one token.

    Source: §1 · p. 1

  • Method

    A prompt-level e-process calibrates whole-response mismatch events on trusted benign traffic and accumulates evidence sequentially under a calibration-transfer assumption.

    Source: §1 · p. 2, §2 · pp. 2–3

  • Results

    The e-process alarms on all seed-blind streams within a median of 6–14 responses across models; Qwen3-32B is detected at 0.0258 bits/token, while Qwen3-30B-A3B is not at 0.0304 bits/token.

    Source: §3 · pp. 3–4

  • Takeaways & Limitations

    The e-process supports monitoring after every response with anytime false-alarm control, but some low-rate seed-aware attacks remain undetected within 250 responses.

    Source: §4 · p. 4

    The guarantee requires the same prompt distribution, model, and serving stack as trusted calibration; shifts may require stratified calibration or recalibration.

    Source: §2 · p. 3

Abstract

from arXiv · show

A compromised LLM inference server can leak model weights by encoding payload bits in otherwise plausible token choices. A replay of the same prompt in a trusted server can expose such deviations, but benign numerical nondeterminism also causes token mismatches. Patient attackers can therefore hide within normal variation unless evidence is combined across responses. We introduce a prompt-level e-process that calibrates whole-response mismatch events on trusted benign traffic and accumulates evidence sequentially while, under a calibration-transfer assumption, controlling the probability of any false alarm over an unbounded monitoring horizon. We evaluate it on four models against a seed-blind attack and a stronger seed-aware attack that hides payload bits only in near-ties to remain stealthy, analyzing the channel capacity vs detectability trade-off. Compared with a hard per-token alarm, the e-process combines weak evidence across responses while providing explicit anytime false-alarm control.

1 Introduction

A compromised inference server can encode data in plausible token choices, while replaying prompts on a trusted server can expose deviations despite benign numerical mismatches. The paper proposes a prompt-level e-process to combine response-level evidence with anytime false-alarm control.

1 Introduction

  • Compromised inference servers can leak weights or other data through plausible token choices, while replay can expose deviations despite benign mismatches from numerical nondeterminism.Seed-blind attackers cannot predict seeded near-ties; seed-aware attackers can confine payload swaps to them.Source: §1 · p. 1
  • A patient attacker may leave too little evidence in any one token, while repeatedly applying hard token alarms can eventually flag benign traffic without whole-path error control.Existing rules classify individual tokens but leave response-level pooling and escalation to deployment.Source: §1 · pp. 1–2
  • The proposed prompt-level e-process combines whole-response mismatch events and, under calibration transfer, controls the probability of any false alarm over an unbounded horizon.Across four models, it detects every resampled seed-blind stream and several seed-aware settings.Source: §1 · p. 2
  • The monitor combines fixed-seed scores with benign-mismatch analysis, complementing prior per-token exfiltration detection and work characterizing numerical nondeterminism.Related e-process work addresses general sequential change detection and watermark detection, not inference-server weight exfiltration.Source: §1 · p. 2

2 Method: a prompt-level e-process

The method tracks response-level mismatch events across seeded-margin thresholds, calibrates their benign rates, then accumulates sequential evidence. Its false-alarm guarantee depends on calibration transfer and holds over an unbounded monitoring horizon.

2 Method: a prompt-level e-process

  • Each response triggers nested binary events when any token mismatch occurs at or above a seeded-margin threshold, preserving margin information without assuming token independence.The verifier compares served tokens with those prescribed by the committed seed.Source: §2 · p. 2
  • The method estimates each event’s benign rate from new calibration responses using a Clopper–Pearson upper bound, with joint coverage of at least 1 − γ across K events.A loose bound may delay detection but cannot make benign responses appear more suspicious.Source: §2 · p. 2
  • Under the prompt-level null, each event occurs no more often than its benign rate conditional on past rounds, averaged over prompts from the calibration distribution.The same prompt distribution, model, and serving stack are required; prompt shift or stack changes may require stratified calibration or recalibration.Source: §2 · pp. 2–3
  • For each threshold, evidence starts at one and updates by a factor that rises on an event and falls otherwise, with the update based only on earlier responses and capped so it cannot multiply evidence by less than 0.5.The monitor combines the K components using a fixed average.Source: §2 · p. 3
  • Conditional on successful calibration, each component is a nonnegative supermartingale and their fixed average is an e-process; Ville’s inequality bounds the chance of ever crossing 1/α by α.Including calibration failure γ gives the overall false-alarm bound.Source: §2 · p. 3

3 Experiments

The experiments compare seed-blind and seed-aware attacks with an e-process and a per-token detector across four models. Detection of seed-aware attacks varies with attack rate and the model–serving stack’s benign margin profile.

3 Experiments

  • Seed-aware attacks reconstruct the prescribed token and restrict payload swaps to seeded near-ties, trading exfiltration capacity for stealth.Increasing dmax makes more positions eligible but exposes mismatches at wider, more detectable margins.Source: §3 · pp. 3–4
  • The evaluation uses 1,000 bootstrap streams of 250 responses and records time to first alarm and whether each stream alarms by response 250.For each model, 125 benign responses fit margin events and another 125 calibrate their rates; the e-process uses a 200 alarm threshold and a 1% overall false-alarm bound.Source: §3 · p. 3, Table 1 · p. 4
  • For seed-blind attacks, the e-process alarms on all 1,000 resampled streams for every model at a median of 6–14 responses, while no matched benign stream alarms.The Rinberg et al. [2025] hard alarm misses the attack but alarms on every benign stream because benign replay mismatches can satisfy its rank gate.Source: §3 · p. 3
  • Qwen3-32B is detected at 0.0258 bits/token, while Qwen3-30B-A3B is not detected at the higher rate 0.0304 bits/token; seed-aware detectability tracks the benign margin profile as well as channel intensity.Mixture-of-experts models show more benign mismatches at wide seeded margins, which can leave little excess for near-tie attack mismatches.Source: §3 · p. 4, Figure 1 · p. 4

4 Conclusion

Inference verification can limit but not remove the exfiltration channel. The prompt-level e-process rapidly detects seed-blind attacks across four models, while seed-aware detection varies with rate and benign margin profile.

4 Conclusion

  • Some seed-aware attacks transmitting only a few hundredths of a bit per token are not detected within 250 responses; such rates may slow bulk exfiltration but do not prevent small-secret leakage.The authors propose future work on an e-process using the continuous fixed-seed likelihood score to detect seed-aware attacks at lower rates.Source: §4 · p. 4

A Why the response statistic matters

A response-level any-mismatch event is too common on benign traffic to leave much detection power. Tracking mismatch margins helps distinguish numerical noise from payload-carrying mismatches, while multi-threshold monitoring and deployment-specific calibration address variation across models and attacks.

A Why the response statistic matters

  • 41.2%–87.2% of complete benign responses contain a mismatch, leaving an any-mismatch event little room to detect attackers matching that rate.The e-process retains false-alarm control, but this event has little power against an attacker matching its benign event rate.Source: A Why the response statistic matters · p. 6
  • Per-token benign mismatch rates fall from roughly 10%–40% in the tightest margin cell to below 0.1% in the widest.Cells range from tight ties (cell 0) to wide margins (cell 6); the monitor calibrates whole-response events on separate benign responses.Source: A Why the response statistic matters · p. 6, Figure 2 · p. 6
  • Table 2 reports token mismatch fraction, the fraction of responses with any mismatch, and mean mismatches per response.Its measures summarize 250 held-out benign responses across model types.Source: Table 2 · p. 6
  • OLMoE-1B-7B has the highest pooled mismatch burden, while the larger dense Qwen3-32B is lower on all three measures.The four checkpoints confound model family, architecture, and serving kernels, so the MoE pattern is a hypothesis rather than a causal architecture result; each deployment is calibrated separately.Source: A Why the response statistic matters · p. 6
  • A seed-aware attacker can better match where benign mismatches fall, but must restrict payload swaps to tighter ties and fewer eligible positions.Higher benign mismatch rates in wide-margin cells give payload-carrying mismatches more cover and add less evidence.Source: A Why the response statistic matters · p. 7
  • Averaging seven threshold-specific e-processes avoids choosing a margin threshold in advance, at a cost of at most log 7 in log evidence.Each event records only whether it fired; richer response statistics would require their own calibration and validity argument.Source: A Why the response statistic matters · p. 7

B Validity of the e-process

The e-process calibrates nested response-level events using independent benign data, then updates evidence sequentially with predictable bets. Under calibration and transfer assumptions, its fixed mixture controls the probability of ever alarming, without requiring independence among tokens within a response.

B Validity of the e-process

  • Independent calibration sets first fix nested margin events and then estimate their rates from benign response-level indicators, rather than individual tokens.A higher-threshold event also triggers every lower-threshold event; no token independence is assumed.Source: B Validity of the e-process · p. 7
  • Calibration and monitoring must use the same prompt distribution, model, hardware, and serving stack; prompt shift or adaptive prompt selection can violate the null.The guarantee is marginal over prompts, not uniform for every prompt.Source: B Validity of the e-process · pp. 7–8
  • The monitor smooths past event rates with half-counts, then chooses each next bet from earlier responses only and clips it to a permitted range.The Krichevsky–Trofimov estimator avoids finite-sample estimates of exactly zero or one; the bet grows with estimated excess over the calibrated bound.Source: B Validity of the e-process · pp. 7–8
  • Clopper–Pearson upper bounds jointly cover the benign event rates with probability at least 1 − γ, and a loose bound can delay detection but cannot make benign responses more suspicious.Each bound misses its true rate with probability at most η; a union bound gives simultaneous coverage.Source: B Validity of the e-process · p. 8
  • Under the prompt-level null and successful calibration, each component and their fixed mixture are nonnegative supermartingales, so Ville’s inequality bounds the conditional probability of ever crossing the alarm threshold.The proof does not require independence among tokens within a response.Source: B Validity of the e-process · p. 8
  • A fixed mixture over thresholds avoids selecting one using attack data, at a cost of at most log K in log evidence relative to the best component.With K = 7, the text describes this as a small fixed price.Source: B Validity of the e-process · p. 8

C.1 Threat model and seed access.

The threat model gives a compromised serving stack control of returned tokens, but not the trusted verifier or its calibration and recorded responses. Seed access separates an attacker restricted to blind choices from one that can confine payload deviations to near-ties; both decoding schemes assume access to sufficient top-candidate logprobs.

C.1 Threat model and seed access

  • The attacker can encode payloads through returned-token choices but cannot alter the verifier, calibration data, or recorded responses.Detection therefore relies on emitted-token patterns rather than potentially corrupted telemetry.Source: §C.1 · pp. 8–9
  • A seed-blind attacker cannot see the verifier’s seed, replayed token, or seeded margin, whereas a seed-aware attacker reconstructs the sampling step and can confine deviations to near-ties.Seed-blind operation requires keeping seed-derived randomness outside the compromised encoding component; seed-aware operation is treated as a stress test.Source: §C.1 · p. 9
  • The evaluated seed-aware code uses a binary choice between the prescribed token and runner-up; increasing dmax admits wider score gaps and raises both rate and exposure to detection.Each eligible position carries one bit, with rate K(dmax)/N bits per token.Source: §C.1 · p. 9
  • The seed-blind code uses arithmetic coding over up to 16 clean candidates within 0.5 logits of the maximum, an efficiency choice that uses a variable-size codebook.Seeded noise can reorder clean candidates, causing wide-margin mismatches; rates are ideal channel-use estimates, not measured payload.Source: §C.1 · p. 9
  • Both decoders require the emitted token and enough top-candidate logprobs to reconstruct the codebook; missing or truncated logprobs block these implementations.A lower-rate codebook recoverable from output tokens alone could still use the sampler channel.Source: §C.1 · p. 9

C.2 Dataset.

The experiments use separate prompt splits for calibration and testing, with matched attack and benign-control prompts. Four checkpoints sample different model families and architectures, while the protocol fixes generation and replay conditions.

C.2 Dataset.

  • 250 even-indexed prompts calibrate and 250 odd-indexed prompts test; each attack and its benign control use the same odd prompts.Each setting has one response per prompt, capped at 512 output tokens.Source: §C.2 · p. 9
  • The four checkpoints span three model families and dense and mixture-of-experts architectures, but do not isolate a factor or constitute a comprehensive survey.The models are Llama-3.2-3B, Qwen3-32B, OLMoE-1B-7B, and Qwen3-30B-A3B.Source: §C.2 · p. 9
  • Table 3 covers seed-aware scored output-token counts across three dmax settings, while Table 4 covers experimental and monitoring settings.Source: Table 3 · p. 10, Table 4 · p. 10
  • Generation and replay use the same bfloat16 checkpoint, temperature 1, top-k 50, top-p 0.95, seed 42, and concurrency 1.The pseudorandom payload uses seed 1234, separate from the sampling seed.Source: §C.2 · p. 10
  • Each model’s 250 even-prompt benign responses split into 125 for margin-threshold fitting and 125 for event-rate calibration; 1,000 streams resample complete test responses.The protocol sets α = γ = 0.005 and c = 0.5, yielding threshold 200 and a 1% bound under the prompt-level null and calibration-transfer assumption.Source: §C.2 · p. 10

C.3 Baseline [Rinberg et al., 2025]

The baseline scores served tokens against the token prescribed by a committed seed, then applies a three-tier token rule. Its response-level hard alarm triggers on any dangerous token and has no claimed stream-level error guarantee.

C.3 Baseline [Rinberg et al., 2025]

  • The verifier recomputes the committed-seed Gumbel perturbation and prescribed token, then scores the perturbed-logit gap between that token and the served token.The score uses the standard normal cumulative distribution function Φ and noise scale σ; the prescribed winner receives log 1/2 ≈ −0.69.Source: §C.3 · p. 10
  • Rinberg et al. [2025] classify tokens as safe at score ≥ τ, suspicious below τ with raw-logit rank ≤ R, and dangerous below τ with rank > R.The evaluation uses τ = −5 and R = 8.Source: §C.3 · p. 10
  • Table 5 reports fitted margin thresholds from 125 edge-fitting responses, with b0 = 0 and the last cell including margins above b6.Source: Table 5 · p. 11
  • The hard-alarm policy triggers on the first response containing a dangerous token, with no claimed stream-level error guarantee.The baseline leaves response-level pooling and deployment escalation unspecified.Source: §C.3 · p. 11
  • The e-process instead tests whether mismatches occur at wider seeded margins more often than under benign serving.Source: §C.3 · p. 11

D Detailed experimental results

Across resampled streams, the seed-blind attack alarms on every model, while seed-aware detection varies by model and rate. The results also show no held-out benign e-process crossings, alongside frequent hard-alarm crossings.

D Detailed experimental results

  • The results use 1,000 audit streams per model and setting, resampling complete responses with replacement from the test prompts.These streams probe resampling and response-order sensitivity within one fixed test pool, not independent runs or confidence intervals for a new deployment.Source: D Detailed experimental results · p. 11
  • Seed-blind streams alarm on every model, typically within the first fifteen responses (medians 6–14); seed-aware alarms increase with dmax on dense models but reach at most 1.7% on mixture-of-experts models.Figure 3 summarizes the fraction of attack streams alarmed by response across 1,000 bootstrap streams of 250 responses.Source: Figure 3 · p. 11, D Detailed experimental results · p. 11
  • Recorded-order seed-blind evidence paths cross within the first few responses, while detectable seed-aware paths cross and matched benign paths remain below threshold.Figure 4 plots accumulated log evidence for the benign control and both attacks, stopping curves at log 200.Source: D Detailed experimental results · p. 12, Figure 4 · p. 12
  • Mixture-of-experts models have several times more benign mismatches at wide seeded margins, raising calibrated bounds and leaving little excess for near-tie seed-aware mismatches.The four-model comparison does not establish an architecture effect.Source: D Detailed experimental results · p. 12
  • Among detected seed-aware settings, higher realized exfiltration rates bring earlier median alarms; the evaluated seed-blind codec operates at a higher rate and alarms within the first dozen responses.Figure 5 plots median first-alarm response against realized rate, censoring streams with no alarm by response 250.Source: D Detailed experimental results · p. 12, Figure 5 · p. 13
  • None of 1,000 resampled benign streams crosses the e-process threshold, while the hard-alarm baseline’s cumulative benign crossing rate approaches 100%.The maximum recorded-order benign log evidence is 0.15, versus an alarm line of log 200 ≈ 5.3.Source: D Detailed experimental results · p. 13, Figure 6 · p. 13
Loading 2610.11843v1…