Source-linked AI summary
Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys
Georgios Politis, Evangelos Pappas
TL;DR
This paper examines a split-LLM privacy evaluation that passed while omitting an observable backward gradient channel. It uses a calibrated, channel-explicit audit and finds exact row-partition disclosure across frames, while mitigation removes the signal at a small held-out cross-entropy cost.
Problem
The gap is an evaluation that returns a privacy pass without testing whether its metrics detect a known leak on every declared channel.
Method
The paper audits a two-node split-LLM system with a fixed protocol covering declared channels, injected-leak calibration, shuffled-label controls, and a pre-set gate.
Results
On both datasets, all six defended open-gradient mitigation runs passed forward privacy and utility gates but failed the joint view; clipping and Gaussian noise removed the signal at ≈0.01 nats.
Takeaways & Limitations
The returned gradient’s zero-support pattern discloses which rows are real, collapsing the decoy padding intended to provide anonymity.
Takeaways & Limitations
The evaluation does not measure five attack classes, including attacks accumulating observations across training steps, and held-out decisions use blocks within one flattened corpus stream rather than held-out documents.
Abstract
from arXiv · showhide
We present a systems-security case study of a two-node split-LLM training system whose privacy evaluation passed while leaving an observable channel untested. The Trusted Local Node (TLN) sends protected activations to the Untrusted Cloud Node (UCN), the UCN returns its output, and TLN, holding the private loss, returns the output gradient. The frame the UCN receives mixes real rows with decoys, and the loss ignores the decoys. Their gradients are exactly zero, so the pattern of zeros reveals which rows were real. We measure it with a protocol fixed in advance: a leak injected at known strength to prove the instrument can see one, a shuffled-label control to prove it does not report absent leaks, and a threshold set before the runs. Across nine seeds, the zeros identified the real rows on every frame, 4,096 of 4,096 per run. An attack on the frame contents recovered about one extra token per hundred over a constant-guess baseline (+0.65 to +1.50 percentage points); the shuffled controls recovered nothing. A second set of runs repeated this on a configuration that keeps model quality within budget, so the finding is not confined to a setting nobody would deploy. On both datasets, every such run passed the forward-channel privacy check and the quality check, yet failed that same check once the returned gradient was included. Clipping and noising each row of the gradient closed the leak for about 0.01 nats of held-out cross-entropy. The system is not thereby safe: five classes of attack, including those accumulating observations across training steps, were never measured.
1 Introduction
This case study finds that a split-LLM privacy evaluation passed because it omitted the backward gradient channel, whose zero-support pattern reveals real and decoy rows. The paper diagnoses this implementation and evaluation failure and bounds content inference to a frequent-token probe.
- System context: The paper studies a concrete two-node split-learning system in which the trusted side holds the private loss and returns gradients needed by the cloud.The privacy claim is that the cloud cannot read the training data, making channel coverage central to the evaluation.
- Evaluation gap: The original evaluation instrumented only the forward activations and passed its privacy gate, leaving the returned output gradient outside the declared privacy surface.The paper treats the zero-support construction as an implementation and system-design defect, and the false assurance as an uninstrumented-channel evaluation failure.
- Structural disclosure: Exact gradient support lets the cloud discard 48 decoys and isolate 32 loss-bearing rows before any content attack.This collapses the anonymity set that padding was intended to create.
- Claim boundary: The content evidence is limited to a structural metadata disclosure and a separately bounded implemented frequent-token probe, not text reconstruction.The paper positions itself as a systems-security case study with a calibrated protocol applied to one system.
2 The system and the threat model
The system splits training between a trusted local node and an untrusted cloud node, protecting the forward activation wire while leaving the returned gradient wire raw. Its threat model treats the cloud as fully compromised and requires privacy evaluation across all observable channels.
- Protocol: TLN sends protected latents to UCN, receives UCN’s output, computes the private loss, and returns the output gradient.Both tensors are held by the untrusted node in the clear once they cross the boundary.
- Defence and framing: Each 80-row frame contains 32 real rows and 48 decoys, with a width-D=64 latent bottleneck, per-request rotation and permutation, clipping, and noise.The original evaluation declared and instrumented only the forward wire; the backward wire remained outside the declared adversary view.
- Experiments: The experimental setup covers the main configuration plus replication and mitigation runs, with full fingerprints identifying the evaluated model weights and corpus.The cited setup passage identifies the scope of Table 1 but does not report additional configuration values.
- Threat model: The fully compromised UCN can read every message, state update, and cross-step observation on its side.Accordingly, the declared adversary view includes multiple channels, and the privacy claim depends on enumerating and testing them.
- Channel status: The forward wire is clipped and noised and passed its gate, whereas the backward wire is unclipped, unnoised, and leaks.The three-hop protocol makes the backward hop’s left-to-right movement explicit despite its role in the backward pass.
- Per-frame anatomy: Figure 2 distinguishes sealed data, active obfuscation or protection layers, and the unprotected backward wire marked as the leak.Its per-frame anatomy shows the forward probe passing while later backward steps leak.
3 Evaluation protocol: declared channels, calibrated metrics, and the gate statistic
The protocol addresses an evaluation gap by declaring every adversary-visible channel, calibrating metrics against injected leaks, and applying a predeclared gate statistic with shuffled-label controls.
- Declared channels: The evaluation must enumerate forward, backward, membership, and timing channels before measurement; undeclared channels are exempt from the gate.The backward channel was the exemption in which the reported leak lived.
- Calibrated metrics: Metrics are calibrated with controlled token leaks so thresholds reflect each metric’s detection curve rather than an untested privacy claim.A metric without a measured detection curve cannot support a primary claim.
- Gate statistic: The gate statistic compares Bonferroni-adjusted Wilson upper-95 accuracy with a constant most-frequent-token baseline and fails when G > +1.0 pp.The baseline is near 5–6%, and +1 pp represents roughly one extra correctly recovered token per hundred.
- Controls and uncertainty: Shuffled-label controls provide a channel-specific false-positive check, while paired effects are bootstrapped over frames and, across seeds, hierarchically over seeds and frames.The control effect N is not subtracted inside G.
- Verdict terms: Detection and gate breaking are distinct verdicts: a paired effect may be detected below the gate, whereas the gate breaks only when the stated upper-bound criterion is exceeded.The paper fixes these verdict terms before interpreting results.
4 Instrument calibration
Calibration shows that the four metrics respond differently to injected token leaks, so thresholds must be metric-specific and supported by representation-matched positive controls.
- The four metrics disagree: Rare token top1 is most sensitive, while token cross entropy remains nearly unchanged until the injected leak dominates.The dose–response sweep spans 19 doses at amplitude 1.0.
- The four metrics disagree: Membership auc misses the low-dose token signal because the injection tests token identity rather than membership, so it is retained only as a generalisation diagnostic.Across the full sweep, AUC−0.5 spans 0.058–0.159.
- Metric-specific thresholds: No single threshold fits all metrics; thresholds are therefore set per metric and remain budget- and frame-invariant along the remaining sweep axes.The historical +1.0 pp Bonferroni–Wilson gate is preserved after calibration.
- Positive control: A floor reading is interpretable only when the same probe succeeds on a representation-matched undefended control.The split-14 positive control uses a naked capture and a probe trained to map released latent rows back to tokens.
- Positive control: +24.19 pp on the naked boundary is a sensitivity demonstration, not a primary leak estimate; the defended comparison tests whether the representation-matched attacker reads through the defence.The naked result selects the best epoch on the evaluation set.
5 Results: structural gradient leakage and bounded content inference
The returned gradient deterministically reveals which frame rows are real, while the measured content inference remains modest and bounded to a frequent-token probe. The finding persists in a four-layer configuration that passes forward privacy and utility gates, and row-wise clipping plus noise removes the structural signal at about 0.01 nats of held-out cost.
- Structural gradient leakage: 4,096/4,096 frames exactly matched real rows to decoys because decoy gradients were identically zero.This is the structural disclosure measured across the evaluated seeds.
- Bounded content inference: The backward wire supplies the partition that makes the forward frame readable, while its direct content contribution is generally small.The gradient’s margin over the oracle-partitioned forward frame was at most +0.18 pp on five exploratory seeds and +0.64 pp on seed 42.
- Bounded content inference: Rare-token recovery, sequence reconstruction, and held-out-text reconstruction were not established.Those claims require different emitters and controls.
- Deployable configuration: On all six defended, gradient-open cells, runs passed forward privacy and utility gates but failed the joint-view +1.0 pp gate.The six cells covered three seeds on each of two datasets; gradient-only readings stayed near guessing.
- Mitigation: Per-row gradient clipping and Gaussian noise removed the structural disclosure in 0/1,024 frames with about 0.01 nats of held-out cost.Agreement was 0.400 at the 32/80 base rate.
- Mitigation: Turning the defence off restored a +34 pp joint-view signal, confirming the defended near-floor readings were not simply instrument blindness.The naked cells also exceeded the forward gate by roughly ten percentage points.
6 When does the structural signal convert to a token advantage? Depth, width, and budget
The structural partition signal is deterministic across audit cells, but its conversion into a token advantage depends on model shape rather than exposure budget in the tested range. The study reports single-seed shape readings and does not establish a monotone dose–response.
- Structural signal: Every audit cell exactly recovered the real-versus-decoy partition, including cells where the token effect was undetected.The nine cells were one-run, seed-42 readings with shuffled-label controls at floor.
- Budget: The paired token effect did not increase from 40k to 100k gradient exposure: +0.9302 versus +0.9066 per row.The tested exposure budget increased 2.5×.
- Depth and width: The token advantage was detected at 12 and 11 delegated layers but not at 8 and 6, bracketing the gate between 8 and 12 layers.The 11-layer point is confounded with budget, and the ladder estimates are not monotone.
- Depth and width: The paired effect was detected at latent widths D=64 and D=96 but not at D=128.These shape thresholds are single-seed readings.
7 External audits and related work
The external audit positions this work as an evaluation-method contribution rather than a claim that prior defences are ineffective. Its distinctive combination is channel coverage, calibrated positive controls, shuffled-label negatives, and predeclared gates.
- Audit scope: The audit selected three recent evaluations as a purposeful sample spanning bidirectional attacks, bidirectional defence, and attack-plus-defence work.It was not intended as a field survey.
- Selected evaluations: None of the three audited works combined all three controls used here.The comparison concerns evaluation semantics, not defence efficacy.
- Selected evaluations: BiSR lacks an acceptance gate and shuffled-label negative control, while DualGuard covers forward, backward, and bidirectional paths without the full calibration-control-gate combination.From Prompts to Responses is forward-only and uses an unmatched random-token baseline in its appendix.
- Related work: The broader literature already covers reconstruction, label leakage, malicious backward-signal control, and intermediate-state leakage.The narrower contribution is integrating calibrated onset, channel-specific negative controls, advance gates, and artefact traceability.
- Comparison boundary: The positioning table should be read as an evaluation-method comparison, because literature rows retain their authors’ own uncalibrated metrics and unmatched-null patterns.The paper’s excess is measured against a matched no-attack control with a confidence bound and predeclared gate.
8 Scope and limitations
The evidence is bounded to one split-training implementation, WikiText-2-centered experiments, diagnostic-scale mitigation runs, and the measured attack families. Several adversarial channels and stronger reconstruction claims remain untested, while some verification artefacts are unavailable for independent re-execution.
- Scope: The study’s findings concern one implementation; the second corpus is a three-seed diagnostic robustness check, not evidence of corpus independence.The external evaluation covers only three targeted recent works.
- Unmeasured attacks: Five adversarial families remain unmeasured, including membership/property inference, response-side recovery, timing metadata, stateful remote state, and accumulated history.Active perturbation was also not executed at mitigation-run scale.
- Utility and scale: The mitigation result comes from 2,000-step diagnostics, so convergence-scale evidence remains out of scope.The main configuration also fails the utility gate, with Δloss 0.9185 originally and 0.896 in the clean rerun versus the 0.35 gate.
- Metric limitation: A near-zero MINE reading is not a privacy certificate: the same cell held a detected +0.758 pp probe effect.Finite-sample lower bounds cannot upper-bound true mutual information.
- Evaluation assumption: Held-out cross-entropy uses blocks within one flattened corpus stream rather than held-out documents, so document-independent claims are out of scope.This applies to the 0.35 utility gate and the ≈0.01 nat mitigation cost.
- Availability: The protocol and thresholds are released, but raw prediction tensors, the complete transcript, and the internal scorer’s verification data are not independently re-executable.The internal scorer matched nine primary seed values to ≤10^-6 pp.
9 Conclusion
The case study shows that a forward-only privacy evaluation can pass while an omitted backward channel reveals real rows among decoys. Per-row gradient clipping and Gaussian noise suppressed the implemented probes below the gate at an approximately 0.01-nat held-out cross-entropy cost.
- The omitted backward channel exposed which rows were real and which were decoys through their returned gradients.The zero-support construction is both an implementation defect and a system-design defect.
- On both datasets, all six defended open-gradient cells passed the forward privacy and utility gates but failed the joint-view privacy gate.The joint-view result held across three seeds on each dataset.
- Per-row gradient clipping plus Gaussian noise removed the zero-support partition signal and suppressed implemented probes below the gate at approximately 0.01 nats of held-out cross-entropy.
- A passing privacy verdict is meaningful only for the channels, attacks, and leak magnitudes that the evaluation instrument has tested.The protocol used controlled injection, predeclared gates, and shuffled-label negative-control falsification.
A Artifact and verification index
The artifact index is organized by claim level rather than directory, distinguishing inspection of committed summaries from end-to-end re-derivability. Availability boundaries are recorded for each result.
- The artifact index is organized by claim level rather than by directory.
- Committed summaries or manifests support inspection of displayed values, while “re-derivable” denotes end-to-end execution from the release alone.
- Availability boundaries are recorded separately for each result.
A.1 Headline results
The verification index records committed summaries, code, manifests, and evidence boundaries, while distinguishing cluster-side raw tensors and non-re-executable checks. It also records a withdrawn forward-membership result and a held-out-split discrepancy.
- The committed manifest covers 3 seeds, 30,000 optimiser steps, and 180,636 events, while raw cluster-side prediction tensors remain uncommitted.The hardened verifier and forward, gradient, and joint-view consumers confirm the committed records.
- The committed verification trace reproduces the seed-44 paired advantage of +0.6929 and requires the shuffled negative control to remain at floor.The associated scorer computes the paired statistic against a constant baseline over frame-clustered evaluation rows.
- An internal scorer reimplementation matched all nine primary seed values to ≤10^-6 pp, but the check is not repository-re-executable.The scorer and its prediction tensors are uncommitted.
- The mitigation and exploratory results are redisplayed from committed derived artefacts, but their raw prediction tensors remain on the cluster.
- The evaluation runner used a sequential fixed-width block split rather than the declared document-level held-out split, leaving document-independence claims out of scope.
- A forward-membership reading of +0.068 AUC was falsified by shuffled-label controls and withdrawn as a non-generalizing coordinate-probe artifact.It decomposed into in-sample memorisation of +0.095 with a corpus-region term of approximately zero.
B The defended stack, mechanism by mechanism
The defended stack combines standard mechanisms whose measured interaction depends on composition and configuration. Forward-only batteries on the defended cell remained below the +1.0 pp gate, but several mechanisms and attack families have limited or withdrawn status.
- The fragmentation cell was invalidated because remote modules never received a gradient, while public-pretraining, capacity framing, and the deep-defended-cloud capacity conclusion were withdrawn.
- The table records each mechanism’s implementation site and current status as core, available, optional, withdrawn, or invalidated.
- The stack’s engineering contribution is the composition order and measured interaction of standard primitives, not novel primitives in isolation.
- The bottleneck alone leaks, the gauges alone leak, and noise alone destroys utility; the validated composite reaches the floor only at the tested settings.
- The forward-only batteries target the defended forward cell and include a representation-matched positive control absent from the original batteries.
- The larger 35B-A3B model was used only as a capacity check, and no body result depends on it.
- None of the compromise-fraction arms approached the +1.0 pp gate, and none improved as the attacker received more secret information.