PaperlayerView exact revision on arXiv

Source-linked AI summary

What Gradients Add to Text Leakage in Split Language Models, Counted per Token and per Document

Georgios Politis, Evangelos Pappas

arXiv:2610.04128v1cs.CRcs.CLcs.LG

TL;DR

Split learning keeps raw text on the client but shares activations, and training also sends gradients across the split. This paper compares attacks using activations alone or activations and gradients, and studies defenses and server-layer placement. On GPT-2, gradients add 3.17 percentage points in token recovery while increasing exact recovery of 32-token documents from 13.71% to 37.77%.

  • Problem

    Split-learning activations and training gradients are computed from private text, but how much gradients add when activations already reveal much of the text—and how scoring affects that assessment—remains unclear.

    Source: §1 · pp. 1–2

  • Method

    The paper compares otherwise controlled activation-only and joint attackers, then evaluates leakage and model quality across server-trained layer runs and defenses.

    Source: §1 · p. 2, §5 · p. 12

  • Results

    On GPT-2 split after layer six, token recovery rises from 94.20% with activations alone to 97.38% with gradients, while exact recovery of 32-token documents rises from 13.71% to 37.77%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Secret mixup can leave at most 1.48% of documents exactly recovered while 83.01–90.67% of tokens remain recoverable, supporting evaluation by token, document, and span recovery.

    Source: §6 · p. 15, §8 · p. 16

    The study covers GPT-2 and Qwen3-0.6B on 32-token WikiText-2 documents; larger models, longer documents, and other text may leak differently, and the second experiment uses a ridge-based decoder with one attack seed.

    Source: §7 · p. 16

Abstract

from arXiv · show

Split learning lets a client train a language model on a server without sending its text. The client runs the first layers itself and sends the server only their output, a vector of numbers for each token. During training, the server sends gradients back. We show that an observer at the split can rebuild most of the client's text from this traffic, and we measure how much the gradients help. On GPT-2, an attacker who holds only the publicly released weights of the client's layers recovers 94.20% of tokens from the activations alone and 97.38% when it also sees the gradients, 3.17 percentage points more 95% interval [2.72, 3.64]. Counted by document, the difference is much larger. The attacker rebuilds 13.71% of 32-token documents exactly without the gradients and 37.77% with them, because a document only counts when every token is right. How we count also changes how good a defence looks. Secret mixup, which blends each outgoing vector with a decoy, stops the attacker from rebuilding almost any document exactly, yet the attacker still recovers 83-91% of tokens. In a second experiment on GPT-2 and Qwen3-0.6B, where the server trains only a run of consecutive layers, the layer at which the run starts changes both model quality and leakage, even when the run's length is fixed. We recommend reporting leakage both per token and per document, and treating what a split model sends as being as sensitive as the text itself.

1 Introduction

Split learning keeps raw text on the client, but activations and training-time gradients crossing the split can reveal it. This section measures the gradients’ added leakage, shows that scoring by document changes the apparent strength of defenses, and examines how server-layer placement relates to model quality and leakage.

1 Introduction

  • The attacker observes activations at the split and, for the joint attack, also receives gradients during training; both signals are computed from private text.Earlier work attacked gradients, internal vectors, and both signals together, but did not answer how much gradients add when activations already reveal much of the text.Source: §1 · pp. 1–2
  • 94.20% of tokens are recovered from activations alone and 97.38% with gradients, while exact recovery rises from 13.71% to 37.77% for 32-token documents.The comparison uses GPT-2 split after layer six and publicly released weights for the client’s layers; Figure 1 reports the recovery measures across evaluation documents and runs.Source: §1 · p. 2, Figure 1 · p. 2
  • Secret mixup leaves at most 1.48% of documents exactly reconstructed while token recovery remains 83–91%, so document-only scoring can make the defense look stronger.Secret mixup blends each outgoing activation with a decoy from earlier training data.Source: §1 · p. 3
  • On GPT-2 and Qwen3-0.6B, the starting layer of a server-trained run affects model quality and leakage beyond run length; starts predict new outcomes only under the 8-bit defense.The experiments train server-side runs of two, four, or six consecutive layers, starting at one of the first four layers.Source: §1 · p. 3
  • Per-token and per-document scores can rank the same defense differently, so leakage reports should include both measures.Exact document recovery requires every token to be correct.Source: §1 · pp. 2–3

2 Related Work

Prior work establishes that gradients and internal vectors can reveal inputs, and studies attacks on split language models. This paper measures recovery across undefended and defended models, scoring units, and placements of server-trained layers.

2 Related Work

  • Split activations are an instance encoding, and prior analyses ask whether such encodings can be private [2] or bound how well they can be inverted [9].Split learning keeps raw data client-side and shares activations at the split [18], but a malicious server may steer client layers and rebuild private inputs [14].Source: §2 · p. 3
  • Prior attacks invert gradients [21] [20], text gradients [1], and internal vectors [12, 13]; the paper implements SipIt as one of its attacks.Property inference can reveal attributes of training data other than the data itself [10].Source: §2 · p. 4
  • BiSR optimizes guesses against activations and gradients, whereas this paper’s joint attacker uses gradients to suggest candidate tokens and activations to choose among them.The paper also adapts a VFLAIR-LLM model-inversion attack [6] and differs from earlier work by measuring text recovery across defenses, scoring units, and offloaded-layer starts.Source: §2 · p. 4

3 Threat Model and Setup

The threat model gives a passive observer split traffic and a copy of the client’s layers, but not the private text, server layers, or loss labels. The attack turns activations—and, for the joint attacker, gradients—into candidate tokens and uses the observed activations to select a reconstruction.

3 Threat Model and Setup

  • The passive attacker observes activations and, during training, gradients; it has a copy of the client’s layers but never the private text, server layers, or loss labels.Both attackers probe the final trained model, whose weights no longer change; the attacker does not collect gradients throughout training.Source: §3 · p. 5
  • Token, whole-document, and span recovery measure correct tokens, documents with every token correct, and annotated names, numbers, and identifiers reconstructed exactly.The paper reports all three because few exact document reconstructions can coexist with high token recovery.Source: §3 · p. 5
  • The main experiment uses GPT-2 at split depths 3 or 6, with 32-token WikiText-2 documents divided into calibration, training, and evaluation sets.The sets contain 32 public calibration documents, 128 client-training documents, and 256 evaluation documents.Source: §3 · pp. 5–6

3.1 How the attack works

  • The attack fits a ridge regression from activations to token embeddings using 32 public calibration documents, then ranks tokens by embedding cosine similarity.It tests candidate budgets k ∈ {8, 16, 32}.Source: §3.1 · p. 7
  • The joint attacker replaces half the activation-derived candidates with gradient-suggested tokens, while the first position receives no gradient help.The gradient signal at a position reflects the document’s next token; overlapping candidates are filled from the activation list.Source: §3.1 · p. 7
  • A width-two beam search extends candidate texts left to right and retains those whose recomputed activations best match the observations.The activation-only and joint attackers check the same number of partial texts; they differ in which candidates they try.Source: §3.1 · p. 7
  • Controls replace the gradient with another document’s gradient or random candidates; SipIt is also tested, but its exact-recovery guarantee does not apply when its budget expires.The backward-only heuristic uses the gradient alone; SipIt returns its best checked candidate when the iteration budget runs out.Source: Table 1 · p. 6, §3.1 · pp. 7–8
  • The main experiment runs 40 attack configurations across ten trained models and two attack seeds, with document-and-model variation assessed by crossed bootstrap.The primary token comparison uses depth 6, public client-layer weights, and k=32; its 95% interval resamples documents and trained models as separate axes.Source: §3.1 · p. 8

4 Gradients Add Little to Token Recovery but Much to Whole-Document Recovery

The section compares attackers who see activations alone with attackers who also see gradients, then tests gradient perturbations and two defences. Activations already expose most tokens, but gradients add substantially to exact document recovery.

4.1 Activations alone reveal almost every token

  • 94.20–97.81% of tokens are recovered from activations alone across both split depths and both client-layer copies.These rates also describe an attacker watching inference traffic, where only activations are sent.Source: §4.1 · p. 9

4.2 Gradients add three points of token recovery

  • +3.17 percentage points: at depth six with public client-layer weights, gradients raise token recovery from 94.20% to 97.38%.The 95% interval is [+2.72, +3.64], and the lower end clears the pre-set two-point mark.Source: §4.2 · p. 9, Table 2 · p. 9

4.3 Whole-document recovery nearly triples

  • 13.71% to 37.77%: at the main setting, gradients raise exact recovery of 32-token documents by 2.75-fold.The same gain appears at both depths and with both client-layer copies; Figure 4 compares the two recovery measures.Source: §4.3 · p. 9, Figure 4 · p. 10, Table 3 · p. 10

4.4 Perturbing the observed gradient changes little

  • Noise and rounding leave most gradient-assisted recovery in place, while shuffled-gradient and random-candidate controls fall below the activation-only baseline.Figure 5 compares token and whole-document recovery across gradient conditions against the activation-only line.Source: §4.4 · p. 10, Figure 5 · p. 11

4.5 Defences separate token recovery from whole-document recovery

  • Secret mixup leaves 83.01–90.67% of tokens recoverable but at most 1.48% of documents exactly recovered; replica-median changes token recovery little while lowering document recovery by up to 7.7 points.Secret mixup raises held-out perplexity by 31.8% at depth three and 26.9% at depth six, while replica-median costs almost nothing in model quality.Source: §4.5 · pp. 11–12, Figure 6 · p. 12

5 Where the Server-Trained Layers Start Matters

The paper tests whether the starting layer of a server-trained run matters beyond its length, measuring model-quality cost and leakage. Start position explains substantial variation, though the results differ by model, defence, and whether seeds are held out.

5 Where the Server-Trained Layers Start Matters

  • Across GPT-2 and Qwen3, server-run start position explains variation beyond run length in seven of eight tests; the exception is GPT-2 leakage under the 4-bit noisy defence.The tests combine start and its interaction with length using partial R2, with a threshold of 0.10 and permutation p < 0.0125.Source: Table 5 · p. 14, §5 · p. 14

5.1 In GPT-2, the start matters in three of four tests

  • In GPT-2, start position matters for utility under both defences and leakage under 8-bit rounding, but 4-bit noisy-defence leakage fails the corrected test.Under 8-bit rounding, runs starting at layer three pay several times the extra loss of earlier starts and leak a fraction as much.Source: §5.1 · p. 14
  • For GPT-2 held-out seeds, adding start lowers utility and leakage-index RMSE beyond 10% under 8-bit rounding, but not under the 4-bit noisy defence.For example, under 8-bit rounding, utility RMSE falls from 1.5698 to 0.7594 and leakage-index RMSE from 0.3042 to 0.1337.Source: §5.1 · p. 14

5.2 In Qwen3, the start passes all four tests but predicts held-out seeds only under the 8-bit defence

  • Qwen3 passes all four start-effect tests under both defences, but on held-out seeds start improves RMSE only under 8-bit rounding.Under the 4-bit noisy defence, held-out utility and leakage-index RMSE rise when start is added.Source: §5.2 · p. 15

6 Discussion

Activations already reveal most tokens, while gradients add a larger gain in exact-document recovery. Defence scores diverge by metric, and server-run placement can affect utility and leakage beyond run length.

6 Discussion

  • Activation-only attacks recover at least 94.20% of tokens at every depth and with both copies of the client’s layers; gradients add little per token but much per document.Source: §6 · p. 15
  • Secret mixup leaves 83.01–90.67% of tokens recoverable while cutting exact-document recovery to near zero; replica-median barely changes token recovery but lowers exact-document recovery by up to 7.7 points.The two scoring units can therefore make the same defence look very different.Source: §6 · p. 15
  • The starting layer explains utility variation under both tested defences, but improves held-out-seed prediction only under 8-bit rounding.Under 4-bit rounding with gradient clipping and noise, it also explains Qwen3 leakage variation but does not improve held-out-seed prediction in either model.Source: §6 · p. 15
  • The joint attacker uses the same activation-and-gradient signals as BiSR, but measures their added value against an activation-only attacker with the same candidate budget.The measured gradient contribution depends on whether recovery is counted per token or per document.Source: §6 · p. 15

7 Scope and Limitations

The measurements cover two small models, WikiText-2, and 32-token documents; larger models, longer documents, and other text may leak more or less. The second experiment also uses a limited leakage-evaluation setup.

7 Scope and Limitations

  • The study measures GPT-2 (124M parameters) and Qwen3-0.6B on 32-token WikiText-2 documents, with splits after the third or sixth layer.The authors note that leakage on larger models, longer documents, or other kinds of text may differ in either direction.Source: §7 · p. 16
  • Defended results show at least what an attacker using the undefended-model methods recovers; attacks designed for a known defence could recover more.The reported defended attacks use the same methods as the undefended setting.Source: §7 · p. 16
  • Second-experiment leakage uses a ridge-based decoder and one attack seed, rather than the published optimisation-based model inversion.Source: §7 · p. 16

8 Conclusion

The joint attacker recovers more than the activation-only attacker, with a much larger difference for exact documents than for tokens. The paper recommends treating split traffic as sensitive and evaluating leakage across multiple scores and layer placements.

8 Conclusion

  • On GPT-2 split after layer six, token recovery rises from 94.20% to 97.38% with gradients, while exact recovery of 32-token documents rises from 13.71% to 37.77%.The joint attacker has the publicly released copy of the client’s layers.Source: §8 · p. 16
  • In partial offloading, the starting layer affects model quality and leakage beyond what run length explains, so placement should be evaluated alongside length.The paper also recommends checking placement rules on held-out seeds under the intended defence.Source: §8 · p. 16
  • Secret mixup leaves 83.01–90.67% of tokens recoverable despite reducing exact-document recovery to at most 1.48%, motivating evaluation by token, document, and span.At tested strengths, observed-gradient noise or rounding leaves token recovery at 96.38–97.33%.Source: §8 · p. 16

A Evidence Availability and Reproduction

The appendix describes a public release intended to support checking and reproducing the experiments. It provides experiment materials and verification steps, including recorded evidence and analysis outputs.

A Evidence Availability and Reproduction

  • The release contains the experiments’ code snapshots, exact inputs, pre-run protocols, per-run results, logs, and analyses in a public Hugging Face dataset frozen at tag v1.0.The appendix also identifies experiment-specific release paths and checksums.Source: A Evidence Availability and Reproduction · pp. 16, 19
  • Reproduction begins by downloading the pinned v1.0 release, checking files against its manifest, and installing the code snapshot in a fresh Python environment.The snapshot pins the research dependencies.Source: A Evidence Availability and Reproduction · p. 16
  • The main GPT-2 experiment comprises 3,680 evaluations across 16 shards, with an official analyser and verification guide for reproducing headline numbers.The appendix describes hash-chained prediction journals and records the evaluation inputs, merged results, comparisons, and binary captures.Source: A Evidence Availability and Reproduction · p. 17
  • The release maps internal file labels to the paper’s names, and Table 6 is identified as the label-mapping table.Source: A Evidence Availability and Reproduction · p. 18, Table 6 · p. 18

B Full Attack Matrix

Across the 40 attack configurations, token recovery generally favors depth three over depth six and a trained copy over a public copy, with the backward-only heuristic reversing the depth pattern. Secret mixup lowers token recovery, while replica-median stays close to undefended rows.

B Full Attack Matrix

  • Depth three generally leaks more than depth six, and trained copies leak more than public copies throughout the 40 configurations.The backward-only heuristic reverses the depth ordering, reaching approximately 24% at depth three versus 50% at depth six.Source: Figure 7 · p. 20
  • Secret mixup lowers token recovery, whereas replica-median stays close to undefended rows.The figure places 28 undefended configurations above the divider and 12 defended configurations below it.Source: Figure 7 · p. 20
  • Table 7 reports token recovery percentages averaged over public and trained client-layer copies, with each entry aggregating 10,240 document-level evaluations.Figure 7 instead aggregates 5,120 evaluations over 256 distinct held-out documents per entry.Source: Figure 7 · p. 20, Table 7 · p. 21

C Partial-Offloading Experiment: Full-Precision Tests and Means per Placement

The partial-offloading results report placement means and test whether the layer-run start explains variation in utility and leakage. The start passes three of four GPT-2 tests and all four Qwen3 tests.

C Partial-Offloading Experiment: Full-Precision Tests and Means per Placement

  • In the released tables, the 8-bit defence is labeled D0 and the 4-bit noisy defence D1.Tables 8 and 9 give the variance tests of Table 5 to four decimal places.Source: C Partial-Offloading Experiment:… · p. 22
  • The start passes three of four GPT-2 variance tests and all four Qwen3 tests.The tests examine leftover variation explained by the start and its interaction with length; the Qwen3 tests report partial R2 and permutation p values.Source: Table 8 · p. 22, Table 9 · p. 22
  • The released GPT-2 results retain per-seed values, three property AUCs, per-document scores, and crossed bootstrap intervals for utility and VMI excess.Means are measured on final trained models.Source: C Partial-Offloading Experiment:… · p. 22
  • Table 10 reports GPT-2 means across eight seeds for every placement, including utility as excess held-out loss in nats and VMI excess as recovery minus the majority control.It also reports the normalized leakage index I.Source: C Partial-Offloading Experiment:… · p. 22, Table 10 · p. 23

D Audit of an Earlier Programme and Withdrawn Claims

The audit recovered and checked part of an earlier programme's archive, but the paper's claims do not rely on that programme. Missing executed source prevents recovered files or corrected arithmetic from reinstating its retired claims.

D Audit of an Earlier Programme and Withdrawn Claims

  • The audit verified 3,720 of 5,167 historical manifest references without mismatches, but did not recover another 772, chiefly tensors and corpus assets.The audit covered five internal execution hosts.Source: D Audit of an Earlier Programme and… · p. 24
  • The retired causal, corpus-frontier, and P5 gate claims cannot be reinstated because the exact executed source is missing and cited hashes cannot reconstruct it.The owning project retired the pre-reset results on 10 September 2026.Source: D Audit of an Earlier Programme and… · p. 24
  • P0's 30 engineering checks did not qualify the later privacy attacks; recovered P4B qualification passed punctuation but failed vowel and newline targets.P4B measured Pearson correlation minus shuffled-null correlation, not a difference in R2; its historical definitions cannot replace partial-offloading token-presence AUCs.Source: D Audit of an Earlier Programme and… · p. 24
  • 0.016701851273 nats is the mean recovered P2 gain, defined as frozen-random loss minus trained loss, and correcting its arithmetic does not repair missing provenance.The two-sided 95% Student interval is [0.014001090665, 0.019402611881].Source: D Audit of an Earlier Programme and… · p. 24
  • Newer external exploratory runs labeled R1 are not held-out confirmation, and the paper's claims do not use them.The audit report and recovered archive are available in the v1.0 dataset release.Source: D Audit of an Earlier Programme and… · p. 24

Related papers

Loading 2610.04128v1…