PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Attack Suite: A Unified Red-Team Framework for Privacy Evaluation of Split and Distributed LLM Systems

Georgios Politis, Evangelos Pappas

arXiv:2610.22722v2cs.CRcs.DCcs.LG

TL;DR

Privacy evaluations of split models are difficult to compare when attacker access, model knowledge, datasets, and recovery measures differ. LLM-Attack Suite records these choices and supports checkable comparisons; its studies show that corpus and defence conditions yield different recovery outcomes. The paper limits its strongest quantitative reconstruction comparisons to GPT-2 124M under stated corpora, lengths, and budgets.

  • Problem

    Privacy results depend on attacker observations and knowledge, public data and computation, dataset, and recovery measure, complicating interpretation and comparison across studies.

    Source: §1 · p. 1

  • Method

    LLM-Attack Suite records evaluation choices, reuses observations across compatible attacks, retains predictions for independent scoring, and requires linked undefended references for defence passes.

    Source: §1 · pp. 1–2, §3 · p. 4

  • Results

    Across corpora, recovery ranged from 63.25% on WikiText-2 to every token on templated clinical notes; stable-compressor depth profiles retained 18.85–68.59% mean-token recovery.

    Source: §5.2 · pp. 6–7

  • Takeaways & Limitations

    Privacy claims for split models should name the attacker, corpus, and scoring rule, report controls, and label untested attacks as untested.

    Source: §8 · p. 10

    The strongest quantitative reconstruction comparisons concern GPT-2 124M under the stated corpora, lengths, and budgets; aggregating different protocols into one success rate would discard their distinctions.

    Source: §6 · p. 9

Abstract

from arXiv · show

Split language models keep input text on a client but send intermediate representations to a server. Attacks can reconstruct text from these messages. Comparing privacy evaluations is difficult when attackers see different messages, know different model weights or use different datasets and recovery measures. We present LLM-Attack Suite, a framework that records these choices and lets researchers check, compare and extend an evaluation. Compatible attacks reuse recorded messages, and their predictions are saved for independent scoring. The scoring rule requires a linked undefended reference before a defence can pass. Reports also identify attacks that remain untested. In a controlled GPT-2 experiment, adding gradients returned during training increased token recovery from 94.20% to 97.38% and exact-document recovery from 13.71% to 37.77%. The 95% interval for the token gain lay above the two-percentage-point threshold fixed before testing. This experiment is shared with a companion study; here it tests how the evaluation responds to a change in attacker access. Further studies measure how corpus choice and compression defences affect recovery, compare an adapted published attack with its reference implementation, and test the measurement code. The framework reports implementation checks separately from measured attack effectiveness. Each privacy claim should state the attacker, dataset and scoring rule, report its controls and identify untested attacks.

1 Introduction

LLM-Attack Suite makes privacy evaluations checkable and comparable by recording attacker access, data, resources, controls, and predictions. Its experiments illustrate how attacker observations and evaluation choices affect reconstruction results.

1 Introduction

  • The suite records attacker access, data, and resources, reuses observations across compatible attacks, and retains predictions for independent scoring.Its scoring combines dataset-specific baselines and linked undefended controls, while reports identify missing tests.Source: §1 · pp. 1–2

2.1 Threat model

  • The passive attacker observes split messages but receives neither private text nor next-token labels, and does not modify traffic.Public client layers are the default knowledge condition; trained layers are a stronger condition.Source: §2.1 · p. 2

2.2 A worked comparison

  • The worked comparison holds search limits fixed while adding gradients; complete-document recovery gains more than token recovery.Predictions and settings are saved so both measures can be checked independently.Source: §2.2 · p. 3

2.3 Protocol, capture and replay

  • The protocol records target, attacker, observations, knowledge, budgets, corpus, measures, and controls, then replays compatible attacks on shared captures.This supports paired comparisons and later attack additions without retraining the victim.Source: §2.3 · pp. 3–4

2.4 Modules, coverage and evidence

  • Modules use common input and output contracts, while scorecards, coverage reports, and validation records distinguish measured outcomes, untested threats, and implementation checks.The workflow supports attack comparisons, defence studies, and reviewer rescoring of retained evidence.Source: §2.4 · p. 4

3 Scoring and Decision Rules

The scoring rule measures tested attacks’ excess recovery above a corpus baseline and gates defence passes on repeated runs and a qualifying undefended reference. Its verdict is limited to the tested attacks and settings.

3 Scoring and Decision Rules

  • The rule takes the largest valid attack’s upper recovery bound above a simple-guess baseline, using the most common calibration-text token for token reconstruction.The envelope is unmeasured when no valid attack ran.Source: §3 · p. 4
  • The upper-bound gate is an operational screening convention, not a distribution-free privacy guarantee or a simultaneous confidence guarantee across all modules.Token correlation also limits a literal binomial interpretation.Source: §3 · p. 4
  • A defence passes only when the largest tested-attack excess is at most one percentage point, with three seed runs and a qualifying linked undefended reference.The reference must cover the same attacker family and show excess above one percentage point.Source: §3 · pp. 4–5
  • At finite sample sizes, even baseline-level recovery has positive excess, so a failed gate must be read with its estimate and interval.The reported verdict names its corpus, attacker set, and settings and does not extend automatically to another protocol.Source: §3 · p. 5

4 Experimental Design

The experimental design pairs controlled reconstruction tests with separate corpus, defence, and implementation comparisons. These protocols specify distinct attacker knowledge, observations, data, and resource conditions.

4.1 Controlled reconstruction

  • At GPT-2 124M layer three or six, the controlled study uses pinned 32-token WikiText-2 passages, held-out documents, and separate public- and trained-layer conditions.Ten seeded models are fine-tuned and frozen before attacks; each run includes calibration, training, and evaluation documents.Source: §4.1 · p. 5
  • The joint attacker combines activation and gradient candidates while matching the activation-only attack’s candidate count and beam width; shuffled-gradient and random-candidate controls test gradient information.The prespecified primary test requires the 95% interval for token gain to exceed two percentage points; 47 secondary contrasts use corrected intervals.Source: §4.1 · p. 5

4.2 Additional protocols and comparators

  • The corpus study applies linear inversion without beam search to code, templated clinical notes, and WikiText-2, with separate calibration-fitted baselines.Each corpus pools three seeds and 3,072 scored positions.Source: §4.2 · p. 5
  • The defence-aware study optimises embeddings through the actual compressor across eleven depths and three compressed widths, separating stable-compressor and co-trained observation protocols.The co-trained protocol uses trained layers, final-compressor captures, and held-out final-state inference.Source: §4.2 · pp. 5–6
  • Table 1 aggregates 5,120 document-level scores from the same 256 held-out documents across ten models and two attack seeds.Highlighting marks the prespecified primary setting.Source: Table 1 · p. 6
  • The adapted BiSR comparison uses fresh inverters and fixed per-attack budgets, measuring specified configurations rather than equal-compute ranking or the original two-cut protocol.Reference conformance is checked separately on fresh held-out documents.Source: §4.2 · p. 6

5 Results

The results show that attacker access, corpus, defence, and protocol choices change measured text recovery, while the suite separately checks attack conformance and measurement procedures. The findings are tied to stated GPT-2 settings and do not establish a universal attack ranking.

5.1 Returned gradients increase reconstruction

  • 97.38% of tokens were recovered by the joint attack versus 94.20% by activation-only reconstruction, a 3.17-point gain; exact-document recovery rose from 13.71% to 37.77%.The token-gain 95% interval was [2.72, 3.64].Source: §5.1 · p. 6
  • Shuffled-gradient and random-candidate controls recovered 93.91% and 93.89%, versus 49.98% for the backward-only heuristic; defended outcomes varied across measures and configurations.Gradient noise and eight-bit quantisation left recovery at 97.31% and 97.33%; secret mixup left 83.01–90.67% token recovery but at most 1.48% exact-document recovery, while name-span recovery remained 47.85–66.15%. Figure 2 shows exact-document recovery ranging from 1.70% to 58.55% across undefended configurations at the same depth-six boundary.Source: §5.1 · p. 6, Figure 2 · p. 7

5.2 Corpus and defence change the conclusion

  • The inversion attack recovered every token of templated clinical notes, compared with 64.94% of code and 63.25% of WikiText-2; these results do not measure recovery from clinical records.Corpus choice changes recovery and its baseline.Source: §5.2 · pp. 6–7, Table 2 · p. 7
  • Across stable-compressor depth profiles, white-box recovery was 18.85–68.59% of mean tokens; co-trained bottlenecks yielded 23.52–67.68% under a separate final-state protocol.Each depth mean averages three widths and three seeds; twenty checks validated the gradient path, work accounting, and capture matching.Source: §5.2 · p. 7
  • In the horizon pilot, no verdict changed between 20 and 100 training steps; at 500 steps, fixed random projection suppressed its declared attacker below majority baseline but largely prevented useful learning.Cross-entropy was 7.42–7.75 with the defence versus 4.39–4.44 without it; Gaussian activation noise yielded 9.96–55.05% recovery in longer-context settings.Source: §5.2 · pp. 7–8

5.3 Published-attack conformance and comparison

  • The adapted BiSR port matched its archived reference on all 3,072 paired predictions; refined BiSR recovered 93.21% versus 97.38% for the joint attack with public layers, but nearly tied with trained layers.The public-layer paired difference was −4.17 points, with 95% interval [−4.79, −3.53]; the trained-layer difference was −0.02 points, interval [−0.63, 0.58].Source: §5.3 · p. 8

5.4 Validation of the measurement procedures

  • All 37 selected measurement cases matched expected outcomes with no failures or skips, including checks of scorer-only text, required controls, correlated documents, and evidence integrity.The cases covered recovery metrics, score calculations, control eligibility, observation boundaries, solver fidelity, resampling, and evidence validation.Source: §5.4 · p. 8

6 Cumulative Evidence and Scope

The evidence spans structural leakage in a deployed Qwen split-training system, GPT-2 text reconstruction, and further corpus-, placement-, depth- and defence-dependent findings. Its strongest quantitative reconstruction comparisons are scoped to GPT-2 124M under stated corpora, lengths and budgets, while agentic results and controlled-system access have explicit boundaries.

6 Cumulative Evidence and Scope

  • The Qwen audit recovered the real/decoy partition on all 4,096 frames in its main run despite a passing forward-only privacy gate.Structural recovery and text reconstruction are distinct outcomes.Source: §6 · pp. 8–9
  • The broader qualification programme exercised six named Qwen and Mistral models across diagnostic, ablation, robustness, obfuscation and agentic roles.The campaigns include expected, negative and privacy-breaking outcomes across multiple models and defences.Source: §6 · p. 9
  • The agentic tier’s Qwen3-0.6B planner acted in all ten fixed settings without policy violations in twenty trials, but blocked all six sanctioned peer-request trials.This qualifies planner/tool integration and identifies over-blocking; it does not establish general agentic security.Source: §6 · p. 9
  • The strongest quantitative reconstruction comparisons concern GPT-2 124M under the stated corpora, lengths and budgets, rather than an aggregate success rate across protocols.Longer contexts, new targets and additional attack budgets can extend the evidence through the same procedure.Source: §6 · p. 9
  • All experiments used author-controlled systems; live endpoint modes require a supplied host, and the agentic planner accepts local model targets only.Source: §6 · p. 9

7 Related Work

Related tools and prior attacks cover configurable split-model evaluation, security probing, privacy measurement and reconstruction, while LLM-Attack Suite focuses on the experimental contract for split and distributed privacy evaluation. The conclusion presents reusable, checkable evaluations; supplementary material retains protocols, evidence and reproduction artifacts.

7 Related Work

  • VFLAIR-LLM [7], garak [4] and ML Privacy Meter [13] address split-model attacks and defences, security probing, and membership-inference privacy measurement, respectively.LLM-Attack Suite instead focuses on explicit observations and knowledge, shared captures, linked controls, execution coverage and versioned evidence.Source: §7 · p. 9
  • The name LLM-Attacker also belongs to Mei et al.’s autonomous-driving framework [10]; this suite was previously released under that name.This paper focuses on split and distributed privacy evaluation.Source: §7 · p. 9
  • Prior work spans gradient and activation reconstruction [22] [3] [1] [12] [14], combined inversion, and split-learning attacks and leakage studies [15] [6] [5] [8].The suite assesses BiSR under a local single-seam contract; its other modules are local implementations or adaptations, not claims to reproduce every cited protocol.Source: §7 · p. 9

8 Conclusion

  • The suite makes privacy evaluation reusable through shared observation contracts, evidence records, baselines, positive controls and reports of untested threats.Researchers can add attacks to retained captures, compare defence revisions and provide predictions for reviewer rescoring.Source: §8 · p. 10

Evidence and Supplementary Material

  • Supplementary material retains the full protocols, per-condition results, 64-module validation matrix, source identifiers and reproduction guide.The v2.0 release reuses unchanged scientific evidence from v1.0 and updates branding and paper packaging only; it adds no experiments or runtime changes.Source: Evidence and Supplementary Material · p. 10

Supplementary Material

The supplementary sections preserve detailed protocols, results and artifacts behind the main text, not an additional independent evidence base.

Supplementary Material

  • The supplementary material retains detailed threat models, attack procedures, experimental results, release information and the complete 64-module validation matrix.It describes the same experiments and artifacts as the main text, with retained campaign and measurement-validation artifacts collected in tag v2.0.Source: Supplementary Material · p. 11

S1 Threat Model and Setup

The setup defines the split, the traffic available to the principal passive attacker, and stronger attacker conditions separately. It also maps additional observation surfaces and identifies the modules that read them.

S1 Threat Model and Setup

  • The client runs the first model layers and sends a per-token activation to the server; during training, the server returns a gradient for each activation.The gradient is the next-token loss differentiated with respect to that activation.Source: S1 Threat Model and Setup · p. 11
  • The principal attacker passively records split traffic without modifying it and does not receive private text, next-token labels or server layers.It may hold a copy of the client layers and public calibration text; the capture harness uses labels for the loss but does not expose them to the predictor.Source: S1 Threat Model and Setup · p. 11
  • An active server is a stronger attacker that alters tensors or returned updates to make the client leak more; only active-attack modules assume this role.Source: S1 Threat Model and Setup · p. 11
  • Attacker conditions can include public or trained client-layer weights, labelled examples or oracles, and key-solving privileges under secret rotation.A per-epoch attacker restarts after each new key, whereas a cross-epoch attacker carries information between keys; the paper’s key-solving experiment uses no rotation.Source: S1 Threat Model and Setup · pp. 11–12
  • The observation model covers nine surfaces: three on the link, one on the client and five on the server side.Most surfaces beyond traffic shape require a stronger attacker; cutting the model twice adds another activation crossing, while mixture-of-experts layers can expose routing decisions.Source: Figure S2 · p. 12, S1 Threat Model and Setup · p. 12
  • Table S1 lists what crosses each observation point and the modules that attack it.Source: Table S1 · p. 13

S2 How the Attacks Work

The strongest reconstruction routine builds a document from token candidates, with the joint attacker adding gradient-derived candidates to the activation-only search. Both attackers share the same beam search, while other evaluated attacks use distinct reconstruction procedures.

Activation-only candidate generation

  • The activation-only attack learns a linear decoder from 32 public documents, mapping client activations back to token embeddings.The attacker fits the map with ridge regression, a size penalty that stabilizes the fit.Source: S2 How the Attacks Work · p. 14
  • The activation-only attacker ranks 32 tokens per position by cosine similarity between their embeddings and the decoded activation.Cosine similarity measures how nearly two vectors point in the same direction.Source: S2 How the Attacks Work · p. 14

Joint candidate generation

  • The joint attack keeps activation-derived candidates and adds candidates suggested by the negated gradient at the previous position.It retains 16 candidates from each source, filling overlaps from the activation list; the first position uses activation candidates only.Source: S2 How the Attacks Work · p. 14

Beam search

  • Both attackers use a width-two, left-to-right beam search, retaining guesses whose recomputed activations best match the observed activations.They try equally many partial texts and differ only in their candidate lists.Source: S2 How the Attacks Work · p. 14

Other attack procedures

  • Other evaluated attacks include gradient-only guessing, SipIt’s position-wise search, white-box embedding optimization, inversion, and BiSR’s two-stage refinement.The backward-only heuristic uses the negated gradient without activations or beam search; BiSR refines an initial activation-based guess by matching gradients and then activations.Source: S2 How the Attacks Work · pp. 14–15

S3 Detailed Experimental Results

The experimental programme tests reconstruction under different attacker observations, corpora, defences, attack implementations, and model settings. It reports measured attack effectiveness separately from implementation and measurement checks.

S3.1 What the attacker observes

  • At depth six with public layers, joint recovery was 97.38% of tokens versus 94.20% for activation-only, a 3.17-point gain (95% interval [+2.72, +3.64]).The prespecified gain test passed; all 47 secondary comparisons favored the joint attacker, and shuffled-gradient and random-candidate controls performed about 0.3 points below activation-only.Source: S3.1 What the attacker observes · p. 16, Table S3 · p. 16

S3.2 Which text the attacker rebuilds

  • Inversion recovered 64.94% of code, 100% of templated clinical notes, and 63.25% of Wikipedia text, with corpus choice shifting recovery by up to 37 points.All undefended conditions exceeded corpus-specific guesses by at least 46 points; in the long-horizon battery, no verdict changed between 20 and 100 steps, and mid-lite recovery at 500 steps was 0.04–0.85% against its 4.83–4.96% majority baseline.Source: S3.2 Which text the attacker rebuilds · pp. 17–18, Table S4 · p. 17, Table S6 · p. 18

S3.3 What the validated white-box attack recovers

  • The validated white-box attack recovered 18.85%–68.59% of tokens on average across depths, while longer-text and dialogue recovery remained measurable under added noise.For longer texts and dialogue, undefended recovery ranged from 76.56% to 90.22%, while added-noise recovery ranged from 9.96% to 55.05%.Source: S3.3 What the validated white-box… · p. 19, Table S7 · p. 19, Table S8 · p. 19

S3.4 Which published attack the evaluator runs

  • The BiSR port matched its reference on all 3,072 paired predictions, but at depth six with public layers it recovered 93.21% versus 97.38% for the joint attack.The adapted comparison also found no attack dominated every setting: the refined port matched the joint attack with trained layers.Source: S3.4 Which published attack the… · p. 20, Table S9 · p. 20

S3.5 What strength is established

  • Evidence spans structural leakage in a deployed Qwen split-training system, high GPT-2 reconstruction, corpus- and defence-dependent failures, and multi-model module qualification.The programme distinguishes confirmatory recovery measurements from independent audits, implementation conformance, and qualification studies.Source: S3.5 What strength is established · pp. 20, 23, Table S10 · pp. 21–22

S3.6 Live agentic qualification

  • In ten live Qwen3-0.6B planner settings, all produced measurable actions and none of 20 fixed scenario trials recorded a policy violation, but six sanctioned peer requests were blocked.The six false blocks identify systematic over-blocking of sanctioned requests in this qualification setting.Source: S3.6 Live agentic qualification · p. 23

S3.7 Framework operation across protocols

  • The workflow reused captures across 3,680 measurements and connected reconstruction, reference conformance, white-box, agentic, Qwen, and Mistral studies to explicit attacker contracts and retained evidence.Together, these studies exercise the framework across reconstruction, robustness, obfuscation, and agentic protocols.Source: S3.7 Framework operation across… · p. 23

S3.8 Validation of the evaluation methodology

  • All 37 selected measurement-validation cases matched their specified outcomes, including dependence stress tests and checks that scorer-only truth changes did not alter predictions.The results establish conformance of the selected measurement procedures on their fixed cases.Source: S3.8 Validation of the evaluation… · p. 24, Table S11 · p. 24

S4 Release and Reproduction

The v2.0 release preserves the unchanged scientific evidence from v1.0 while separating the manuscript, reviewed taxonomy, and code snapshot for reproducibility. It retains experiment and validation artifacts, with tagged release paths identified as the navigation targets.

S4 Release and Reproduction

  • The Hugging Face dataset holds predictions, logs, experiment captures, analyses, controls, code, protocols, and inputs.The release tag also collects retained campaign evidence, the revised manuscript, and measurement-validation records.Source: S4 Release and Reproduction · p. 24
  • v2.0 reuses unchanged scientific evidence from v1.0 and changes branding and paper packaging only, with no new experiments or runtime changes.Experimental files retain their original hashes, and source snapshots preserve the code needed for reproduction independently of the former Git history.Source: S4 Release and Reproduction · pp. 24–25
  • The GitHub remote is the canonical source release, while the tagged Hugging Face release is the reproducibility artifact.The released command-line tool offers self-test, module listing, scorecard, coverage, and profile reports; the synthetic tier needs no deep-learning stack.Source: S4 Release and Reproduction · p. 25, Table S12 · p. 25
  • The measurement-validation package preserves its original selection, outcomes, manifest, and source snapshot for reproduction without Git history.The source snapshot matches all 123 retained source-file hashes.Source: S4 Release and Reproduction · p. 26

S5 Names in the Released Files

The application is named LLM-Attack Suite, though older software identifiers may retain LLM-Attacker. Released files also use internal experiment, attack, benchmark, and observation-point names.

S5 Names in the Released Files

  • LLM-Attacker may remain in software identifiers and historical release metadata, but refers to this suite rather than Mei et al.'s autonomous-driving framework [10].The naming change does not alter the frozen experimental evidence.Source: S5 Names in the Released Files · p. 26
  • Released files call the main experiment the v3 campaign and the adapted comparison the v4 adaptation, with distinct internal names for attacks and controls.For example, the joint attack is joint_true_k32 and the activation-only attack is forward_k32.Source: S5 Names in the Released Files · p. 26
  • Benchmark profiles are named for split privacy, split integrity and availability, GenAI applications, and agentic systems; Figure S2 and Table S1 use observation points S1–S14.The agentic benchmark profile is out of scope by default.Source: S5 Names in the Released Files · pp. 26–27

S6 Full Module List and Full-Precision Numbers

The section distinguishes attack-family coverage from implementation evidence and supplies full-precision experiment records. Its synthetic order-and-decoy battery reports excess order recovery over chance separately from token recovery.

S6 Full Module List and Full-Precision Numbers

  • The reviewed taxonomy has 64 modules grouped across split privacy, integrity and availability, GenAI application, and agentic threat profiles.The agentic profile is out of scope by default; Table S14 lists the modules by family.Source: S6 Full Module List and Full-Precision… · p. 27, Table S14 · pp. 27–28
  • Validation tiers distinguish reproduced, adapted, real-model-qualified, synthetic-qualified, and unmeasured or helper modules; deployment claims require Q-REAL evidence.Protected and null outcomes qualify a module when they match its controls, while registration alone is not real-model validation.Source: S6 Full Module List and Full-Precision… · pp. 28, 30, Table S15 · p. 30
  • Artifact keys identify evidence packages, including the shared v3 campaign, positive controls, domain and horizon studies, and deterministic module fixtures.The artifact column names evidence, not merely the source file that registers the module.Source: S6 Full Module List and Full-Precision… · p. 29
  • 3.1714 percentage points was the main experiment’s primary difference; its bootstrap used 100,000 replicates with seed 93281.The experiment used 336.1 hours of summed online attack time and wrote 17 hash-chained logs.Source: S6 Full Module List and Full-Precision… · p. 40
  • All 32 BiSR implementation-check comparisons gave the interval [0, 0], indicating agreement on those fixed documents.The adapted comparison used two attack seeds in 80 joint executions and 160 scored runs.Source: S6 Full Module List and Full-Precision… · p. 40
  • 0.52 to 5.21 points was the range of highest excess order recovery over chance across six synthetic matching reports.The battery ran for three seeds and scored row order separately from tokens using a local supervised baseline.Source: S6 Full Module List and Full-Precision… · p. 40

Related papers

Loading 2610.22722v2…