Fine-Tuning Large Language Models with User-Level Differential Privacy

1 Introduction

• Because users may contribute correlated examples, example-level DP can fail to protect against user-level membership inference, motivating guarantees over all data contributed by each user.The motivating applications include sensitive agent interactions, emails, and text; LLMs have also been shown to leak training data.Source: §1 · pp. 1–2
• Prior theoretical user-level DP methods rely on subroutines that are challenging to scale to high-dimensional LLMs, while prior federated-learning studies focused on small edge-device models.Datacenter LLM training instead requires carefully batched operations to maximize accelerator throughput.Source: §1 · p. 2
• ELS applies example-level sampling and per-example clipping to at most G_ELS examples per user, then translates the resulting guarantee to the user level.ULS instead samples users and clips gradients averaged over G_ULS examples from each sampled user.Source: §1 · p. 2
• The study asks how to configure the methods’ group sizes for privacy and compute budgets and which method performs better when both are configured well.The motivation is that DP adds computational cost and privacy, utility, and computational efficiency must be balanced under fixed budgets.Source: §1 · p. 2
• The paper develops tight user-level privacy accounting for ELS through analysis of group privacy loss in iterated compositions of a subsampled Gaussian mechanism.The bounds provide user-level guarantees for ELS and are presented as a tighter alternative to generic example-to-user reductions.Source: §1 · p. 2
• In fixed-compute experiments, ULS often outperforms ELS, with the largest improvements when users have high gradient variance; the study also scales to 350-million-parameter models.The work reports LLM-compatible implementations scaling to models with hundreds of millions of parameters and datasets with hundreds of thousands of users.Source: §1 · p. 2

User-level DP

• User-level adjacency adds or removes one user together with that user’s dataset, recovering example-level add-or-remove adjacency when each user contributes one example.The paper defines user-level DP with respect to this adjacency relation.Source: §2 · p. 3

Algorithms for training with user-level DP

• ELS limits each user to G_ELS examples before example-level sampling and clipping; ULS samples users and clips gradients averaged over G_ULS examples.The ELS example-level guarantee is translated to a user-level guarantee; ULS directly applies DP-SGD to user-level gradients.Source: §2 · p. 3

User-level DP accountants

• The MoG accountant gives near-linear ε scaling in G_ELS in the studied settings, while generic user-to-example reductions scale exponentially and black-box accounting diverges for sufficiently large G_ELS at σ_ELS ∈ {0.5, 1}.The comparison uses T = 2000 iterations, p = 10^-2, and δ = 10^-6.Source: §2 · p. 4, Figure 1 · p. 4

Comparisons under fixed compute budgets

• The ELS and ULS comparison fixes compute by counting gradient computations, with expected per-iteration costs B for ELS and G_ULS M for ULS.The analysis excludes gradient averaging, clipping, noise generation, and communication costs.Source: §2 · p. 4
• The experiments use shuffling and deterministic batch and cohort sizes, which can make reported privacy bounds differ from the algorithm’s actual privacy.The authors caution against using this practice to report guarantees on actual privacy-sensitive datasets.Source: §2 · p. 4

Variance comparison

• When L_ULS is sufficiently smaller than L_ELS, ULS adds noise with smaller variance than ELS.The variance ratio is useful for comparison but does not fully predict relative performance because the methods sample data differently.Source: §3 · p. 5
• With L_ULS = L_ELS/√G_ULS, ULS has smaller variance when ε is small, while varying G_ELS has almost no effect on ELS variance.The comparison is shown in Figure 2 under the captioned compute-budget and group-size settings.Source: §3 · p. 5, Figure 2 · p. 6

Privacy comparison

• For G_ELS = G_ULS = K and p = q, the authors conjecture σ_ELS ≤ Kσ_ULS for all ε, δ, T, p, and K.They report empirical support for the conjecture by comparing variance at the largest compute budget when L_ELS = L_ULS.Source: §3 · p. 5

Scope

• Privately choosing a clip norm is beyond the scope of the analysis, and L_ELS and L_ULS are data-dependent.The paper notes that tuning clip norms on the dataset is common practice but non-private.Source: §3 · p. 5

Privacy comparison

• For integer α, the paper proves a weaker version of the conjecture for one-sided α-RDP and relates the bound to ELS requiring noise multiplier at most K times ULS’s.The comparison follows from the lemma’s correspondence between one iteration of ULS at σ and ELS at Kσ.Source: §3 · p. 6

Experimental design

• The experiment estimates a population mean from users with individual means and examples sampled around those means, using square-distance loss.The authors vary ε, compute budget, and within-user variance, and sweep learning rate and clip norm across 128 random trials.Source: §4 · p. 6

Results

• ULS with G_ULS = 1 is comparable to or better than ELS across all tested settings.The study evaluates mean loss across varying ε, compute budget, and within-user variance.Source: §4 · p. 6, Figure 3 · p. 7
• ULS benefits from larger G_ULS when within-user variance or compute budget is large, or ε is small.In these regimes, ULS improves significantly on ELS; the larger-group benefit is also corroborated theoretically for Lipschitz losses.Source: §4 · p. 6

Model

• The experiments use a 350-million-parameter decoder-only transformer with sequence length 128 and causal next-token prediction loss.The model uses a WordPiece tokenizer and is implemented in Praxis.Source: §5 · p. 7

Datasets

• Fine-tuning uses Stack Overflow data partitioned by user and CC-News articles partitioned by URL base domain.The Stack Overflow training split contains 135,818,730 examples across 342,477 users; CC-News contains 708,241 examples across 8,759 users.Source: §5 · p. 7

Pretraining

• To reduce overlap with fine-tuning data, pretraining uses C4-- filtered by near-duplicate detection and URL exclusion.The exclusions remove examples associated with stackoverflow.com and URLs in CC-News.Source: §5 · p. 7

Training

• The model is pretrained for 400,000 steps and fine-tuned for 10,000 Stack Overflow steps or 2,000 CC-News steps.Fine-tuning uses Adafactor, with learning rate and clip norm tuned using normalized clipping.Source: §5 · p. 7

Accounting and sampling

• Privacy accounting assumes amplification by subsampling, while implementation samples shuffled fixed-size example batches for ELS and user cohorts for ULS.For CC-News accounting, the setup assumes ten times as many users as are present in the dataset.Source: §5 · p. 7

Software and compute

• ULS uses Dataset Grouper and parallelized computation in FAX, while experiments run in PAX on TPU v3 pod slices.The listed topologies are 4 × 4, 8 × 8, and 16 × 16 for small, medium, and large compute budgets.Source: §5 · pp. 7–8

• ELS group size balances dataset size and privacy noise; GELS values of 27 for Stack Overflow and 24 for CC-News, near each dataset’s median, work across budgets and ε.Source: Figure 4 · p. 8, §6 · p. 8
• The ULS tuning heuristic doubles GULS or M according to which reduces stochastic variance more, comparing the effects on LULS and σULS.Source: §6 · pp. 8–9
• For a fixed Stack Overflow compute budget, increasing cohort size M up to 512 is preferable before increasing GULS; compute-optimal GULS remains at least tenfold smaller than M.Source: §6 · p. 8, Figure 5 · p. 9
• ULS performs at least as well as ELS on Stack Overflow and often outperforms it on CC-News, especially at small ε or large compute budgets.Source: Figure 6 · p. 9, Figure 7 · p. 9, §6 · p. 9
• As compute increases, ULS loss decreases more; as the number of users in the privacy accounting increases, ELS loss decreases more.Source: Figure 8 · p. 10, §6 · p. 10
• Both ELS and ULS appear to benefit comparably from personalization, without changing the general trade-offs.Source: §6 · p. 10

7.1 Comparison to Chua et al. [20]

• Compared with Chua et al. [20], this work adds tighter accounting, compute-normalized comparisons, larger-scale experiments, and explicit group-size selection methodology.Source: §7.1 · p. 10

7.2 Theoretical Advances in User-Level DP

• Stable-subset reductions by Ghazi et al. [49] [15] can require super-polynomial search, whereas this work gives tight user-level accounting specifically for DP-SGD-ELS.Source: §7.2 · p. 10
• Prior work optimizes per-example weights, while this paper fixes GELS globally and randomly selects examples; utility bounds for adapting those methods to LLMs are unavailable.Source: §7.2 · p. 11
• Other theoretical approaches combine user-gradient clipping with robust mean estimation [13, 14, 16, 21], while this work evaluates an accelerator-friendly unweighted average; related theory also studies stylized tasks [52] [53] [56].Source: §7.2 · p. 11

7.3 Federated Learning and User-Level DP

• Federated-learning work introduced user-level gradient clipping and related variants [17] [60] [61] [62], while similar ULS methods have been applied in other domains [63] [64] [66] [68].Source: §7.3 · p. 11

• Tight group-level accounting makes ELS practical for user-level DP, but ULS often outperforms ELS across synthetic and realistic settings.Source: §8 · p. 11
• No general theorem compares ELS and ULS utility across settings, and the practical group-size heuristics may not transfer beyond the experiments.Source: §8 · p. 11
• Compute constraints led the study to explore hyperparameter breadth rather than maximally scaling model size, leaving scaling behavior for future work.Source: §8 · p. 12

User-level Privacy Attacks

• Example-level DP may not adequately mitigate user-inference attacks, which can infer whether a user's data was used even without access to that user's exact training samples.The cited work reports this concern across several application settings and notes that user-level DP provides formal upper bounds on attack success.Source: User-level Privacy Attacks · p. 17

A Tight Accounting for DP-SGD-ELS

• Theorem 3 gives a tight user-level privacy bound for T-step ELS with Poisson sampling, noise multiplier σ, and at most K added examples.The bound is tight for some loss function and datasets.Source: A Tight Accounting for DP-SGD-ELS · p. 17

A.1.1 Computing ε and δ

• The provided dp_accounting workflow composes a group-level event in a PLD accountant to compute ε from δ or δ from ε for ELS.The event uses a Binom(K, p) mixture of Gaussian mechanisms and T composed steps.Source: §A.1.1 · p. 19

A.1.2 Computing σELS

• The ELS noise multiplier σ can be calibrated to meet a target (ε, δ)-DP guarantee using dp_accounting's mechanism-calibration routine.The calibration routine uses the group-level event for the specified T, p, and K.Source: §A.1.2 · p. 20

B Comparing Variances of ELS and ULS

• When LULS = LELS, ELS has lower noise variance in nearly all tested settings; when LULS = LELS/√GULS, ULS often has lower variance, especially at small ε or large compute budgets.These comparisons use the fixed-compute setup summarized in Figure 9.Source: B Comparing Variances of ELS and ULS · p. 20, Figure 9 · p. 21

C Proof of Lemma 1

• The proof decomposes each binomial count into Bernoulli variables and uses a Gaussian moment-generating identity.The supplied proof passage does not state the lemma's conclusion.Source: C Proof of Lemma 1 · p. 22

D Datasets

• Stack Overflow uses pre-2018 training examples and a later test split, while CC-News holds out part of each user's data for evaluation and trains on the remainder.The pretraining C4 dataset is filtered for near-duplicates and examples associated with Stack Overflow or CC-News URLs.Source: D Datasets · p. 23, Table 1 · p. 23, Figure 10 · p. 23

E Configuring DP-SGD-ELS

• Setting GELS to the median user dataset size works well across the tested compute budgets and privacy levels on Stack Overflow and CC-News.GELS trades off training on a smaller fraction of the dataset against potentially over-representing users with more examples.Source: E Configuring DP-SGD-ELS · p. 24, Figure 11 · p. 24, Figure 12 · p. 24

F Configuring DP-SGD-ULS

• For ULS, the authors estimate LULS(GULS) from sampled users' gradients, use the median estimator, and balance GULS against cohort size M while keeping GULS M = B.Their Estimate-and-Double heuristic compares whether doubling GULS or M more reduces the noise-variance proxy vULS; the exact dependence of LULS on GULS is data-dependent.Source: F Configuring DP-SGD-ULS · p. 25

F.1 Validating Algorithm 3

• The validation sweeps GULS, M, and ε, and compares the heuristic with random, maximum-cohort, and maximum-group-size configurations.For Stack Overflow, heatmap anti-diagonals correspond to fixed compute budgets.Source: §F.1 · pp. 25, 27, Figure 13 · p. 26
• Estimate-and-Double configures DP-SGD-ULS by initializing GULS = 1 and M = 32, estimating LULS(GULS) via (8), and computing σ(M) via (3) under compute budget B.Source: §F.1 · p. 26
• On Stack Overflow and CC-News, Estimate-and-Double performs as well as the oracle strategy for most compute budgets across the tested privacy levels.Suboptimality is measured as the loss difference from optimally configured ULS; the figures average results across ε values and show standard deviation.Source: §F.1 · p. 27, Figure 14 · p. 27, Figure 15 · p. 27

G.1 Compute-Loss Trade-offs

• ULS is more capable of improving performance with increased compute budgets because it can trade off reductions in clip norm and noise multiplier as their benefits saturate.ELS can use additional compute to reduce its noise multiplier, with diminishing returns.Source: §G.1 · p. 28, Figure 16 · p. 28, Figure 17 · p. 28

G.2 Personalizing Fine-Tuned Models

• Personalization reduces loss for both ELS and ULS, while ULS with personalization matches or outperforms ELS at the same privacy levels and compute budgets.The comparison evaluates each personalized model on the reserved half of each test user's examples.Source: §G.2 · p. 28, Figure 18 · p. 29
• Personalization's loss reduction is roughly the same for both checkpoints and does not appear to change the fundamental shape of the trade-off curves.The authors personalize on half of each Stack Overflow test user's examples and evaluate on the reserved half.Source: §G.2 · p. 28