Source-linked AI summary
Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning
Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Daogao Liu, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang
TL;DR
Fine-tuning on domain data risks retaining sensitive information, and record-level privacy does not ensure equal protection when users contribute multiple records. This paper evaluates Group Privacy and User-wise DP-SGD for user-level private language-model fine-tuning. User-wise DP-SGD consistently outperforms Group Privacy, especially at smaller privacy budgets.
Problem
Fine-tuning on domain data risks retaining sensitive information, while users who contribute multiple records need user-level privacy for equal guarantees.
Source: §1 · p. 2, §8 · p. 10Method
The paper evaluates Group Privacy and User-wise DP-SGD for user-level differential privacy in language-model fine-tuning, including data-selection strategies.
Source: §1 · p. 2, §8 · p. 10Results
User-wise DP-SGD consistently outperforms Group Privacy, especially for smaller privacy budgets.
Source: §6 · p. 9Takeaways & Limitations
Selection outcomes were dataset-dependent: longest records performed best on Enron, while BookSum differences across methods were relatively small.
Source: §4.1 · p. 6User-wise DP-SGD fixes k, the number of gradients sampled per user, for simplicity, leaving dynamic per-user choices unexplored.
Source: §8 · p. 10
Abstract
from arXiv · showhide
Large language models (LLMs) have emerged as powerful tools for tackling complex tasks across diverse domains, but they also raise privacy concerns when fine-tuned on sensitive data due to potential memorization. While differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit, current evaluations on LLMs mostly treat each example (text record) as the privacy unit. This leads to uneven user privacy guarantees when contributions per user vary. We therefore study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users. We present a systematic evaluation of user-level DP for LLM fine-tuning on natural language generation tasks. Focusing on two mechanisms for achieving user-level DP guarantees, Group Privacy and User-wise DP-SGD, we investigate design choices like data selection strategies and parameter tuning for the best privacy-utility tradeoff.
1 Introduction
Record-level DP can give users with more records weaker privacy, while user-level DP aims to protect users uniformly. The paper evaluates user-level DP fine-tuning for language generation and reports that assumptions behind a theoretically optimal algorithm may be too restrictive in practice.
1 Introduction
- Fine-tuning LLMs on domain data risks retaining sensitive information that could potentially be extracted later.The paper notes that larger models may amplify memorization risks.Source: §1 · p. 2
- User-level DP aims to give users equal privacy guarantees regardless of how many records they contribute, unlike record-level DP.Under record-level DP, users contributing more records obtain weaker privacy protection.Source: Figure 1 · p. 1, §1 · p. 2
- The study evaluates Group Privacy and User-wise DP-SGD, data-selection strategies, and their privacy-utility trade-offs for language-generation fine-tuning.It does not assume a distributed or federated learning setting.Source: §1 · p. 2
- The Asi & Liu (2024) algorithm’s assumptions could be too restrictive for real-world user-level DP applications, motivating more realistic assumptions.The paper presents this as the first practical study of the algorithm’s application to language-model fine-tuning tasks.Source: §1 · p. 2
- The paper notes that concurrent work by Charles et al. (2024) also studies user-level DP for LLM fine-tuning, with largely consistent observations.The authors say they discuss differences in Section 7.Source: §1 · p. 2
2 User-level Differential Privacy Mechanisms
The section defines user-level privacy in terms of adding or removing all records belonging to one user, then presents two mechanisms for obtaining that guarantee. Group Privacy adapts record-level DP, while User-wise DP-SGD samples and clips at the user level.
2 User-level Differential Privacy Mechanisms
- For user-level DP, neighboring datasets differ by adding or removing all text belonging to one user, rather than one record.Source: §2 · p. 3
- Group Privacy adapts an example-level DP algorithm to user-level DP by limiting each user to at most k records and applying group-privacy accounting.The section describes using tighter bounds for the specific case of DP-SGD.Source: §2 · p. 4
- User-wise DP-SGD samples users, averages sampled records’ gradients per user, then clips, aggregates, and adds noise to the user gradients.It can sample contiguous text chunks during training and does not require preprocessing each user’s data to a fixed record count.Source: §2 · p. 4
- The Asi & Liu (2024) method removes outlier user gradients when the batch gradients are sufficiently concentrated, but halts without a meaningful output otherwise.Its optimal privacy-utility trade-off is proven under an i.i.d. assumption on records.Source: §2 · p. 4
3 Experimental Setup
The evaluation uses Enron Email and BookSum, with privacy units defined at the sender and book levels, respectively. It uses GPT-2 small with LoRA and explores record-selection criteria and the number of records selected per unit.
3 Experimental Setup
- Enron Email contains about 500,000 employee emails, while BookSum contains human-written summaries of books.The authors describe Enron emails as potentially privacy-sensitive and note copyright considerations for BookSum.Source: §3 · pp. 4–5
- The evaluation uses Enron Email and BookSum, defining each unique email sender and each book as the respective privacy unit.Book-level privacy is used because author names are unavailable in BookSum.Source: §3 · pp. 4–5
- The experiments use pretrained GPT-2 small (125M) with LoRA fine-tuning and compare record-selection criteria and the number of records selected per privacy unit.The authors report that LoRA does not significantly impact model performance in Appendix C.1.Source: §3 · p. 5
4 Group Privacy
This section evaluates Group Privacy design choices for selecting records per user, including selection strategy and the number of records k. Results show that the better choice depends on dataset and privacy budget.
4 Group Privacy
- The choice of DP fine-tuning recipe is described as orthogonal to user-level DP algorithms, with consistent results reported using another recipe.The alternative fine-tuning method is Li et al. (2022).Source: §4.1 · p. 5
4.1 Data selection strategy
- Longest-record selection performs best across privacy levels on Enron Email, while BookSum has only small differences among strategies.For BookSum, random selection performs best at ε = 1.0 and 8.0, while lowest-perplexity selection works slightly better at ε = 3.0.Source: Table 2 · p. 6, §4.1 · p. 6
4.2 Number of selected records (k) per unit
- Smaller k generally performs better at ε = 1.0 or 3.0, while larger ε usually favors larger k.The study attributes this trade-off to increased privacy noise versus greater training-data diversity; similar patterns were observed on GPT-2 medium and large models.Source: Table 3 · p. 6, §4.2 · p. 7
4.1 Data selection strategy
- Simple length-based heuristics can sometimes outperform perplexity-based selection, with more pronounced performance gaps on Enron Email than BookSum.The authors suggest this difference could reflect greater variation in record lengths and content in email data.Source: §4.1 · p. 6
4.2 Number of selected records (k) per unit
- In Group Privacy, increasing k raises both noise and training-data diversity, creating opposing effects on utility.The section frames choosing k as a trade-off between these effects.Source: §4.2 · pp. 6–7
- For User-wise DP-SGD, Random Chunk consistently performs best across datasets and privacy budgets.The reported benefit may reflect greater training-data diversity and spanning record boundaries.Source: Table 4 · p. 7
5 User-wise DP-SGD
This section evaluates User-wise DP-SGD’s data-selection strategies and choices of k, then examines whether an advanced method’s noise advantage applies in the evaluated settings.
5 User-wise DP-SGD
- The section evaluates User-wise DP-SGD’s record-selection strategies, the number of selected records, and an advanced gradient-estimation method.Source: §5 · p. 7
5.1 Data selection strategy
- Random Chunk consistently outperforms other selection methods across datasets and privacy budgets, likely through increased data diversity and spanning record boundaries.Source: §5.1 · p. 7
5.2 Number of selected records (k) per unit
- Larger k monotonically improves User-wise DP-SGD utility, but returns diminish; on BookSum, k = 10 appears to balance utility and efficiency.The reported ε = ∞ setting bypasses aggregation and noise addition.Source: §5.2 · pp. 7–8, Table 5 · p. 8
5.3 Analysis of the advanced User-wise DP-SGD
- For most k and ε, Asi & Liu’s advanced method yields lower noise only when τ/C < 0.1, a threshold unlikely in the evaluated settings.The evaluated datasets have τ/C around 1.Source: §5.3 · p. 8, Figure 3 · p. 8
6 Comparing Group Privacy and User-wise DP-SGD
The comparison finds that User-wise DP-SGD generally outperforms Group Privacy, especially at smaller privacy budgets, while the methods differ in clipping sensitivity and implementation demands.
6 Comparing Group Privacy and User-wise DP-SGD
- User-wise DP-SGD generally outperforms Group Privacy, especially at smaller privacy budgets.The comparison uses Longest with the best reported k for Group Privacy and Random Chunk with k = 10 for User-wise DP-SGD.Source: §6 · pp. 8–9, Table 6 · p. 9
- Group Privacy is generally more sensitive to clipping norms than User-wise DP-SGD.Figure 4 compares perplexity across clipping norms and batch sizes on BookSum at ε = 1.0; larger batches generally improve both methods.Source: Figure 4 · p. 9, §6 · p. 9
- The authors suggest User-wise DP-SGD’s minibatch data diversity may contribute to its performance, while Group Privacy may repeat records for users with fewer than k records.Source: §6 · p. 9
- Group Privacy requires minimal changes to record-level private training, whereas User-wise DP-SGD may require data-loading pipeline modifications.Source: §6 · p. 9
7 Related Work
Related work has largely studied language-model differential privacy at the record level, alongside user-level approaches in federated learning and theory. A concurrent empirical study shares the finding that User-wise DP-SGD usually has higher utility, while exploring different datasets and design choices.
7 Related Work
- DP-SGD clips per-record gradients and adds noise to average gradient updates, and prior language-model applications have primarily focused on record-level privacy.Source: §7 · pp. 9–10
- Prior user-level language-model DP work studied federated LSTMs, while this paper considers central training and modern transformers; theoretical work commonly assumes i.i.d. data.The authors describe that assumption as restrictive and call for more realistic assumptions.Source: §7 · p. 10
- A concurrent study reports a generally consistent utility advantage for User-wise DP-SGD, while this work explores broader data-selection strategies, books and email datasets, and models of various sizes.Charles et al. (2024) offer an explicit group-size selection methodology, whereas this work mainly ablates group sizes.Source: §7 · p. 10
8 Conclusions
The conclusion evaluates Group Privacy and User-wise DP-SGD for user-level private language-model fine-tuning and identifies fixed k, data-selection choices, and task coverage as areas for further study.
8 Conclusions
- The paper evaluates Group Privacy and User-wise DP-SGD for user-level DP in language-model fine-tuning on natural language generation tasks.It investigates data-selection strategies for the privacy-utility trade-off because users generally contribute multiple text records.Source: §8 · p. 10
- User-wise DP-SGD fixes k for simplicity; exploring dynamic per-user k may improve utility while preserving privacy guarantees.Source: §8 · p. 10
- The evaluation focuses on language generation, while user-level DP applications in recommendation systems and structured prediction remain future research directions.The authors also propose investigating gradient-based or active-learning data selection for User-wise DP-SGD.Source: §8 · p. 10
Appendices
The appendix outlines an adaptation of Asi & Liu (2024) and its filter-based user-level training procedure, which checks gradient concentration before aggregating selected users’ gradients and updating the model.
A User-Level Algorithm of Asi & Liu (2024)
- Asi & Liu (2024) assume equal records per user and full-batch training; this adaptation draws k′ records per user.The algorithm takes the training set, privacy budget, iteration count, concentration threshold, and per-user record counts as inputs.Source: A User-Level Algorithm of Asi & Liu… · p. 16
- The procedure computes per-user averaged gradients, scores their concentration, and uses AboveThreshold to decide whether to remove outliers before aggregation.The algorithm then averages gradients from the selected users, adds Gaussian noise, and updates the model; it halts if the threshold test fails.Source: A User-Level Algorithm of Asi & Liu… · p. 16
- AboveThreshold is a classic algorithm in the differential privacy literature.Here, the algorithm applies it to concentration scores before aggregating gradients.Source: A User-Level Algorithm of Asi & Liu… · p. 16
B.1 Dataset details
These appendices describe dataset preparation, model and privacy-accounting choices, and checks of LoRA’s effect on utility. The reported LoRA impact is relatively minor.
B.1 Dataset details
- Enron preprocessing parses email fields, expands CC/BCC recipients into separate records, removes duplicates and forwarded emails, and leaves 240,173 emails.The deduplication rule retains records whose maximum repetition of any 20-gram is 1.Source: §B.1 · p. 17
- Figure 5 presents the record-count distribution per privacy unit in Enron Email and BookSum, while Table 7 reports selected-record statistics including sequence length and perplexity.The table also reports the fraction of selected sequences shorter than the maximum allowed length.Source: Figure 5 · p. 17, §B.1 · p. 17, Table 7 · p. 18
B.2 Model details
- Experiments primarily use GPT-2 small with 125 million trainable parameters, LoRA rank 32, and perplexity as the performance metric.The appendices also report results for other GPT-2 sizes; the paper says LoRA does not significantly affect performance.Source: §B.2 · p. 17
B.3 Privacy Accounting
- Accounting assumes Poisson subsampling although batches are implemented by randomly ordering and traversing users or records; the authors consider comparisons informative in a relative sense despite the noted gap.The paper notes that privacy guarantees can differ between Poisson subsampling and shuffling.Source: §B.3 · pp. 17–18
- Privacy accounting uses Poisson-subsampled Gaussian-mechanism composition for User-wise DP-SGD and a mixture-of-Gaussians mechanism for Group Privacy.For User-wise DP-SGD, accounting substitutes the number of users and user-batch size for the record count and batch size; Group Privacy analyzes sensitivity under contributions of up to k records.Source: §B.3 · p. 18
C.1 LoRA v.s. Full fine-tuning
- LoRA fine-tuning has a relatively minor impact on perplexity compared with full fine-tuning on Enron Email and BookSum.Table 8 covers full fine-tuning and LoRA at different rank values.Source: §C.1 · p. 19, Table 8 · p. 19
C.2 Compatibility with a different DP fine-tuning method
The appendix tests compatibility with another DP fine-tuning pipeline, model-size trends, and computational costs. It reports that selection-strategy findings persist under the alternate pipeline and that both methods add little runtime overhead.
C.2 Compatibility with a different DP fine-tuning method
- With the Li et al. (2022) pipeline, random chunking works best for User-wise DP-SGD and longest-sequence selection for Group Privacy; Table 9 reports User-wise DP-SGD generally outperforms Group Privacy.The authors describe the DP fine-tuning recipe as orthogonal to the user-level algorithms.Source: Table 9 · p. 19, §C.2 · p. 19
C.3 Results under different models
- Across GPT-2 sizes, larger k consistently improves User-wise DP-SGD, while Group Privacy favors smaller k at smaller ε and larger k at larger ε.Tables 10 and 11 report these k trends on Enron Email and BookSum.Source: §C.3 · p. 19, Table 11 · p. 20, Table 10 · p. 20
C.4 Computational overhead
- 4%–9%: both privacy methods add this little computational overhead over non-private training across the tested model sizes.Table 12 compares per-step runtimes for k = 10 on Enron Email, using LoRA rank 32 and batch size 1024.Source: Table 12 · p. 21, §C.4 · p. 21
C.5 Cost of k in User-wise DP-SGD
- Increasing k raises Group Privacy training time through larger datasets and User-wise DP-SGD runtime through sampling more records per step.Figure 6 shows User-wise DP-SGD runtime across k values on BookSum.Source: §C.5 · p. 21, Figure 6 · p. 21
C.6 More results for comparing User-wise DP-SGD and Asi & Liu (2024)
The appendix compares noise in User-wise DP-SGD and Asi & Liu (2024)’s method. The latter’s lower-noise condition is not met at the reported concentration-to-clipping ratio in the evaluated datasets.
C.6 More results for comparing User-wise DP-SGD and Asi & Liu (2024)
- For Asi & Liu (2024) to yield lower noise than standard User-wise DP-SGD, τ/C must be below 0.1; the evaluated datasets have τ/C around 1.Figure 7 compares the methods’ noise under different ε values and numbers of records per user.Source: §C.6 · p. 21, Figure 7 · p. 22