PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Can Be Strong Differentially Private Learners

Xuechen Li, Florian Tramèr, Percy Liang, Tatsunori Hashimoto

arXiv:2110.05679v6cs.LGcs.CL

TL;DR

DP learning has struggled with useful large language models because straightforward DP-SGD can cause performance loss and high overhead. The paper studies direct DP fine-tuning of large pretrained models with DP-suited hyperparameters and aligned objectives, and reports strong results alongside a memory-saving clipping method. Its conclusions are bounded by reliance on particular publicly pretrained models and their potentially sensitive pretraining data.

  • Problem

    DP learning has typically struggled to produce useful large language models, yielding vacuous privacy guarantees or performance far below non-private baselines.

  • Method

    The paper directly fine-tunes public pretrained language models with DP-SGD or DP-Adam, studying hyperparameters, objectives, model sizes, and memory-efficient ghost clipping.

  • Results

    Large pretrained models fine-tuned with suitable DP optimization achieve strong NLP performance under modest privacy budgets, surpassing heuristic-privacy models and some strong non-private baselines.

  • Takeaways & Limitations

    Private NLP practitioners can treat properly configured DP fine-tuning as a competitive baseline before adopting less formal privacy notions.

  • Takeaways & Limitations

    The scaling conclusion depends on the particular pretrained models, whose internet-collected pretraining data may contain sensitive information.

Abstract

from arXiv · show

Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead. We show that this performance drop can be mitigated with (1) the use of large pretrained language models; (2) non-standard hyperparameters that suit DP optimization; and (3) fine-tuning objectives which are aligned with the pretraining procedure. With the above, we obtain NLP models that outperform state-of-the-art DP-trained models under the same privacy budget and strong non-private baselines -- by directly fine-tuning pretrained models with DP optimization on moderately-sized corpora. To address the computational challenge of running DP-SGD with large Transformers, we propose a memory saving technique that allows clipping in DP-SGD to run without instantiating per-example gradients for any linear layer in the model. The technique enables privately training Transformers with almost the same memory cost as non-private training at a modest run-time overhead. Contrary to conventional wisdom that DP optimization fails at learning high-dimensional models (due to noise that scales with dimension) empirical results reveal that private learning with pretrained language models doesn't tend to suffer from dimension-dependent performance degradation. Code to reproduce results can be found at https://github.com/lxuechen/private-transformers.

1 INTRODUCTION

The paper shows that appropriately configured DP fine-tuning of large pretrained language models can produce strong NLP performance under modest privacy budgets, while also addressing the memory cost of DP-SGD.

  • Core findings: Large pretrained language models can be effectively and efficiently fine-tuned under modest privacy leakage for sentence classification and language generation.The study uses datasets containing tens to hundreds of thousands of examples.
  • Core findings: For sentence classification, the models surpass approaches based on heuristic privacy notions that lack formal privacy guarantees.For text generation, the models also surpass strong non-private baselines.
  • Optimization choices: Appropriate hyperparameters and downstream objectives substantially improve DP-Adam performance and can match some strong non-private learning results.The paper uses these analyses to explain poor results from straightforward DP optimization in prior work.
  • Core findings: Fine-tuning pretrained language models with DP-SGD or DP-Adam yields strong performance across NLP tasks at privacy levels ϵ ∈{3, 8}.Some models outperform strong non-private baselines and models trained under heuristic privacy notions.
  • Memory efficiency: Ghost clipping makes fine-tuning large Transformers under DP memory efficient without instantiating per-example gradients for linear layers.Combined with layer-by-layer clipping, it approaches the memory cost of non-private training at the cost of one additional backward pass per batch.
  • Scaling behavior: Larger pretrained models produce better private fine-tuning results, contrary to the expectation that dimensionality necessarily harms private optimization.Parameter-efficient methods that reduce update dimensionality do not necessarily outperform full fine-tuning.

2 PROBLEM STATEMENT

The paper formulates private NLP as learning sentence classification and language generation models under approximate DP, then describes gradient clipping, noise addition, and pretrained-model fine-tuning.

  • Setting: The study targets sentence classification and language generation with modest-sized datasets under central approximate (ϵ, δ)-DP.The datasets contain tens to hundreds of thousands of examples.
  • Setting: Two datasets are adjacent when one can be obtained from the other by adding one record, with the record definition depending on the task.This adjacency notion underlies the privacy guarantee.
  • DP optimization: DP optimizers clip each per-example gradient to norm bound C and add Gaussian noise z ∼N(0, C2σ2Ip) to the sum of clipped gradients.The noise multiplier σ depends on the privacy budget, update count, and sampling rate.
  • DP optimization: Clipping bounds each example’s influence on updates, while Gaussian noise prevents exact tracing of particular examples through their gradients.The isotropic noise norm scales as Cσ√p in p dimensions.
  • Modeling approach: The paper starts from public pretrained language models, fine-tunes them with DP-Adam, and tracks privacy loss using Rényi DP.It also reports conversions from Gaussian DP and composed tradeoff functions.
  • Tasks: Sentence classification uses labeled input sentences, whereas language generation predicts text from contexts such as tables, metadata, or dialogue histories.The study evaluates BERT and RoBERTa families for classification and generation tasks with structured or conversational inputs.

3 EFFECTIVE DIFFERENTIALLY PRIVATE FINE-TUNING

DP fine-tuning performance depends strongly on optimization choices, especially batch size, learning rate, clipping, and task-objective alignment. Larger batches can improve signal-to-noise ratio and performance, while pretrained objectives help preserve task alignment.

  • Optimization sensitivity: DP-Adam performance can improve from near-random-initialization levels to near state-of-the-art through appropriate hyperparameters.The paper presents hyperparameter and objective choices as explanations for poor straightforward-baseline results.
  • Batch size and learning rate: With fixed epochs, GPT-2 models reached BLEU score ~62 on E2E using both large batch sizes and large learning rates.Small batches combined with small learning rates performed considerably worse; the non-private seq2seq baseline reached test BLEU ~65.
  • Batch size and update budget: With fixed updates, larger batches almost always improved performance at a given privacy budget, but required processing more examples and compute.This conclusion holds after fixing S, η, C, ϵ, and δ.
  • Signal-to-noise ratio: Larger batches increase early-training signal-to-noise ratio because the effective noise multiplier decreases with batch size, correlating log-linearly with final performance.The relationship is shown for E2E and is described for moderately large batches up to a threshold.
  • Clipping: Small clipping norms that keep almost all gradients clipped throughout training performed best in the reported experiments.The noise standard deviation scales with the clipping norm C, so overly large thresholds add more noise than necessary.
  • Objective alignment: For classification, text-infilling objectives reduce the mismatch between language-model pretraining and downstream fine-tuning.The method reframes sentiment classification as predicting a missing token, closely matching masked-language-model pretraining.

4 GHOST CLIPPING: CLIPPING WITHOUT PER-EXAMPLE GRADIENTS

Ghost clipping addresses the memory cost of DP-SGD by computing per-example gradient norms without materializing full per-example gradients. This makes Transformer fine-tuning nearly as memory-efficient as non-private training while adding a backward pass.

  • Motivation: Naive DP-SGD can be prohibitively memory-intensive because clipping instantiates a giant gradient vector for each example.This problem is particularly costly for Transformer embedding layers and language-modeling heads.
  • Clipping procedure: Per-example clipping rescales each example’s loss by c_i = min(1, C/∥∇L_i∥_2) before the usual backward pass.The resulting backward pass produces the sum of clipped gradients, so the key task is obtaining per-example gradient norms.
  • Ghost clipping: For sequential linear layers, per-example gradients factor into input and output-gradient matrices, enabling their norms to be computed without explicitly instantiating the gradients.The procedure generalizes the Goodfellow trick to sequential inputs and targets linear and embedding layers.
  • Memory complexity: The squared Frobenius norm can be computed from matrix products, with memory complexity O(BT^2) instead of O(Bpd) for naive per-example-gradient instantiation.The identity is applied to the per-example gradient norm rather than constructing the full gradient tensor.
  • Memory savings: For GPT-2’s embedding layer, the method theoretically reduces memory cost by at least a factor of 22 compared with naive per-example-gradient instantiation.The example uses d ≈ 50,000, p = 768, and context window T ≤ 1024.
  • Empirical comparison: Ghost clipping was the most memory-friendly tested approach and processed ~10% more examples than Lee & Kifer (2020) for GPT-2-large.GPT-2-large could not be handled by the Opacus or JAX baselines in the reported comparison.

5 LOW DIMENSIONAL UPDATES ARE NOT NECESSARILY BETTER

The experiments test whether reducing update dimensionality improves private fine-tuning, finding that larger pretrained models and full fine-tuning are strong choices despite dimension-dependent noise concerns. They also introduce ghost clipping to reduce the memory cost of private Transformer training.

  • 5 LOW DIMENSIONAL UPDATES ARE NOT NECESSARILY BETTER: Ghost clipping reduces DP training memory by 3 times versus Opacus and approaches non-private training efficiency across model sizes.For GPT-2-large, Opacus and JAX could not fit single-example microbatches with gradient accumulation on a 24 GB GPU.
  • 5.1 LARGER PRETRAINED MODELS RESULT IN BETTER PERFORMANCE: Larger pretrained models lead to better private fine-tuned performance across language generation and sentence classification tasks at the same privacy budget.Hyperparameters were tuned on the smallest model of each type and reused across model sizes; similar gains appeared on seven tasks.
  • 5 LOW DIMENSIONAL UPDATES ARE NOT NECESSARILY BETTER: Full fine-tuning is a strong DP baseline that matches specialized low-dimensional methods for both classification and generation.The comparison directly tests whether optimizing fewer parameters improves private results.
  • 5.2 FULL FINE-TUNING WITH DP-ADAM MATCHES STATE-OF-THE-ART: Full fine-tuning with text infilling outperforms well-tuned RGP on all evaluated sentence-classification tasks.RGP is faster per update but requires more than 3 times as many epochs, making overall wall time comparable.
  • 5.2 FULL FINE-TUNING WITH DP-ADAM MATCHES STATE-OF-THE-ART: Private DP-Adam models achieve strong performance for table-to-text and chit-chat generation, including on the 130k-entry Persona-Chat dataset.Persona-Chat evaluation reports F1, perplexity, and human-rated generation quality.

6 RELATED WORK

Related work on private NLP includes differentially private federated training of small RNNs and production-scale next-word prediction, alongside other privacy-preserving language-model efforts.

  • 6 RELATED WORK: Prior work trained small word-level RNNs with 1.35 million parameters under a global DP guarantee in a federated setting with more than 700k users.The reported privacy guarantee was (ϵ, δ) = (4.6, 10^-9).
  • 6 RELATED WORK: Other work trained production-grade next-word prediction models using DP-FedAvg with millions of users.
  • 6 RELATED WORK: Related studies also examined private fine-tuning of BERT for language understanding.The supplied passage introduces this line of work but does not provide further results.

7 SCOPE AND LIMITATIONS

The paper demonstrates practical DP fine-tuning for large pretrained language models, while identifying boundaries around pretraining data, hyperparameter coverage, scaling evidence, and experimental scope.

  • Scope and implications: DP fine-tuning is presented as a competitive baseline for private NLP and may require substantially less private data than training from scratch.The authors connect this to reducing private-data collection and use, while noting broader privacy concerns remain.
  • Public pretraining: Off-the-shelf BERT and GPT-2 models rely on internet-collected pretraining data that may contain sensitive information such as personally identifiable information.The authors chose these models for accessibility and reproducibility, but recommend more curated public pretraining corpora for real-world applications.
  • Hyperparameter tuning: The hyperparameter study is incomplete: weight decay, learning-rate, clipping-norm, and batch-size schedules were not evaluated.The paper reports that hyperparameter values can significantly affect private fine-tuning and calls for more transparent reporting and privacy accounting for tuning.
  • Pretraining and scaling: The apparent benefit of larger pretrained models depends on the particular pretrained models selected, leaving how pretraining helps private learning unresolved.The authors identify better pretrained models for private learning as a future research direction.
  • Experimental scope: The paper does not establish precise private-learning scaling laws, and its experiments use public, well-studied datasets rather than private datasets.The authors frame model dimensionality, pretraining, compute, and model quality as open trade-offs for future study.
  • Societal considerations: Successful private learning may encourage more aggressive user-data collection, while harms such as representation bias are not captured by Differential Privacy.These are societal risks acknowledged alongside the paper’s technical results.

C ADDITIONAL RESULTS ON MODEL SCALING

Additional experiments show that larger and better pretrained models improve private fine-tuning across tasks, while small-batch privacy accounting affects scaling rules and label-word alignment benefits private classification.

  • Model scaling: Larger and better pretrained models consistently improve private fine-tuned performance across sentence classification and language generation tasks.The trend holds across the paper’s other evaluated tasks.
  • Hyperparameter scaling: The linear learning-rate and batch-size scaling rule does not generalize to small batches because its square-root noise relationship fails there.The paper illustrates this failure at (ϵ, δ) = (3, 10^-5), N = 50k, and E = 50.
  • Label-word choice: Increasing label-word alignment improves private SST-2 performance by 1−2%, unlike the non-private setting where label-word choice has no statistically significant effect.Alignment is measured by zero-shot infilling performance.
  • Frobenius-norm identity: The Frobenius-norm identity rewrites linear-layer gradient quantities using input and output-gradient factors.The derivation applies to a linear layer with input a and output gradient g.

G PROTOCOL FOR EXPERIMENTS IN SECTION 4.2

The experimental protocol uses mock E2E-format sequences, optimized implementations, batch-size searches, throughput comparisons, timed warmups, and full-precision runs.

  • Mock data: Mock examples use E2E-format inputs with sequence length 100, near the actual dataset’s maximum training-example length.The mock inputs are used for the experiments described in this protocol.
  • Implementations: The JAX implementation adapts prior code and optimizes the Opacus baseline by replacing einsum operations with direct tensor primitives.The changes target fair comparison and unnecessary runtime overhead.
  • Batch-size protocol: Maximum batch sizes are found by binary search, and private-method throughput is compared with non-private training using least-common-multiple update batch sizes.The search continues until the bounds match and out-of-memory errors do not occur.
  • Timing: PyTorch timing uses torch.profiler after three warmup gradient updates to remove JAX compilation time from measurements.The warmup precedes the steps used for timing.
  • Numerical precision: All experimental runs in this section use full precision.

H DETAILS AND ADDITIONAL RESULTS FOR STUDIES IN SECTION 3.1

The additional-results section includes a table of default ablation hyperparameters and a figure addressing hyperparameter sensitivity.

  • Ablation studies: Table 4 lists the default hyperparameters used for ablation studies.
  • Hyperparameter sensitivity: Figure 8 presents additional results on hyperparameter sensitivity.

I HYPERPARAMETER SEARCH RANGES FOR EXPERIMENTS IN SECTION 5

The experiment details report task-specific evaluation settings and provide hyperparameter search ranges for the different methods.

  • Evaluation protocol: Adaptation methods are evaluated with task-specific test metrics using hyperparameters that maximize validation BLEU on E2E.Sentence-classification settings reuse the same hyperparameters with adjustments for epochs and batch size.
  • Search ranges: Table 5 provides hyperparameter search ranges for different methods.
  • Search ranges: Table 6 continues the hyperparameter search ranges for different methods.

J FULL RESULTS FOR EXPERIMENTS IN SECTION 5.2

This section presents full fine-tuning results on the E2E and DART datasets using GPT-2. The DART results follow the same trend as those on E2E.

  • Table 7 reports full E2E results from fine-tuning GPT-2.
  • Table 8 reports full DART results from fine-tuning GPT-2.
  • The DART trend is consistent with the results on E2E.

K DETAILS FOR EXPERIMENTS IN SECTION 5.2

The experimental details cover evaluation procedures, datasets, model choices, privacy-related analyses, and comparisons with alternative private-learning approaches.

  • E2E and DART generations are evaluated using beam search with beam size 5 and their respective official evaluation pipelines.
  • DialoGPT-medium is additionally fine-tuned because its conversation-like pretraining data is more aligned with conversational downstream data than WebText.
  • Human ratings supplement F1 evaluation for dialogue generation because F1 can be gamed by predicting a predetermined phrase.
  • The study transfers sampling rate q across datasets rather than batch size to avoid discrepancies in effective noise multipliers across workloads.
  • The related-work discussion contrasts simplifying private learning, reducing noise scale, and accelerating DP-SGD with the paper’s ghost clipping approach.
  • The authors leave alternative clipping strategies for NLP tasks to future work and report difficulties with parameter-efficient dialogue fine-tuning when embeddings and the language-modeling head are frozen.
Loading 2110.05679v6…