PaperlayerView exact revision on arXiv

Source-linked AI summary

DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass

Minxin Du, Xiang Yue, Sherman S. M. Chow, Tianhao Wang, Chenyu Huang, Huan Sun

arXiv:2309.06746v2cs.CR

TL;DR

DP-SGD is costly for fine-tuning large language models and leaves inference-time embedding attacks uncovered. DP-Forward perturbs embedding matrices in the forward pass. Across three tasks it often outperforms DP-SGD, while also reducing the success of embedding attacks.

  • Problem

    DP-SGD is costly for fine-tuning large language models and does not perturb inference-time embeddings, leaving them vulnerable to recovery attacks.

    Source: §1 · pp. 1–2

  • Method

    DP-Forward perturbs embedding matrices in the language model's forward pass using the analytic matrix Gaussian mechanism.

    Source: §8 · p. 14

  • Results

    Across three tasks, at least half of DP-Forward instances outperform DP-SGD on each task, with a largest accuracy gain of ∼7.7pp for QQP; embedding inversion success falls by up to 88pp and attribute-inference success by ∼41pp.

    Source: §1.2 · p. 2, §5.4 · p. 10

  • Takeaways & Limitations

    Users can share sanitized data for LM-as-a-Service APIs while protecting privacy, and need download only pipeline parts for deriving noisy embeddings.

    Source: §8 · p. 14

    The aMGM's singular-value allocation is empirical; optimizing allocations for different utility functions remains a theoretical alternative.

    Source: §4.2.1 · p. 8

Abstract

from arXiv · show

Differentially private stochastic gradient descent (DP-SGD) adds noise to gradients in back-propagation, safeguarding training data from privacy leakage, particularly membership inference. It fails to cover (inference-time) threats like embedding inversion and sensitive attribute inference. It is also costly in storage and computation when used to fine-tune large pre-trained language models (LMs). We propose DP-Forward, which directly perturbs embedding matrices in the forward pass of LMs. It satisfies stringent local DP requirements for training and inference data. To instantiate it using the smallest matrix-valued noise, we devise an analytic matrix Gaussian~mechanism (aMGM) by drawing possibly non-i.i.d. noise from a matrix Gaussian distribution. We then investigate perturbing outputs from different hidden (sub-)layers of LMs with aMGM noises. Its utility on three typical tasks almost hits the non-private baseline and outperforms DP-SGD by up to 7.7pp at a moderate privacy level. It saves 3$\times$ time and memory costs compared to DP-SGD with the latest high-speed library. It also reduces the average success rates of embedding inversion and sensitive attribute inference by up to 88pp and 41pp, respectively, whereas DP-SGD fails.

1 INTRODUCTION

DP-Forward perturbs forward-pass embeddings locally rather than aggregated gradients, extending protection to inference-time embeddings while addressing the cost and utility challenges of DP-SGD. The paper introduces aMGM for matrix-valued noise and reports task, efficiency, and attack-resistance comparisons with DP-SGD.

1 INTRODUCTION

  • DP-SGD protects training examples by clipping per-example gradients and adding Gaussian noise, but requires a trusted curator for central DP and is costly for large pipelines.The cited example is BERT-Base, with more than 110M parameters.Source: §1 · p. 1

1.1 Natural Privacy via Perturbing Embeddings

  • aMGM combines analytic-GM’s necessary and sufficient DP condition with matrix Gaussian noise, allowing potentially non-i.i.d. noise calibration.The paper notes that i.i.d. noise may already be optimal for general applications such as DP-Forward.Source: §1.1 · p. 2
  • The method applies aMGM noise to embeddings from hidden sub-layers, normalizing outputs to a fixed Frobenius or L2 norm to estimate sequence-level sensitivity.For the first few sub-layers, the paper also describes token-level relaxations to improve accuracy.Source: §1.1 · p. 2
  • DP-Forward lets users locally perturb sequence embeddings before sharing them, providing sequence-level LDP for training and standard LDP for unlabeled test sequences.Subsequent computation on noisy embeddings incurs no extra privacy loss under post-processing.Source: §1.2 · p. 2, §1.1 · p. 2

1.2 Our Contributions

  • Up to 7.7 percentage points of accuracy gain over DP-SGD or its recent improvements is reported for deep-layer DP-Forward at a similar privacy level; DP-SGD incurs >3× time and GPU-memory costs.The comparison uses randomized response for labels and shuffling to compare standard LDP with example-level CDP.Source: §1.2 · p. 2
  • DP-SGD leaves inference-time embeddings unperturbed, while DP-Forward reduces embedding-inversion success by up to 88pp and attribute-inference success by about 41pp.DP-SGD fails in two embedding inversion attacks; its average reduction for neural-network attribute inference is 15pp.Source: §1.2 · p. 2, §1 · p. 2

2 PRELIMINARIES AND NOTATIONS

The preliminaries outline BERT’s encoder computations and establish the differential-privacy notions and mechanisms used later.

2.1 Transformer Encoders in BERT

  • BERT maps tokens to embeddings, processes them through attention and feed-forward sub-layers, and reduces the final encoder output to a sequence feature.The sub-layers use residual connections and layer normalization; sequence features can use mean pooling or the [CLS] embedding.Source: §2.1 · p. 3

2.2 (Local) Differential Privacy

  • Central DP protects neighboring datasets through a curator, whereas LDP lets individuals perturb inputs before sending them to an untrusted aggregator.DP also permits free post-processing and composes across repeated mechanisms, with privacy loss increasing under sequential composition.Source: §2.2 · pp. 3–4
  • The Gaussian mechanism perturbs matrix-valued outputs with i.i.d. Gaussian noise, while privacy loss can be analyzed through privacy loss random variables.The section also introduces PLRVs as log-ratios of output probabilities and uses their tail bounds in the proposed mechanism.Source: §2.2 · p. 4

3 DP-FORWARD

DP-Forward protects sequences by perturbing their embeddings locally, rather than perturbing aggregated gradients, and extends this approach to inference. The section develops sequence-level privacy guarantees and describes normalization, fine-tuning, shuffling, and noisy pre-training as parts of the approach.

3 DP-FORWARD

  • Users locally perturb sequence embeddings before sharing them for fine-tuning or pre-training, and DP-Forward also protects inference sequences unlike DP-SGD.The provider processes noisy embeddings, with free post-processing allowing minimal pipeline changes for inference.Source: §3 · p. 4

3.1 Notions of Sequence (Local) DP

  • SeqDP protects sequences rather than sequence-label pairs, while label DP protects labels only; the paper notes SeqDP complements label DP because sequences can reveal labels.Sequence-only protection is meaningful when labels are public or absent, and SeqLDP is its local counterpart.Source: §3.1 · p. 4, §3.1.1 · p. 4, §3.1.2 · p. 4

3.1.2 Sequence Local DP (SeqLDP).

  • SeqLDP bounds indistinguishability between sequences, but attaining meaningful privacy parameters is difficult; early-layer perturbation can hurt utility and motivates weaker token-level SeqLDP.The general approach normalizes outputs, and the section identifies deeper pre-noise functions as more favorable for utility.Source: §3.1.3 · p. 5, §3.1.2 · p. 5

3.2 Our Approach for Sequence LDP

  • DP-Forward splits the pipeline at a hidden layer and normalizes or clips its outputs before adding calibrated Gaussian noise, making sensitivity tractable even when attention is not Lipschitz.Optional dimension-reduction maps can alleviate high-dimensional noise, but may lose useful information and require pipeline modifications.Source: §3.2 · p. 5

3.3 DP-Forward Fine-tuning

  • During fine-tuning, users send noisy embeddings and labels to the provider, which updates post-noise layers; normalized or clipped outputs give (𝜖,𝛿)-SeqLDP.Updating pre-noise layers requires either a trusted party, which makes the setup central DP, or costly secure aggregation; freezing those layers avoids raw-input access but may reduce utility.Source: §3.3 · pp. 5–6

3.4 DP-Forward with Shuffling versus DP-SGD

  • Perturbing labels separately and shuffling noisy embedding-label pairs lets DP-Forward claim example-level central DP, and the paper reports a qualitative SNR advantage over DP-SGD.Label perturbation uses randomized response; shuffling amplifies the local guarantee without extra noise addition.Source: §3.4 · p. 6

3.5 DP-Forward Inference

  • At inference, users can send noisy test-sequence embeddings that satisfy (𝜖,𝛿)-LDP, and matching inference noise to fine-tuning noise can benefit task accuracy.The labels are absent for test sequences, so the guarantee applies to the sequences themselves.Source: §3.5 · p. 6

3.6 DP-Forward Pre-training

  • Noisy pre-training modifies the MLM objective to predict raw masked tokens from noisy embeddings, while retaining sequence-level LDP through free post-processing.The paper describes this as enabling noisy BERT to be offered as a service.Source: §3.6 · p. 6

4 OPTIMIZING MATRIX GAUSSIAN NOISE

The section explains why the classical Gaussian mechanism and MVG are suboptimal for matrix-valued outputs, then develops aMGM using a necessary-and-sufficient DP condition. It derives covariance constraints and noise choices for the mechanism, including the uniform-noise choice used for DP-Forward.

4.1 Matrix-Variate Gaussian (MVG) Mechanism

  • MVG draws potentially non-i.i.d. matrix Gaussian noise using row and column covariances, allowing less noise on more important rows or columns.Source: §4 · p. 7, §4.1 · p. 7
  • MVG’s sufficient-condition and loose trace-based analyses are suboptimal, and its bound can drive summed noise upward as nd grows or ε→0.The discussion identifies two sources of slack: the sufficient DP condition and the matrix-trace-based privacy analysis.Source: §4.1 · p. 7

4.2 Analytic Matrix Gaussian Mechanism

  • aMGM replaces a sufficient DP condition with the necessary-and-sufficient condition from aGM, adapting calibration to two covariance matrices; its PLRVs follow N(η, 2η).This adaptation is non-trivial because aMGM uses two covariance matrices rather than a single variance.Source: §4.2 · p. 7
  • Because aMGM’s lower bound is independent of nd, it can yield orders-of-magnitude variance reduction over MVG, as experiments confirm.Theorem 6 constrains only the product of the two smallest singular values, leaving other covariance choices open.Source: §4.2.1 · p. 8
  • For DP-Forward, equal singular values give i.i.d. noise—the aGM case—and pilot experiments found this choice best for task utility; application-specific allocation is future work.Larger singular values can allocate more noise to rows or columns that harm utility.Source: §4.2.1 · p. 8

5 EXPERIMENTS

The experiments test SeqLDP configuration and layer placement, compare DP-Forward with private and non-private baselines, and assess efficiency and noisy pre-training.

5.1 Experimental Setup

  • The experiments used SST-2, IMDb, and QQP, instantiating DP-Forward with GM, MVG, or aMGM and comparing it with DP-SGD and a non-private baseline.The reported DP-SGD baseline used private fine-tuning but non-private inference; the non-private baseline used neither private fine-tuning nor inference.Source: §5.1 · pp. 8–9

5.2 Configuring Matrix Dimensions

  • SST-2 accuracy was often best at n=64 or 128, so the authors chose n=128 for later experiments.With fixed normalization and privacy parameters, n and d determine embedding-entry signal magnitude and configure signal-to-noise ratios.Source: §5.2 · p. 9, Figure 2 · p. 9

5.3 Fine-tuning with Sequence LDP

  • DP-Forward performed better with deeper encoders, while noise position within an encoder caused large accuracy variation, including a ~20pp last-versus-first sub-layer gain in Encoder 1.The comparison is for SST-2 at ε=16 for SeqLDP.Source: Figure 3 · p. 9, §5.3 · p. 10
  • As ε increased, accuracy rose on all three tasks for GM and aMGM; aMGM beat GM, while MVG performed near random guessing.The authors report that effective models could not be fine-tuned at ε<1; under n=128 and d=768 at ε=8, MVG noise variance was >10^8 versus ~0.6 for aMGM.Source: §5.3 · p. 10, Table 4 · p. 10

5.4 DP-Forward versus DP-SGD

  • For each task, at least half of eight DP-Forward instances beat DP-SGD; the largest gain was ~7.7pp on QQP, and noisy output embeddings often approached non-private accuracy.The noisy output embeddings often gave the best accuracy, attributed to dimension reduction at the last encoder output.Source: §5.4 · p. 10
  • DP-Forward’s fine-tuning time and storage overheads were nearly those of the non-private baseline and ~3× smaller than DP-SGD’s.The approach allows batch processing without per-example gradients, and its normalization and noise operations work on smaller embeddings than gradients.Source: §5.4 · p. 10

5.5 Noisy Pre-training

  • Noisy pre-training on WikiCorpus yielded 1–2pp accuracy gains for most ε choices over fine-tuning on the original BERT.The reported pre-training used the 2006 English WikiCorpus dump, about 600M words, for one epoch; it also consumed fewer resources.Source: §5.5 · p. 10

6 DEFENSE AGAINST PRIVACY THREATS

The section evaluates membership inference, embedding inversion, and sensitive-attribute inference under stated black-box threat models. DP-Forward and DP-SGD are compared across these threats, with DP-Forward also evaluated for protection of inference-time embeddings.

6.1 Threat Models

  • MIAs use black-box prediction access without pipeline weights or architecture; embedding attackers can access clear or noisy inference embeddings and limited auxiliary data.The MIA adversary queries prediction results, while the embedding attacker has auxiliary data with attributes similar to those of the targets.Source: §6.1 · p. 11

6.2 Membership Inference Attacks

  • Confidence-above-threshold and entropy-below-threshold rules infer membership; entropy can flag confidently wrong predictions, a caveat addressed by encoding the ground-truth label.Thresholds are selected on one split; test tokens are randomly dropped or replaced to make MIAs easier.Source: §6.2 · p. 11
  • About 0.5: deeper-layer DP-Forward success rates for both MIAs, outperforming DP-SGD by >6pp at the same privacy level.This pattern holds for all choices of ε.Source: Table 7 · p. 12, §6.2 · p. 12

6.3 Embedding Inversion Attacks

  • Token-wise inversion uses nearest-neighbor search on input embeddings; a general gradient-based attack maps observed embeddings to lower-layer representations and optimizes token choices.The attacks aim to recover raw text as unordered tokens.Source: §6.3 · p. 12
  • Within 0.2: most embedding-inversion attack success rates under DP-Forward; DP-SGD fails to defend.The evaluation concerns two inversion attacks on the lowest-layer input embeddings at translated central ε≈8.Source: Table 8 · p. 12, §6.3 · p. 12

6.4 Sensitive Attribute Inference Attacks

  • Table 9: DP-Forward reduces the attribute classifier to majority-class prediction, while DP-SGD only moderately lowers success rates versus the non-private baseline.The classifier returns the majority class, “action,” on all inputs.Source: Table 9 · p. 12, §6.4 · p. 13
  • The attribute attack trains a supervised classifier on labeled auxiliary sequences, then predicts sensitive attributes from target embeddings; IMDb supplies 20k auxiliary examples and SST-2 supplies 3.3k test examples.The reported sensitive attribute is film genre, and success is measured using recall.Source: §6.4 · pp. 12–13

7 RELATED WORK

Related work addresses privacy risks in language models, sanitization of text and embeddings, DP-SGD variants, and mechanisms for matrix-valued functions. The paper distinguishes its focus on perturbing hidden-layer embeddings from prior approaches.

7.1 Privacy Threats on LMs and Embeddings

  • Taxonomies cover inversion, membership, and sensitive-attribute attacks on embeddings, while adversarial training is a common defense; related work also studies memorization and black-box text extraction, including at k=1.The account also describes metrics for update leakage and identifies DP as a promising approach to memorization.Source: §7.1 · p. 13

7.2 Input (Text/Feature) Perturbation for LDP

  • Prior work sanitizes term-frequency vectors [71], tokens, or sentence embeddings; perturbing hidden-layer embeddings inside LM pipelines remains unexplored.The approaches include token-level metric-LDP and sequence-level mechanisms.Source: §7.2 · p. 13

7.3 DP-SGD (Variants) in Training LMs

  • DP-SGD variants address cost through low-rank reparameterization, plug-in parameters, and ghost clipping, yet still protect training data by perturbing gradients.An earlier effort applies DP-SGD to federated LSTM language models; DP-SGD/Adam has also been used to pre-train BERT-Large.Source: §7.3 · p. 13

7.4 DP Mechanisms for Matrix Functions

  • Matrix Gaussian mechanisms use directional, potentially non-i.i.d. noise with singular-value bounds; specialized mechanisms address linear counting queries and binary matrix outputs.The follow-up tightens the sufficient bound on singular values described for.Source: §7.4 · pp. 13–14

8 CONCLUSION

The conclusion identifies privacy exposure in training and inference inputs, alongside the cost of vanilla DP-SGD for large language models. It presents DP-Forward’s forward-pass perturbation and matrix Gaussian mechanism as an alternative, and positions forward-pass sanitization as a direction for further privacy research.

8 CONCLUSION

  • Fine-tuning corpora and inference inputs face privacy attacks, while vanilla DP-SGD is costly in GPU memory and computation and cannot be batched.The passage notes that raw tokens or sensitive attributes can be recovered or inferred from forward-pass embeddings.Source: §8 · p. 14
  • DP-Forward adds noise directly to embedding matrices derived from raw training or inference data during the forward pass, using the analytic matrix Gaussian mechanism for matrix-valued noise.The mechanism draws optimal matrix-valued noise under a necessary-and-sufficient condition for DP.Source: §8 · p. 14
  • Perturbing embeddings at different pipeline positions lets users download only the pipeline parts needed to derive noisy embeddings, which is more storage- and time-efficient than deriving noisy gradients.The conclusion connects these options with prior sanitization of input tokens and output sentence embeddings for sharing sanitized data with LM-as-a-Service APIs.Source: §8 · p. 14
  • The paper presents forward-pass signal sanitization as a new local-DP paradigm and a basis for further privacy research, including possible extension to transformer-based computer vision.Its evaluation includes baselines across three typical NLP tasks and studies efficiency, accuracy, and defenses against diverse attacks.Source: §8 · p. 14

A.1 Definition and Related Notions

This appendix defines token-level SeqLDP for sequences differing in one token while retaining the same label. It reports empirical MIA mitigation for this setting and explains the protection target motivating the notion.

A.1 Definition and Related Notions

  • Token-level SeqLDP considers neighboring sequences that differ in one token but have the same label.The definition applies for ε≥0 and 0≤δ≤1.Source: §A.1 · p. 15
  • With only the input embedding layer as f(·), token-level SeqLDP designs can mitigate sequence-level MIAs, with up to 20pp accuracy gains at the same ε choices.The appendix says this guarantee covers pairs ranging from very similar sequences to radically different ones.Source: §A.1 · p. 16
  • The paper studies token-level SeqLDP to narrow the gap between formal privacy notions and protection targets, distinguishing first-layer protection from whole-pipeline protection.It contrasts this goal with metric-DP token-level protection, which it says requires much larger ε, such as 175.Source: §A.1 · p. 16

A.2 Two Token-level SeqLDP Designs

The section presents two token-level SeqLDP designs that estimate tighter sensitivity bounds for the input embedding layer and early attention processing. Both designs satisfy token-level (𝜖,𝛿)-(Seq)LDP for fine-tuning or inference.

A.2 Two Token-level SeqLDP Designs

  • For the input embedding layer, independently normalizing each row to norm C gives sensitivity S2(f) = C, whether or not the embedding layer is updated.Noise is drawn as Z ∈ R^(n×d).Source: §A.2 · p. 16
  • MHA(·) is not Lipschitz because of its nonlinear Softmax activation, complicating sensitivity estimation for noise added after attention.The section notes that proposed alternatives—L2-MHA and LipschitzNorm—have estimation or suitability problems for high-dimensional BERT attention.Source: §A.2 · p. 16
  • Instead, the second design shifts noise inside Softmax to the linear maps deriving Q, K, and V, whose largest singular value gives the 2-norm Lipschitz constant.The noisy matrices can be formed by adding noise to XW_Q, XW_K, and XW_V; this requires estimating the maps’ largest singular values or fixing the maps during each forward pass.Source: §A.2 · p. 16
  • Both row-wise-normalized designs satisfy token-level (𝜖,𝛿)-(Seq)LDP for fine-tuning or inference.The designs use separate sensitivity estimates for the input embedding layer and early attention processing.Source: §A.2 · p. 16
  • For deeper pipeline components, composed-layer sensitivity bounds can become looser, so the section recommends adding noise early when a tight sensitivity estimate is feasible.The discussion gives ReLU a Lipschitz constant of 1 and describes estimating linear-map and layer-normalization sensitivity using rank and Lipschitz bounds.Source: §A.2 · p. 16

A.3 More Experiment Results

The additional experiments examine token-level SeqLDP accuracy and attack success, while the matrix-algebra appendix establishes equivalence between two formulations of the Gaussian mechanism’s density.

A.3 More Experiment Results

  • ∼0.7 vs. 0.5 for IMDb: task accuracy increases with local 𝜖, and input-embedding token-level SeqLDP outperforms sequence-level SeqLDP in this example.Figure 5 presents the privacy-accuracy tradeoff for token-level SeqLDP instances as local 𝜖 is tuned.Source: §A.3 · p. 16, Figure 5 · p. 17
  • ∼14pp and ∼11pp: token-level SeqLDP reduces confidence-based and entropy-based attack success rates, respectively, versus the non-private baseline on SST-2.Table 10 reports success rates for the two sequence-level MIAs on the token-level SeqLDP instances.Source: §A.3 · p. 16, Table 10 · p. 17

B RELEVANT MATRIX ALGEBRA

  • The PDF in Eq. (5) is equivalent to the matrix-trace-based PDF used in MVG.The appendix proves the equivalence by comparing the density expressions and using invariance of matrix trace under similarity transformation.Source: B RELEVANT MATRIX ALGEBRA · p. 17

C PROOFS FOR OUR ANALYTIC MATRIX GAUSSIAN MECHANISM

The proofs derive likelihood-ratio and privacy bounds for the matrix Gaussian mechanism, then connect a norm condition to the stated DP guarantee. They use a change of coordinates that makes the transformed noise standard Gaussian.

C PROOFS FOR OUR ANALYTIC MATRIX GAUSSIAN MECHANISM

  • For Lemma 1, the proof transforms the noise and sensitivity difference by U^-1 and V^-⊤, yielding standard i.i.d. Gaussian entries for the transformed noise.It uses this representation to express the likelihood ratio through an inner product involving the transformed sensitivity difference.Source: C PROOFS FOR OUR ANALYTIC MATRIX… · p. 18
  • Lemma 2 applies the standard normal CDF and symmetry to express the privacy-bound probability using tail probabilities.The proof also applies the same argument with the neighboring inputs reversed.Source: C PROOFS FOR OUR ANALYTIC MATRIX… · p. 18
  • Theorem 6’s reverse direction is reduced to showing ||Δ′||F ≤ B for neighboring inputs, using singular-value bounds and Theorem 6.The proof applies a matrix norm inequality to the transformed difference U^-1ΔV^-⊤.Source: C PROOFS FOR OUR ANALYTIC MATRIX… · p. 18

Related papers

Loading 2309.06746v2…