Source-linked AI summary
A General Approach to Adding Differential Privacy to Iterative Training Procedures
H. Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, Peter Kairouz
TL;DR
Privacy-preserving iterative training must handle multiple heterogeneous vector queries while keeping privacy integration practical and reliable. The paper introduces configurable Gaussian mechanisms and a ledger-based accounting framework that extends subsampled-Gaussian Moments Accountant analysis to these queries. It concludes that this modular design supports heterogeneous statistics and separates privacy concerns, while noting unresolved sampling and floating-point implementation limitations.
Problem
Iterative training procedures estimate many heterogeneous vectors from shared records, creating practical challenges for configuring mechanisms and computing reliable overall privacy guarantees.
Method
The paper represents iterative training as Gaussian sum queries, supports flexible clipping and noise configurations for vector groups, and records privacy events in a ledger for later accounting.
Results
The approach provides privacy-preserving estimation for vectors with different norm bounds and noise standard deviations, including gradients, classifier accuracy, and class counts.
Takeaways & Limitations
Separating training, mechanism configuration, and accounting allows privacy estimates to remain insulated from hyperparameter-selection bugs and permits later reprocessing with improved accounting.
Takeaways & Limitations
Quantitatively tight differential-privacy analysis is not known for equally sized disjoint minibatches, and a provable floating-point DP-SGD implementation remains open.
Abstract
from arXiv · showhide
In this work we address the practical challenges of training machine learning models on privacy-sensitive datasets by introducing a modular approach that minimizes changes to training algorithms, provides a variety of configuration strategies for the privacy mechanism, and then isolates and simplifies the critical logic that computes the final privacy guarantees. A key challenge is that training algorithms often require estimating many different quantities (vectors) from the same set of examples --- for example, gradients of different layers in a deep learning architecture, as well as metrics and batch normalization parameters. Each of these may have different properties like dimensionality, magnitude, and tolerance to noise. By extending previous work on the Moments Accountant for the subsampled Gaussian mechanism, we can provide privacy for such heterogeneous sets of vectors, while also structuring the approach to minimize software engineering challenges.
1 Introduction
The paper presents a modular approach for integrating differential privacy into iterative training while separating training, mechanism configuration, and privacy accounting. It extends subsampled Gaussian privacy analysis to heterogeneous vector queries arising from shared records.
- Modular design: The approach decouples training-procedure specification, privacy-mechanism configuration, and computation of the final (ε, δ)-DP guarantee.This separation isolates privacy calculations so implementation or hyperparameter-selection bugs do not alter the privacy estimate.
- Iterative query model: Iterative procedures sample records and consume multiple vector queries, including layer gradients, batch-normalization updates, and accuracy metrics.The framework treats model training as one instance of a broader iterative template.
- Heterogeneous queries: The analysis generalizes the Moments Accountant from one vector query per round to heterogeneous collections of vectors and groups.Different vectors or groups can use distinct clipping thresholds and noise multipliers.
- Gaussian averaging mechanism: For a sampled average, records are selected with probability q, vectors are clipped to L2 threshold S, and Gaussian noise is added before scaling by the expected sample size qn.The noise multiplier z = σ/S controls the privacy–utility tradeoff.
- Privacy accounting: The mechanism’s privacy cost is determined by the sampling probability q and the privacy tuple (S, σ), where S bounds vector norms and σ is noise standard deviation.The paper focuses on unweighted averages with a fixed expected denominator, while noting straightforward extensions to weighted averages and sums.
- Implementation: The techniques are implemented in the open-source TensorFlow Privacy framework.The implementation is described as supporting the paper’s privacy-preserving training approach.
2 Privacy mechanisms for a group of vectors
The paper develops mechanisms for privately estimating groups of heterogeneous vectors, allowing separate noise and clipping choices or joint clipping across differently scaled vectors. Joint clipping uses per-vector scales while preserving a single Gaussian-sum privacy analysis.
- Privacy analysis: The group mechanisms permit individual noise standard deviations for separate groups, with the Moments Accountant applied after resolving the resulting nonspherical-noise issue.When k = 1, the two described mechanisms are identical.
- Separate clipping and noise parameters: Separate clipping and noise parameters treat a group as one concatenated vector with a group clipping threshold Sg and noise parameter σg.This mechanism is equivalent to a Gaussian mechanism for sums with privacy tuple (Sg, qnσg).
- Separate clipping and noise parameters: Applying the separate-parameter mechanism to all vectors recovers flat clipping, while applying it per vector recovers per-layer clipping.A dimensionality-aware choice uses Sg = S∗√(dg/D) for vector dimensionality dg.
- Joint clipping: Joint clipping rescales each vector by its supplied scale αj before clipping the combined vector and rescales outputs afterward.The scales represent bounds or reasonable individual clipping parameters for the component vectors.
- Privacy analysis: After scaling and joint clipping, the output is a post-processing of a subsampled Gaussian mechanism for sums with privacy tuple (Sg, qnσg).If no clipping occurs, scaling and rescaling recover each original vector component.
- Joint clipping: The joint mechanism supports differently scaled vectors while exploiting the fact that one vector may be zero for each record.The paper gives α1 = 1, α2 = 100, σg = 0.01, and Sg = 1 for vectors with norm bounds 1 and 100.
3 Composing privacy guarantees for multiple vector groups
The paper transforms multiple Gaussian mechanisms applied to vector groups into an equivalent single Gaussian sum query. This enables the privacy accountant to analyze the composite mechanism.
- 3 Composing privacy guarantees for multiple vector groups: Each vector group receives its own privacy tuple, while the groups share one sampling step.The tuples specify clipping bounds and noise levels for the respective groups.
- 3 Composing privacy guarantees for multiple vector groups: The composite mechanisms can be analyzed by imagining a single Gaussian sum query with unit noise and rescaling each group’s output.The rescaling uses the group-specific noise factors.
- 3 Composing privacy guarantees for multiple vector groups: The resulting expression is post-processing of one Gaussian sum query with parameters (S∗, σ = 1).The privacy accountant can therefore bound privacy loss for iterative applications of the composite mechanism.
4 Hyperparameter selection strategies
The paper frames hyperparameter selection as choosing sampling, clipping, and noise settings to balance privacy and utility. It describes proportional and dimensionality-adjusted strategies for distributing noise across vector groups.
- 4 Hyperparameter selection strategies: The key quantity for selecting q, Sg, and σg is the noise multiplier z, with ˜σg = qnσg.These hyperparameters are selected to achieve a desired privacy versus utility tradeoff.
- 4 Hyperparameter selection strategies: z ≈1 typically provides a reasonable privacy guarantee.If z is too small, users can decrease clipping thresholds, increase noise, or increase the sampling rate q.
- 4 Hyperparameter selection strategies: Increasing q is generally preferable when datasets are large and the added cost of processing larger samples is affordable.The paper notes this preference in the setting where more computation is acceptable.
- 4 Hyperparameter selection strategies: Clipping thresholds can be chosen from norm bounds, low-clipping heuristics, or parameter-tuning grids, while noise levels can be tuned for acceptable utility loss.If private data informs clipping or noise tuning, its privacy cost must be accounted for.
- 4 Hyperparameter selection strategies: Proportional noise sets ˜σg ∝ Sg, whereas dimensionality-adjusted noise scales with the group dimension dg and bound Sg.The dimensionality-adjusted strategy uses the maximum root mean squared component value implied by each group’s bound and dimensionality.
5 Sampling policies
The paper distinguishes several minibatch sampling policies because their privacy analyses and practical properties differ. It emphasizes that equally sized, disjoint minibatches are common and efficiently implemented, but lack quantitatively tight DP-SGD analysis.
- 5 Sampling policies: Standard SGD uses small minibatches, and the original Moments Accountant analyzes the regime where minibatches are i.i.d. samples.The convergence guarantees of standard optimization theory also assume i.i.d. minibatches.
- 5 Sampling policies: Alternative minibatch policies are practically relevant and have different implications for privacy analysis.The paper introduces three common sampling policies.
- 5 Sampling policies: I.i.d. minibatches support privacy analysis from prior work but can make sample-size secrecy and hardware implementation problematic.Variable sample sizes are a poor fit for hardware that expects fixed-size batches.
- 5 Sampling policies: Equally sized, independent minibatches make multi-iteration privacy analysis straightforward through differential privacy composition rules.This policy includes basic SGD as the cardinality-1 case.
- 5 Sampling policies: Equally sized, disjoint minibatches are efficiently implemented and give epochs the semantics of visiting every example once, but quantitatively tight DP-SGD analysis is not known.The policy repeatedly permutes and partitions the dataset into fixed-size subsets.
6 Privacy ledger
The privacy ledger separates mechanism execution from privacy accounting by recording sampling and Gaussian sum-query events. The accountant can later process this ledger to compute guarantees and reprocess it with improved accounting methods.
- 6 Privacy ledger: The ledger records sampling events with q and n, and sum-query events for Gaussian aggregates.Accounting is performed after the events are recorded rather than necessarily online.
- 6 Privacy ledger: The accountant converts each sampling event plus its sum-query events into an equivalent single event with parameters (S∗, σ = 1).This conversion uses Equation (1) before producing the privacy guarantee.
- 6 Privacy ledger: Separating accounting from configuration prevents hyperparameter-selection bugs from affecting the privacy estimate.The separation isolates the critical privacy calculations from configuration logic.
- 6 Privacy ledger: The ledger can be reprocessed when a tighter privacy-loss bound becomes available after data processing.This allows the accounting mechanism to change without rerunning the data-processing stage.
7 TensorFlow Privacy
TensorFlow Privacy operationalizes the paper’s decoupled design for differentially private training, supporting optimizer wrapping, configurable mechanisms, federated use, and privacy accounting.
- TensorFlow Privacy implements TensorFlow optimizers for training machine learning models with differential privacy.It also provides tutorials and analysis tools for computing privacy guarantees.
- The library wraps existing optimizers with gradient clipping and noising logic to obtain privacy.The illustrated use case wraps TensorFlow’s stochastic gradient descent optimizer.
- Microbatches are clipped before minibatch gradients are averaged and noised, while increasing their number often improves utility but typically slows training.The number of microbatches is an additional optimizer hyperparameter.
- TensorFlow Privacy supports federated training by treating user-supplied model updates as the gradients provided to the DPQuery.
- Its RDP accountant sums per-step privacy costs from the sampling fraction q and noise multiplier z to estimate ε for fixed δ.
8 Floating-point arithmetic and randomness source
The paper distinguishes ideal differential privacy from practical implementations using finite-precision arithmetic and pseudorandomness, identifying corresponding scope boundaries.
- Differential privacy is defined against adversaries with unrestricted knowledge and capabilities, imposing strong requirements on the output distribution.
- Floating-point arithmetic: The paper leaves open a provable floating-point implementation of DP-SGD integrated into a machine-learning library.
- Randomness source: Cryptographically secure pseudorandom generators approximate true entropy only against computationally bounded distinguishers, so implementations provide computational differential privacy variants.
- Randomness source: Only randomness recorded in the privacy ledger must use a cryptographically secure PRNG; minibatch sampling and Gaussian noise therefore require it, whereas microbatch selection need not be randomized.
9 Conclusion
The conclusion presents a modular privacy framework for heterogeneous vector queries and separates iterative-training implementation, configuration, and privacy accounting.
- The Gaussian mechanism handles heterogeneous vectors with different norm bounds and noise standard deviations.This supports privacy-preserving estimation of gradients and other statistics such as classifier accuracy and class counts.
- Recording Gaussian sum-query privacy events in a ledger for an accountant separates the three major concerns of privacy-preserving iterative training.The separation preserves flexibility in clipping strategies and noise allocation.
- The techniques can be implemented using the TensorFlow Privacy library.
A Differential Privacy
The appendix defines differential privacy through indistinguishability of outputs on adjacent datasets, with the protected unit determined by how adjacency is defined.
- A randomized mechanism is (ε, δ)-differentially private when output probabilities on any measurable set differ between adjacent datasets by the stated multiplicative and additive bounds.
- The choice of adjacent datasets determines the unit of information protected by the algorithm.Common machine-learning cases include example-level privacy.