Source-linked AI summary
Differentially Private Average Treatment Effect Estimation by Propensity Score Blocking
Duncan Stewardson, Grayson W. White, Adam Groce
TL;DR
Observational ATE estimation often uses sensitive data, creating a need for private methods that work beyond restrictive settings and at small sample sizes. The paper proposes SeqIPW and DPBlocking, two propensity-score-based ε-DP estimators. Both improve error and bias over prior work, with DPBlocking often reducing error by 75% or more.
Problem
Differentially private ATE estimation needs methods for sensitive observational data that work at small sample sizes and beyond restrictive prior settings.
Method
The paper proposes SeqIPW, an improved IPW estimator, and DPBlocking, a BPS-based estimator for nonparametric ATE estimation.
Results
Both algorithms achieve lower observed error and bias than prior work, with DPBlocking often reducing error by 75% or more.
Takeaways & Limitations
The proposed methods provide differentially private ATE point estimates with improved accuracy across the tested settings.
Takeaways & Limitations
Generating confidence intervals on top of the private point estimates remains an important future direction.
Abstract
from arXiv · showhide
Average treatment effect (ATE) estimation in observational studies is a fundamental statistical tool used frequently in social science, medicine, and other fields. These fields often work with sensitive data where privacy protections are important, so a differentially private mechanism for ATE estimation is highly desirable. Here we present two propensity score-based algorithms for ATE estimation on observational data, one improving the inverse probability weighting (IPW) method used in prior work, and the other using blocking on the propensity score (BPS). Both show lower error and less bias than prior work, with the BPS-based algorithm frequently reducing error by 75% or more compared to prior work.
1 Introduction
Observational studies often estimate treatment effects when randomized trials are impractical, but sensitive data make privacy protection important. This work develops private ATE methods for small samples and evaluates concrete performance.
- Motivation: Observational data are used when randomized controlled trials are expensive or unethical, but treatment assignment may confound outcome comparisons.Propensity scores help account for confounding by estimating each individual’s treatment probability for stratification, matching, or weighting.
- Motivation: Sensitive or legally protected attributes make privacy particularly important in ATE studies using existing observational data.The paper targets differential privacy, which protects whether an individual is present in the data.
- Contributions: The paper presents two differentially private, nonparametric ATE algorithms requiring only approximately accurate bounds on covariates and outcomes.SeqIPW improves inverse probability weighting, while DPBlocking applies blocking on the propensity score.
- Contributions: 75% or more: the BPS-based algorithm often reduces error compared with prior work, while both algorithms improve accuracy and bias.The algorithms are evaluated on simulated and real data.
2 Background
The background defines differential privacy, potential-outcomes ATE estimation, propensity-score methods, and the assumptions needed for observational causal inference. Prior private nonparametric ATE work remains limited, motivating the proposed methods.
- Differential privacy: Differential privacy limits how much one individual’s information can affect a randomized mechanism’s output distribution.The paper uses ε-DP, whereas prior ATE work uses the weaker (ε, δ)-DP formulation.
- Differential privacy: Laplace noise achieves ε-DP, and its scale is calibrated to a function’s sensitivity, the maximum output change from altering one row.The Laplace mechanism adds noise proportional to sensitivity divided by ε.
- Average treatment effect: ATE is the expected difference between each unit’s potential outcomes, although only one potential outcome is observed per individual.The potential-outcomes dataset contains covariates, treatment assignment, and an observed outcome.
- Average treatment effect: Observational ATE estimation requires SUTVA, unconfoundedness, and overlap because treatment assignment is not generally independent of covariates.The propensity score e(X) is the treatment probability conditional on covariates and can reformulate unconfoundedness.
- ATE estimators: IPW weights outcomes by inverse estimated propensity scores, whereas BPS stratifies observations into propensity-score blocks and averages within-block mean differences.Five propensity-score blocks are often sufficient to account for most confounding.
- Related work: Prior differentially private nonparametric ATE methods are limited to two works allowing broader outcomes and minimal covariate assumptions.Earlier work either splits data for private propensity modeling and IPW or privately aggregates many non-private estimators.
3 Proposed Algorithms
The paper proposes SeqIPW and DPBlocking as ε-DP estimators using propensity scores. SeqIPW avoids data splitting and uses Laplace noise, while DPBlocking privatizes bin counts and outcome sums to reduce noise.
- Sensitivity: The IPW sensitivity bound is 2Cy(nω)^−1 when outcomes are bounded and estimated propensity scores lie in [ω, 1 −ω].This bound calibrates Laplace noise for private IPW estimation.
- SeqIPW: SeqIPW trains the propensity-score model and estimates ATE on the entire dataset instead of splitting the data as prior work does.This uses sequential composition while reducing statistical error and bias associated with smaller samples.
- SeqIPW: Laplace noise gives SeqIPW ε-DP for both propensity estimation and final ATE estimation, with privacy budgets split between the two stages.The algorithm clips propensity scores to [ω, 1 −ω] and satisfies ε-DP by composition.
- DPBlocking: DPBlocking uses propensity-score blocking because BPS can have lower non-private bias and error than IPW, while histogram queries permit lower-noise privatization.The method adds noise to bin counts and outcome sums rather than naively perturbing the final estimator.
- DPBlocking: Equal-spaced, data-independent bins replace the usual propensity-score quintiles, and bins with noisy treatment counts below threshold T are omitted.The threshold prevents potentially highly inaccurate estimates caused by noisy counts near zero.
- Privacy analysis: DPBlocking satisfies ε-DP by privatizing propensity modeling, bin counts, and bin outcome sums under separate privacy-budget shares, then composing them.The final estimator is post-processing of the private quantities.
4 Experiments
The experiments evaluate observed error and bias for SeqIPW and DPBlocking across synthetic settings, privacy levels, sample sizes, and binary or real-data comparisons. DPBlocking generally performs best as sample size or privacy budget increases, while SeqIPW can be preferable in small-sample settings.
- Goals and Metrics: Observed error and bias are the primary evaluation metrics, with observed error emphasized because it captures both bias and estimator variance.The experiments compare synthetic-data estimates against the true treatment effect and include LGPM19 plus non-private IPW and BPS estimates.
- Well-Specified Setting: At ε = 0.25, DPBlocking reduces observed error versus LGPM19 by 63% on average; the reduction rises to 75% at ε = 1.0 and 84% at ε = 3.0.At n = 25,000 and ε = 3.0, the reduction reaches 90%.
- Unbalanced and Low Overlap: For low overlap and unbalanced data at ε = 1.0, DPBlocking has lower observed error than both private IPW-based methods, and it can beat non-private IPW.In low overlap, DPBlocking is better for all tested n and beats non-private IPW when n > 2,000.
- Binary Outcomes: For binary outcomes, DPBlocking gives more accurate point estimates than Guha and Reiter’s method, although that method also provides confidence intervals.The comparison uses translated outcomes in {-0.5, 0.5} for DPBlocking’s privacy analysis.
- National Supported Work Data: On real-data benchmarks, DPBlocking has the lowest error and bias for every privacy level on the largest dataset, while SeqIPW can have lower error on smaller datasets.DPBlocking has lower bias than SeqIPW on PSID2 for ε = 1.0 and 3.0, and lower error than LGPM19 in all cases.
5 Conclusion
The paper presents two differentially private ATE algorithms and reports lower observed error and bias than prior work in most tested settings. It also identifies confidence intervals and privatized bias-reduction methods as future directions.
- SeqIPW and DPBlocking are two new differentially private algorithms for nonparametric ATE estimation.
- DPBlocking achieves lower observed error than prior work for all tested values of n and ε.
- For most values of n and ε, both algorithms achieve lower observed bias than LGPM19.
- For low n and ε without low overlap or unbalanced datasets, SeqIPW has lower error; otherwise DPBlocking has lower error and bias.
- Generating confidence intervals and privatizing augmented IPW or normalized-weight IPW are identified as future directions.
C Other Tested Estimators
This section extends prior private ATE estimators to broader outcome settings and describes their privacy-preserving constructions. It also reports that SeqIPW performs well when its privacy budget is allocated with α near 0.2 or 0.3.
- SeqNIPW generalizes prior work by using logistic regression and supporting any bounded outcomes instead of a restrictive propensity model and binary outcomes.
- SeqNIPW splits the privacy budget between propensity-score training and final estimation, then uses calibrated noise and post-processing to preserve ε-DP.
- AggNIPW generalizes a binary-outcome method to bounded outcomes by averaging noisy non-private normalized-IPW estimates across randomly partitioned subsets.
- AggNIPW satisfies ε-DP through the Laplace mechanism using the normalized-IPW sensitivity bound.
- α = 0.2 or α = 0.3 achieves low SeqIPW error across the tested settings.
D.2 DPBlocking
DPBlocking parameters were selected through a restricted grid search and achieved low error across tested privacy levels and sample sizes. The appendix also reports statistical significance tests for NSW results.
- The parameter search was not exhaustive, because α3 was set to 1 −(α1 + α2) and the search space was reduced using SeqIPW observations.
- α1 = 0.1, α2 = 0.35, α3 = 0.55, and T = 1.5 achieved low error for all tested ε and n values.
- Increasing the number of bins generally accompanied increases in n and ε, but no clear winner for m emerged.
- For datasets larger than 10,000, an increase in error at ε = 0.25 and 1.0 motivated capping m based on ε.
E.1 Statistical Significance of Results
The NSW experiment’s absolute-error comparisons were statistically significant under two-sided Wilcoxon signed-rank tests at α = 0.01.
- All NSW experiment results were statistically significant at α = 0.01 under two-sided Wilcoxon signed-rank tests.
E.2 Comparison to LEBJ25
Compared with LEBJ25, the presented methods achieve lower error, while LEBJ25 shows no clear bias pattern and remains high-error at the tested sample sizes.
- LEBJ25 has significantly higher error than the other tested methods, despite AIPW potentially achieving lower bias and error non-privately.Its higher sensitivity, including sensitivity that relies on K, increases error under privacy.
- At n ≤40,000, LEBJ25 still has high error, suggesting that setting K = √n alone would not help for low n.
- The experiments found no clear pattern for LEBJ25's bias across 500 trials.The authors suggest this may reflect the greater noise added by the method.
E.3 Gaussian v. Laplace
The experiments compare Gaussian and Laplace noise and examine the presented algorithms across privacy levels and against prior methods. Laplace noise consistently produces lower observed error, especially when n and ε are small.
- For every tested combination of n and ε, Laplace noise achieves lower error than Gaussian noise.The difference is especially apparent for lower values of n and ε.
- At sufficiently low n and ε, simply switching to Laplace noise outperforms Gaussian noise for SeqIPW.
- Figure 5 compares observed error and bias with LEBJ25 at ε = 1.0 for n ranging from 5,000 to 10,000.The restricted range reflects LEBJ25's requirement for large datasets.
- Figure 6 compares observed error for algorithms using Gaussian versus Laplace noise.
- Figure 7 reports observed error for the presented algorithms across additional ε values on log-scaled axes.
- Figure 8 compares the presented algorithms with two generalizations of prior algorithms under restrictive outcome spaces using log-scaled axes.
E.4 Other Values of ε
Across additional privacy levels and sample sizes, the presented algorithms generally achieve lower observed error than the compared methods. DPBlocking performs especially consistently, while SeqNIPW becomes competitive mainly for large datasets and high ε.
- Across ε = 0.05, 0.1, and 5.0, SeqIPW beats LGPM19, and usually also beats DPBlocking at ε = 0.05 and 0.1.At ε = 5.0, DPBlocking has error comparable to or lower than non-private IPW estimation.
- Across most tested settings, DPBlocking achieves lower observed error than the compared algorithms.SeqNIPW approaches DPBlocking's error around n > 10,000 and ε = 1.0, but performs poorly when both n and ε are low.
- For n ≤2500 and ε = 0.25, SeqNIPW has observed error of 5 or greater when the true value of τ is 2.