Source-linked AI summary
Fair Decision Making using Privacy-Protected Data
Satya Kuppam, Ryan Mckenna, David Pujol, Michael Hay, Ashwin Machanavajjhala, Gerome Miklau
TL;DR
Sensitive statistics increasingly inform consequential allocation decisions, yet prior accuracy-focused analysis leaves open whether privacy-induced errors affect populations equally. This paper studies differential privacy in three allocation settings and finds that noise can create disparities, while sometimes improving ex-ante equity in apportionment. It concludes that outcome fairness should be evaluated alongside aggregate error.
Problem
Prior work has focused largely on aggregate accuracy, leaving limited evidence about whether privacy-induced decision errors affect populations equally, a question with consequences for welfare and representation.
Source: §1 · p. 1Method
The paper empirically studies Laplace and DAWA privacy mechanisms in simulated funds allocation, voting-benefit assignment, and parliamentary apportionment.
Source: §1 · p. 2, §2.2 · p. 3Results
Across the studied allocation settings, privacy noise can cause disparities, especially at smaller privacy-loss budgets, although some apportionment settings are more equitable ex ante than deterministic apportionment.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
Designers should assess fairness of outcomes alongside conventional aggregate-error metrics, since disparities can reflect algorithm features, decision thresholds, or population size.
Source: §7 · p. 10The proposed repair relies on a known distribution for the added noise, which holds for Laplace and some mechanisms but not DAWA.
Source: §4.3 · p. 6
Abstract
from arXiv · showhide
Data collected about individuals is regularly used to make decisions that impact those same individuals. We consider settings where sensitive personal data is used to decide who will receive resources or benefits. While it is well known that there is a tradeoff between protecting privacy and the accuracy of decisions, we initiate a first-of-its-kind study into the impact of formally private mechanisms (based on differential privacy) on fair and equitable decision-making. We empirically investigate novel tradeoffs on two real-world decisions made using U.S. Census data (allocation of federal funds and assignment of voting rights benefits) as well as a classic apportionment problem. Our results show that if decisions are made using an $\epsilon$-differentially private version of the data, under strict privacy constraints (smaller $\epsilon$), the noise added to achieve privacy may disproportionately impact some groups over others. We propose novel measures of fairness in the context of randomized differentially private algorithms and identify a range of causes of outcome disparities.
1 INTRODUCTION
The paper asks whether privacy-induced error affects populations equally when sensitive statistics guide consequential decisions. It studies this question across three allocation settings and examines both disparities and possible remedies.
1 INTRODUCTION
- Across three simulated decisions, privacy noise can produce substantial disparities, especially under strict privacy or for small populations.The paper also reports that impacts may be negligible when privacy noise is modest relative to the underlying statistics.Source: §1 · p. 2
- At ϵ = 10−3, some districts receive over 500× their proportional share of federal funds while others receive less than half; at ϵ = 10, the disparity is much smaller.The disparity remains observable under the weaker privacy setting.Source: §1 · p. 2
- For minority-language voting benefits, privacy noise can create disparities in correct-identification rates, particularly under stricter privacy settings.The benefits concern jurisdictions meeting conditions based on minority-language speakers and illiteracy rates.Source: §1 · pp. 1–2
- For Indian-state apportionment, some ϵ settings produce more equitable ex-ante seat allocations from noisy data than standard deterministic apportionment.Seats are allocated to states in proportion to population.Source: §1 · pp. 1–2
- The paper proposes remedies for inequities in funds and voting-benefit allocation that do not require changing the private release mechanism.It frames the concern around differential privacy’s growing use and the Census Bureau’s planned methods for protecting census data.Source: §1 · p. 2
- Disparities may depend on privacy-algorithm features, decision structure, or input data, so aggregate error alone may miss unequal group effects.The study uses public data and simulations; it does not measure impacts of agency practices currently in use.Source: §1 · p. 2
2 PROBLEM SETTING
The paper models assignments from population statistics, then studies how privatizing those statistics makes outcomes randomized. Its fairness assessment focuses on changes from the ground-truth assignment and on equal treatment among populations with equal entitlements.
2.1 Assignment Problems
- An assignment method maps each population to a resource or benefit, typically using statistics such as population totals to determine its ground-truth outcome.The paper formalizes populations, outcomes, queries, and assignment rules; its three example rules are specified in Tables 1–3.Source: §2.1 · p. 2
2.2 Differential Privacy
- Differential privacy bounds output-probability changes when one individual’s record changes; smaller ϵ provides greater privacy but adds more noise.The paper studies the Laplace mechanism and DAWA: Laplace estimates are unbiased with input-independent error, while DAWA adapts complex noise to the input and may introduce statistical bias.Source: §2.2 · pp. 2–3
2.3 Assignment Using Private Inputs
- The pipeline privatizes query answers first, then applies the assignment method to those noisy statistics, yielding randomized outcomes that retain the privacy guarantee.The paper distinguishes expected error in statistics from expected error in outcomes.Source: §2.3 · p. 3
- The initial study follows agencies’ data-publishing practice by applying standard assignment methods to privatized statistics, while also testing remedies that adjust assignments for noise.Changing the privacy mechanism itself is identified as future work.Source: §2.3 · p. 3
2.4 Methodology
- Fairness is assessed as change from the legally specified ground-truth assignment, seeking equal treatment in expectation or with high probability for populations with equal entitlements.The study does not question the declared entitlement or ground-truth allocation itself.Source: §2.4 · p. 3
3 RELATED WORK
The related work spans differential privacy, social choice, fair allocation, machine-learning fairness, and statistical-agency practice. The paper distinguishes its focus on privacy-induced changes to fixed assignment rules from these adjacent lines of work.
Fairness and privacy
- Prior privacy-and-fairness work includes warnings about discrimination, a fair-classification framework, questions about privacy-fairness and accuracy-fairness, and disparate impacts on learned models.The paper describes these settings as distinct from its focus on allocation decisions.Source: §3 · p. 3
Economics and Social Choice Theory
Fair allocation
- Funds allocation is proportional fair division when rights are known, but noisy information about rights creates a variant the authors say lacks a direct fairness definition.Xue studies a related setting with uncertain rights and discounts allocations accordingly.Source: §3 · pp. 3–4
Apportionment
- For apportionment, the authors identify a lack of existing approaches for noisy inputs and cite Grimmett as a related randomized method with a fairness criterion.Apportionment divides indivisible seats among states according to population-based rights.Source: §3 · p. 4
Fairness in Machine Learning
- Machine-learning fairness work primarily studies biased scores or classifications, whereas this paper examines noise-induced outcome changes under a fixed, legally mandated decision rule.Its voting-benefit metric relates to error-rate balance, but the paper also considers arbitrary pairs of jurisdictions.Source: §3 · p. 4
Statistical Agency Practices
- Agency products have long tracked sampling uncertainty, but historical margins of error did not account for privacy-protection transformations; existing products may also have income-correlated errors.The Census balances privacy obligations with accurate decision support and uses special variance-reduction methods for some benefits.Source: §3 · p. 4
4 PROBLEM 1: MINORITY LANGUAGE VOTING RIGHTS
Section 203 identifies jurisdictions that must provide minority-language election materials using population and illiteracy criteria. The section finds that privacy noise can produce unequal classification accuracy across jurisdictions, while a repair method can trade disparity against misclassification costs.
4.1 Problem Definition
- Coverage requires sufficient language-minority population counts and an illiteracy rate above the national average; classification accuracy is measured separately for covered and not-covered jurisdictions.Population thresholds can be met by either a percentage or absolute-count route.Source: §4.1 · p. 4
4.2 Empirical Findings
- For the Hispanic minority language group, the study used county or minor-civil-division data for 5,180 jurisdictions, 175 covered, and ran 1,000 trials of each algorithm for each ε value.The algorithms were D-Laplace and DAWA; D-Laplace decomposed queries of sensitivity 3 into parallel sensitivity-1 queries.Source: §4.2 · p. 5
- For D-Laplace, classification success tracks distance from the threshold: Maricopa is correct 100% of the time at ϵ = .1, versus 63% for Knox.The paper attributes the difference to threshold distance under equivalent-magnitude unbiased noise; threshold distance also correlates with population size.Source: §4.2 · p. 5
- D-Laplace also has substantial errors for uncovered jurisdictions: correct-negative rates range from 54% to 100% at ϵ = .1, with false positives creating administrative and financial burdens.Such errors can require minority-language materials in jurisdictions that do not qualify.Source: §4.2 · p. 5
- At ϵ = .1, 61% of jurisdictions under DAWA versus 74% under D-Laplace exceed a 95% correct-classification rate, and DAWA exacerbates disparities.DAWA's underlying-statistic error is approximately equivalent at ϵ = .1 and 30% lower at ϵ = .01, yet its classification disparities are greater.Source: §4.2 · p. 5, Figure 1 · p. 5
4.3 Mitigating unfairness
- A posterior-probability threshold can reduce disparity by prioritizing correct positive classification, but raises the cost of false positives.At ϵ = 1, the minimum correct-classification rate can rise from 0.54% to .80% as expected false positives increase from 334 to 870; the approach could let policymakers weigh disenfranchisement risk against oversupply costs.Source: §4.3 · p. 6
- The repair method leaves privatized counts unchanged and does not require changing the publisher's release mechanism, but depends on knowing the noise distribution, which DAWA lacks.The paper's cost measure counts misclassified jurisdictions; a more nuanced evaluation could account for affected individuals and voting-age population.Source: §4.3 · p. 6
5 PROBLEM 2: TITLE I FUNDS ALLOCATION
This section studies Title I basic-grant allocation, which distributes funding among qualifying school districts using eligible-child counts weighted by state per-student expenditures. It assesses how privacy noise affects district-level fairness and tests a post-processing repair. [sic]
5.2 Empirical Findings
- 500×: at ϵ = 0.001, the smallest districts’ allocations increase 500× while the largest districts’ allocations decrease by 50%.At ϵ = 10, the smallest districts see a 1.01× increase; at ϵ = 0.1, they see a 10× increase.Source: Figure 2 · p. 7, §5.2 · p. 7
- At ϵ = 0.001, DAWA gives some districts 555× their true allocation and others one-tenth, with at least one inversion in 381 of 888 districts.Under Laplace, every district receives at least 0.48x of its true allocation.Source: Figure 2 · p. 7, §5.2 · p. 8
5.1 Problem Definition
- Fairness is assessed ex ante by comparing noisy and true allocations across districts, using multiplicative error and dollars misallocated per million.Ex ante measures capture disparities due to the randomized allocation mechanism.Source: §5.1 · p. 7
5.2 Empirical Findings
- The experiment uses free-lunch eligibility counts as a proxy for Title I-eligible students, with per-student expenditures treated as public.Data covers 15,650 of 18,609 school districts; Laplace and DAWA estimate eligible counts.Source: §5.2 · p. 7
- Rounding negative Laplace-noisy counts to zero creates upward bias, which increases total counts and lowers larger districts’ allocation weights.At ϵ = 0.001, both Laplace and DAWA allocate roughly 0.001 in expectation to districts whose true allocation is below 0.001.Source: §5.2 · pp. 7–8
- Florida shows milder effects than Michigan: at ϵ = 0.001, larger districts are taxed less, consistent with fewer small counts and districts.At ϵ = 10, Florida shows almost no difference between true and noisy allocations.Source: Figure 3 · p. 8, §5.2 · p. 8
5.3 Mitigating unfairness
- The inflationary repair adds slack to each district’s count and subtracts slack from the total, satisfying the no-penalty allocation definition.The theorem states that the repair algorithm satisfies Definition 5.1.Source: §5.3 · pp. 8–9
- At ϵ = 0.1, repair requires 1.082× the original budget; at ϵ = 0.001, it requires 5.715×, and unequal inflation can still leave inequity.The approach is designed for Laplace noise; adapting it to DAWA’s data-dependent error may require additional privacy-budget consumption.Source: §5.3 · p. 9
6 PROBLEM 3: APPORTIONMENT OF LEGISLATIVE REPRESENTATIVES
The apportionment study tests whether Laplace noise changes how closely state seat allocations match population quotas. Its two fairness measures move differently: noise can improve expected quota deviation while worsening the disparity between the most- and least-favored states.
6 PROBLEM 3: APPORTIONMENT OF LEGISLATIVE REPRESENTATIVES
- The experiments consider noise sufficient to affect apportionment; comparable effects could arise with ε = .1 for a population of 10^3, such as a small school district.The paper relates this to ε = 10^-4 for a population of 10^6, such as a small state.Source: §6 · p. 9
6.1 Problem Definition
- The baseline rounds each state's population quota to the nearest integer, subject to a one-seat minimum, and is not guaranteed to allocate exactly 543 seats.The paper does not compare apportionment algorithms; it studies how Laplace noise affects representation.Source: §6.1 · p. 9
- Fairness is assessed through quota satisfaction, using both maximum pairwise multiplicative disparity and average expected absolute deviation from each state's quota.The first measure averages an ex-post disparity over randomized outcomes; the second aggregates each state's expected deviation.Source: §6.1 · p. 9
6.2 Empirical Findings
- The study uses Laplace-perturbed populations for 35 Indian states and union territories, and does not use DAWA because it does not outperform Laplace on this small set of counts.The experiments use 1971 population totals published in the Indian Parliament's 2006/07 budget.Source: §6 · p. 9, §6.2 · p. 10
- Between ε = 1.4×10^-6 and 3.8×10^-4, noise improves average expected deviation from quota by offsetting deviation caused by the integrality constraint.Figure 4(a) shows the measure across ε; the expected apportionment may not be attainable in any trial.Source: §6.2 · p. 10, Figure 4 · p. 10
- As ε decreases, the max-multiplicative disparity between the most- and least-favored states worsens, with some receiving substantially more than quota and others less.Figure 4(c) shows the measure as ε varies.Source: §6.2 · p. 10, Figure 4 · p. 10
7 CONCLUSION
The conclusion reports that more protective privacy budgets can bring disparities in allocation outcomes. Those disparities have multiple potential sources, and the authors call for fairness evaluation alongside aggregate error while noting that mitigation remains an open challenge.
7 CONCLUSION
- A privacy-loss budget must cover all public releases, including statistics supporting allocation decisions, so the budget available for any single problem may be small.The conclusion notes that high budgets in some deployments could diminish impacts, but the total budget covers multiple releases.Source: §7 · p. 10
- Disparities can arise particularly at smaller, more protective privacy-loss budgets, and their causes include privacy-algorithm bias, decision thresholds, and different treatment of small and large populations.Designers should evaluate fairness outcomes alongside conventional aggregate error metrics.Source: §7 · p. 10
- The authors proposed remedies for funds allocation and voting benefits, but say further investigation of mitigation technology is needed.Customizing mechanisms for each assignment problem could require agencies to design algorithms for thousands of problems.Source: §7 · p. 10
Laplace mechanism.
- Some algorithms, including DAWA, do not directly support releasing error bounds, and modified allocation procedures may conflict with governing regulations.The authors identify both issues as matters for future work.Source: Laplace mechanism · p. 10
8 APPENDIX
The appendix supplies background on the privacy algorithms used in the experiments. It describes Laplace noise and the D-Laplace query decomposition, alongside domain-specific processing choices.
8 APPENDIX
- The appendix provides additional algorithm background and a proof omitted from the body of the paper.The privacy algorithms were implemented using the open-source Ektelo framework.Source: §8 · p. 11, §8.1 · p. 11
8.1 Algorithm background
- The Laplace mechanism adds mean-zero Laplace noise calibrated to ε and query sensitivity; the paper uses a vector variant with automatically calculated sensitivity.Negative counts are set to zero in the Title I allocation and apportionment applications.Source: §8.1 · p. 11
- For voting-rights queries, D-Laplace decomposes the original sensitivity-3 queries into queries with sensitivity one, estimates those components, and derives estimates of the original queries.The decomposition enables parallel composition because adding or removing an individual changes only one component query answer.Source: §8.1 · p. 11
- In experiments, D-Laplace performed consistently better than a standard Laplace application; apportionment used a standard Laplace application.The appendix provides background and details to aid intuition and support reproducibility.Source: §8.1 · p. 11
DAWA.
DAWA groups and smooths statistics, which can reduce total error but introduces practical complexities, including data-dependent bias. The appendix also states that the Inflation repair algorithm is a no-penalty allocation for any δ > 0.
DAWA.
- DAWA groups similar statistics into contiguous intervals and smooths their estimates, which can significantly reduce total error in some settings.It uses the Laplace mechanism to measure each group; the approach was shown to outperform competing mechanisms on real datasets [25].Source: DAWA · pp. 11–12
- DAWA's lower mean-squared error comes with bias that is hard to predict in advance; grouping and smoothing can bias an outlier region's count downward.Expected error for a count depends on the data and is itself sensitive, so releasing it requires accounting for its privacy cost.Source: DAWA · p. 12
- DAWA was not used for apportionment because, for a small count set such as 35 Indian states, it does not outperform Laplace.The appendix also notes that geographic ordering affects DAWA output and describes the experiment's ordering choices.Source: DAWA · p. 12
8.2 Proofs
- For any δ > 0, the Inflation repair algorithm described in Section 5.3 is a no-penalty allocation.The appendix provides the proof of this theorem.Source: §8.2 · p. 12