Source-linked AI summary
Fairness Without Demographics in Repeated Loss Minimization
Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, Percy Liang
TL;DR
The paper asks how to prevent average-loss training from producing and amplifying high minority-group risk when demographic identities are unavailable. It analyzes ERM’s failures and develops DRO, showing theoretical risk control and improved minority satisfaction in autocomplete, while noting important fairness and robustness limitations.
Problem
Average-loss minimization can create representation disparity and amplify it through retention-driven population changes, while group identities and proportions may be unavailable.
Method
The paper uses distributionally robust optimization to upweight high-loss examples and bound unknown groups’ risks without requiring their identities.
Results
DRO prevents disparity amplification where ERM fails and improves median user satisfaction from 3.7 to 4.0 and retention from 0.7 to 0.85 in autocomplete.
Takeaways & Limitations
DRO provides an upper bound on minority-group risk and offers a straightforward way to introduce distributional robustness into loss-minimization systems.
Takeaways & Limitations
The authors caution that observational DRO controls only limited aspects of fairness and is unsuitable for high-stakes applications such as loans, criminality, or admissions.
Abstract
from arXiv · showhide
Machine learning models (e.g., speech recognizers) are usually trained to minimize average loss, which results in representation disparity---minority groups (e.g., non-native speakers) contribute less to the training objective and thus tend to suffer higher loss. Worse, as model accuracy affects user retention, a minority group can shrink over time. In this paper, we first show that the status quo of empirical risk minimization (ERM) amplifies representation disparity over time, which can even make initially fair models unfair. To mitigate this, we develop an approach based on distributionally robust optimization (DRO), which minimizes the worst case risk over all distributions close to the empirical distribution. We prove that this approach controls the risk of the minority group at each time step, in the spirit of Rawlsian distributive justice, while remaining oblivious to the identity of the groups. We demonstrate that DRO prevents disparity amplification on examples where ERM fails, and show improvements in minority group user satisfaction in a real-world text autocomplete task.
1. Introduction
The paper frames high minority-group loss as representation disparity and shows how retention-driven feedback can amplify it over time. It proposes DRO as a group-oblivious Rawlsian alternative to ERM.
- Motivation: Average-loss optimization can produce high overall accuracy alongside systematically higher errors for minority accents, a phenomenon the paper calls representation disparity.The paper treats this disparity as unfairness and situates it across several machine-learning applications.
- Motivation: Higher minority-group error can discourage users from continuing, shrinking the group’s future data representation and potentially increasing later error rates.This feedback loop is termed disparity amplification and can arise in deployed systems retrained on user data.
- Problem and approach: The paper studies sequential loss minimization where users may quit after experiencing loss, while group identities and the number of groups are unavailable.The objective is to minimize the worst-case group risk across time despite missing demographic information.
- Problem and approach: ERM can fail to control worst-case group risk and can turn initially fair models unfair, whereas DRO bounds group risk using an appropriate divergence ball.The proposed method is computationally efficient and compatible with many stochastic-gradient-descent models.
- Results: DRO improves minority satisfaction and reduces disparity amplification in a Mechanical Turk autocomplete task.The paper presents this as an empirical demonstration alongside its theoretical guarantee.
- Framing: The approach is presented as an application of Rawlsian distributive justice, allocating predictive accuracy with attention to the worst-off group.The paper distinguishes this focus from fairness methods based on protected demographic labels and classification constraints.
2. Problem setup
The problem setup models data as a mixture of unknown latent groups whose proportions evolve with user retention. The goal is to control group-wise risk over time without observing group labels, motivating robust loss minimization.
- Representation disparity: Representation disparity is defined as low overall risk but high worst-group risk caused by a group with small population proportion.Observations arise from unknown latent group distributions, and neither group proportions nor distributions are assumed known.
- Disparity amplification: The sequential setting allows group proportions to change across rounds as a function of past model losses.This makes the training distribution endogenous to earlier model performance.
- Disparity amplification: User retention is modeled by a differentiable, strictly decreasing function of group risk, so higher risk corresponds to a lower retained-user fraction.New-user rates also contribute to the expected number of users and samples from each group.
- Optimization goal: The objective is to control every group’s risk at every time step, despite unknown group proportions, new-user rates, and retention behavior.Without group labels, the worst-case group risk cannot be directly measured or minimized.
- Optimization goal: If a group’s population proportion approaches zero, no algorithm can control its future risk.The setup therefore includes a fundamental feasibility boundary tied to group disappearance from the data.
- Optimization goal: The paper first analyzes how ERM can amplify initially low disparity, then proposes DRO to mitigate this effect experimentally.This establishes the problem setup as a dynamic fairness problem rather than static average-risk minimization.
3. Disparity amplification
Under repeated retraining, average-risk minimization can amplify small group imbalances: groups that shrink receive higher losses, causing further shrinkage. The resulting dynamics can make initially fair solutions unstable, whereas DRO is designed to control worst-case group risk over time.
- Disparity amplification: ERM can amplify disparity because reduced minority retention produces fewer samples and still higher minority loss after retraining.This feedback loop can occur even when population-risk minimization is used instead of empirical-risk minimization.
- 3.1. Motivating example: In the two-class example, random accuracy fluctuations reduce right-cluster samples, after which ERM increasingly favors the left cluster.After 500 rounds, nearly no samples remain from the right cluster, which then suffers high loss.
- 3.2. Conditions for disparity amplification: Finite-sample fluctuations or estimation noise can move the system away from a fair fixed point, and fairness also depends on the unknown retention function.The paper motivates more stable loss-minimization schemes because a fair fixed point alone does not ensure fair evolution.
- 3.2. Conditions for disparity amplification: The dynamics’ sensitivity to risk is governed by retention behavior and cross-group risk interactions encoded by a K × K gradient-Hessian matrix.The relevant condition compares model stability, represented through the Hessian and per-group gradients, with population sensitivity to risk.
- 3.2. Conditions for disparity amplification: A fair fixed point can be unstable when improving one group’s risk sufficiently worsens other groups’ risks, leading eventually to a possibly unfair fixed point.The stability analysis studies the population-size update map and characterizes stability through the Jacobian’s eigenvalues.
- 3.2. Conditions for disparity amplification: For mean estimation, perturbations can destabilize equal group representation, while median estimation is degenerate because its minimizer changes immediately when group proportions differ.For the mean example, the instability expression is approximately 3.4; for the median example, no retention function can induce stability.
4. Distributionally robust optimization (DRO)
DRO addresses unknown-group fairness by minimizing worst-case loss over a chi-square neighborhood around the data distribution, yielding bounds on group risks and improved stability over time. Its loss reweighting emphasizes high-loss examples, while experiments show tighter fairness estimates and reduced disparity amplification compared with ERM.
- Interpreting the dual: High-loss examples are upweighted in DRO, biasing the model toward fewer high-loss examples rather than toward a fixed parameter value.Examples below the optimal dual threshold η⋆ can be ignored, while larger losses are upweighted through a squared term.
- Bounding the risk over unknown groups: DRO’s group-risk bound is tighter when between-group loss variation exceeds within-group variation, especially for distinct, concentrated group loss supports.The bound can be loose when a small group has low expected loss but a high-loss tail.
- Bounding the risk over unknown groups: DRO controls worst-case group risk by optimizing over a chi-square divergence ball, without observing group memberships.With a lower bound αmin on the smallest group proportion, radius rmax := (1/αmin −1)^2 provides the relevant upper bound.
- Optimization: The chi-square ball is used because it is straightforward to optimize and empirically outperformed other f-divergence balls.The dual objective can be optimized with stochastic gradient descent for models such as deep neural networks.
- Stability of minority loss minimization: Repeated median dynamics make the fair estimate unstable under ERM but uniquely stable under DRO, which reduces disparity amplification across time.The paper’s guarantees control worst-case risk over time without requiring a fair fixed point, and experiments report significant reductions versus ERM.
5. Experiments
Experiments show that DRO stabilizes minority-group performance and improves minority retention and satisfaction, while ERM can amplify disparity over time.
- DRO controls worst-case risk over time and improves minority retention in both the motivating example and the autocomplete evaluation.
- 5.1. Simulated task: 0.8 accuracy is maintained by DRO, whereas ERM becomes unstable and the minority group rapidly loses accuracy beyond 300 rounds.
- 5.1. Simulated task: 75% accuracy is maintained by DRO even when the minority fraction falls to 1%.
- 5.2. Autocomplete task: The autocomplete evaluation uses Mechanical Turk users, held-out AAE and SAE tweets, ten models, and 1,000 total feedback responses.
- 5.2. Autocomplete task: DRO increases minority satisfaction and retention, producing corresponding increases in inferred user count; error bars show bootstrap quartiles.
- 5.2. Autocomplete task: The higher minority satisfaction under DRO may reflect users valuing slang completion more than common-word completion, revealing a mismatch with training loss.
6. Discussion
The discussion frames DRO as minority-risk minimization with Rawlsian motivation, while recognizing philosophical, causal, applicability, and robustness limitations.
- DRO provides an upper bound on minority-group risk and performs well in practice, whereas ERM can amplify disparity and unfairness.
- The proposed algorithm is straightforward to implement and induces distributional robustness as an additional benefit.
- The Rawlsian framing inherits critiques concerning focus on an abstract worst-off group, extreme risk-aversion, and alternatives based on diminishing returns.
- Because fairness is fundamentally causal, observational DRO controls only limited aspects of fairness and is unsuitable for high-stakes applications such as loans, criminality, or admissions.
- Adversarial or high-noise settings can substantially loosen the DRO upper bound, leaving open whether unknown-group fairness and robustness can be achieved together.
- The results are reproducible using code released on the CodaLab platform.
A.1. Proof of Proposition 1
The proof analyzes stability of the population-update dynamics through the Jacobian and derives an eigenvalue condition at a fixed point.
- The proof studies a fixed point λ∗ = Φ(λ∗) with model parameter θ∗ minimizing population risk.
- The analysis uses the positive-definite Hessian HR(α∗) of expected risk and per-group parameter gradients ∇L.
- Stability holds when the absolute value of the maximum Jacobian eigenvalue ρmax is strictly below 1.
- The Jacobian calculation separates derivatives of risk with respect to θ, θ with respect to α∗, and α∗ with respect to λ∗.
- Applying the chain rule to these Jacobians yields the overall stability claim.
A.2. Proof of Corollary 2
The generalized proof extends the stability analysis to population-size updates defined by a differentiable function h of population size and risk.
- The generalized dynamics use h(λ, R) to define the evolution of expected user population size.
- For differentiable h in both arguments, the generalized analysis obtains an essentially identical stability result.
- The proof evaluates stability at a fixed point λ∗ = Φ(λ∗), with θ∗ minimizing population risk and HR(α∗) positive definite.
- Stability again follows from requiring the forward-map Jacobian eigenvalues to have absolute values strictly less than 1.
- The interpretation requires further assumptions on h, and the authors generally expect ∂h/∂R to scale with λ when risk affects users independently.
- The derivative ∂h/∂λ is upper-bounded by the maximum implied retention rate.
A.4. Proof of Proposition 2
Because P_k is a mixture component of P, it lies within the relevant divergence ball around P, yielding the stated upper bound.
- P_k is a mixture component of P, expressed as P = α_kP_k + · · ·.
- Since P_k belongs to B(P, r_k), the supremum over all Q in that ball provides the upper bound.
A.5. Proof of Proposition 4
The proof reduces Proposition 4 to establishing a sufficient condition.
- A sufficient condition for the proposition is identified as the next step of the proof.
B. Amazon Mecahnical Turk task description
The Mechanical Turk study evaluates autocomplete systems through tweet-retyping tasks followed by surveys measuring satisfaction, retention, and task or interface feedback.
- The experiment comprises 1000 HITs combining user replicates, five α values, two models, tweet sets, and AAE/SAE test sets.
- Users receive task instructions, retype 10 tweets with a software keyboard, and use dropdown next-word predictions from the autocomplete system.
- The survey separates feedback on autocomplete suggestions from feedback on the task interface.
- Survey questions include quality-control verification, retention, and satisfaction with the HIT.
- A unique code is requested at the end of the task to support completion verification.