Source-linked AI summary
The Social Cost of Strategic Classification
Smitha Milli, John Miller, Anca D. Dragan, Moritz Hardt
TL;DR
Strategic classification can improve institutional decision-making when individuals adapt to classifiers, but it may impose costs on those being classified. The paper defines social burden and proves a trade-off between institutional utility and individual impact, including disproportionate effects on disadvantaged groups.
Problem
Existing strategy-robust classification work emphasizes institutional utility while giving limited attention to the costs imposed on classified individuals and subpopulations.
Method
The paper introduces social burden as the expected cost positive individuals incur to receive positive classification and analyzes it under outcome-monotonic cost functions.
Results
Any increase in institutional accuracy necessarily increases social burden, which can disproportionately affect disadvantaged subpopulations and widen their burden gap.
Takeaways & Limitations
Strategy-robustness decisions should weigh institutional accuracy against social welfare, individual impact, and fairness.
Takeaways & Limitations
The analysis restricts costs to outcome-monotonic functions, and under alternative social utility measures the social gap need not increase with the institution’s threshold.
Abstract
from arXiv · showhide
Consequential decision-making typically incentivizes individuals to behave strategically, tailoring their behavior to the specifics of the decision rule. A long line of work has therefore sought to counteract strategic behavior by designing more conservative decision boundaries in an effort to increase robustness to the effects of strategic covariate shift. We show that these efforts benefit the institutional decision maker at the expense of the individuals being classified. Introducing a notion of social burden, we prove that any increase in institutional utility necessarily leads to a corresponding increase in social burden. Moreover, we show that the negative externalities of strategic classification can disproportionately harm disadvantaged groups in the population. Our results highlight that strategy-robustness must be weighed against considerations of social welfare and fairness.
1 INTRODUCTION
Strategic classification aims to protect institutional decision-making from manipulation, but more robust decision rules can increase the burden on individuals. The paper formalizes this trade-off and examines its implications for disadvantaged groups.
- Strategic behavior can reduce a classifier’s original predictive power as individuals adapt to its decision rule.This concern is associated with Campbell’s Law or Goodhart’s Law.
- More conservative decision boundaries can improve institutional utility while raising the success threshold for honest individuals.
- The social burden measures the expected cost that a positive individual incurs to be classified correctly.
- For a broad class of cost functions, every increase in institutional accuracy entails an increase in social burden.The Stackelberg equilibrium attains maximal institutional accuracy and maximal social burden.
- Social burden can fall disproportionately on disadvantaged subpopulations, and greater institutional accuracy can widen the burden gap.The paper illustrates these concerns with a FICO-data case study.
- Strategy-robustness should therefore be considered alongside fairness and individual impact rather than institutional utility alone.
2 MODEL
The model treats strategic classification as a binary decision problem in which individuals may modify their features at a cost to improve classification outcomes. It defines institutional utility and social burden under outcome-monotonic costs.
- Individuals choose feature modifications that maximize classification benefits minus modification costs in response to the institution’s classifier.The institution models these choices through individual best responses.
- Institutional utility measures classification accuracy after individuals respond strategically.The paper also calls this quantity strategic utility.
- Individual burden is the minimum cost required for an individual to receive a positive classification.
- Social burden is the expected individual burden among positive individuals.
- Outcome-monotonic costs require that improving one’s outcome becomes progressively more costly, while worsening one’s outcome costs nothing.These assumptions capture settings such as repaying a loan being harder than going bankrupt.
- The model assumes every individual has positive outcome likelihood, ℓ(x) > 0, and permits costs to be represented over outcome likelihoods.
3 INSTITUTIONAL UTILITY VERSUS SOCIAL BURDEN
The paper characterizes a trade-off between institutional utility and social burden in strategic classification. Raising the acceptance threshold can improve institutional utility, but increases the costs imposed on individuals, with intermediate thresholds offering context-dependent compromises.
- Any classifier that improves institutional utility beyond the non-strategic optimum causes a corresponding increase in social burden.
- The Pareto-optimal trade-off set is an interval I, with non-strategic classification and Stackelberg classification at opposite extremes.The Stackelberg equilibrium provides maximal strategic utility and maximal social burden, while intermediate thresholds permit context-specific balancing.
- Under outcome-monotonic costs, strategic classifiers can be represented by outcome threshold classifiers accepting individuals with ℓ(x) ≥ τ.Equivalent threshold classifiers preserve both institutional utility and social burden.
- Institutional utility is quasiconcave in τ, while social burden is monotonically non-decreasing in τ.The utility maximum occurs at a threshold τ* at least as large as the non-strategic optimum τ0 = 0.5.
- The appropriate threshold depends on balancing institutional objectives, robustness, and broader social interests.The paper identifies gradual movement from τ0 toward τ* as one possible response when classifier dynamics are difficult to model.
- Nash equilibria give institutions latitude to choose thresholds in [τN, τ*], rather than requiring the Stackelberg threshold τ*.This interval can provide more favorable social-burden outcomes while retaining equilibrium reasoning.
4 FAIRNESS TO SUBPOPULATIONS
Strategic classification can distribute social burden unevenly across subpopulations. The paper shows that disadvantaged groups may bear greater burden, and that improving institutional utility can widen this disparity under feature- or cost-based disadvantage.
- The social gap G(f) measures how much more costly acceptance is for group b than for group a.It is defined as G(f) = B+,b(f) − B+,a(f).
- Strategic classification can exacerbate social gaps that already arise under non-strategic classification.The paper analyzes disadvantage based on either lower outcome likelihoods or higher feature-adaptation costs.
- FICO case study: The FICO case study finds that the minority group, identified as Black individuals, is disadvantaged in features relative to the majority White group.
- Sources of disadvantage: The feature-based pattern can arise from group membership explaining features or from features predicting different group base rates.The paper gives grammatical errors and proxy variables such as zip code or name as examples.
- Feature disadvantage: When group b is disadvantaged in features and the likelihood condition holds, G(τ) is positive and monotonically increasing in τ.Feature disadvantage means positive individuals from group b have outcome-likelihood distributions shifted lower than group a's.
- Feature disadvantage: If institutional utility exceeds the non-strategic optimum, the social gap also increases under the stated feature-disadvantage conditions.This result assumes the relevant derivative of the likelihood cost is monotonically non-decreasing in τ.
- Cost disadvantage: When group b's adaptation costs are systematically higher, group b incurs greater social burden than group a.
- Cost disadvantage: Even with identical feature distributions among positive individuals, higher adaptation costs for group b make G(τ) non-negative and non-decreasing in τ.Thus, threshold increases can widen disparities through cost differences alone.
5 CASE STUDY: FICO CREDIT DATA
The FICO case study shows that strategic classification can impose greater burdens on black borrowers because their positive-score distribution is lower and, plausibly, their score-improvement costs are higher. Raising decision thresholds increases the social gap, especially when score changes are costly.
- Data and setup: 301,536 FICO scores from TransUnion TransRisk data are normalized to 0–100, with repayment defined as avoiding a 90-day default over 18–24 months.The two groups are white and black borrowers, and the analysis considers threshold-based lending decisions.
- Data and setup: Repayment probability P(Y = 1 | x) is monotonically increasing in the FICO score, so likelihood-based results apply to score thresholds.A threshold of τ = 58 is typically used for prime-rate loan eligibility.
- Different feature distributions: Credit-worthy black borrowers tend to have lower FICO scores than credit-worthy white borrowers, satisfying the disadvantaged-in-features condition.For every score x, F+,black(x) ≥ F+,white(x).
- Different feature distributions: For any score-changing cost α, the social gap increases with the threshold τ, and its growth rate becomes larger as α increases.The cost model is c(x,x′) = max(α(x′ − x), 0), where α is the cost of increasing the score by one point.
- Different cost functions: When groups face different score-changing costs, increasing the cost ratio κ rapidly enlarges the social cost gap and can make small threshold increases disproportionately burdensome.The setting assumes group B has cost coefficient β ≥ α and κ = β/α > 1.
6 RELATED WORK
The paper broadens strategic-classification research beyond institutional utility by analyzing burdens imposed on classified individuals and their unequal distribution across groups. It also offers a social-welfare rationale for preferring less conservative equilibria or decision rules.
- Strategic Classification: Unlike prior strategic-classification work focused primarily on institutional utility, this paper studies the trade-off between institutional utility and individual burden.Prior work seeks high-utility solutions, including Stackelberg-equilibrium algorithms.
- Strategic Classification: The Stackelberg equilibrium achieves maximal institutional utility but also causes high social burden, making it undesirable for some institutions.The paper gives several examples where this burden motivates alternative solutions.
- Strategic Classification: The paper provides a complementary reason to prefer Nash equilibria over Stackelberg solutions, beyond the concern that people may not respond optimally in practice.The strategic setting creates a utility–burden trade-off for broad classes of cost functions.
- Fairness: Strategic classification can create unequal impacts when groups differ in feature distributions or in the costs of adapting their features.This adds a fairness concern distinct from disparities arising solely from feature or label distributions.
- Fairness: As classifiers become more robust to strategic behavior, the gap between the costs incurred by disadvantaged and advantaged groups can increase.The paper characterizes this as a side effect that can exacerbate unfairness.
- Related impact measures: The paper differs from non-strategic impact analyses by measuring individual impact through the cost of strategic behavior induced by the classifier.Liu et al. study a dynamics model of how individuals are affected by received classifications.
7 DISCUSSION OF SOCIAL BURDEN
The discussion argues that social burden is a broadly applicable measure of individual impact and that the institutional-accuracy trade-off persists across related measures. However, some subgroup-gap monotonicity results depend on using social burden rather than social utility.
- Alternative measures: Social burden measures the expected cost positive individuals need to incur to be classified positively, while social utility measures expected individual utility.The paper prefers social burden because it requires fewer assumptions about how individuals behave.
- Alternative measures: Social burden applies regardless of the policies individuals actually follow, although the institutional analysis assumes individuals respond optimally.The authors regard optimal response as a strong assumption in practice.
- Alternative measures: The institutional utility–social burden trade-off also holds when social utility replaces social burden, with the direction expressed as utility being monotonically non-increasing.The paper states that Theorem 3.1 remains valid under this alternative measure.
- Limitations and scope: For subgroup analyses, alternative social-utility measures preserve a non-negative social gap but do not necessarily make that gap increase with the institution’s threshold.Thus, threshold monotonicity is not fully measure-agnostic.
- Limitations and scope: The analysis does not incorporate potential long-term harms to negative individuals who receive positive classifications such as loans.The authors state that the social benefits of reducing negative individuals’ costs are uncertain.
- Implications: Across complementary impact measures, the supported takeaway is that strategic classification requires considering institutional accuracy alongside individual impact.The discussion frames this as a choice relevant to strategy-robustness decisions.
A PROOF OF LEMMA 3.2
The proof establishes three properties of institutional Nash equilibrium strategies: the Stackelberg threshold is itself a Nash equilibrium, all Nash strategies lie within a bounded interval, and equilibrium strategies form an upper interval.
- All Nash equilibrium strategies lie in the interval [τ0, τ*].Thresholds below τ0 fail because the accepted population has likelihood below 0.5; thresholds above τ* also fail the Nash condition.
- Together, these properties imply that the institution’s equilibrium strategies are exactly [τN, τ*] for some τN ∈ [τ0, τ*].The lemma follows after establishing the three properties and their interval structure.
- A Nash equilibrium requires the institution’s threshold to remain a best response after individuals strategically respond to it.Given the strategic responses, acceptance must coincide with strategic outcome likelihood at least 0.5.
- The Stackelberg threshold τ* is a Nash equilibrium strategy.The proof verifies the institution’s best-response condition by considering individuals above, below, and at the threshold.
- If τN is a Nash equilibrium strategy, every threshold τ ∈ [τN, τ*] is also a Nash equilibrium strategy.The proof checks the three possible cases for individuals’ responses under τ and τN.