PaperlayerView exact revision on arXiv

Source-linked AI summary

The Disparate Effects of Strategic Manipulation

Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan

arXiv:1808.08646v4cs.LGcs.GTstat.ML

TL;DR

Algorithmic classifiers can induce strategic responses, but standard models often assume agents have equal ability to manipulate their features. This paper models groups with unequal manipulation costs in a Stackelberg game and finds that equilibrium errors can reinforce inequality, while subsidies can sometimes benefit only the learner and harm both groups.

  • Problem

    Existing strategic-classification models commonly treat agents as equally able to manipulate, while consequential algorithmic decisions operate amid unequal social resources and opportunities.

  • Method

    The paper adapts strategic-classification models into an agent-centric Stackelberg framework with group-specific features, labels, manipulation costs, and candidate welfare outcomes.

  • Results

    Unequal manipulation costs produce equilibrium classifiers that exclude some qualified disadvantaged candidates and admit some advantaged candidates, while subsidies can make both groups worse-off in some cases.

  • Takeaways & Limitations

    Evaluating algorithmic quality without accounting for unequal adaptive capacity can have adverse social consequences and exacerbate existing inequalities.

  • Takeaways & Limitations

    The model does not capture the irreducible complexity of social stratification and restricts the learner from adapting classifications to group identities.

Abstract

from arXiv · show

When consequential decisions are informed by algorithmic input, individuals may feel compelled to alter their behavior in order to gain a system's approval. Models of agent responsiveness, termed "strategic manipulation," analyze the interaction between a learner and agents in a world where all agents are equally able to manipulate their features in an attempt to "trick" a published classifier. In cases of real world classification, however, an agent's ability to adapt to an algorithm is not simply a function of her personal interest in receiving a positive classification, but is bound up in a complex web of social factors that affect her ability to pursue certain action responses. In this paper, we adapt models of strategic manipulation to capture dynamics that may arise in a setting of social inequality wherein candidate groups face different costs to manipulation. We find that whenever one group's costs are higher than the other's, the learner's equilibrium strategy exhibits an inequality-reinforcing phenomenon wherein the learner erroneously admits some members of the advantaged group, while erroneously excluding some members of the disadvantaged group. We also consider the effects of interventions in which a learner subsidizes members of the disadvantaged group, lowering their costs in order to improve her own classification performance. Here we encounter a paradoxical result: there exist cases in which providing a subsidy improves only the learner's utility while actually making both candidate groups worse-off--even the group receiving the subsidy. Our results reveal the potentially adverse social ramifications of deploying tools that attempt to evaluate an individual's "quality" when agents' capacities to adaptively respond differ.

1 Introduction

Algorithmic classifiers can reproduce social inequality not only through their features, but also by inducing unequal opportunities to strategically respond. This paper models those disparities and shows that heterogeneous manipulation costs can produce inequality-reinforcing errors and paradoxical subsidy effects.

  • Motivation: Algorithmic decision systems influence access to resources and opportunities while often reproducing existing social inequalities.The paper frames this concern around prediction-based models used in consequential decisions.
  • Motivation: Classifiers can actively shape behavior when individuals alter their features to increase the chance of approval.This reactivity turns static data into strategic responses to the algorithm’s preferences.
  • Related work: Strategic manipulation is the machine-learning term for agents’ reactive efforts to change features in response to a classifier.Earlier models commonly treat these interactions as antagonistic and assume agents face equal manipulation costs.
  • Approach: The paper models a Stackelberg game in which the learner publishes a classifier before candidates best-respond, while groups may differ in features, true labels, and manipulation costs.This extends prior strategic-classification models toward candidate welfare and social consequences.
  • Results: Whenever groups face unequal manipulation costs, equilibrium classification reinforces inequality by excluding some qualified disadvantaged candidates and admitting some advantaged candidates through cheaper manipulation.The result holds even when the learner knows the groups’ costs.
  • Results: Subsidies for disadvantaged candidates can improve the learner’s performance and reduce inequality-reinforcing errors.The intervention lowers the burden of manipulation for the disadvantaged group.
  • Results: In some cases, subsidies benefit only the learner while both candidate groups become worse-off, including the subsidized group.The analysis further identifies cases where all parties would prefer manipulation to be impossible.
  • Discussion: The paper connects its analysis to prior strategic-classification, signaling-theory, and social-impact work while emphasizing candidate welfare beyond learner utility.Its broader aim is to examine adverse consequences of quantification in unequal social settings.

2 Model Formalization

The model represents strategic classification as a Stackelberg game with group-specific features, labels, and manipulation costs. It restricts candidates to upward feature changes and studies how a non-group-adaptive learner evaluates welfare and classification errors under unequal opportunities to manipulate.

  • 2 Model Formalization: The Strategic Classification Game has a learner publish a binary classifier before candidates best-respond by changing their feature vectors.Candidates begin with innate features and manipulate inputs to seek positive classifications.
  • 2 Model Formalization: Candidates may move only from x to y ≥ x, reflecting the assumption that higher feature values signal higher quality to the learner.The comparison is component-wise, and manipulation costs increase as feature vectors move farther apart.
  • Remark on unequal group costs: Group B’s manipulation is always at least as costly as group A’s for the same feature change, modeling systematically unequal access to resources and opportunity.The cost condition captures more than monetary expense and can reflect effort, valuations, or other factors affecting manipulation.
  • 2 Model Formalization: The model includes groups A and B with distinct distributions over unmanipulated features and potentially different true labeling functions.Group populations are represented by distributions D_A and D_B, with true classifiers h_A and h_B.
  • 2 Model Formalization: A group-m candidate moving from x to y pays the cost difference c_m(y) − c_m(x), while the learner incurs penalties for false-positive and false-negative errors.The learner seeks correct labels relative to original features, whereas candidates trade positive-classification value against manipulation cost.
  • Remark on unequal group costs: Candidates manipulate only when the change flips classification from 0 to 1 and costs less than the normalized value of a positive classification.Setting positive-classification utility to 1 is treated as a scaling choice without loss of generality.
  • 2 Model Formalization: The learner must publish one classifier that does not adapt policies to candidates’ group identities.This restriction reflects settings where group membership, especially protected-class status, cannot be used to issue distinct policies.
  • 2 Model Formalization: With heterogeneous costs, prior near-optimal-error guarantees do not carry over, and unequal costs create unavoidable uncertainty when the classifier cannot distinguish groups.The paper also argues that learner-centered performance analysis gives only a partial view of total welfare.

3 Equilibrium Analysis

With unequal manipulation costs, the learner’s undominated and equilibrium classifiers systematically trade off false positives for the lower-cost advantaged group against false negatives for the higher-cost disadvantaged group. This inequality-reinforcing result holds in one-dimensional threshold models and extends to multidimensional linear-cost settings.

  • Model and one-dimensional setup: Candidates in group B face greater manipulation costs than candidates in group A, with higher feature values treated as higher quality and manipulation restricted to y ≥ x.The model studies non-negative monotone costs and two groups with potentially different feature distributions, labels, and manipulation costs.
  • One-dimensional equilibrium: The learner’s undominated threshold strategies lie in the interval [σB, σA], where σA and σB are the thresholds that perfectly classify groups A and B separately.Thresholds below σB or above σA are dominated for any false-positive and false-negative penalties.
  • One-dimensional equilibrium: Every threshold σ ∈ [σB, σA] produces false positives on group A and false negatives on group B, so errors systematically favor the lower-cost group.The learner’s cost contains only these two error types; no false negatives occur for group A or false positives for group B in this interval.
  • One-dimensional equilibrium: When costs are strictly concave, the equilibrium threshold is σB; when strictly convex, it is σA; with affine costs, all thresholds in [σB, σA] are equivalent.The equilibrium location depends on the shape of the cost functions even though the inequality-reinforcing error direction remains unchanged.
  • General d-dimensional feature vectors: In d dimensions with linear costs, perfect classifiers exist for each group, but every undominated classifier still has no false negatives for group A and no false positives for group B.The multidimensional result generalizes the one-dimensional asymmetry: potentially optimal classifiers trade off undue optimism toward A against undue pessimism toward B.
  • General d-dimensional feature vectors: Thus, whenever a perfect classifier for the entire population does not exist, equilibrium classification reinforces existing inequality by admitting some unqualified lower-cost candidates and excluding some qualified higher-cost candidates.The result follows from asymmetric manipulation costs rather than from the learner’s ignorance of those costs.

4 Learner Subsidy Strategies

The paper formalizes monetary subsidies that reduce disadvantaged candidates’ manipulation costs and examines their effects on learner performance and group welfare. Although subsidies can improve classification, they can also induce stricter classifiers that leave both groups worse off.

  • 4.2 Group Welfare Under Subsidy Plans: Subsidies can improve the learner’s classification performance and reduce inequality-reinforcing errors, but they can also make both groups worse off.The paper identifies cases in which the intervention lowers candidate welfare without improving any candidate’s outcome.
  • 4.1 Subsidy Formalization: The learner can subsidize group B by reducing each candidate’s manipulation cost proportionally or by absorbing a flat cost amount.Under proportional subsidies, group B candidates pay a β fraction of their original cost; flat subsidies absorb up to α.
  • 4.1 Subsidy Formalization: Under proportional subsidies, the learner jointly chooses a subsidy parameter β and classifier f to minimize misclassification and subsidy costs.The learner’s objective weights false positives, false negatives, and monetary subsidy costs through λ.
  • 4.2 Group Welfare Under Subsidy Plans: In one example, the no-subsidy threshold σ∗=σB≈0.398 perfectly classifies group B but permits false positives for group A with x∈[0.272,0.4).This establishes the comparison baseline for the subsidy equilibrium.
  • 4.2 Group Welfare Under Subsidy Plans: With subsidies, the stricter classifier correctly classifies group A but creates false negatives for group B with x∈[0.3,0.348).The higher threshold removes some prior false-positive benefits for group A and imposes greater manipulation costs on group B than the subsidy offsets.
  • 4.2 Group Welfare Under Subsidy Plans: Subsidies can make both groups worse off even when false negatives receive twice the penalty of false positives, and non-manipulation can be preferred by all parties.The opposing effects are greater access to manipulation for group B and a potentially stricter learner classifier.

5 Discussion

The discussion situates strategic manipulation within broader social inequality and emphasizes that classification systems can reward apparent merit while deepening exclusion. It also cautions that subsidies may backfire, although consistent classification can avoid the paradox in some cases.

  • 5 Discussion: The model does not capture the full complexity of overlapping social stratification, but it shows how classification can exacerbate existing inequalities.Systems may grant rewards to those who appear meritorious under a chosen standard while justifying exclusions of those who do not meet it.
  • 5 Discussion: Subsidizing disadvantaged candidates can encourage a learner to raise the classification standard, further excluding the group the intervention was intended to help.The discussion states that this unintended consequence does not always arise.
  • 5 Discussion: Some signaling and strategic-classification models treat manipulation as desirable when it improves candidate quality, but differential access to manipulation remains a social concern.The paper notes that quality-improving manipulation may create additional problems for machine-learning systems through feedback effects.
  • 5 Discussion: The paper uses a theoretical learning perspective to investigate adverse effects of algorithmic quantification in socially unequal environments.It calls for perspectives from other disciplines to inform machine-learning research on these domain-specific concerns.

A.1.1 Proof of Proposition 1

The proof characterizes the learner’s undominated one-dimensional threshold strategies when groups differ in costs and true-label thresholds. It shows that thresholds outside [σB,σA] are dominated, leaving an interval that trades off group-specific errors.

  • A.1.1 Proof of Proposition 1: A learner facing only group A can perfectly classify it at threshold σA, while a learner facing only group B can do so at threshold σB.These thresholds arise from the maximum feature values candidates are willing to reach through manipulation.
  • A.1.1 Proof of Proposition 1: Threshold σB dominates every σ<σB because lower thresholds create additional false positives without reducing false negatives.The argument uses monotonicity of both groups’ cost functions.
  • A.1.1 Proof of Proposition 1: Threshold σA dominates every σ>σA because higher thresholds create additional false negatives without reducing false positives.This holds for any error function with CFN>0.
  • A.1.1 Proof of Proposition 1: The interval [σB,σA] is undominated because thresholds within it trade off false negatives on group B against false positives on group A.The proof first establishes σB≤σA under the assumed ordering of true-label thresholds and cost conditions.

A.1.2 Proof of Proposition 2

The proof computes the learner’s cost for threshold strategies in the undominated interval by identifying which candidates cannot profitably manipulate to the threshold. These candidates generate false negatives in group B and false positives in group A.

  • A.1.2 Proof of Proposition 2: For σ∈(σB,σA], group B incurs false negatives because σB is the threshold that perfectly classifies group B.Candidates with unmanipulated features below the manipulation cutoff cannot reach σ at acceptable cost.
  • A.1.2 Proof of Proposition 2: Group B candidates with x∈[τB,ℓB(σ)) receive negative classifications despite hB(x)=1.This interval identifies the group B candidates contributing false-negative error cost.
  • A.1.2 Proof of Proposition 2: For σ∈[σB,σA), group A incurs false positives because σA is its optimal threshold.Candidates with features below τA can nevertheless be positively classified after the threshold and manipulation response are accounted for.
  • A.1.2 Proof of Proposition 2: Combining the group-specific error calculations yields the total cost of any threshold classifier with σ∈[σB,σA].The result combines the false-negative and false-positive contributions derived for the two groups.

A.1.3 Proofs of Corollaries 1 and 2

The proof compares two strategies: one avoids errors for group B, while the other avoids errors for group A, with each bearing only its corresponding cost.

  • Strategies σB and σA respectively commit no errors on groups B and A, so each strategy bears only the cost associated with that group.

A.1.4 Proof of Proposition 3

Under uniform feature distributions and proportional or strictly concave group costs, the optimal threshold depends on how the groups’ loss difference changes with the threshold. The learner may select σB, σA, or any threshold in between.

  • Under uniform feature distributions, minimizing classification error reduces to choosing a threshold σ.
  • With proportional group costs cA(x) = qcB(x) for q ∈(0, 1), the analysis characterizes the threshold choice through the relative group costs.
  • When the loss difference ℓB(σ) − ℓA(σ) increases over [σB, σA], the optimal classifier threshold is σ∗= σB.
  • When the loss difference decreases over [σB, σA], the optimal classifier threshold is σ∗= σA.
  • When the loss difference is constant over [σB, σA], the learner is indifferent among all thresholds in that interval.

A.2.1 Proof of Lemma 1

The lemma characterizes optimal manipulations under linear decision boundaries and monotone costs: candidates use the learner-valued, low-cost components, while excessively costly positive classifications are rejected.

  • A candidate with unmanipulated feature x ∈[0, 1]d chooses a manipulated feature y ≥ x to respond to a linear classifier.
  • If the original feature is already positively classified, the candidate’s best response is y = x because further manipulation only adds cost.
  • When achieving positive classification requires cost greater than 1, the candidate does not manipulate because the resulting utility is negative.
  • Optimal manipulations increase only components in K, the components with the highest learner value relative to manipulation cost.
  • Any manipulation reaching the decision boundary with lower cost than an alternative positive-classification manipulation yields higher candidate utility and rules out the alternative as optimal.

A.2.2 Proof of Theorem 1

The proof establishes perfect single-group classifiers and shows that, with heterogeneous manipulation costs, undominated classifiers make only inequality-reinforcing errors. It then develops subsidy comparisons showing that subsidies can improve the learner while worsening candidate welfare.

  • Perfect classifiers: A learner with the true decision boundary can construct a perfect classifier for candidates from either single group.The construction commits neither false positives nor false negatives for the targeted group.
  • Inequality-reinforcing errors: All undominated classifiers avoid false negatives on group A and false positives on group B when candidates best respond.Thus their remaining errors are false positives on group A and false negatives on group B.
  • Inequality-reinforcing errors: Perfect classifiers for one group can therefore commit only false negative errors on group B or only false positive errors on group A.The classifiers f A 1 and f B 1 are examples, but they are not unique.
  • Subsidy comparisons: Without subsidies, one equilibrium threshold perfectly classifies group B but permits false positives on group A with features x ∈[0.217, 0.4).The cited equilibrium uses σB = 0.55.
  • Subsidy comparisons: With a proportional subsidy, the equilibrium still perfectly classifies group B while reducing, but not eliminating, false positives on group A.The equilibrium uses σ∗B ≈0.552 and β∗= 0.994, with false positives remaining for x ∈ [0.219, 0.4).
  • Subsidy comparisons: There exist subsidy settings in which both groups have lower welfare, even though the learner’s outcome improves relative to manipulation regimes.In one case, group B candidates receive the same classifications while manipulation to reach the higher threshold costs more; group A candidates can also lose false-positive benefits.
Loading 1808.08646v4…