PaperlayerView exact revision on arXiv

Source-linked AI summary

Algorithmic Monoculture and Social Welfare

Jon Kleinberg, Manish Raghavan

arXiv:2101.05853v2cs.GTcs.CYcs.LG

TL;DR

The paper examines whether shared reliance on a more accurate screening algorithm improves collective outcomes absent unexpected shocks. Using a model of firms choosing between private and common rankings, it shows that monoculture can produce lower average decision quality and socially worse equilibria. The analysis frames this as a broader risk of correlated evaluations that can let valuable options go unselected.

  • Problem

    It asks whether convergence on one more accurate screening algorithm necessarily improves average outcomes when multiple decision-makers use it without unexpected shocks.

  • Method

    The paper models firms choosing between independent private rankings and a shared algorithmic ranking, analyzed through probabilistic properties of noisy rankings.

  • Results

    Algorithmic monoculture can reduce overall decision quality: algorithm use can be strictly dominant for each firm while both firms would generate higher social welfare using human evaluators.

  • Takeaways & Limitations

    Monoculture can diminish social welfare by reducing heterogeneity in evaluations and allowing valuable options to slip through the cracks, even when individual decisions are more accurate.

  • Takeaways & Limitations

    The paper leaves open whether its main theorem extends to larger candidate sets under some noise models, because relevant definitions can fail for particular candidate distributions.

Abstract

from arXiv · show

As algorithms are increasingly applied to screen applicants for high-stakes decisions in employment, lending, and other domains, concerns have been raised about the effects of algorithmic monoculture, in which many decision-makers all rely on the same algorithm. This concern invokes analogies to agriculture, where a monocultural system runs the risk of severe harm from unexpected shocks. Here we show that the dangers of algorithmic monoculture run much deeper, in that monocultural convergence on a single algorithm by a group of decision-making agents, even when the algorithm is more accurate for any one agent in isolation, can reduce the overall quality of the decisions being made by the full collection of agents. Unexpected shocks are therefore not needed to expose the risks of monoculture; it can hurt accuracy even under "normal" operations, and even for algorithms that are more accurate when used by only a single decision-maker. Our results rely on minimal assumptions, and involve the development of a probabilistic framework for analyzing systems that use multiple noisy estimates of a set of alternatives.

1 Introduction

The paper asks whether shared use of a more accurate screening algorithm necessarily improves outcomes without unexpected shocks. It shows that algorithmic monoculture can instead reduce overall decision quality, creating a Braess’-paradox-like outcome for society.

  • Motivation and contribution: Algorithmic monoculture can worsen overall screening performance even without unexpected shocks.The result concerns average decision quality, not only correlated failures under adverse conditions.
  • Main result: The result resembles Braess’ paradox: introducing a more accurate algorithm can drive firms to an equilibrium worse for society than the pre-algorithm outcome.The comparison is between the equilibrium induced by shared algorithm use and the earlier equilibrium without the algorithm.
  • Implications: The harm can affect both particular applicants and the average quality of decisions, rather than one necessarily offsetting the other.The paper distinguishes individual exclusion from globally lower performance.
  • Analysis: The analysis develops probabilistic properties of noisy rankings, including cases where incremental ranking construction yields less error than constructing the ranking directly.This probabilistic framework supports the counterintuitive comparison between individual accuracy and collective outcomes.
  • Model: Each firm chooses between an independent private ranking and a common algorithmic ranking, then hires the highest-ranked available candidate in random order.The shared algorithm produces the same ranking for firms that adopt it, while private rankings remain independent.

2 Algorithmic hiring as a case study

The paper models algorithmic hiring as a choice between independent human rankings and a more accurate but shared algorithmic ranking. Under minimal conditions, firms may rationally converge on the algorithm even though independent rankings would yield higher individual and social welfare.

  • Hiring model: Firms rank candidates and hire the highest-ranked remaining candidate, choosing between private human evaluations and a common algorithmic ranking.Human rankings are independent across firms, whereas all firms using the algorithm receive the same ranking.
  • Ranking model: The model represents candidate values as an ordered vector and rankings as noisy permutations generated by randomized mechanisms.The accuracy parameter θ increases as noisy rankings converge toward the true candidate ordering.
  • Welfare objective: The analysis evaluates both each employer’s utility and social welfare, defined as the sum of utilities across hired candidates.Social welfare is highest when firms hire the highest-value available candidates.
  • Conditions: The main result holds under two conditions: preference for the first position and preference for weaker competition.These conditions concern the value of the top-ranked candidate and the expected quality of the remaining candidate after different rankings remove candidates.
  • Main theorem: For any human accuracy θH, some higher algorithm accuracy θA makes algorithm use a strictly dominant strategy for both firms, although both firms would generate higher social welfare with human evaluators.The result also implies that the algorithmic equilibrium can be worse for each individual firm, not only for society overall.

3 Instantiating with Ranking Models

The paper instantiates its monoculture framework with Random Utility Models and the Mallows Model, finding that shared rankings can reduce social welfare under broad conditions. The result is robust for Mallows rankings but depends on noise model, candidate count, and candidate distribution for RUMs.

  • The Mallows Model: Under the Mallows Model, a common algorithmic ranking can decrease social welfare for any candidate distribution D and human evaluator accuracy θH.For any human accuracy, some algorithm accuracy θA satisfies the conditions producing reduced welfare.
  • Random Utility Models: For n = 15, Laplacian noise can make UAH(θ, θ) − UAA(θ, θ) < 0, so Definition 2 is not met.Figure 2 uses candidate utilities drawn from a unit-variance uniform distribution and compares n = 3, n = 5, and n = 15.
  • Random Utility Models: For three candidates, Gaussian and Laplacian RUMs satisfy the theorem’s conditions for every candidate distribution D.These RUM families use noise with standard deviation 1/θ.
  • Random Utility Models: The RUM result does not generally extend beyond three candidates or all noise and candidate distributions.Some distributions violate the theorem’s conditions, and an open question remains for larger candidate sets under Gaussian noise.
  • Random Utility Models: The Plackett-Luce model never meets Definition 2, so monoculture has no effect and firms should use the best available ranking.The model is equivalent to the Gumbel RUM family.
  • The Mallows Model: The Mallows Model’s welfare reduction appears in the shaded AA region, where both firms use the algorithm although human evaluators would yield higher social welfare.Figure 3 characterizes equilibrium behavior in the (θH, θA) plane.

4 Models with Multiple Firms

With more than two firms, algorithmic monoculture can still produce Braess’ Paradox, while sequentially optimal strategies can vary non-monotonically across firms and accuracy parameters. Computational region maps reveal additional structure, including a binary-counter pattern that remains mathematically unproved.

  • Braess’ Paradox for k > 2 firms: Braess’ Paradox persists for k > 2: algorithmic evaluation can be dominant for every firm while human evaluation yields higher social welfare.For n = 4, k = 3, φA = 2, and φH = 1.75, computation gives equilibrium utility ≈.551 versus ≈.552 when all firms use humans.
  • Sequential decision-making: When φH ≥φA, all firms optimally choose H; when φA > φH, every optimal sequence begins with A.The first firm chooses the more accurate mechanism, and subsequent choices depend on the effects of earlier selections.
  • Sequential decision-making: For k = 5, all 16 binary strategy sequences beginning with A appear in some region of the (φH, φA)-plane.The strategy sequence records each firm’s optimal choice, conditional on the choices made by earlier firms.
  • Sequential decision-making: Along vertical lines of fixed φH, strategy regions appear in increasing binary-label order, but this binary-counter property is supported only computationally.The authors do not have a proof or know how generally the property holds, leaving its mathematical analysis open.

5 Conclusion

The paper argues that algorithmic monoculture can lower globally averaged decision quality even without shocks, while emphasizing that adverse outcomes are conditional rather than inevitable. It identifies quantitative, multi-firm, multi-algorithm, and cross-domain extensions as important directions.

  • Conclusion: Algorithmic monoculture can produce globally lower average-quality decisions even in the absence of unexpected shocks.This extends concerns beyond correlated shocks and harms to particular individuals.
  • Future work: Quantifying monoculture’s possible negative effects requires relating the ranking noise model to the numerical qualities of candidates.The paper leaves comprehensive bounds on how much equilibrium outcomes can worsen relative to socially optimal decisions for future work.
  • Future work: The analysis focuses on two firms and one shared algorithm, leaving generalizations to more firms and multiple algorithms unresolved.Multiple algorithms could create clusters balancing ranking accuracy against correlation in firms’ decisions.
  • Broader implications: The findings are presented in algorithmic hiring but may also apply to settings involving job candidates, potential hit songs, or budding entrepreneurs.The stated concern is that monoculture can allow valuable options to slip through the cracks.

A Random Utility Models satisfying Definition 1

The appendix verifies that a random utility model family satisfies differentiability, asymptotic optimality, and monotonicity under its stated assumptions. Increasing accuracy makes adjacent ordering errors arbitrarily unlikely, yielding a high-probability guarantee for the full ranking.

  • Verification of Definition 1: The proof establishes that the random utility family satisfies differentiability, asymptotic optimality, and monotonicity.Differentiability follows by expressing each permutation probability as an integral of differentiable functions over a fixed region.
  • Asymptotic optimality: For sufficiently large θ, the probability of incorrectly ordering an adjacent pair is at most δ.The argument considers two candidates whose value difference is positive and shows the error probability decreases with increasing accuracy.
  • Asymptotic optimality: The probability of outputting the correct ranking is at least 1 −(n −1)δ after applying a union bound over adjacent pairs.Because δ is arbitrary, the correctness probability can be made arbitrarily close to 1.
  • Monotonicity: Higher θ improves the probability that the best candidate is ranked first, including after other candidates are removed from consideration.The monotonicity argument uses the fact that removing elements leaves the remaining distribution within the relevant random utility family.

B.1 Violating Definition 2

The appendix constructs a counterexample in which using the same algorithmic ranking performs worse than using independent human rankings. A smooth approximation preserves the effect despite the initial construction violating the paper’s regularity definition.

  • Counterexample: The constructed example satisfies UAH < UAA, showing that shared algorithmic use can yield lower utility than the comparison strategy.The example chooses a specific noise distribution, accuracy parameter, and candidate distribution.
  • Counterexample: The initial noise distribution violates Definition 1 because it is neither differentiable nor supported on (−∞, ∞).The authors state that a smooth approximation formed from tightly concentrated Gaussians can achieve the same results.
  • Counterexample: For δ = .1, the example gives UAH(θ, θ) −UAA(θ, θ) ≈−0.00076.The difference is negative for sufficiently small δ under the selected candidate-value condition.

B.2 Violating Definition 3

A three-candidate random utility model shows that a better algorithm can make the algorithm-versus-human comparison reverse when agents choose sequentially. The reversal arises because the algorithm changes which candidate is selected first, affecting the human evaluator’s remaining choice.

  • For parameters θA = 1.1θ and θH = 0.9θ, UAH(θA, θH) > UHH(θA, θH).The example therefore makes choosing after a better opponent better than choosing after a worse opponent.
  • When choosing first, the algorithm is more likely than the human evaluator to choose x2 rather than x3, while both select x1 with identical probabilities.
  • When choosing second, the human evaluator’s utility is higher if x2 is unavailable than if x3 is unavailable.With x2 unavailable, the human evaluator is almost guaranteed to get x1; with x3 unavailable, it chooses x2 with probability approximately 1/4.
  • Conditioned on x2 being unavailable, the human evaluator’s expected utility is approximately 3, compared with approximately 2.75 when x3 is unavailable.
  • The utility comparison reduces to probability differences for selecting x2 and x3, weighted by the utilities obtained when each candidate is unavailable.

C Proof of Theorem 2

Theorem 6 establishes that, for three candidates with well-ordered i.i.d. noise, a more accurate algorithm does not improve the algorithm-human welfare comparison when θA > θH. The proof uses ordering properties of noisy rankings and candidate-removal utilities.

  • Well-ordered noise: Gaussian and Laplacian distributions are well-ordered, so the theorem applies to both noise distributions.
  • Well-ordered noise: A well-ordered noise model makes correctly ordering two candidates more likely than inverting them when their realized noisy values are conditioned in either order.
  • Theorem 6: For three candidates with unique values x1 > x2 > x3 and well-ordered i.i.d. noise, θA > θH implies UAH(θA, θH) < UHH(θA, θH).
  • Proof setup: The proof defines u−i as the expected utility of the maximum human-ranked candidate when candidate i is unavailable.For three candidates, u−1, u−2, and u−3 are expressed using pairwise ranking probabilities λ1, λ2, and λ3.
  • Proof comparison: The argument compares how the algorithm and human evaluator alter first-choice probabilities and how those changes affect the remaining human choice.The probability differences satisfy a conservation relation, ∆p1 + ∆p2 + ∆p3 = 0, and are analyzed in two cases.

C.3 Supplementary Lemmas for Random Utility Models

The supplementary lemmas verify the ordering properties needed for the random utility model analysis. They show that Gaussian and Laplacian noise satisfy the relevant monotonicity condition for conditional pairwise ordering.

  • Contraction: The contraction operation pulls noisy values toward their means and can correct existing ranking inversions without introducing new ones.
  • Contraction: A well-ordered noise model supports the contraction-based comparison between algorithmic and human-generated rankings.The supplementary lemmas establish the needed properties for ranking transitions under contraction.
  • Laplacian noise: For Laplacian noise, the derivative of the conditional ordering probability is nonnegative in the relevant cases and strictly positive in some cases.
  • Proof strategy: The proofs analyze the conditional probability through separate ranges of a relative to xi and xj, then establish the derivative sign in each range.
  • Conditional ordering: For both Gaussian and Laplacian noise, Pr[Xi > Xj | Xi < a, Xj < a] is nondecreasing in a and strictly increasing for some a.This establishes the conditional ordering property used in the random utility model analysis.
  • Gaussian noise: For Gaussian noise, the derivative of the conditional ordering probability is positive.

D Verifying that the Mallows Model Satisfies Definition 1

The Mallows Model with Kendall tau distance satisfies the noisy permutation family conditions required by the paper. The verification checks differentiability and monotonicity properties through its permutation probabilities.

  • The Mallows Model with Kendall tau distance and θ = φ −1 satisfies the conditions of Definition 1.
  • The verification must establish differentiability, asymptotic optimality, and monotonicity.
  • Differentiability: Mallows permutation probabilities are differentiable with respect to θ because their numerator and denominator are differentiable.
  • Monotonicity: The monotonicity argument uses permutation swaps whose inversion counts differ by at least one and whose mapping is bijective.
  • Monotonicity: The relevant probability sum is a polynomial in φ with nonnegative weights, giving it a positive derivative with respect to φ.

E Proof of Theorem 3

The proof establishes that selecting first from an algorithmic ranking can outperform selecting second when the algorithm and human rankings differ, and compares human-only with algorithm-assisted utility. It uses Mallows-model ordering properties to show that more accurate pairwise rankings improve expected utility and that human-first selection exceeds human-second selection.

  • Theorem 3 proof: The proof concludes that E[π1 − π2 | π1 ≠ τ1] > 0, so the first human-ranked candidate has greater expected value than the second human-ranked candidate under disagreement.The result follows from summing pairwise value-weighted probability differences.
  • Theorem 3 proof: Conditioned on the algorithm and human rankings disagreeing at the top choice, higher-valued candidates are weakly more likely to appear first in the algorithmic ranking than in the human ranking.The inequality is strict except in the boundary case involving the first and last candidates.
  • Utility comparison: Because CA > CH while UHH − UH < 0, the proof derives UAH − UH < UHH − UH for the compared algorithm-assisted and human-only utilities.The argument combines the probability-weight comparisons with the negative difference between human-second and human-first utility.
  • Pairwise ordering: Lemma 5 shows that for xi > xj, the ordering xi before xj is φ times more likely than the reverse ordering.This follows by comparing permutations with the same remaining candidates and positions.
  • Human-only comparison: Lemma 7 establishes that human-only utility from selecting first exceeds human-only utility from selecting second: UH(θA, θH) > UHH(θA, θH).Selecting first is better because independent human rankings tend to order candidates correctly when their top choices agree.

F.1 Proof of Theorem 4

The proof of Theorem 4 analyzes pairwise ordering under the Mallows model. It shows that the probability of correctly ranking any pair increases with the accuracy parameter and uses this monotonicity to establish firms’ preference for the more accurate algorithm when φH ≥ φA.

  • Pairwise accuracy: Lemma 8 states that under the Mallows model, the probability of correctly ranking any pair i < j increases monotonically with φ.The proof compares permutations placing i before j with those placing j before i and differentiates their probability ratio.
  • Pairwise accuracy: The proof reduces the pairwise comparison to the extreme candidates 1 and n by showing that inversions outside the interval from i through j contribute a constant factor.This permits analysis of the relevant inversion count without loss of generality.
  • Pairwise accuracy: The probability that candidate 1 precedes candidate n increases strictly with φ because numerator terms increase while denominator terms weakly decrease.Consequently, for any i < j, d/dφ Pr[i ≻ j] > 0.
  • Firm choice: When φH ≥ φA, every firm rationally chooses H because H is at least as likely as A to correctly order each pair of candidates.The argument applies first to the initial firm and then conditionally to subsequent firms after candidates are removed.
Loading 2101.05853v2…