PaperlayerView exact revision on arXiv

Source-linked AI summary

Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?

Rishi Bommasani, Kathleen A. Creel, Ananya Kumar, Dan Jurafsky, Percy Liang

arXiv:2211.13972v1cs.LGcs.AIcs.CLcs.CVcs.CY

TL;DR

Machine-learning systems increasingly share components, raising the question of whether reuse creates systemic exclusion through homogeneous outcomes. This paper formalizes and tests that risk across shared data and foundation-model settings, finding reliable data-sharing effects but mixed model-sharing effects that depend on adaptation method.

  • Problem

    The paper asks whether machine-learning sharing practices create outcome homogenization, where the same individuals or groups receive negative outcomes from every decision-maker.

  • Method

    It formalizes homogenization as systemic failure and tests the component-sharing hypothesis using controlled data-sharing experiments and foundation models.

  • Results

    Data sharing often homogenizes individual and racial-group outcomes across 3 fairness datasets, while foundation-model results are mixed and adaptation method matters.

  • Takeaways & Limitations

    The findings support scrutiny of sharing practices because homogenization can represent a systemic harm, but the relationship between sharing and homogenization is incomplete.

  • Takeaways & Limitations

    The study focuses on settings where monoculture is present and notes that stronger future analyses should consider denial of most opportunities, not only all opportunities.

Abstract

from arXiv · show

As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e.g. training data), are deployed by multiple decision-makers. While sharing offers clear advantages (e.g. amortizing costs), does it bear risks? We introduce and formalize one such risk, outcome homogenization: the extent to which particular individuals or groups experience negative outcomes from all decision-makers. If the same individuals or groups exclusively experience undesirable outcomes, this may institutionalize systemic exclusion and reinscribe social hierarchy. To relate algorithmic monoculture and outcome homogenization, we propose the component-sharing hypothesis: if decision-makers share components like training data or specific models, then they will produce more homogeneous outcomes. We test this hypothesis on algorithmic fairness benchmarks, demonstrating that sharing training data reliably exacerbates homogenization, with individual-level effects generally exceeding group-level effects. Further, given the dominant paradigm in AI of foundation models, i.e. models that can be adapted for myriad downstream tasks, we test whether model sharing homogenizes outcomes across tasks. We observe mixed results: we find that for both vision and language settings, the specific methods for adapting a foundation model significantly influence the degree of outcome homogenization. We conclude with philosophical analyses of and societal challenges for outcome homogenization, with an eye towards implications for deployed machine learning systems.

1 Introduction

The paper asks whether machine-learning sharing practices create algorithmic monoculture and introduces outcome homogenization as a systemic risk. It tests whether shared components, especially training data and foundation models, make decision-makers produce more homogeneous outcomes.

  • 1 Introduction: Machine learning routinely shares datasets, models, libraries, optimizers, and evaluations, allowing repeated reuse of costly assets and broad improvements.The paper asks whether this sharing ethos also creates undesirable outcomes.
  • 1 Introduction: Outcome homogenization occurs when individuals or groups exclusively receive negative outcomes from all decision-makers they encounter.The paper frames this as a systemic harm that requires examining treatment across multiple decision-makers.
  • 1 Introduction: The component-sharing hypothesis predicts that decision-makers sharing training data or models will systematically fail the same individuals or groups.The paper broadens monoculture to include similar systems constructed from shared components, not only identical systems.
  • 1 Introduction: Across 3 algorithmic fairness datasets, data sharing often homogenizes outcomes for individuals and racial groups.The introduction reports this result before examining foundation-model sharing.
  • 1 Introduction: Foundation-model experiments find mixed results: adaptation methods significantly influence homogenization, with linear probing consistently more homogeneous than finetuning across vision and language.Thus, foundation-model sharing does not always exacerbate outcome homogenization.

2 Outcome Homogenization in Hiring

The hiring example illustrates how shared or similar screening systems could repeatedly reject the same applicants across companies. The paper therefore broadens algorithmic monoculture beyond identical systems and notes that outcome homogenization can also arise without monoculture.

  • 2 Outcome Homogenization in Hiring: In resume screening, maximum homogenization occurs when every company makes the same decision about each candidate, leaving some interviewed everywhere and others nowhere.Candidates receiving no interviews are described as experiencing systemic failure.
  • 2 Outcome Homogenization in Hiring: Outcome homogenization may arise even without algorithmic monoculture, although this work restricts its analysis to settings where monoculture is present.This limits the scope of the paper’s empirical and conceptual treatment.
  • 2 Outcome Homogenization in Hiring: Human hiring agreement can reflect identical company needs, shared incentives, and common decision patterns rather than algorithms alone.The paper presents these as possible sources of homogeneous outcomes before algorithmic deployment.
  • 2 Outcome Homogenization in Hiring: Different company needs, bias prevalence, and manager preferences can preserve heterogeneity in screening outcomes, so systemic failures are not inevitable.The cited discussion notes that bias is not uniform across companies.
  • 2 Outcome Homogenization in Hiring: Algorithmic screening and dominant third-party vendors may increase the possibility of homogeneous outcomes across companies.The paper contrasts a stylized identical-system case with a more complex vendor-mediated status quo.
  • 2 Outcome Homogenization in Hiring: The paper expands algorithmic monoculture to include similar, non-identical systems whose consistent processes may yield consistently good or poor outcomes for different students.This broader conception is applied beyond exact algorithm sharing.
  • 2 Outcome Homogenization in Hiring: The component-sharing hypothesis predicts increased outcome homogenization when deployed systems share components, including training data or a foundation model.The paper empirically tests both forms of sharing.
  • 2 Outcome Homogenization in Hiring: Interpretation of homogeneous outcomes is contextual and value-laden because some individuals may be justifiably rejected from every opportunity.The paper gives failing a professional bar exam as an example.

3 Formalizing Outcome Homogenization

The paper formalizes outcome homogenization as systemic failure across decision-makers, measuring observed failure relative to an expected baseline. It extends the framework from individuals to groups and distinguishes homogenization from accuracy, fairness, and robustness metrics.

  • 3 Formalizing Outcome Homogenization: The paper provides the first mathematical formalism of outcome homogenization, beginning with individual systemic failures and extending to group exclusion.It also relates the new metrics to established measures of correlation, fairness, robustness, and accuracy.
  • 3.1 Homogeneous Outcomes for Individuals: The setup models a social system of individuals and decision-makers, with each individual interacting with every decision-maker represented by a model.The formalism is general about the nature of the decision-maker model.
  • 3.1 Homogeneous Outcomes for Individuals: An individual experiences systemic failure when every interacting model produces a failure for that individual.In classification experiments, failures are classification errors; other settings can use harms such as rejection from opportunities.
  • 3.1 Homogeneous Outcomes for Individuals: The individual homogenization metric normalizes the observed rate of systemic failure by the expected rate based on model failure rates.It compares observed systemic failure with the probability that independently sampled model outputs would all be failures.
  • 3.2 Homogeneous Outcomes for Groups: Group-level homogenization replaces the individual average with a weighted average over groups and can focus on average frequency or the worst-off group.The worst weighting assigns all weight to the group with the highest systemic failure rate.
  • 3.2 Homogeneous Outcomes for Groups: The weighting schemes expose a trade-off: frequency-based averages can obscure systemic exclusion experienced by minority or worst-off groups.The paper introduces multiple weights because different contexts may require different group emphases.
  • 3.3 Understanding Our Metrics: Homogenization ratios range over [0,∞), with 0 meaning no systemic failures, 1 matching the expected rate, and values above 1 indicating excess homogeneous outcomes.The individual metric assumes each person interacts exactly once with each decision-maker.
  • 3.4 Relationship with Other Metrics: Homogenization differs from fairness and robustness because it concerns correlated outcomes across a social system, whereas those metrics are usually defined for one model.Unlike many fairness and robustness formalisms, homogenization can be defined for singular individuals.

4 Data-sharing Experiments

Controlled experiments compare models trained on shared versus disjoint data, finding that greater data sharing generally produces more homogeneous outcomes. Individual-level homogenization exceeds group-level homogenization, while accuracy can remain nearly unchanged across sampling settings.

  • Experimental design and results: The fixed and disjoint settings sample training data differently: fixed reuses n points across k models, whereas disjoint partitions kn points across them.Both settings share the same training distribution, but only fixed shares the exact data inputs.
  • Experimental design and results: Fixed training data, which shares exact inputs across task-specific models, generally produces more homogeneous outcomes than disjoint training data.The comparison spans three datasets and four model families, supporting the component-sharing hypothesis.
  • Experimental design and results: Data sharing does not fully determine homogeneity: some disjoint settings match or exceed fixed settings, reflecting randomness and instability in systemic-failure counts.Systemic failures are sensitive because they identify data points failed by every model.
  • Experimental design and results: Near-identical error-rate trends make fixed and disjoint settings indistinguishable by accuracy alone, despite differing in observed systemic failures.The outcome-homogenization measures reveal discrepancies that error rates do not.
  • Experimental design and results: Individual-level homogenization consistently exceeds racial group-level homogenization in ACS PUMS, where group-level values remain near 1 with little change as dataset size varies.The comparison uses average and uniform group-level metrics against the individual-level metric.

5 Model-sharing Experiments

The experiments test whether sharing foundation models homogenizes outcomes across vision and language tasks. Results are mixed: adaptation method matters substantially, and model sharing alone does not reliably predict homogenization.

  • Experimental design: Foundation models are evaluated on CelebA vision tasks and separate language tasks using multiple adaptation strategies.Vision compares scratch training, CLIP probing, and CLIP finetuning; language compares probing, finetuning, and BitFit with RoBERTa-base.
  • Vision results: Vision results reverse the predicted sharing pattern: scratch models are most homogeneous, followed by probing and finetuning.The authors conjecture that CLIP’s broader WebImageText training may regularize models against CelebA-specific idiosyncrasies.
  • Language results: Language results broadly follow the hypothesis: probing is most homogeneous, while BitFit and finetuning are roughly equally homogeneous.BitFit shares 99.92% of foundation-model parameters across tasks, yet parameter counts alone do not explain the observed ordering.
  • Metric relationships: Across the reported metric analyses, homogenization is generally distinct from accuracy but related to fairness, especially for the worst-performing group.For language and vision, the metrics are generally uncorrelated or weakly correlated with accuracy; H_worst is strongly correlated with fairness in both modalities.
  • Discussion: Overall, sharing provides evidence for homogeneous outcomes but is an incomplete explanation, requiring attention to data distributions and distribution shifts.The paper contrasts foundation-model experiments with lower-dimensional data-sharing experiments and emphasizes their differing regimes.

6 Societal Considerations

The paper frames outcome homogenization as a context-dependent social harm, particularly when repeated exclusion prevents equal participation. It also identifies practical barriers to diagnosing and rectifying such harms.

  • Why are Homogeneous Outcomes Harmful?: The moral significance of homogeneous outcomes depends on context, with relational egalitarianism focusing on whether people can participate as equals.The analysis links exclusion from employment, education, or credit to social hierarchy and domination.
  • Why are Homogeneous Outcomes Harmful?: Harm is most severe when individuals are excluded from all opportunities rather than denied a single opportunity.The paper studies this strongest form of exclusion while acknowledging that exclusion from a significant fraction may also matter.
  • Diagnosis: Identifying shared vendors, datasets, or foundation models is difficult because algorithmic systems are often opaque.Observed homogenization could nevertheless motivate transparency that helps auditors identify latent monoculture.
  • Measurement: Individual-level measurement requires linking outcomes across deployments, which privacy constraints may make difficult or impossible.Group-level homogenization is therefore presented as more generally accessible.
  • Rectification: Organizations may lack incentives to reduce homogenization because responsibility and remedies are distributed across entities.The paper suggests regulation, policy, or compliance mechanisms may be needed to address collective-action problems.

7 Limitations and Conclusion

The paper presents outcome homogenization as an understudied systemic harm associated with machine-learning sharing and monoculture. It calls for further work while emphasizing the strongest exclusion setting studied here.

  • Limitations and Conclusion: Outcome homogenization remains poorly understood in definition, interpretation, estimation, mitigation, and its connection to monoculture.The authors encourage future work across each of these directions.
  • Limitations and Conclusion: The paper argues that homogenization deserves attention as algorithmic monoculture grows, including through foundation models.It frames early intervention as important for preventing harms from becoming entrenched.
  • Limitations and Conclusion: The paper reports that code, data, and experiments are publicly available for reproducibility.Materials are hosted on GitHub and CodaLab Worksheets.
  • Limitations and Conclusion: The experiments study the strongest exclusion case, while future work should examine denial from most opportunities rather than all opportunities.The authors note that exclusion from a significant fraction of opportunities may also be harmful in some contexts.

Checklist

The checklist reports coverage of claims, limitations, societal impacts, reproducibility, training details, error bars, compute, and asset attribution. It also records disclosures concerning consent and personally identifiable information.

  • Review checklist: The checklist states that the paper describes its contributions, scope, limitations, societal impacts, and ethics compliance.It points readers to the metrics, experimental, societal-considerations, and conclusion sections.
  • Reproducibility: The authors report that code, data, and instructions needed to reproduce the experiments are available, with training details and error bars documented.The checklist directs readers to CodaLab Worksheets, Appendix B, and Figure 3.
  • Reproducibility: The checklist reports that compute resources and existing-asset citations are documented.Compute details are in Appendix B, while asset attribution is covered in the relevant paper sections.
  • Data and ethics: The authors disclose that some datasets contain personally identifiable information, face images, or hate-speech-related content.They state that these materials are established datasets and that prior work discusses the associated issues.
  • Data and ethics: The checklist marks participant research and compensation disclosures as not applicable.The relevant entries cover IRB approvals, hourly wages, total compensation, and participant instructions.

A.1 Group Homogenization Metrics

The paper defines group-level homogenization metrics by specifying how groups are weighted and by formalizing average, uniform, and worst weightings.

  • Group-level homogenization metrics differ according to how groups are weighted.The paper considers average, uniform, and worst weightings.
  • The frequency of group g in dataset D_i is denoted p_i(g), while the joint probability of the group across datasets is denoted p(g).

A.2 Relating Individual and Group Homogenization

Individual- and group-level homogenization need not coincide: depending on how errors overlap across decision-makers, either level can show greater systemic failure.

  • Individual-level outcome homogenization can exceed racial-group-level homogenization.The paper demonstrates this empirically and notes that finer-grained individual groupings do not necessarily produce greater homogenization.
  • Scenario 1: In a two-model scenario where no individual is misclassified by both models, individual systemic failure is zero while group-level systemic failure is positive.Here, H_individual = 0, whereas H_avg = H_unif > 0.
  • Scenario 2: In a contrasting scenario, two individuals are misclassified by both models, yielding H_individual = 0.5 and H_avg = H_unif = H_worst = 1.The individual-level rate of systemic failure is 0.5, while the group-level rate is 1 under all three weightings.
  • The individual-level formalism assumes each individual produces an input for every decision-maker, although inputs may differ across decision-makers.The framework does not initially support multiple inputs per decision-maker or absent interactions with some decision-makers.
  • The notation c_j represents the subset of decision-makers with whom individual j interacts, enabling homogenization metrics to accommodate missing interactions.

A.4 Alternative Metrics

The paper compares its homogenization metrics with alternative measures and extends the framework beyond binary failures, while acknowledging unresolved questions about metric validity and choice.

  • Outcome homogenization is a new construct, so the paper acknowledges that its metrics may require validation in real systems.The authors state that their understanding of the construct is incomplete and that precise measurement desiderata remain under study.
  • The metrics’ structural fidelity, predictive validity for long-term harms, and usefulness for scientific and social hypotheses remain unclear.
  • Related metrics: For two decision-makers, the homogenization metric closely resembles covariance and pointwise mutual information evaluated when both models fail.
  • Related metrics: The paper favors its metric over covariance because it is more comparable across settings with different social-system failure rates.
  • Beyond binary failures: The framework can accommodate different tasks by binarizing outcomes as failures or non-failures, but comparable tasks may instead use graded loss functions.For comparable losses, MinExp and ExpExp provide alternative homogenization measures.
  • Alternative loss metrics: MinExp compares average best-case individual loss with the loss of the best model, whereas ExpExp uses average model loss in the denominator.
  • Alternative loss metrics: Under 0-1 classification loss, MinExp and H_individual have similar numerators but different normalizations and ranges.MinExp lies in [0,1], whereas H_individual lies in [0,∞).
  • Metric choice: The paper recommends considering MinExp when losses are comparable but defaulting to H_individual because it requires less structure and has a simple probabilistic interpretation.The authors expect the metrics to be strongly correlated in many settings but acknowledge that they can diverge.

B.1.1 Data

The experiments use several fairness, vision, and language datasets with models trained or adapted for classification tasks, while grouping outcomes using demographic attributes available across settings.

  • ACS PUMS contains US Census survey data for 3.6 million individuals with 286 recorded features.The dataset supports research on algorithmic fairness and measurement of harms from algorithmic systems.
  • For ACS PUMS, the experiments train logistic regression models across fixed and disjoint training-data settings, using 25 models per setting and data amount.Models span three tasks, five random subsamples, and five random seeds.
  • The ACS PUMS analysis uses self-identified racial categories supplied by the Census, including White, Black, Asian, and other listed categories.
  • CelebA supplies facial-attribute annotations for apparel-prediction tasks involving Earrings and Necklace.The dataset is used analytically to study homogenization rather than for face recognition.
  • The vision experiments compare released CLIP weights with a randomly initialized model using the same modified ViT-B/16 architecture.The released model has 150M parameters, and training required approximately 1000 GPU hours in aggregate.
  • CelebA groups are defined primarily by hair color, with an additional individual-level measure based on whether a person has a beard.Small or missing hair annotations are combined into an Other category.
  • The language experiments use IMDB, AGNews, Yahoo, and HateSpeech18 datasets.
  • Language models adapt RoBERTa-base using Hugging Face weights and a learning rate of 2e-5.The implementation uses the Transformers Trainer defaults apart from the learning-rate setting.

C Additional Experiments

Additional experiments across datasets and model families support the paper’s core findings: greater data sharing reliably produces greater outcome homogenization, with individual-level effects exceeding group-level effects. The absolute magnitude varies with dataset structure and task relationships, while shared data can increase homogenization even when model accuracies are unchanged.

  • Results and Analysis: Individual-level homogenization reliably exceeds the corresponding racial group-level homogenization when group metadata is available.This pattern transfers across the additional evaluations, including ACS PUMS.
  • Datasets: The additional datasets include LSAC, GC, and ACS PUMS, using paired prediction tasks attached to the same individuals.LSAC predicts bar passage and above-mean law-school GPA; GC predicts loan quality and whether credit amount exceeds 2000.
  • Model Families: Data-sharing effects generalize beyond logistic regression to gradient boosted trees, support vector machines, and neural networks.The additional experiments vary model families on ACS PUMS and replicate the core comparisons on LSAC and other datasets.
  • Results and Analysis: Across datasets and model families, fixed data partitions reliably produce greater homogenization than disjoint partitions.The comparison concerns relative change from sharing; absolute homogenization differs across datasets and task relationships.
  • Results and Analysis: Even when expected systemic failure rates and model accuracies converge across partitions, fixed data sharing still yields higher observed homogenization.This creates a social-systems Rashomon effect: models with the same accuracies can differ in outcome homogeneity.
Loading 2211.13972v1…