PaperlayerView exact revision on arXiv

Source-linked AI summary

Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes

Connor Toups, Rishi Bommasani, Kathleen A. Creel, Sarah H. Bana, Dan Jurafsky, Percy Liang

arXiv:2307.05862v2cs.LGcs.AIcs.CY

TL;DR

Research often evaluates individual models, although people’s outcomes can depend on their interactions with multiple deployed systems. This paper analyzes those outcomes at the ecosystem level and finds homogeneous outcomes across settings, improvements that rarely reduce systemic failures, and racial patterns in dermatology model predictions absent from human predictions.

  • Problem

    Research on machine learning’s societal impact typically focuses on individual models, although people’s impact depends on their aggregate interactions with deployed systems.

    Source: §1 · p. 1

  • Method

    Ecosystem-level analysis uses a failure matrix to represent individuals’ outcomes across decision-makers and examines deployed systems across modalities and datasets.

    Source: §1 · pp. 1–2

  • Results

    Across settings, commercial systems show homogeneous outcomes; most model improvements miss systemic failures, and dermatology models show racial outcome patterns not present in humans.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Ecosystem-level analysis offers a tool for characterizing machine learning’s cumulative societal impact on individuals.

    Source: §7 · p. 11

    The comparison baseline assumes independent model failures, without assumptions about correlations among models in the ecosystem.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Machine learning is traditionally studied at the model level: researchers measure and improve the accuracy, robustness, bias, efficiency, and other dimensions of specific models. In practice, the societal impact of machine learning is determined by the surrounding context of machine learning deployments. To capture this, we introduce ecosystem-level analysis: rather than analyzing a single model, we consider the collection of models that are deployed in a given context. For example, ecosystem-level analysis in hiring recognizes that a job candidate's outcomes are not only determined by a single hiring algorithm or firm but instead by the collective decisions of all the firms they applied to. Across three modalities (text, images, speech) and 11 datasets, we establish a clear trend: deployed machine learning is prone to systemic failure, meaning some users are exclusively misclassified by all models available. Even when individual models improve at the population level over time, we find these improvements rarely reduce the prevalence of systemic failure. Instead, the benefits of these improvements predominantly accrue to individuals who are already correctly classified by other models. In light of these trends, we consider medical imaging for dermatology where the costs of systemic failure are especially high. While traditional analyses reveal racial performance disparities for both models and humans, ecosystem-level analysis reveals new forms of racial disparity in model predictions that do not present in human predictions. These examples demonstrate ecosystem-level analysis has unique strengths for characterizing the societal impact of machine learning.

1 Introduction

Because people encounter multiple deployed systems, their outcomes depend on the ecosystem, not just one model. The paper introduces ecosystem-level analysis and finds homogeneous outcomes, improvements that rarely reduce systemic failure, and racial patterns in dermatology predictions not seen in human predictions.

1 Introduction

  • Ecosystem-level analysis centers individuals’ cumulative outcomes across decision-makers, complementing group-centric methods and extending findings across NLP, vision, speech, and medical imaging.The paper frames the approach as a tool for holistic evaluations of machine learning’s societal impact.Source: §1 · pp. 1, 3
  • Systemic failure means an individual receives no correct outcome from any system, a consequential distinction that average error rates can miss.The passage’s hiring example has the same 20% false-negative rate whether firms reject the same candidates or different candidates, but only the first creates systemic failures.Source: §1 · p. 2
  • Across HAPI settings, many instances are classified correctly by all three commercial systems and many incorrectly by all three, indicating homogeneous outcomes.The analysis spans three modalities and eleven datasets.Source: §1 · p. 2
  • Only 10% of a system’s instance-level improvements occur on cases misclassified by all other models, despite systemic failures comprising 27% of potential improvements.In one example, Amazon’s WAIMAI error rate fell by 2.5%, yet systemic failures did not decrease.Source: §1 · p. 2
  • On DDI, models are more homogeneous on darker skin images, while human outcome homogeneity is consistent across skin tones.Both models and humans perform worse for darker skin tones under conventional group-level analysis.Source: §1 · p. 3
  • Systemic failures can block access to jobs, benefits, or correct diagnoses, while a person failed by only one model may seek another option.The paper also argues that reliance on similar algorithms or shared pipeline components could increase systemic failure prevalence.Source: §1 · p. 3

2 Ecosystem-level Analysis

Ecosystem-level analysis centers the person’s outcomes across multiple decision-makers. It represents those outcomes in a failure matrix and focuses on cases where every decision-maker fails an individual.

2 Ecosystem-level Analysis

  • Ecosystem-level analysis centers people’s outcomes across the multiple models they encounter or choose among.Exclusively negative outcomes can have more severe consequences than receiving even one positive outcome.Source: §2 · p. 3

2.1 Definition

  • The failure matrix F records each individual’s outcome for each decision-maker, with columns ordered from lowest to highest failure rate.An individual’s outcome profile is that individual’s row of F.Source: §2.1 · p. 4
  • Systemic failure occurs when an individual experiences failure across all decision-makers in the domain of interest.The paper identifies it as a consequential subset of homogeneous outcomes.Source: §2.1 · p. 4

3 Homogeneous Outcomes in Commercial ML APIs (HAPI)

Across HAPI datasets, all-model failures and all-model successes occur more often than an independence baseline predicts. The analysis compares observed outcome patterns with a baseline adjusted for model failure rates.

3 Homogeneous Outcomes in Commercial ML APIs (HAPI)

  • The HAPI audit tracks nine commercial systems across text, image, and speech tasks on eleven datasets over three years.The datasets span sentiment analysis, facial emotion recognition, and spoken command recognition.Source: §3 · p. 4, §3.1 · p. 4

3.2 Ecosystem-level Behavior

  • The independence baseline adjusts for model error rates and compares observed outcome distributions without assuming correlations between models.The authors choose independence to avoid making judgments about whether particular instances are inherently hard or easy.Source: §3.2 · p. 5
  • In randomly sampled FER+ examples, systemic failures are not believed to be inherently harder for humans to label than examples all models classify correctly.The authors report no difficulty labeling the failure examples themselves.Source: §3.2 · p. 5, Figure 3 · p. 6
  • Across all datasets, observed all-correct and all-incorrect outcomes exceed baseline rates, while intermediary outcomes are mostly less common.The pattern also appears in the DIGIT spoken-command dataset.Source: §3.2 · p. 5, Figure 2 · p. 6

4 Do Model Improvements Improve Systemic Failures?

The authors examine whether improvements in individual models reduce systemic failures across model ecosystems. Across eleven datasets, improvements tend to benefit instances already handled by other models, with little progress on systemic failures.

4 Do Model Improvements Improve Systemic Failures?

  • Because deployed systems and users change over time, the authors ask whether single-model improvements reduce systemic failures across the ecosystem.They note that societal-impact analyses often consider static models.Source: §4 · p. 7
  • The analysis compares outcome profiles of other models on instances an improving API initially misclassified with profiles on the instances it later corrected.Year-over-year comparisons were included when one API improved by at least 0.5% accuracy.Source: §4 · p. 7
  • For Amazon on WAIMAI, 64.7% of improvements were on instances both other APIs classified correctly, versus 34.1% of potential improvements.Only 11.6% of potential systemic-failure cases improved, and 78 new systemic failures offset 78 resolved cases, yielding no net reduction.Source: §4 · pp. 7–8
  • Across eleven datasets, observed improvements underrepresent systemic failures and overrepresent instances where both other models already succeed.Reductions in systemic failures are consistently marginal relative to the previous year's distribution.Source: §4 · p. 8

5 Ecosystem-level Analysis in Dermatology (DDI)

On DDI, ecosystem-level analysis finds homogeneous outcomes for both dermatologists and models, with greater homogeneity among humans overall. Stratifying by skin tone reveals a model-specific disparity in homogeneity that is not significant in human predictions.

5.1 Data

  • DDI contains 656 biopsied skin-lesion images labeled for malignancy and grouped into three Fitzpatrick skin-tone categories.The section compares predictions from two prominent machine-learning models with those of two board-certified dermatologists.Source: §5.1 · p. 8

5.2 Results

  • Both models and board-certified dermatologists produced homogeneous outcomes on DDI, with human outcomes more homogeneous than model outcomes.The comparison uses observed and baseline ecosystem-level outcomes for model and human predictions.Source: §5.2 · p. 8, Figure 5 · p. 9
  • For models, systemic failure was 8.2% above baseline for the darkest skin tones and 1.5% below baseline for the lightest; humans showed no significant skin-tone variation.The analysis stratifies DDI outcomes by Fitzpatrick skin-tone categories and compares observed with baseline rates.Source: §5.2 · p. 9, Figure 6 · p. 9
  • Ecosystem-level analysis identifies a racial disparity in model homogeneity that is not present in human predictions.Human predictions have higher absolute homogeneity across racial groups, despite showing no significant differences in homogeneity by group.Source: §5.2 · pp. 9–10

6 Commentary

The commentary considers why homogeneous outcomes arise and what they imply for researchers and policymakers. Its analyses suggest example difficulty is a partial explanation, while policy responses may be needed to monitor ecosystem-level outcomes and provide recourse.

6.1 Explanations for Homogeneous Outcomes

  • Homogeneous outcomes across settings are mostly unabated even when individual models improve.The paper frames this as evidence from its earlier findings on deployed machine learning.Source: §6.1 · p. 10
  • In FER+, systemic failure rises monotonically with annotator disagreement, but also occurs on images with little disagreement.The result suggests ambiguity or image difficulty is only a partial explanation of homogeneous outcomes.Source: §6.1 · p. 10
  • On DDI, model outcomes remain homogeneous when both dermatologists fail, but not when both succeed.The observed and baseline distributions nearly match for instances both dermatologists classify correctly.Source: §6.1 · p. 10
  • A two-parameter difficulty model fits the observed trends when roughly 25% of examples are about 3.5 times harder than overall performance.The model assumes shared hard and easy instances while retaining instance-level independence.Source: §6.1 · p. 10
  • The paper cannot test whether shared training data, architectures, components, or methods explain correlated outcomes because details of deployed commercial systems are scarce.It expects these model-centric factors may contribute and encourages future work on them.Source: §6.1 · p. 11
  • Ecosystem-level research applies to human-only, machine-only, and intertwined decisions in contexts where people face decisions from multiple actors.The authors identify job applications, medical treatment, loan applications, and rent pricing as examples.Source: §6.1 · p. 11
  • Policy could support ecosystem-level monitoring and improvement, alongside mechanisms for recourse or redress for people systemically failed by machine learning.The authors note that individual decision-makers may not observe other actors' decisions and may lack incentives to act alone.Source: §6.1 · p. 11

7 Conclusion

The paper introduces ecosystem-level analysis to assess cumulative impacts on individuals, finding homogeneous outcomes that often persist despite model improvement and new racial disparities in dermatology model predictions. It also outlines data-centric tests of whether example difficulty helps account for these outcomes.

7 Conclusion

  • The paper introduces ecosystem-level analysis to characterize cumulative societal impacts, reports homogeneous outcomes largely unaddressed by model improvement, and finds new racial disparities in dermatology model predictions but not human predictions.The conclusion calls for further theoretical backing, causal explanations, and sociotechnical mitigations.Source: §7 · p. 11

A Data-centric Explanations for Homogeneous Outcomes

  • The appendix motivates these tests with the hypothesis that shared properties of underlying data may contribute to homogeneous outcomes across the examined datasets.It contrasts this data-centric explanation with prior model-centric explanations.Source: A Data-centric Explanations for… · p. 16
  • The appendix tests three data-centric hypotheses using annotator disagreement, dermatologist accuracy, and a more expressive theoretical model.The theoretical model uses two difficulty parameters and computes a different baseline for ecosystem-level outcomes.Source: A Data-centric Explanations for… · p. 16

A.1 Annotator disagreement

Across these analyses, homogeneous outcomes persist across annotator-disagreement levels and track dermatologist errors on DDI. A theoretical model with shared hard and easy instances can better recover observed ecosystem-level distributions, under the assumption that models agree about which instances are hard.

A.1 Annotator disagreement

  • Homogeneous outcomes exceed baseline rates at every FER+ annotator-disagreement level, so instance ambiguity does not fully explain the pattern in that dataset.More ambiguous examples have higher model error and systemic failure rates.Source: §A.1 · p. 17
  • The relationship between disagreement and homogeneity intensity changes with the measure: the difference grows with disagreement, while the ratio effect is strongest at low disagreement.The ratio is the homogenization metric introduced by Bommasani et al. [2022].Source: §A.1 · p. 17

A.2 Human accuracy

  • On DDI, model outcomes are homogeneous when both dermatologists fail, but observed and baseline distributions match when both succeed.For instances where both dermatologists fail, models jointly fail and jointly succeed more than baseline rates.Source: §A.2 · p. 18
  • The DDI human-accuracy analysis is limited to 155 instances where both dermatologists failed and 356 where both succeeded, and uses only two domain experts.The authors caution against overgeneralizing these findings to larger pools of non-expert annotators.Source: §A.2 · p. 18

A.3 More expressive theoretical models

  • The model divides examples into a hard fraction α and an easy fraction 1 − α, with Δ adjusting expected difficulty and baseline failure rates.It forms the baseline distribution of total model failures as a weighted sum of two Poisson-Binomial distributions.Source: §A.3 · p. 18
  • The difficulty model's best-fitting parameters use α values of 0.2 or 0.3 and Δ values from 1 to 4, with hard-instance failure rates up to five times higher.The model selects the α and Δ combination minimizing the L1 distance between observed and baseline distributions; Figure 10 shows the distributions for the distance-minimizing pair.Source: §A.3 · p. 19, Figure 10 · p. 19
  • The difficulty framework assumes all models regard the same instances as hard, an agreement that could reflect properties of the data or model construction.High Δ and low α correspond to a small group of very difficult instances on which all models struggle.Source: §A.3 · p. 19

B HAPI Experiments

The HAPI experiments analyze ecosystem outcomes across classification tasks and datasets, including how failures cluster across models and how that pattern responds to model improvements. A single failure by a more accurate model is associated with a larger rise in systemic-failure probability than a failure by the least accurate model.

B.1 Data

  • The analysis covers 11 datasets across text, image, and speech classification, excluding COMMAND because duplicate IDs have conflicting predictions from the same API provider in the same period.The selected HAPI tasks are spoken command recognition, sentiment analysis, and facial emotion recognition; AFNET uses 2021 Face++ predictions where 2022 predictions are unavailable.Source: §B.1 · p. 20

B.2 Leader Following Effects in Systemic Failure

  • Across various thresholds, the patterns discussed in §4 remain consistent; using net rather than gross improvements, overall trends are also consistent, with himp under-improving on systemic failures.Source: Figure 13 · p. 21, Figure 14 · p. 22
  • A failure by the most accurate model raises systemic-failure probability by 37%, versus around 16% after a failure by the least accurate model.The authors describe this graded relationship as leader following: users failed by the most accurate model likely have few alternative models that work for them.Source: §B.2 · p. 21, Figure 12 · p. 21

B.3 Model improvement analysis is insensitive to threshold

The appendix tests whether the model-improvement findings depend on the improvement threshold or on counting gross rather than net gains, and also examines model declines. The reported patterns persist across thresholds, net gains make less progress on systemic failures, and model declines disproportionately add new systemic failures.

B.3 Model improvement analysis is insensitive to threshold

  • The finding that models under-improve on systemic failures holds across six improvement thresholds, so the qualitative pattern is not particularly sensitive to the chosen threshold.The analysis varies the threshold for including a model improvement around the paper’s 0.5% accuracy threshold.Source: §B.3 · p. 22

B.4 Net Improvements

  • Gross improvements count instances a model changes from incorrect to correct; net improvements subtract instances that change from correct to incorrect.The authors recommend analyzing both because their qualitative trends may be similar while their interpretations differ.Source: §B.4 · pp. 22–23
  • The net-improvement outcome-profile distribution subtracts gross declines from gross improvements within each profile.Its denominator is the difference between total gross improvements and total gross declines across profiles.Source: §B.4 · p. 22
  • WAIMAI had no net improvements on systemic failures despite a 2.5% decrease in model error at the population level.Across the analysis, net improvements make even less progress on systemic failures than gross improvements.Source: §B.4 · p. 22

B.5 When Models Get Worse

  • When models get worse, they disproportionately introduce new systemic failures by over-declining on instances other models already fail.The authors note that single-model measurements often fail to align with ecosystem-level outcomes.Source: §B.5 · p. 23

C.1 Data

The dermatology analysis uses lesion images with predictions from models and board-certified dermatologists, and tests whether excluding HAM10K affects the findings. Including HAM10K exacerbates homogeneous model outcomes and racial disparities in model outcomes.

C.1 Data

  • DDI contains 656 skin-lesion images, with three model predictions and two board-certified dermatologists’ predictions for malignant-versus-benign classification.Biopsy provides the ground-truth label, and none of the evaluated models were trained on DDI; the dataset serves as a test set.Source: §C.1 · p. 23
  • DDI images are grouped as light, medium, or dark skin tone using Fitzpatrick categories, and low-quality images were discarded after dermatologist review.The authors report no significant difference in image-quality ratings between Fitzpatrick skin-tone classifications.Source: §C.1 · pp. 23–24

C.2 Analyses are insensitive to including/excluding HAM10k

  • Including HAM10K exacerbates homogeneous model outcomes and racial disparities in model outcomes.The authors report this after replicating the analyses with HAM10K predictions included.Source: Figure 16 · p. 24, §C.2 · p. 24
  • The analysis excludes HAM10K because its near-universal negative predictions were considered unlike expected clinical deployment behavior and would complicate systemic-failure and human comparisons.HAM10K has 0.99 precision and 0.06 recall; including it would leave three models but only two human comparators.Source: §C.2 · p. 24

Related papers

Loading 2307.05862v2…