Source-linked AI summary
Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging
Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, Christopher Ré
TL;DR
Medical imaging models can perform well overall while failing on clinically important subsets left unidentified by incomplete labeling, a problem termed hidden stratification. The paper evaluates human- and algorithm-based measurement approaches across medical imaging datasets and finds clinically meaningful subclass differences, including differences of up to 20%. It argues that hidden-stratification evaluation should accompany medical imaging deployment.
Problem
Incomplete schemas can leave clinically meaningful imaging subsets unidentified, making aggregate model performance potentially insufficient for assessing clinical utility and safety.
Method
The paper evaluates schema completion, error auditing, and algorithmic methods for detecting and measuring hidden stratification on medical imaging test datasets.
Results
Up to 20% performance differences occurred on clinically important subsets, with hidden stratification associated with low prevalence, reduced label accuracy, subtle features, and spurious correlates.
Takeaways & Limitations
Overall task performance may fail to identify the most clinically important cases, while simple unsupervised approaches can help reveal unrecognized subclasses.
Abstract
from arXiv · showhide
Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model still consistently misses a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring and describing hidden stratification effects, and characterize these effects on multiple medical imaging datasets. We find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we explore the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.
1 Introduction
Medical imaging models can appear strong overall while failing on clinically meaningful subgroups that coarse labels do not distinguish. The paper calls this hidden stratification and evaluates ways to measure its effects.
- Hidden stratification occurs when incomplete labels coarsely describe meaningful clinical and visual variation within a population.A single class may contain distinct subsets, such as solid versus subsolid or central versus peripheral tumors.
- Aggregate accuracy, sensitivity, and ROC AUC can be dominated by larger subsets and obscure poor performance on unidentified minority subsets.Underperformance on rare serious diseases may therefore cause disproportionate patient harm.
- The paper evaluates schema completion, error auditing, and algorithmic measurement to characterize hidden stratification in medical imaging.It reports performance differences of up to 20% on clinically important subsets and evidence that simple unsupervised methods can help identify them.
2 Methods for Measuring Hidden Stratification
The paper presents three complementary approaches for measuring hidden stratification on test data: prospective subclass labeling, retrospective error review, and automated subclass discovery.
- Schema Completion: Schema completion prospectively defines a more complete subclass schema and labels test data for those subclasses.It can support consensus definitions, reporting standards, and model development, but depends on the schema author not omitting important subclasses.
- Error Auditing: Error auditing reviews model outputs for recognizable subclasses with unexpected, consistently incorrect predictions.It is informed by model behavior and more labor-efficient than exhaustive labeling, but may miss low-prevalence, clinically salient strata.
- Algorithmic Measurement: Algorithmic measurement automatically searches for underperforming subclasses, often using unsupervised clustering, before human review.This reduces dependence on an auditor’s initial ability to recognize stratification patterns.
3 Experiments
Experiments across radiograph datasets show hidden stratification associated with prevalence, label quality, subtle features, and spurious correlates. Schema completion, error auditing, and clustering reveal clinically important differences that aggregate metrics can conceal.
- Schema Completion: 0.900 and 0.911 sensitivity on subtle and cervical hip fractures were lower than 0.981 on the overall task.The differences were significant at p < 0.01, supporting effects from subtle features and low prevalence.
- Schema Completion: 0.98 ROC AUC for hardware cases exceeded aggregate ROC AUC of 0.91, while degenerative disease reached 0.76 on MURA.The hardware subclass had low prevalence, whereas degenerative disease combined low-sensitivity labels with subtle visual features.
- Error Auditing: 0.94 ROC AUC for pneumothorax with chest drains contrasted with 0.77 without drains, despite overall ROC AUC of 0.87.Chest-drain cases comprised 80% of test-set pneumothoraces, and positive predictive value was 0.90 versus 0.60 without drains.
- Algorithmic Measurement: Unsupervised k-means clustering produced clusters with a high proportion of drains on CXR14 but did not always yield meaningful separation, such as on MURA.The approach can assist auditors and help confirm schema completion.
4 Discussion
Hidden stratification produces substantially different performance across clinically meaningful subsets, especially when labels are uneven, features are subtle, or spurious correlates are present. The CXR14 example shows how overall results can obscure failures in untreated cases and potentially contribute to harmful deployment decisions.
- Hidden stratification can produce markedly different superclass and subclass performance when subclass labels vary in accuracy, subclasses are imbalanced, features are subtle, or spurious correlates are present.
- In CXR14, chest drains were common among pneumothorax cases, while performance was far worse on clinically important cases without drains.Chest drains reflect a healthcare process variable rather than a causal pneumothorax feature, and drained cases were already treated.
- Overall task results alone could incorrectly justify clinical use or regulatory approval when performance is poor in clinically important subsets.The authors warn that such a scenario could ultimately cause patient harm.
- Simple unsupervised methods produced clusters containing different proportions of previously identified hidden subclasses.The authors therefore suggest considering hidden stratification when planning, building, evaluating, and regulating clinical image analysis systems.
Appendix
The appendix tables summarize subclass prevalence and performance for hip-fracture and musculoskeletal abnormality tasks. They highlight that subclass-level performance can differ from overall performance and that high-prevalence subclasses may still have low sensitivity.
- Table 2 compares superclass and subclass performance for hip-fracture detection from frontal pelvic x-rays.Bolded subclasses have significantly worse performance than the overall task.
- Table 3 reports MURA abnormal-label prevalence and sensitivity for fracture, hardware, and degenerative joint disease subclasses.Degenerative joint disease has the highest prevalence but the lowest sensitivity relative to board-certified radiologist review.