Change is Hard: A Closer Look at Subpopulation Shift

Yuzhe Yang; Haoran Zhang; Dina Katabi; Marzyeh Ghassemi

Source-linked AI summary

Change is Hard: A Closer Look at Subpopulation Shift

Yuzhe Yang, Haoran Zhang, Dina Katabi, Marzyeh Ghassemi

arXiv:2302.12254v3Feb 23, 2023cs.LGcs.AIcs.CV

TL;DR

Subpopulation shift causes models to fail on underrepresented groups, but its mechanisms and cross-shift algorithmic behavior remain poorly understood. The paper introduces a unified framework and benchmarks 20 algorithms on 12 datasets using over 10,000 models, finding shift-specific robustness, effective group-free model selection, and a tradeoff between worst-group accuracy and other metrics.

Problem
Existing understanding of subpopulation shift is limited across its diverse mechanisms and how algorithms generalize across them.
Method
The paper decomposes subpopulation shift into attribute and class components and benchmarks 20 algorithms on 12 real-world datasets across multiple domains.
Results
Algorithms improve subgroup robustness for some shift types but not others, while worst-class accuracy causes an average accuracy drop of less than 2% without validation attributes.
Takeaways & Limitations
Worst-class accuracy can support model selection without group-annotated validation data, while evaluation should consider metrics beyond worst-group accuracy.
Takeaways & Limitations
Using worst-group accuracy alone can overlook low worst-case precision, including in critical applications such as medical diagnosis.

Abstract

from arXiv · show

Machine learning models often perform poorly on subgroups that are underrepresented in the training data. Yet, little is understood on the variation in mechanisms that cause subpopulation shifts, and how algorithms generalize across such diverse shifts at scale. In this work, we provide a fine-grained analysis of subpopulation shift. We first propose a unified framework that dissects and explains common shifts in subgroups. We then establish a comprehensive benchmark of 20 state-of-the-art algorithms evaluated on 12 real-world datasets in vision, language, and healthcare domains. With results obtained from training over 10,000 models, we reveal intriguing observations for future progress in this space. First, existing algorithms only improve subgroup robustness over certain types of shifts but not others. Moreover, while current algorithms rely on group-annotated validation data for model selection, we find that a simple selection criterion based on worst-class accuracy is surprisingly effective even without any group information. Finally, unlike existing works that solely aim to improve worst-group accuracy (WGA), we demonstrate the fundamental tradeoff between WGA and other important metrics, highlighting the need to carefully choose testing metrics. Code and data are available at: https://github.com/YyzHarry/SubpopBench.

1. Introduction

The paper addresses subpopulation shift as a broad, underdefined source of subgroup failure and introduces a unified framework and large-scale benchmark to study it systematically.

Subpopulation shift changes subgroup proportions between training and deployment, allowing high overall performance alongside poor rare-subgroup performance.
Prior work emphasizes spurious correlations, but subpopulation shift includes wider mechanisms whose occurrence and algorithmic generalization remain insufficiently understood.
The proposed framework decomposes attributes and classes, explains common subgroup shifts, and quantifies each shift’s type and degree.
The benchmark covers 20 state-of-the-art algorithms and 12 real-world datasets across vision, language, and healthcare.
The evaluation varies attribute availability, validation and model-selection strategies, and metrics across more than 10K trained models.
The study finds that algorithms improve robustness for some shift types but not others, while worst-class accuracy can support selection without group-annotated validation data.

2. Related Work

Related work treats subpopulation shift through several narrower settings, whereas this paper unifies those scenarios and benchmarks methods across more datasets, algorithms, shift components, and evaluation choices.

Prior subpopulation-shift work largely studies shortcut learning, where attribute-label correlations produce poor minority-subgroup performance.
Class-imbalanced learning defines subpopulations using labels alone and has developed extensive methods for that setting.
When subpopulations are defined by attributes such as demographic groups, worst-group optimization coincides with minimax fairness.
This paper presents a unified framework spanning shortcut, class-imbalance, and attribute-defined subpopulation shifts.
Compared with prior benchmarks, the work evaluates more algorithm categories and real-world datasets while quantifying shift components and studying model selection and metrics.

3. Unified Framework of Subpopulation Shift

The framework models subpopulation shift through core label-relevant features, attributes, and class distributions, then defines four basic shift types that can combine in real datasets.

General formulation: Source and target distributions differ through group mixture weights, motivating worst-group risk as the subpopulation-shift objective.
General formulation: Inputs are represented by invariant core features x_core and attributes a, with subpopulations defined from attributes and labels.
General formulation: The framework separates a robust core-feature term from attribute and class terms that can introduce deployment bias.
General formulation: The attribute decomposition assumes mutual and conditional independence across attributes, with benign attributes contributing no label-dependent bias.
Basic shift types: Four basic shift types are defined: spurious correlations, attribute imbalance, class imbalance, and attribute generalization.
Basic shift types: Spurious correlations arise when attribute-label associations in training disappear at test time, biasing predictions toward correlated labels.
Basic shift types: Attribute imbalance underrepresents some attributes during training, reducing prediction confidence for those attributes.
Basic shift types: Class imbalance biases the class term toward majority labels, while attribute generalization requires robustness to attributes absent during training but present at test time.

4. Benchmarking Subpopulation Shift

The benchmark evaluates diverse algorithms, datasets, shift settings, metrics, and model-selection strategies to characterize subpopulation robustness under realistic conditions.

The benchmark covers 12 real-world datasets across multiple modalities and tasks, including vision, language, and healthcare settings.
The study evaluates 20 algorithms spanning subgroup robustness, data augmentation, domain-invariant learning, imbalanced learning, and vanilla ERM.
The evaluation includes average, worst, precision, and calibration metrics because worst-group accuracy alone does not capture all relevant aspects of subgroup shift.
The benchmark considers whether attributes are available during training and validation, including the hardest setting where both are unknown.When training attributes are unavailable, subgroup methods operate over classes; the main reported setting is unknown attributes in both training and validation.
Model selection is rigorously analyzed because choosing hyperparameters by average validation accuracy without group annotations can substantially reduce worst-group test accuracy.The implementation uses random hyperparameter search, validation-based selection, and reruns with three random seeds for final reporting.

5. A Fine-Grained Analysis

The benchmark shows that subpopulation-shift algorithms generalize unevenly across shift types, while model-selection choices and evaluation metrics materially affect conclusions. Representation and classifier learning contribute differently by shift, and worst-group accuracy alone can be misleading.

Shift characterization: Shift types vary substantially across datasets, so the benchmark categorizes each dataset by its dominant shift before comparing algorithms.The framework measures shift components and their degrees, while experiments evaluate algorithms with unknown attributes in training and validation.
Performance across shifts: Existing algorithms improve worst-group accuracy consistently for spurious correlations and class imbalance, but show little improvement for attribute imbalance and worse performance for attribute generalization.These gains over ERM occur even without validation attributes for spurious correlations and class imbalance.
Representation and classifier: Balanced classifier learning substantially helps spurious correlations and class imbalance, whereas balanced representation learning provides gains for attribute imbalance.Under attribute generalization, no stratified learning approach improves over vanilla ERM.
Representation and classifier: ERM features are insufficient for all subpopulation shifts, with improved feature extraction especially benefiting attribute imbalance and explaining weak two-stage performance under attribute imbalance and attribute generalization.The findings indicate that ERM-learned features may be adequate for some shifts but not others.
Model selection and attribute availability: Training attributes help group-dependent methods such as GroupDRO, but not attribute-agnostic methods such as ERM and JTT.The usefulness of attribute availability therefore depends on the algorithm's information requirements.
Model selection and attribute availability: Worst-class accuracy selection loses less than 2% average accuracy relative to group-aware selection, despite requiring no validation attributes.The paper contrasts this small drop with degradation exceeding 20% when average accuracy is used without validation attributes.
Metrics beyond worst-group accuracy: Adjusted accuracy correlates positively with worst-group accuracy, whereas worst-case precision has a strong negative correlation, revealing an intrinsic metric tradeoff.The paper reports that many metrics exhibit this inverse-line relationship, so testing should not rely on worst-group accuracy alone.
Additional analyses: Supervised pretraining generally outperforms self-supervised pretraining, while larger and more diverse pretraining datasets consistently improve worst-group performance.The dataset-size comparison examines ImageNet-1K, ImageNet-21K, and SWAG.

6. Conclusion

The paper formalizes and quantifies subpopulation shifts and benchmarks methods across diverse real-world settings. Its large-scale experiments reveal shift-dependent algorithm performance, model-selection effects, and limitations of relying on a single evaluation metric.

Conclusion: The unified framework defines and quantifies different subpopulation-shift types, and the benchmark evaluates 20 state-of-the-art methods across 12 real-world datasets and domains.The study trains over 10,000 models to analyze these settings systematically.
Conclusion: The findings reveal divergent algorithm performance across shifts, alongside important effects from model-selection criteria and evaluation metrics.The authors present these observations as implications for future research and more rigorous evaluations.

A. Limitations and Broader Impacts

The benchmark omits mislabeled attributes and labels, multiple spurious attributes, and the interactions among them. The work also notes potential accuracy, environmental, and clinical-deployment risks.

Limitations: The benchmark does not account for potential mislabeling in attributes and labels.The authors suggest examining this effect in a synthetic setting.
Limitations: The benchmark considers only one attribute, although reducing reliance on one spurious attribute may increase reliance on another.The authors identify multiple spurious attributes as an important direction for future evaluation.
Broader Impacts: Improving subpopulation robustness can reduce overall accuracy on the training distribution.The authors note that some practitioners may instead prioritize overall accuracy regardless of spurious correlations.
Broader Impacts: The large grid of trained deep models likely produced a significant carbon footprint.The authors hope the resulting insights will reduce future training steps and emissions.
Broader Impacts: The clinical models should not be deployed blindly before real-world clinical testing, privacy, fairness, and interpretability issues are resolved.

B.1. Dataset Details

The benchmark covers 12 real-world datasets across vision, language, and healthcare, representing several subpopulation-shift settings. Dataset construction includes spurious correlations, attribute generalization, and controlled splits or subsampling.

Benchmark Coverage: The benchmark includes 12 real-world datasets spanning computer vision, natural language processing, and healthcare applications.Examples are provided for image and text datasets, with some clinical examples omitted under data-use agreements.
Spurious Correlations: Waterbirds, CelebA, MetaShift, CivilComments, and MultiNLI encode spurious correlations between attributes and labels.Examples include backgrounds, gender, demographic identity references, and negation cues.
Healthcare Datasets: MIMIC-CXR uses race-by-gender attributes for No Finding prediction, while CXRMultisite uses hospital site as a pneumonia-related spurious correlation.MIMIC-CXR is randomly split into 85% training, 5% validation, and 10% test data.
Attribute Generalization: NICO++ creates attribute generalization by removing training pairs with fewer than 75 samples, leaving some attributes absent for certain classes.
Attribute Generalization: ImageNetBG evaluates models trained with original backgrounds on MIXED-RAND, where background attributes are unavailable during training.
Algorithms: The benchmark groups algorithms into vanilla ERM, subgroup-robust, data-augmentation, and imbalanced-learning strategies.Examples include GroupDRO, Mixup, ReSample, ReWeight, and related methods.

C. Experimental Settings

The experiments standardize pretrained backbones, hyperparameter search, shift-severity metrics, and model-selection settings across the benchmark. Results show that gains vary substantially by shift type, while worst-class selection can work without attribute annotations.

Training Setup: Image experiments use pretrained ResNet-50, text experiments use pretrained BERT, and MIMICNotes uses a three-layer MLP.Living17 is trained from scratch.
Training Setup: Each algorithm receives 16 random hyperparameter-search trials, followed by three random-seed reruns using selected validation hyperparameters.This produces a best-versus-best comparison with reported averages and standard deviations.
Shift Quantification: Shift severity is quantified with entropy, class-probability gaps, mutual-information measures, Cramer’s V, and Tschuprow’s T.Attribute generalization is assessed by whether test subpopulations are absent during training.
Results: Algorithmic improvements are consistent for spurious correlations and class imbalance, smaller for attribute imbalance, and nearly absent for attribute generalization.These results indicate limitations of current methods across shift types.
Model Selection: Worst-class accuracy selection averages only a 2.4% degradation in worst-group accuracy versus oracle selection and performs best on 10 of 12 datasets.

D.4. Rethinking Evaluation Metrics in Subpopulation Shift

The paper evaluates how metrics relate to worst-group accuracy across datasets. Adjusted and balanced accuracy track worst-group accuracy, while worst-case precision and ECE trade off against it and other metrics behave inconsistently.

Positive Correlation: Adjusted accuracy and balanced accuracy show consistent positive correlation with worst-group accuracy across diverse datasets.Both metrics exhibit the paper’s “accuracy on the line” property.
Negative Correlation: Worst-case precision and ECE show clear negative correlation with worst-group accuracy.This establishes an intrinsic tradeoff between worst-group accuracy and these metrics.
Implications: The findings support evaluating subpopulation shift with metrics beyond worst-group accuracy.
Inconsistent Correlation: Other metrics show inconsistent correlations with worst-group accuracy across datasets.Optimizing worst-group accuracy alone may therefore produce poor performance on some metrics.

D.5. Impact of Architecture, Pretraining Method, and Pretraining Dataset

The section evaluates how architecture, pretraining method, and pretraining dataset affect worst-group accuracy across text and image settings. Results indicate that architecture is dataset dependent, supervised pretraining is usually strongest, and larger, more diverse pretraining datasets generally improve image robustness.

Experimental settings: The evaluation varies text and image architectures, initialization methods, and pretraining datasets using CivilComments, Waterbirds, CheXpert, and NICO++.The experiments include ERM, JTT, and DFR, with both known- and unknown-attribute settings represented.
Text models: BERTBASE remains competitive on CivilComments and outperforms RoBERTaBASE on many tasks, whereas DistilBERT and GPT-2 perform much worse, especially with ERM.These comparisons use test-set worst-group accuracy across text architectures and pretraining methods.
Image architectures: ResNets perform better on CheXpert and Waterbirds, while vision transformers perform better on NICO++, showing that the optimal architecture is dataset dependent.The section reports mixed worst-group-performance results rather than a universal advantage for either architecture.
Pretraining methods: Supervised pretraining generally outperforms self-supervised learning, although some self-supervised methods remain competitive.The findings motivate better self-supervised schemes for subgroup shifts.
Pretraining datasets: Larger and more diverse pretraining datasets generally improve worst-group accuracy, with ImageNet-1K-to-ImageNet-21K-to-SWAG gains especially evident on NICO++ and Waterbirds.The largest impact on worst-group accuracy appears to come from the dataset used to derive the initial weights, although only publicly available combinations are evaluated.

E. Complete Results

The complete-results analysis compares model-selection and attribute-information regimes, from oracle selection with known attributes to worst-class selection when attributes are unavailable. Results are reported across accuracy, precision, F1, calibration, and ranking metrics.

Evaluation settings: The analysis studies three regimes: attributes known in training and validation, unknown in training but known in validation, and unknown in both.These settings are designed to assess how model selection and attribute availability affect subpopulation-shift evaluation.
Evaluation settings: When attributes are known in training and validation, test-set worst-group accuracy serves as an oracle for selecting each algorithm’s best possible performance.This is described as the most ideal scenario.
Evaluation settings: When attributes are unknown in both training and validation, validation worst-group accuracy degenerates to worst-class accuracy for model selection.Algorithms requiring attribute information during training are excluded from the settings where training attributes are unknown.
Reported metrics: The reported metrics include average and worst accuracy, precision, and F1, plus adjusted accuracy, balanced accuracy, AUROC, and ECE.The tables organize these metrics by algorithm.