PaperlayerView exact revision on arXiv

Source-linked AI summary

Underspecification Presents Challenges for Credibility in Modern Machine Learning

Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, D. Sculley

arXiv:2011.03395v2cs.LGstat.ML

TL;DR

The paper addresses why models with strong training-domain performance can behave unpredictably in deployment, focusing on underspecification as multiple equivalent predictors. Across practical pipelines and case studies, it shows divergent deployment-relevant behavior and argues for explicitly testing and addressing this ambiguity.

  • Problem

    ML pipelines can return predictors with equivalently strong training-domain performance even though their deployment-relevant behavior differs.

  • Method

    The paper probes underspecification by perturbing small pipeline choices, retraining predictor ensembles, and evaluating them across multiple domains and stress tests.

  • Results

    Underspecification appears across practical ML applications, and arbitrary choices can produce substantially different robustness, clinical generalization, shortcut reliance, and other deployment-relevant behavior.

  • Takeaways & Limitations

    Models intended for real-world deployment should be explicitly tested for required behaviors and trained or evaluated with application-appropriate inductive biases.

  • Takeaways & Limitations

    The methodology detects but does not fully characterize underspecification, mainly exploring variation from random-seed changes rather than the broader range of pipeline choices.

Abstract

from arXiv · show

ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We show that this problem appears in a wide variety of practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

1. Introduction

Standard iid evaluation can treat predictors as equivalent while leaving their application-relevant inductive biases unconstrained. The paper argues that underspecification makes deployment behavior sensitive to arbitrary training choices and appears across practical ML pipelines.

  • Motivation: Deployment may require models to encode causal or application-specific structure beyond predicting well in the training domain.These structural requirements determine whether models generalize as expected in deployment scenarios.
  • Motivation: Standard ML pipelines evaluate iid predictive performance while remaining agnostic to the inductive biases encoded by trained models.The pipeline is built around a model specification, training data, and iid evaluation.
  • Problem: Predictors with the same iid generalization can show widely divergent behavior in real-world settings, beyond structural training–deployment mismatch.This motivates underspecification as a distinct deployment challenge.
  • Definition: An ML pipeline is underspecified when many distinct weight configurations achieve equivalent held-out iid performance despite fixed model specification and training data.The paper connects this ambiguity to established ideas including deep ensembles, double descent, Bayesian deep learning, and loss landscape analysis.
  • Main claims: When pipelines choose among near-optimal predictors using only iid performance, arbitrary choices can substantially affect new-domain behavior even if a credible predictor exists.The paper demonstrates this sensitivity in simulated, theoretical, and empirical examples, including medical genomics.
  • Main claims: The study finds underspecification across computer vision, medical imaging, NLP, and EHR-based prediction, with stress tests revealing variation rather than uniform degradation.The protocol perturbs iid-performance-preserving choices such as random seeds and probes practically relevant inductive biases.
  • Implications: The authors recommend explicitly testing required behaviors and developing methods that constrain pipelines toward application-appropriate inductive biases without necessarily compromising iid performance.The paper closes with examples and a discussion of implications for deployment practice.

2. Preliminaries and Related Work

The paper defines underspecification through risk-equivalent near-optimal predictors whose inductive biases may differ outside the training distribution. It distinguishes this practical pipeline problem from structural failure modes and reviews stress tests for exposing application-relevant behavior.

  • Core concepts: An ML pipeline maps training data from distribution P to a predictor f in function class F by approximately minimizing predictive risk on P.The predictive risk is defined as the expected loss over training-distribution examples.
  • Core concepts: Underspecification means that many predictors have similar predictive risk, while encoding different inductive biases and generalizing differently on distributions other than P.The near-optimal risk-equivalent set is denoted F* ⊂ F.
  • Related work: The paper studies underspecification as a practical-pipeline phenomenon related to, but distinct from, analyses of loss landscapes, non-unique minimizers, and overparameterization.Its focus is how ambiguity affects inductive biases for generalization beyond the training distribution.
  • Related work: Related work on Rashomon sets, predictive multiplicity, and right-for-the-right-reasons methods likewise recognizes multiple near-optimal solutions with differing interpretability or fairness properties.The paper complements these studies with empirical examples from common machine learning practice.
  • Structural failure modes: Structural failure modes arise when credible inductive biases cannot achieve optimal iid generalization because spurious features are associated with labels during training but not in important settings.Medical examples include models relying on markers of doctors’ diagnostic judgments, such as surgical ink around cancerous skin lesions.
  • Underspecification beyond structural conflict: Underspecification also matters when many predictors generalize well iid but only some align with the intended solution, including in settings without structural conflict.Arbitrary pipeline choices such as optimization procedures can affect which inductive biases are represented.
  • Stress tests: The paper uses contracts and stress tests to examine observable behavior not covered by iid evaluation, including subgroup and shifted-distribution performance.Stress tests probe broader contracts, while shifted evaluations define a test distribution P′ ≠ P.

3. Warm-Up: Underspecification in Simple Models

Simple epidemiological, random-feature, and polygenic-risk models show that predictors with equivalent training-domain performance can behave very differently under forecasting or distribution shift. These examples demonstrate that underspecification can prevent a pipeline from reliably selecting a predictor with the required inductive bias or transfer behavior.

  • Overview: Three simple models show that underspecification obstructs learning predictors with the inductive biases required for deployment.The examples cover epidemic forecasting, random features, and medical genomics.
  • Epidemiological model: Early epidemic data identify only the growth rate β −1/D, leaving many equivalent parameter pairs that produce divergent forecasts beyond Tobs.Despite identical fits, predicted peak infection numbers can differ by orders of magnitude.
  • Epidemiological model: Changing the initialization point or its sampling distribution changes which observation-equivalent epidemic parameters and trajectories the learning procedure returns.The training objective cannot distinguish parameter sets with equivalent growth rates.
  • Random-feature model: Overparameterized random-feature models can have indistinguishable in-distribution errors yet nearly orthogonal prediction functions with different errors on shifted distributions.The analysis studies minimum-norm interpolation and an adversarial mean shift.
  • Random-feature model: The adversarially targeted random-feature predictor exhibits a 3-fold risk increase under the shift, while an independent model experiences very little risk inflation.The shift depends on random training choices and exposes vulnerability that is not shared by all equivalent-risk predictors.
  • Polygenic-risk model: Distinct polygenic-risk predictors using correlated-feature representatives perform comparably in British data but vary much more on non-British data.British and non-British performance is only weakly associated, with Spearman ρ = 0.135 and 95% CI 0.070-0.20.
  • Polygenic-risk model: Because acceptable transfer predictors coexist with others in the underspecified set, this PRS pipeline cannot reliably return a predictor that transfers between populations.The non-British evaluation still shows some average improvement over the demographic baseline, but the margins are highly variable.

4. Underspecification in Deep Learning Models

The paper tests underspecification in production-grade deep learning pipelines by perturbing training choices, generating near-optimal predictor ensembles, and probing their application-specific inductive biases. Across all considered applications, important inductive biases are underspecified, producing meaningful and sometimes unpredictable variation in stress-test behavior.

  • Scope: Across computer vision, medical imaging, NLP, and EHR prediction, pipelines admit near-optimal predictors with different inductive biases and generalization behavior.The protocol empirically tests whether these pipelines return a non-trivial set F∗ of near-optimal predictors.
  • Protocol: The protocol perturbs small pipeline components, such as random seeds or recurrent units, and retrains models to sample different predictors from F∗.This conservative strategy identifies properties constrained or left underspecified by the pipeline without exhaustively exploring F∗.
  • Protocol: Ensemble members are first checked for near-equivalent iid performance, then evaluated on application-specific stress tests probing relevant inductive biases.Variation in stress-test performance provides evidence that the modeling pipeline is underspecified.
  • Validation: The authors assess stress-test variation by its magnitude, its unpredictability from iid performance, and whether it reflects systematic differences between predictors.Unpredictability is assessed with Spearman rank correlation when sample size permits; qualitative analyses may supplement quantitative comparisons.
  • Findings: All considered applications provide evidence that important inductive biases are underspecified, although the evidence is sometimes subtle because the exploration of F∗ is conservative.The case studies discuss application-specific implications after evaluating each modeling pipeline.

5. Case Studies in Computer Vision

Computer-vision pipelines can produce models with similar iid performance but substantially different behavior on distribution-shift stress tests. This ambiguity appears across synthetic corruptions and natural changes in object settings.

  • Experimental protocol: 50 ResNet-50 and 30 BiT models differing in random seeds reveal substantial ambiguity in stress-test performance despite fixed iid training performance.The ensembles differ only in random seed or fine-tuning initialization choices.
  • Stress-test design: Deep image-classification models are evaluated on ImageNet-C synthetic corruptions and ObjectNet changes in object settings to probe robustness under distribution shift.ImageNet-C applies realistic corruptions at varying intensities, while ObjectNet varies the settings and configurations in which objects are observed.
  • ImageNet-C results: On some ImageNet-C tasks, variation in accuracy is an order of magnitude larger than variation in iid performance, with weak correlations between iid and stress-test results.All 95% confidence intervals for Pearson correlations between iid performance and stress-test performance contain zero.
  • ObjectNet results: ObjectNet accuracy variability is 2x the standard-test variability for ResNet-50 and 5x for BiT, although the difference is less striking than for ImageNet-C.The relationship between standard-test and ObjectNet accuracy is slightly stronger than for ImageNet-C.
  • ObjectNet results: Models disagree substantially more often on ObjectNet than on ImageNet, showing that identical training pipelines can yield different predictions under changed object settings.This disagreement persists when restricting ImageNet to classes also appearing in ObjectNet.
  • Implications: Underspecification persists in scaled-up BiT models, suggesting that scaling data and model size does not eliminate ambiguity in out-of-distribution behavior.The authors suggest underspecified dimensions may account for a larger share of available out-of-distribution performance headroom as models scale.

6. Case Studies in Medical Imaging

Medical-imaging models differing only by fine-tuning random initialization can behave differently across cameras and skin types. These differences are largely hidden by iid evaluation and matter for clinical deployment.

  • Clinical settings: The medical-imaging case studies examine diabetic-retinopathy and skin-condition classifiers for deployment settings involving different cameras and skin types.The retinal model uses an ImageNet-pretrained Inception-V4 backbone and retrospective fundus images from the United States and India.
  • Ophthalmological imaging: Two retinal classifiers with different fine-tuning seeds have nearly identical calibration for camera types 1–4 but qualitatively different calibration for held-out camera type 5.The figure reports error bars of ±2 standard errors.
  • Ophthalmological imaging: Ten retinal models differing only in fine-tuning initialization are evaluated on images from a camera type absent during training.The held-out camera test contains 287 images, compared with 3,712 images in the standard test set.
  • Ophthalmological imaging: AUC variability on the held-out camera type exceeds variability on the standard test set, with z = 2.47 and one-sided p = 0.007.The comparison uses jackknife standard errors to assess whether sample-size differences explain the larger variability.
  • Dermatological imaging: Across skin-type strata, accuracy varies more between identically trained models than overall test accuracy, particularly for skin types II and IV.Skin types II and IV comprise 10.7% and 19.6% of the test data, respectively.
  • Clinical implications: These findings indicate that clinical models should be tailored and tested for the settings and populations in which they will be deployed rather than expected to generalize automatically.The authors identify underspecified inductive biases as a source of complications for deployment across new clinical settings or populations.

7. Case Study in Natural Language Processing

The NLP case study shows that underspecification affects shortcut learning in both pretraining and fine-tuning: predictors with similar task performance can encode substantially different gendered and stereotypical correlations. These correlations vary with random seeds, so individual stress-test results do not necessarily generalize to an architecture or learning algorithm.

  • Experimental setup: Five BERT checkpoints and repeated fine-tunings differing only in random seeds were evaluated on shortcut-focused stress tests.The study used five BERT large-cased pretraining instances and fine-tuned each checkpoint 20 times for tasks requiring fine-tuning.
  • Task performance and gender correlations: STS predictors achieved correlation-based accuracy ranging from 0.87 to 0.90 despite differing shortcut behavior.The metric was correlation with human-provided similarity scores.
  • Task performance and gender correlations: Gendered correlations ranged from 0.3 to 0.7 on STS and from 0.26 to 0.51 on pronoun resolution.These ranges describe correlations with U.S. Bureau of Labor Statistics gender and occupation statistics.
  • Task performance and gender correlations: Test accuracy had only weak relationships with gendered correlation on STS-B and pronoun resolution.Spearman ρ was 0.21 for STS-B and 0.08 for pronoun resolution.
  • Stereotype stress tests: Stereotype stress-test scores varied substantially across identical BERT checkpoints, with an overall ICAT range of 3.35.The range exceeded the gap between the top six public leaderboard models, while aggregate metrics could appear more stable than demographic metrics.
  • Conclusions: Shortcut reliance was sensitive to both fine-tuning and pretraining seeds, indicating that pretraining can encode more or less robustness.The authors conclude that individual stress-test results describe individual checkpoints rather than architectures or learning algorithms.

8. Case Study in Clinical Predictions from Electronic Health Records

The EHR case study finds that clinically similar AKI predictors can rely differently on operational signals such as time of day and lab ordering. Interventions exposing this reliance produced dispersed performance and changed clinical decisions, showing that the extent of reliance is underspecified.

  • Experimental setup: The study trained ensembles of AKI RNN predictors from multiple random seeds and recurrent cell types using EHR data.The data included de-identified records from 703,782 patients across multiple U.S. sites, and analyses focused on a 48-hour lookahead.
  • Experimental setup: Ensemble predictors achieved tightly constrained normalized PRAUC values between 34.59 and 36.61.Normalized PRAUC was adjusted for the prevalence of AKI events.
  • Reliance on operational signals: Stress tests shifted feature times and removed lab tests to probe reliance on operational signals in the EHR.The interventions targeted the timing and number of recorded labs, including a combined time-shift and lab-selection perturbation.
  • Reliance on operational signals: Intervened-data performance was worse and more dispersed than standard-test performance, especially under both interventions.Reliance on time of day and lab content varied with recurrent unit and random initialization.
  • Reliance on operational signals: Two LSTM predictors differing only in random seed showed substantially different risk changes under time-shift and lab interventions.Both changed their predicted risks, but one was far more sensitive and produced different risk distributions across shifted time ranges.
  • Clinical consequences: Changed risk scores crossed calibrated thresholds at different patient-timepoints across models, potentially causing tardy or missed AKI predictions.The models also differed substantially in the number of flipped decisions.
  • Conclusions: The extent of reliance on lab-order patterns was underspecified, and timestamp ablation preserved identical iid predictive performance in a preliminary result.This suggests that training constraints can modulate reliance without sacrificing iid performance.

9. Discussion: Implications for ML Practice

Underspecification is widespread and can make deployment behavior depend on arbitrary training choices despite similar iid performance. The paper argues for application-specific stress tests and selection criteria that encode required inductive biases.

  • Underspecification is a distinct failure mode across practical ML pipelines and can prevent models from encoding generalizable inductive biases.
  • The study’s methodology likely underestimates underspecification because it mainly varies random seeds rather than systematically characterizing all pipeline choices.
  • Application-specific stress tests should assess whether model behavior remains stable across deployment-relevant dimensions, rather than relying only on standardized benchmarks.
  • Selecting predictors with application-specific constraints may be preferable to averaging all near-optimal predictors when stress-test behavior varies substantially.
  • Application-specific regularization may incorporate domain expertise without necessarily sacrificing iid predictive performance.

Appendix A. Computer Vision: Marginaliztaion versus Model Selection

The computer-vision analysis compares selecting the best ensemble member with averaging subsets of 50 ResNet-50 predictors. When stress-test variability is large, selecting one model can outperform large ensemble averages.

  • Iid performance improves with ensemble size, and this improvement is associated with correlated stress-test improvements.
  • When ensemble members vary substantially on stress tests, selecting the best single model can outperform large ensemble averages.
  • Figure 16 compares the best individual ResNet-50 predictor with averages formed from differently sized subsets of a 50-model ensemble.

Appendix B. Natural Language Processing: Analysis of Static Embeddings

Static word2vec embeddings exhibit underspecification: models trained with different random seeds achieve nearly identical analogy performance but differ in socially relevant associations. Race-association significance can depend on the seed.

  • Twenty 500-dimensional word2vec models trained with different random seeds score between 76.2% and 76.7% on word analogy.
  • The analysis uses WEAT scores to quantify relative similarities between target-word sets and attribute-word sets.
  • WEAT scores closer to zero indicate smaller relative-similarity differences, while permutation tests assess statistical significance.
  • Gender associations are strong and consistent across models, whereas race-related association significance varies substantially with random seed.

C.1 Lab Order Patterns and Time of Day

Creatinine sampling varies by time of day even though AKI prevalence is largely stable, creating a spurious relationship between sampling time and AKI risk. Lab-count distributions also differ across time buckets.

  • AKI prevalence remains largely constant across time windows, but creatinine is sampled more frequently between 12am and 6am.
  • Conditional on a creatinine sample being taken, AKI prevalence is higher in the latter time windows because sampling frequency differs across windows.
  • Figure 18 shows distinct lab-count distributions between the first two and second two time buckets.

C.2 Details of Predictor Performance on Intervened Data

The evaluation stratifies model performance and decision changes by time of day under interventions affecting time and lab-order composition. These shifts reduce performance and affect ensemble models differently despite comparable training-domain evaluation.

  • Stratified evaluation: Performance is evaluated separately across time-of-day buckets for all 15 models on standard and intervened test data.The interventions are the Shift and Shift+Labs datasets.
  • Time-of-day analysis: The time-of-day plots show lab-value counts and risk-prediction distributions for morning, afternoon, and evening patient-timepoints.The risk distributions compare original test data with time-shift and combined time-shift-plus-lab perturbations.
  • Intervened performance: Perturbing correlations among AKI labels, time of day, and lab counts decreases performance and widens the performance bounds across models.The widening indicates differential effects across model instances.
  • Decision changes: Decision flips under time-shift and lab-order interventions are counted by direction for patient-timepoints with creatinine measurements.The table compares models that differ only in random seed.

C.3 Preliminary Ablation Experiment

The ablation removes timestamp information while preserving similar test-set performance, then probes whether time-of-day information remains recoverable through correlated features. The results support collinearity as a source of underspecification in this clinical prediction setting.

  • Ablation: Removing the timestamp feature yields normalized PRAUC of 0.368, compared with 0.346 to 0.366 for the ensemble models retaining it.The ablated LSTM therefore has similar test-set population performance to the other predictors.
  • Residual signal: The ablated model predicts time of day with 85% accuracy using an auxiliary head without backpropagation.This indicates that time-related signal remains available through other features.
  • Mechanism: The authors interpret these findings as evidence that collinear features carry time-of-day signal that the training pipeline does not separate.The interpretation parallels the genomics example’s treatment of correlated features.
  • Genomics context: Polygenic risk scores are weighted sums of associated genotypes, but linkage disequilibrium creates highly correlated genetic variants that require representative selection.This provides the genomic context for feature collinearity discussed in the ablation.
  • Genomics context: Prediction accuracy from one ancestry group does not necessarily generalize to other ancestry groups, raising concerns about clinical use of PRS and GWAS.The passage notes that widely used GWAS are disproportionately based on European-ancestry individuals.

D.2 Methods

This section constructs genomic feature sets by varying representatives of correlated variant clusters, evaluates them within and outside the training ancestry, and develops a random-features framework for sensitivity to training choices. The results show that equivalent in-distribution error can coexist with substantial predictor sensitivity.

  • Genomics methods: The study varies genomic predictors by selecting one representative from each of 129 highly correlated variant clusters.It creates 1,000 feature sets, including one set of index variants and 999 sets sampled uniformly from the clusters.
  • Genomics methods: Models are trained on British individuals and evaluated on British and held-out “non-British” individuals using genomic and demographic features.The non-British group is excluded from training and used solely for evaluation.
  • Genomics results: Genomic models show drastically greater sensitivity in the “non-British” evaluation set, where their improvement over the demographic baseline is smaller and highly variable.Sensitivity is lower in the British evaluation and training settings.
  • Genomics results: The original index-variant model outperforms alternative cluster representatives in British evaluation but not in the “non-British” evaluation.Thus, the preferred representative depends on the evaluation domain.
  • Genomics results: Across this analysis, prediction errors become more sensitive and larger when the evaluation domain differs from the training domain.The passage frames robust PRS generalization across underrepresented ancestries as an open question.
  • Random-features theory: The theoretical model treats training-procedure choices as parameters that can change the learned predictor, test distribution, and in-distribution risk.The setup includes arbitrary choices such as initialization and step-size schedule.
  • Random-features theory: For large N, n, and d, the in-distribution error is essentially the same for most choices of first-layer weights W.This formalizes equivalent training-domain performance across many predictors.
  • Random-features results: As neuron count increases, normalized sensitivity S/R grows rapidly and approaches 2 over a broad interval, corresponding to nearly orthogonal model deviations.The analytical prediction is compared with simulations at d = 40.

E.5 Random features model: Distribution shift

The distribution-shift analysis constructs small, model-specific mean shifts and evaluates how they affect random-features predictors. The resulting risk expression isolates a predictor-dependent shift term, showing how a small perturbation can selectively increase error.

  • Distribution-shift setup: The analysis studies a mean shift where test inputs are formed by adding a perturbation to a random spherical component.The shift is designed relative to one model’s first-layer weights.
  • Risk under shift: The perturbation is chosen adversarially for a reference weight matrix W0 under an ℓ2-norm constraint, while the worst-case perturbation problem remains future work.The construction aims to increase risk for W0 while changing risk little for a typical independent W.
  • Risk under shift: For a predictor with weights W, the shifted risk equals the baseline risk plus a squared predictor-dependent alignment term, up to oP(1).The additional term involves β0 − μ1W^Tθ̂(W) and the perturbation x0.
  • Derivation: The reported shifted-test-error expression includes the baseline risk plus a term labeled T1(W, W0) and an oP(1) remainder.The expression is presented as the final distribution-shift result.
  • Derivation: The derivation replaces random-feature nonlinearities with a noisy linear-features model and uses universality results to argue that the substitution is asymptotically negligible.The analysis then derives the claimed expressions through conditional expectations and algebraic substitution.
Loading 2011.03395v2…