Underspecification Presents Challenges for Credibility in Modern Machine Learning

• Credible ML predictors must encode application-relevant structure, because this structure determines whether their behavior generalizes as expected in deployment.The required structure may involve invariant causal phenomena in medical diagnostics or task-specific properties in natural language processing.Source: §1 · p. 2
• Standard iid evaluations assess predictive performance but are agnostic to the inductive biases that models use, leaving concerns such as spurious correlations and shortcut learning.Source: §1 · p. 2
• Underspecification occurs when fixed model specifications and training data permit many distinct predictors with equivalent held-out iid performance, whose deployment behavior can nevertheless diverge widely.This differs from structural-conflict failures, where training and deployment domains are misaligned and iid-optimal predictors necessarily incorporate spurious associations.Source: §1 · pp. 2–3
• When a pipeline selects among many near-optimal iid predictors using only iid performance, arbitrary choices can determine whether it returns a predictor that encodes the right structure and behaves well in deployment.The paper demonstrates this mechanism in simulated, theoretical, and medical-genomics examples where iid performance remains fixed while new-domain performance changes substantially.Source: §1 · p. 3
• The paper finds evidence of underspecification across production-grade deep-learning applications in computer vision, medical imaging, NLP, and EHR prediction.Stress tests reveal variation between predictors under iid-performance-preserving choices such as random seeds, with downstream effects on robustness, fairness, and causal grounding.Source: §1 · p. 3
• The findings show that underspecification can degrade predictor credibility even when prediction is aligned with application goals, motivating explicit tests of required behaviors and training methods that constrain models toward credible inductive biases.Convenience choices such as initialization schemes or step-size schedules may alter substantive real-world behavior without affecting iid performance, while such constraints need not compromise iid performance.Source: §1 · p. 3

2.1 Underspecification

• Underspecification allows multiple near-optimal predictors with similar training-distribution risk but different inductive biases and generalization behavior outside that distribution.Validation on an iid hold-out set establishes aggregate performance on future data from P, but cannot distinguish among risk-equivalent predictors in F∗.Source: §2.1 · p. 4
• This work complements loss-landscape, overparameterization, Rashomon-set, predictive-multiplicity, and “right for the right reasons” research by studying practical pipelines and their empirical equivalence classes.The practical pipelines include ubiquitous techniques such as early stopping, while the paper focuses on inductive biases relevant to generalization beyond P.Source: §2.1 · pp. 4–5

2.2 Shortcuts, Spurious Correlations, and Structural vs Underspecified Failure Modes

• Structural failures reflect unavoidable tension between iid-optimal prediction and credible inductive biases, whereas underspecified failures arise when training data cannot distinguish credible predictors from spurious shortcuts.Identical pipelines can produce distinct shortcuts across random seeds, and these underspecified problems occur across machine-learning applications.Source: §2.2 · p. 5

2.3 Stress Tests and Credibility

• Stress tests operationalize predictor credibility by falsifying explicit behavioral contracts on designed inputs, probing behaviors that iid evaluations do not test.They are intended to reveal failures to generalize as required for real-world application and are increasingly part of evidence standards in applied fields.Source: §2.3 · pp. 5–6
• Stratified, shifted, and contrastive evaluations test subgroup consistency, performance under distribution changes, and localized responses to input modifications, respectively.Contrastive evaluations use matched transformed inputs and can detect disagreement when transformations should preserve labels; shifted evaluations include transformations, collection protocols, or instruments that define P′ ≠ P.Source: §2.3 · p. 6

3.1 Underspecification in a Simple Epidemiological Model

• In the SIR epidemic model, early infection data identify only the growth rate β − 1/D, so observation-equivalent parameter pairs produce forecasts with peak infections orders of magnitude apart.Changing the initialization point D0 or its sampling distribution changes the returned trajectory distribution.Source: §3.1 · p. 7

3.2 Theoretical Analysis of Underspecification in a Random Feature Model

• In the random-feature model, independently trained predictors have indistinguishable in-distribution error but can be nearly orthogonal and respond very differently to shifted test distributions.A small adversarial mean shift can produce a 3-fold risk increase for the targeted predictor while causing very little risk inflation for an independently sampled predictor.Source: §3.2 · p. 9

3.3 Underspecification in a Linear Polygenic Risk Score Model

• For polygenic risk scores, correlated genetic features create near-optimal predictors with comparable training and iid performance but widely varying transfer to a non-British population.British and non-British test performance are only weakly associated (Spearman ρ = 0.135; 95% CI 0.070-0.20), and the PLINK heuristic performs no better than the ensemble under shift.Source: §3.3 · pp. 10–11, Figure 3 · p. 11

Scope and contribution

• Across three domains, near-optimal deep-learning predictors achieve similar iid performance yet encode different inductive biases that produce different generalization behavior.The domains are computer vision, including medical imaging, natural language processing, and clinical risk prediction from electronic health records.Source: §4 · p. 11

Experimental protocol

• Perturbing small pipeline components, such as training random seeds or RNN recurrent units, and retraining constructs a conservative ensemble of predictors from the near-optimal set F∗.This strategy does not exhaustively explore F∗; it indicates which predictor properties are well-constrained versus underspecified.Source: §4 · p. 11
• Ensemble members are first verified to have near-equivalent iid performance, then evaluated on application-specific stress tests whose performance variability signals practically important underspecification.The stress tests probe whether predictors encode appropriate inductive biases for the application.Source: §4 · p. 11

Validation criteria

• Stress-test variability is treated as genuine evidence of underspecification when it is substantial, unpredictable from iid performance, and systematic across application-relevant predictor differences.The protocol uses magnitude comparisons, Spearman rank correlation when sample size permits, and quantitative or qualitative analyses of stress-test outputs.Source: §4 · p. 12

Results

• Every examined case provides evidence that important inductive biases are underspecified, though the strength of evidence varies because the exploration of F∗ is conservative.The section closes each case study with an application-specific discussion of how the results relate to research in that field.Source: §4 · p. 12

5. Case Studies in Computer Vision

• The study trained 50 ResNet-50 and 30 BiT predictors differing only in randomization, achieving 75.9% ± 0.11 and 86.2% ± 0.09 iid top-1 accuracy, respectively.The BiT models shared a JFT-300M-trained checkpoint and varied fine-tuning seeds and initialization distributions, while ResNet-50 models used identical pipelines with different random seeds.Source: §5 · pp. 12–13
• The ensembles were evaluated on ImageNet-C’s high-intensity synthetic corruptions and ObjectNet’s natural changes in object settings and configurations.These benchmarks stress robustness beyond the standard ImageNet validation distribution.Source: §5 · p. 13, §5.2 · p. 13

5.1 ImageNet-C

• On ImageNet-C, accuracy variation across predictors was an order of magnitude larger than iid variation for both ResNet-50 and BiT, with weak correlations across tasks.All 95% confidence intervals for Pearson correlations between iid and stress-test performance, and between stress tests, contained zero.Source: §5.1 · p. 13

5.2 ObjectNet

• On ObjectNet, stress-test accuracy standard deviations were 2x larger for ResNet-50 and 5x larger for BiT than on the matched ImageNet subset, with p = 0.002 and p = 0.000.Predictors also disagreed substantially more on ObjectNet than ImageNet, while ObjectNet accuracy correlations with standard accuracy were 0.22 (−0.06, 0.47) and 0.47 (0.13, 71) for ResNet-50 and BiT.Source: §5.2 · p. 13, Table 1 · p. 14, Table 2 · p. 14

5.3 Conclusions

• Random-seed changes produced substantially different stress-test performance despite similar iid performance, showing that underspecification persists even in scaled-up BiT models.The relevant inductive biases are weakly differentiated by the iid task, and underspecified dimensions may occupy more out-of-distribution performance headroom as models scale.Source: §5.3 · p. 14

6. Case Studies in Medical Imaging

• The section examines underspecification in two high-stakes medical imaging models: retinal classification for diabetic retinopathy and edema, and clinical skin-condition classification.The retinal model uses an ImageNet-pretrained Inception-V4 backbone fine-tuned on fundus images from the United States and India; the dermatology model uses the same backbone strategy.Source: §6 · p. 14, §6.1 · p. 14, §6.2 · p. 17

6.1 Ophthalmological Imaging

• AUC variability was larger for retinal models on an unseen camera type than on standard testing, with z = 2.47 and one-sided p = 0.007.Two models had similar calibration across training cameras but markedly different calibration on the held-out camera, indicating systematically different behavior revealed only by stress testing.Source: §6.1 · pp. 16–17, Figure 6 · p. 16, Figure 7 · p. 16

6.2 Dermatological Imaging

• Skin-type accuracy varied more across identically trained models than overall accuracy, particularly for types II and IV, with type IV variation appearing systematic.Permutation tests found variation consistent with sampling noise for types III, V, and II, but not type IV (p = 0.03, n = 798).Source: Figure 6 · p. 16, §6.2 · p. 17

6.3 Conclusions

• The case studies show that underspecification complicates deployment because inductive biases enabling generalization to new clinical settings or populations are not uniquely determined.The authors therefore argue that models should be tailored and tested for their intended clinical settings and populations.Source: §6.3 · p. 18

7. Case Study in Natural Language Processing

• The pretrain/fine-tune NLP pipeline can produce shortcut-dependent predictors from identical training procedures, as shown across gender, NLI, and pretrained-language-model stress tests.The experiments use five BERT large-cased pretraining seeds and, where applicable, 20 fine-tuning seeds per checkpoint.Source: §7 · p. 18

7.1.2 Pronoun resolution

• On pronoun resolution, predictors achieve accuracy from 0.960 to 0.965, yet their gendered correlations differ substantially under templates where linguistic cues already identify the correct antecedent.The stress-test metric is the difference in predictive probability for the profession conditioned on the gender of the pronoun.Source: §7.1.2 · p. 19, §7.1.3 · p. 19

7.1.3 Gender correlations and underspecification

• Gender-shortcut correlations vary widely across equally trained predictors: correlations range from 0.3 to 0.7 on STS and from 0.26 to 0.51 on pronoun resolution.Test accuracy has only weak relationships with gendered correlation—Spearman ρ = 0.21 for STS-B and 0.08 for pronoun resolution—so accurate predictors need not encode strong gendered correlations.Source: §7.1.3 · p. 19, Pronoun resolution · p. 20
• Pretraining random seeds systematically affect gendered shortcut reliance, with predictors from different pretrainings clustering into weaker or stronger gender-correlation patterns.PCA further shows that the first principal component explains 22% of score-deviation variation and aligns with stereotypical profession-gender associations.Source: Pronoun resolution · p. 20, Figure 9 · p. 21

7.2 Stereotypical Associations in Pretrained Language Models

• Identical BERT checkpoints differ by 3.35 in overall StereoSet ICAT score, exceeding the gap among the public leaderboard’s top six models despite differing sizes, architectures, and training data.Variation is narrower for LM Score than Stereotype Score, and aggregate metrics can appear stable while demographic-specific stereotypes differ.Source: Figure 10 · p. 22, §7.2 · p. 22

7.3 Spurious Correlations in Natural Language Inference

• NLI stress-test performance varies substantially within and between pretraining checkpoints, is weakly predicted by MNLI validation accuracy, and is typically weakly correlated across different stress tests.The study replicates fine-tuning sensitivity and extends it to pretraining sensitivity, showing that underspecified inductive biases span multiple dimensions.Source: §7.3 · p. 23, Figure 11 · p. 24, Figure 12 · p. 26

7.4 Conclusions

• Random-seed changes at both pretraining and fine-tuning induce large variation in shortcut reliance, while standard benchmark performance can remain nearly unchanged.This variation appears across gendered correlations, stereotypical associations, and NLI stress tests, indicating that individual stress-test results characterize checkpoints rather than architectures or learning algorithms.Source: §7.4 · pp. 23, 26

8. Case Study in Clinical Predictions from Electronic Health Records

• 55.8% accuracy across all episodes and 90.2% for dialysis-associated episodes were achieved when detecting AKI up to 48 hours in advance.These results motivate examining whether strong discrimination reflects physiological signals or institution-specific operational factors.Source: §8 · p. 26

8.1 Data, Predictor Ensemble, and Metrics

• The study analyzed 15 AKI RNN predictors across three recurrent cell types and five random seeds, whose normalized PRAUC values ranged from 34.59 to 36.61.The data comprised de-identified EHRs from 703,782 patients across multiple US sites, and the analysis focused on a 48-hour lookahead horizon.Source: §8.1 · p. 27

8.2 Reliance on Operational Signals

• Operational-signal perturbations caused worse, more dispersed performance, model-dependent risk changes, and different threshold-crossing decisions, demonstrating underspecified reliance on lab-order patterns.Reliance varied with recurrent unit and random initialization, and changed risks could lead to tardy or missed AKI predictions.Source: §8.3 · p. 28, §8.2 · p. 28

8.3 Conclusions

• Because timestamp-feature ablation achieved identical iid predictive performance, training constraints could reduce reliance on lab-ordering patterns without sacrificing iid performance.Operational signals include test timing and lab-panel composition, which may change across institutions or over time.Source: §8.2 · p. 27, §8.3 · p. 29

Implications

• Underspecification is widespread and distinct from structural train–deployment mismatch, allowing arbitrary choices such as random initialization to determine substantive model decisions.The authors argue that better interfaces for incorporating domain knowledge are needed in ML pipelines.Source: §9 · p. 29

Measurement

• The study likely underestimates underspecification because it mainly varies random seeds, while initialization, parameterization, optimization, data representation, and hyperparameters can impose additional inductive biases.The authors conjecture that systematically mapping the iid-equivalent predictor set could measure underspecification more fully, drawing on loss-landscape exploration and sensitivity analysis.Source: §9 · pp. 29–30

Evaluation

• Models should be tested on application-specific tasks with stable performance, using domain-informed stress tests that cover deployment requirements because distinct stress-test results may correlate weakly.Medical risk prediction may require customized tests across temporal, demographic, and operational dimensions, favoring evolving stress-test practices over fixed benchmarks.Source: §9 · pp. 30–31

Training

• Resolving underspecification requires application-specific criteria or constraints for selecting predictors from F*, rather than assuming that marginalizing across F* improves stress-test performance.Causal DAGs (Schölkopf, 2019) and explanations are proposed as flexible frameworks for expressing such requirements.Source: §9 · p. 31
• Application-specific regularization may improve inductive biases without sacrificing iid generalization, since robustness–accuracy and bias-reduction tradeoffs need not be fundamental (Raghunathan et al., 2020; Webster et al., 2020).The authors suggest constraining predictors toward approximate causal structure as a way to incorporate domain expertise while retaining modern ML prediction ability.Source: §9 · p. 31

Empirical comparison

• ImageNet iid performance improves as more ResNet-50 ensemble members are averaged, with correlated improvements on ImageNet-C and ObjectNet stress tests.Figure 16 compares the best member of a 50-model ensemble with averages over differently sized subsets.Source: Figure 16 · p. 43, Appendix A · p. 43

Empirical support

• Selecting the best single model can outperform large ensemble averages when stress-test performance varies substantially across the ensemble.The section argues that averaging underspecified predictors does not guarantee outperforming the best member, unlike the expected iid benefit of marginalization.Source: Appendix A · p. 43

Empirical comparison

• Higher stress-test performance variance requires averaging more predictors to surpass the best single model, and the full 50-model ensemble never does so for the pixelate task.Marginalization surpasses the best predictor more slowly on stress tests than on the iid evaluation.Source: Appendix A · p. 43, Figure 16 · p. 43

Appendix B. Natural Language Processing: Analysis of Static Embeddings

• The static-embedding analysis extends the study of underspecification beyond BERT-based NLP models to word2vec embeddings Mikolov et al. (2013).It examines stereotypical associations involving race, gender, and age, previously studied by Bolukbasi et al. (2016).Source: Appendix B · p. 44
• Static word2vec models achieved nearly identical word-analogy performance despite differing random seeds.Twenty 500-dimensional models scored between 76.2% and 76.7%.Source: Appendix B · p. 44
• The analysis used WEAT to quantify relative associations between demographic target and attribute word sets.Lower absolute differences in relative similarity indicate smaller encoded bias, with tests covering gender, race, disability, and age.Source: Appendix B · p. 44
• Despite consistent analogy scores, embeddings showed strong and consistent gender associations across random seeds.Statistical significance was assessed with permutation tests comparing observed scores with scores after randomly shuffling target words.Source: Appendix B · p. 44
• Race-related associations varied substantially across seeds, with statistical significance often depending on the random seed.The sensitive demographic axes also depended on the training dataset and could differ for embeddings trained on other corpora Babaeianjelodar et al. (2020).Source: Appendix B · p. 44

Appendix C. Clinical Prediction with EHR: Additional Details and Supplementary Ablation Experiment

• The EHR appendix adds descriptive statistics on AKI prevalence to the clinical prediction analysis.It provides additional details and results for the model analyzed by Tomašev et al. (2019a).Source: Appendix C · p. 44
• The EHR appendix also summarizes model performance across different time slices and dataset shifts.These are presented as additional summaries accompanying the clinical prediction analysis.Source: Appendix C · p. 44

Lab Order Patterns and Time of Day

• The number of laboratory tests follows distinctly different distributions across the first versus second pair of time-of-day buckets.Figure 18 compares lab-count distributions across four time buckets and shows a clear separation between the first two and second two buckets.Source: §C.1 · p. 45
• Stable AKI prevalence contrasts with more frequent overnight creatinine sampling, producing higher conditional AKI prevalence in the latter two time windows.The test-set AKI prevalence is 2.269%, while creatinine samples are taken more frequently from 12am to 6am.Source: Table 5 · p. 45, §C.1 · p. 45

C.2 Details of Predictor Performance on Intervened Data

• The analysis stratifies each of 15 model instances by time bucket on standard and intervened test data, including windows containing creatinine samples.Table 6 reports normalized PRAUC for Test, Shift, and Shift+Labs conditions and the maximum inter-instance difference.Source: §C.2 · p. 45, Table 6 · p. 47
• The accompanying distribution describes the average number of observed lab values across time of day in sampled test-set trajectories.Violin plots summarize lab-value counts across 100000 time steps.Source: Figure 18 · p. 46
• Performance decreases and bounds widen after perturbing AKI label, time of day, and lab-count correlations, with shifts affecting ensemble models differentially.The effects appear in individual patient-timepoint risk predictions under time-shift and combined time-shift/lab perturbations.Source: §C.2 · p. 48, Figure 19 · p. 48, Figure 20 · p. 48
• Time-shift and lab-order interventions produce random-seed-dependent decision flips, with direction counts changing and flipped patient-timepoints largely disjoint across seeds.Table 7 reports changes between at-risk and not-at-risk decisions for patient-timepoints with creatinine measurements.Source: §C.2 · p. 48, Table 7 · p. 49

C.3 Preliminary Ablation Experiment

• The ablation experiment tests whether a predictor’s signals can be modified without affecting iid performance.The pipeline was rerun with an LSTM after removing the timestamp feature entirely.Source: §C.3 · pp. 49–50
• Normalized PRAUC was 0.368 after timestamp ablation, comparable to the 0.346–0.366 range of predictors retaining that feature.The ablated LSTM therefore achieved similar test-set population performance to the rest of the ensemble.Source: §C.3 · p. 50
• 85% auxiliary-head accuracy for predicting time of day shows that timestamp-related information remains encoded through alternative feature correlations.The auxiliary head used no backpropagation, suggesting that the training pipeline cannot disentangle these correlated signals.Source: §C.3 · p. 50

Appendix D. Genomics: Full Experimental Details

• Appendix D provides full details of a random-featurization experiment using linear models in genomic medicine.It also briefly reviews the relevant research areas.Source: Appendix D · p. 50

D.1 Background

• GWAS identifies genetic variants associated with traits, and PRS combine associated genotypes as weighted sums derived from GWAS (Wray et al. (2007); International Schizophrenia Consortium et al. (2009)).PRS construction must account for correlated variants caused by linkage disequilibrium, commonly by selecting one representative from each correlated cluster (International Schizophrenia Consortium et al. (2009); CARDIoGRAMplusC4D Consortium et al. (2013)).Source: §D.1 · p. 50
• PRS prediction accuracy derived in one ancestry group does not necessarily generalize to others (Martin et al. (2017); Duncan et al. (2019); Berg et al. (2019)).This limitation is especially relevant because most widely used GWAS participants are of European ancestry, despite that group representing less than a quarter of the global population (Morales et al. (2018)).Source: §D.1 · p. 50
• Limited ancestry diversity in GWAS has raised scientific and ethical concerns about the clinical use of PRS and GWAS (Martin et al. (2019); Need and Goldstein (2009); Popejoy and Fullerton (2016)).PRS show potential for identifying high-risk individuals for some common diseases when derived and tested within one ancestry group (Khera et al. (2018)).Source: §D.1 · p. 50

D.2 Methods

• Linear regression models predicted IOP from demographics plus one 129-variant feature set, or demographics alone, using L2 regularization selected by 10-fold cross-validation.The phenotype was intraocular pressure, and genomic models were compared with a demographic baseline.Source: §D.2 · pp. 50–51
• Researchers identified 4,054 IOP-associated variants, clustered them into 129 groups, and created 1,000 feature sets by selecting one representative variant per cluster.The clustering used PLINK with index-variant p-value threshold 5 × 10−8, associated-variant threshold 5 × 10−6, r2 threshold 0.5, and 250 kb radius.Source: §D.2 · pp. 50–51, Table 8 · p. 51
• Models trained on 82,309 British individuals were evaluated on 9,662 British and 14,898 non-British individuals, isolating ancestry-domain effects.Britishness was defined using z-scored distance in the top-10 genetic principal-component space; non-British individuals were excluded from training.Source: §D.2 · p. 51
• Genomic models showed drastically greater sensitivity and more variable improvement over demographics in non-British than British evaluation, with mostly uncorrelated performance (r = 0.131).This indicates that prediction errors become more sensitive and larger when evaluation shifts from the training domain.Source: §D.2 · pp. 51–52
• The PLINK index-variant model outperformed alternative cluster representatives in British evaluation but not non-British evaluation, so training-domain representatives were not consistently best elsewhere.The analysis used 129 clusters of highly correlated IOP-associated variants distributed across 16 chromosomes.Source: §D.2 · pp. 50–52, Table 8 · p. 51

Appendix E. Random Feature Model: Complete Theoretical Analysis

• The analysis tests whether arbitrary training choices strongly affect learned models and whether nearby test distributions can selectively increase one model’s error.These hypotheses motivate the appendix’s definitions of model sensitivity and distribution-shift risk.Source: Appendix E · p. 52

E.1 General Definitions

• The framework defines predictors fτ for regression under square loss, distinguishes in-distribution risk R(τ, P) from shifted-distribution risk, and measures dependence on training choice τ.The choice τ can represent initialization or optimization choices, while shifted risk considers distributions in a neighborhood of the training distribution.Source: §E.1 · p. 52

E.2 Random Featurization Maps

• Random featurization maps turn overparameterized prediction into linear fitting on φτ(x), encompassing lazy neural networks, selected biological features, and random features with random first-layer weights.The asymptotic analysis treats N, n, and d as jointly large under proportional scaling.Source: §E.2 · p. 53

E.3 Random features model: Risk

• For large N, n, and d, the in-distribution error is essentially the same for most realizations W of the random first-layer weights.The asymptotic risk characterization is obtained for a more general noisy ridge-regression random-features model.Source: §E.3 · p. 54

E.4 Random features model: Sensitivity to random featurization

• As neuron count increases, normalized sensitivity S/R grows rapidly and remains near 2 over a broad interval, corresponding to nearly orthogonal errors from two random models.Sensitivity is small in the classical underparametrized regime N/n → 0.Source: §E.4 · p. 57

E.5 Random features model: Distribution shift

• An adversarial mean shift of norm at most ∆ can greatly increase the selected model’s risk while causing only a small change for an independent typical random feature realization.The construction projects the perturbation orthogonally to β0 to avoid increasing risk similarly for other choices, and its ℓ∞ norm is approximately ∆(2 log d)/d.Source: §E.5 · pp. 57–58

E.6 Random features model: Derivation Eq. (29)

• The derivation of Eq. (29) replaces nonlinear random features with a noisy linear-features model, uses conditional Gaussian calculations and universality, and then substitutes the resulting expression into earlier equations.In the overparametrized regime ψ1 > ψ2, the key matrix simplifies because U has full column rank and its minimum singular value stays bounded away from zero.Source: §E.6 · pp. 58–59