PaperlayerView exact revision on arXiv

Source-linked AI summary

Leakage and the Reproducibility Crisis in ML-based Science

Sayash Kapoor, Arvind Narayanan

arXiv:2207.07048v1cs.LGcs.AIstat.ME

TL;DR

The paper investigates reproducibility problems caused by data leakage in ML-based science through a literature survey, leakage taxonomy, model info sheets, and a civil-war prediction reproducibility study. It finds leakage across many fields and shows that corrected complex models do not outperform Logistic Regression in the case study.

  • Problem

    Data leakage and other methodological pitfalls threaten the reproducibility of increasingly widespread ML-based science.

  • Method

    The paper surveys reproducibility errors across scientific fields, develops a leakage taxonomy and model info sheets, and tests reproducibility in civil-war prediction.

  • Results

    17 fields and 329 papers are affected by reported errors, while corrected complex ML models perform no better than baseline Logistic Regression in the civil-war prediction cases.

  • Takeaways & Limitations

    Model info sheets enable detection of leakage in the studied cases, supporting changes to ML research and reporting practices.

  • Takeaways & Limitations

    Model info sheets cannot be verified without the code, data, and computing environment needed for computational reproducibility.

Abstract

from arXiv · show

The use of machine learning (ML) methods for prediction and forecasting has become widespread across the quantitative sciences. However, there are many known methodological pitfalls, including data leakage, in ML-based science. In this paper, we systematically investigate reproducibility issues in ML-based science. We show that data leakage is indeed a widespread problem and has led to severe reproducibility failures. Specifically, through a survey of literature in research communities that adopted ML methods, we find 17 fields where errors have been found, collectively affecting 329 papers and in some cases leading to wildly overoptimistic conclusions. Based on our survey, we present a fine-grained taxonomy of 8 types of leakage that range from textbook errors to open research problems. We argue for fundamental methodological changes to ML-based science so that cases of leakage can be caught before publication. To that end, we propose model info sheets for reporting scientific claims based on ML models that would address all types of leakage identified in our survey. To investigate the impact of reproducibility errors and the efficacy of model info sheets, we undertake a reproducibility study in a field where complex ML models are believed to vastly outperform older statistical models such as Logistic Regression (LR): civil war prediction. We find that all papers claiming the superior performance of complex ML models compared to LR models fail to reproduce due to data leakage, and complex ML models don't perform substantively better than decades-old LR models. While none of these errors could have been caught by reading the papers, model info sheets would enable the detection of leakage in each case.

1. Overview

ML-based predictive modeling is spreading across quantitative science, but methodological pitfalls and unresolved leakage risks threaten reproducibility. The paper surveys this problem and motivates standardized reporting practices.

  • Predictive modeling has spread across quantitative science through widespread use of ML methods, increasing the need to examine reproducibility.
  • The survey examines reproducibility issues in scientific claims that use ML-model performance as evidence, distinct from methods, ethics, engineering, and modeling-contest research.
  • The paper defines reproducible findings as those obtained from available code and data that are correctly analyzed.This definition is broader than computational reproducibility, which requires reproducing results with the authors’ exact code and dataset.
  • Leakage in ML-based science had not been comprehensively investigated, leaving mitigation strategies understudied.
  • The paper systematically investigates leakage-related reproducibility issues and presents three main contributions.

1) A survey and taxonomy of reproducibility issues due

The paper surveys leakage problems across ML-based science, develops a taxonomy, proposes model info sheets, and tests their value in civil war prediction. Correcting leakage removes the reported substantive advantage of complex ML over Logistic Regression.

  • A survey and taxonomy of reproducibility issues due to leakage: The survey identifies 20 papers across 17 fields describing errors that collectively affect 329 papers, with leakage present in every field studied.
  • A survey and taxonomy of reproducibility issues due to leakage: The taxonomy covers 8 leakage types spanning data collection, preprocessing, modeling, and evaluation, from textbook errors to open research problems.
  • A survey and taxonomy of reproducibility issues due to leakage: Model info sheets require precise arguments that models used for scientific claims do not suffer from leakage and address all surveyed leakage types.
  • A survey and taxonomy of reproducibility issues due to leakage: In civil war prediction, all reviewed papers claiming ML superiority over Logistic Regression fail to reproduce because of data leakage.

2. Evidence of a reproducibility crisis

The evidence indicates a systemic reproducibility crisis: leakage inflates ML performance estimates and recurs across scientific fields. Existing mitigations often do not transfer from engineering or competitions to scientific research.

  • Evidence of a reproducibility crisis: Data leakage is a spurious predictor–target relationship created by data collection, sampling, or preprocessing that usually inflates performance estimates.
  • Evidence of a reproducibility crisis: A cross-disciplinary analysis of prior field-specific reviews highlights the scale and scope of failures and identifies common patterns.
  • Evidence of a reproducibility crisis: The survey is a lower bound because differing terminology hinders systematic searching and paper-only reviews miss errors in code and data.
  • Evidence of a reproducibility crisis: 20 papers across 17 fields identify errors collectively affecting 329 papers, and every surveyed field exhibits data leakage.
  • Evidence of a reproducibility crisis: Mitigations developed for modeling competitions and engineering applications often do not translate to ML-based science, where research goals and data access differ.
  • Evidence of a reproducibility crisis: The crisis is systemic because recurring pitfalls affect many fields while communities lack deployed systemic solutions and converged best practices.
  • Evidence of a reproducibility crisis: The taxonomy organizes leakage arising during data collection, preprocessing, modeling, and evaluation to inform potential solutions.
  • Evidence of a reproducibility crisis: Leakage includes inadequate train-test separation, preprocessing or feature selection on the full dataset, duplicate records, and illegitimate proxy features.

3. Model info sheets for detecting and preventing leakage

Model info sheets are proposed as standardized reporting tools for detecting and preventing leakage in ML-based science. They require researchers to justify train-test separation, feature legitimacy, and the relationship between test data and the scientific distribution, while remaining limited by unverifiable or incorrect claims and insufficient expertise.

  • Model info sheets provide precise arguments needed to justify the absence of leakage and address every leakage type identified in the survey.
  • Existing checklists and model cards provide reporting standards but do not address leakage-specific pitfalls in ML-based science.
  • Researchers must justify clean train-test separation, legitimate features, and that test data represent the distribution of scientific interest without selection or sampling bias.
  • Model info sheets can help researchers detect and prevent leakage while transparently reporting model details as supplementary material.
  • Journal guidelines could require model info sheets when papers do not transparently report model creation, enabling closer scrutiny without requiring reviewers to reproduce results fully.
  • Model info sheets cannot be verified without computational reproducibility, may contain incorrect claims, and can miss subtle leakage when authors or reviewers lack ML expertise.
  • Because understanding of leakage may evolve, the model info sheets are versioned and planned for updating.

4. A case study of civil war prediction

The civil war prediction case study examines whether complex ML models outperform Logistic Regression and whether reported findings reproduce. Reviewing available studies, the authors find leakage-related failures and no substantive advantage over LR after correction, alongside weak uncertainty reporting.

  • 124 papers were identified, narrowed to 12 civil-war prediction papers with train-test evaluation and complete code and data for reproducibility analysis.
  • 4 of 12 papers contained errors, exactly matching the papers claiming that complex ML models outperformed baseline Logistic Regression models.
  • After correction, complex ML models performed no better than baseline Logistic Regression in each case except Wang (2019), where the AUC difference fell from 0.14 to 0.01.
  • None of the identified errors could be detected by reading the papers, whereas model info sheets enabled detection of leakage in each case.
  • 9 of 12 papers with complete code and data reported no significance tests or uncertainty quantification for classifier performance comparisons.
  • For Blair and Sambanis (2020), the reported smoothed AUC was 0.85, but bootstrapped 95% confidence intervals ranged from 0.66 to 0.95.

5. Beyond leakage: enhancing the reproducibility of ML-based science

The paper identifies five reproducibility diagnoses beyond leakage and pairs them with recommendations, including communicating predictive limits, treating results cautiously, interdisciplinary collaboration, standardization, and computational reproducibility.

  • Limits to prediction: Predictive performance has strong limits in some social-outcome tasks, so results should not generalize optimism from image classification to other domains.The authors propose identifying upper bounds on task accuracy to avoid futile efforts and scrutinize claims that exceed known bounds.
  • Hype and publication bias: Results from ML-based science should be treated with caution until reproducibility issues are widely addressed and resolved.The recommendation responds to overoptimism, citation feedback loops, and publication biases affecting ML-based science.
  • Expertise and collaboration: Interdisciplinary teams combining domain and ML expertise can help avoid errors caused by inadequate expertise in either area.The paper also calls for clearer communication of ML best practices to scientific researchers.
  • Standardization: Standardized train-test splits, evaluation metrics, and modeling tasks can reduce subtle errors that are otherwise hard to detect.The common task framework adds agreed training data and metrics, a secret holdout dataset, and a public leaderboard, but may encourage optimizing one accuracy metric exclusively.
  • Computational reproducibility: Computational reproducibility facilitates independent verification by removing the need to spend time getting code to run.The authors recommend platforms such as CodeOcean that replicate the original computational environment for long-term reproducibility.

6. Conclusion

The paper calls for fundamental changes to ML-based scientific research and reporting practices, supported by interdisciplinary efforts and model info sheets to address leakage. It also provides reproducible materials for its civil-war case study and documents additional methods and review details in the appendices.

  • Conclusion: Off-the-shelf ML tools enable model creation without ML expertise, but this laissez-faire approach spreads common pitfalls across scientific fields.The authors warn that repeated rediscovery of pitfalls and unchanged practices could undermine public trust.
  • Conclusion: Model info sheets are presented as a first step toward detecting and preventing leakage in ML-based science.The conclusion frames their adoption as part of broader interdisciplinary efforts to develop and drive best practices.
  • Conclusion: The civil-war prediction case study's reproduction materials are uploaded to a CodeOcean capsule.Appendix B describes the methods and additional robustness checks, while Appendix C provides the model-info-sheet template and leakage coverage.

A. Why do we call these reproducibility issues?

The paper uses a broad definition of reproducibility centered on available code and data and correct analysis, while distinguishing it from exact computational replication. Predictive modeling errors qualify as reproducibility issues when they produce incorrect estimates of out-of-sample performance.

  • Definition: A finding is reproducible when its code and data are available and the data are correctly analyzed.This definition is broader than computational reproducibility, which requires replication with the authors' exact code and dataset.
  • Predictive modeling: Violating the train-test split is an error because the test set must estimate performance on data not used for training.The paper uses consensus in the ML community, supplemented by its own arguments, to identify analytical errors.

B. Materials and Methods: Reproducibility issues in civil war prediction

The civil-war prediction reproducibility study corrects leakage while keeping other implementation factors as close as possible, then reassesses out-of-sample performance. It finds that leakage affects both relative and absolute performance estimates, while corrected results do not validate every original methodological choice.

  • Reproducibility issues: Data leakage makes the findings of four civil-war prediction papers inaccurate for both relative and absolute out-of-sample performance.The issue applies regardless of whether researchers seek absolute performance or comparisons between models.
  • Correction procedure: The authors corrected leakage in four papers while keeping other implementation factors as close to the originals as possible.The purpose was to obtain more accurate estimates of out-of-sample performance.
  • Scope of corrections: Corrected results do not endorse all methodological choices in the original papers, including imputation on a test dataset with 95% missing values.Although training-only imputation avoids leakage, the authors do not treat that as validating use of such a highly incomplete dataset.
  • Paper selection: The review identified relevant civil-war prediction papers through title and abstract searches plus citations from a recent field review, then re-analyzed papers with errors.The paper states that it subsequently addresses the reproducibility issues found in each paper in detail.

B.2. Muchlinski et al. (2016)

Muchlinski et al. (2016) jointly imputed training and test data, allowing target-related information to shape the test features and inflate Random Forests’ apparent out-of-sample performance. Correcting the imputation removes the claimed advantage over Logistic Regression.

  • Leakage mechanism: Over 95% of the out-of-sample test values are missing, including 70 of 90 variables, magnifying the impact of leakage.The paper notes that this missingness makes the inflated evaluation especially severe.
  • Corrected results: After correcting the imputation, the Random Forests model performs no better than the Logistic Regression models used for comparison.The corrected analysis uses an imputation procedure that excludes test rows from the imputation model.
  • Caveat: Correct leakage does not validate every remaining design choice: the corrected out-of-sample dataset still contains 95% missing values.The authors specifically caution that avoiding leakage does not make this evaluation dataset desirable.
  • Leakage mechanism: Jointly imputing training and test datasets leaks target-related information into test features and produces overoptimistic performance claims.The imputation model uses test labels and training correlations to fill missing test values.
  • Visualizing the leakage: Imputation separates war and peace observations in agexp, creating concentrated test patterns that Random Forests can learn more readily than Logistic Regression.The imputed war and peace distributions become separated, unlike the original overlapping distributions.
  • Simulation: Imputing training and test sets together increasingly raises purported out-of-sample accuracy in simulation, yielding artificially high estimates.The simulation reports repeated runs with mean accuracy and 95% confidence intervals.

B.3. Colaresi & Mahmood (2017)

Colaresi & Mahmood (2017) reused an already imputed dataset for training and testing, creating the same leakage problem identified in Muchlinski et al. (2016). Correcting the imputation dramatically reduces the reported ML advantage over Logistic Regression.

  • Data and leakage: Colaresi & Mahmood (2017) reuse Muchlinski et al.’s imputed dataset for both training and testing, reproducing the same leakage pitfall.They evaluate through a train-test split rather than the separate out-of-sample test set.
  • Corrected results: Correcting the imputation method dramatically reduces the performance of the proposed ML models.The corrected results address the use of the imputed dataset in both training and testing.
  • Models: The study compares two Random Forests models with the Logistic Regression model from Fearon & Laitin (2003).The comparison uses three civil-war-onset models in the final round of reported results.
  • Data and leakage: The reused imputed dataset contains 17.4% missing values overall, while its test set contains 19% missing values.Missingness varies across individual variables.
  • Data provenance: The imputed dataset assigns peace to the missing target values of 777 original rows, although the onset counts remain the same.Because the original imputation code was not shared, the mechanism producing these target assignments is unclear.
  • Robustness analysis: The robustness analysis excludes milper because it is missing for every instance in the test set and compares separate out-of-sample testing with cross-validation.The authors also caution that temporal cross-validation can itself create leakage across train-test splits.

B.5. Kaufman et al. (2019)

The reproducibility analysis of Kaufman et al. (2019) identifies multiple methodological issues, including leakage from proxy variables and an overly weak baseline, that alter the reported comparison with Logistic Regression.

  • Leakage corrections: Data leakage arose from independent variables that were proxies for civil war outcomes or affected by civil wars.The correction uses lagged versions of affected variables and removes outcome proxies.
  • Model corrections: The original Lasso parameter selection made the model always predict peace, so standard parameter selection was used instead.The corrected procedure uses cv.glmnet to select a suitable parameter value.
  • Evaluation design: K-fold cross-validation can place later observations in training than in testing when temporal data are shuffled.The corrected analysis retains k-fold cross-validation to test the same instances as the original results.
  • Data preprocessing: Replacing missing values with zeros prevents models from distinguishing missing data from true zero values and risks underestimating performance.The paper characterizes this as methodologically unsound handling of missing data.
  • Robustness: The cutoff-selection procedure does not change the main claim: the best model remains worse than a baseline predicting the previous year’s outcome.This robustness result holds when cutoffs are chosen using the best training-set cutoff.
  • Baseline comparison: A previous-year-outcome baseline is stronger than always predicting peace, with accuracy 97.5% vs. 86.1%.McNemar’s test gives χ2=633.7 and p = 7.836e-140.

B.6. Blair & Sambanis (2020)

The analysis finds no significant performance advantage for Blair and Sambanis’s escalation model over comparison models and shows that reported uncertainty is substantial and results depend on ROC-curve smoothing.

  • Statistical comparison: There is no significant difference between the smoothed AUC of the escalation model and the comparison models.The comparisons include quad, goldstein, cameo, and average models.
  • Uncertainty: The reported 95% confidence intervals for sensitivities and specificities are large for both 1 month and 6 month forecasts.The intervals cover escalation and the comparison models across the displayed forecast horizons.
  • Interpretation: A singular focus on statistical significance at a predefined threshold can be harmful.The paper distinguishes significance testing from robustness to sampling variance.
  • Metric construction: Computing AUC on a smoothed rather than empirical ROC curve is a modeling choice that highly affects the results.The paper states that this issue completely changes the original results.

B.7. Overview of papers in Table A6

Table A6 reviews papers using out-of-sample classifier performance as evidence, documenting their evaluation practices and the availability of model-info-sheet guidance for detecting leakage.

  • Table A6 overview: Table A6 records confidence intervals, significance tests, reported metrics, test-set size, positive instances, and reliance on out-of-sample evaluation.AUC denotes Area Under ROC, while MAE denotes Mean Absolute Error.
  • Reported comparisons: The review also documents out-of-sample comparisons involving Random Forests, iterative model improvement, relative ISAF support, and population models.These examples span AUC, performance gain, and normalized MAE comparisons.
  • Figure A3: Figure A3 visualizes the escalation and cameo models for 1 month and 6 month forecasts in the base specification.The figure focuses on the reported sensitivities and specificities and their confidence intervals.
  • Out-of-sample evidence: Several reviewed papers rely on out-of-sample evaluation as primary evidence for improved model performance.Examples include Hegre et al. (2019a), Kaufman et al. (2019), Wang (2019), Blair & Sambanis (2020), and Colaresi & Mahmood (2017).
  • Model info sheets: The model info-sheet template explains how reporting requirements address leakage types found in the survey and civil-war case study.The template requires explanations of train-test splitting during all modeling steps.
Loading 2207.07048v1…