Source-linked AI summary
Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations
Nick McGreivy, Ammar Hakim
TL;DR
ML-based PDE solvers aim to produce sufficiently accurate solutions faster than standard numerical methods, but the literature's evaluation practices have been questioned. This Analysis systematically reviews fluid-related PDE research and finds weak baselines and reporting biases, including evidence that negative results are underreported. It concludes that these issues contribute to overoptimism and calls for cultural and structural reforms.
Problem
ML-based PDE solvers seek to produce sufficiently accurate solutions faster than standard numerical methods, motivating scrutiny of how performance is evaluated.
Method
The Analysis systematically reviews ML research on fluid-related PDEs and examines weak baselines and reporting biases.
Results
79% (60/76) of articles claiming to outperform standard numerical methods compare against a weak baseline, while evidence indicates reporting biases underreport negative results.
Takeaways & Limitations
ML-for-PDE solving research is overoptimistic, supporting calls for cultural changes and structural reforms to reduce biased reporting.
Takeaways & Limitations
The review covers only forward problems and fluid-mechanics-related PDEs, and the evidence for reporting bias does not prove causation.
Abstract
from arXiv · showhide
One of the most promising applications of machine learning (ML) in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of ML-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. We first perform a systematic review of the ML-for-PDE solving literature. Of articles that use ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) compare to a weak baseline. Second, we find evidence that reporting biases, especially outcome reporting bias and publication bias, are widespread. We conclude that ML-for-PDE solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to underreporting of negative results. To a large extent, these issues appear to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting as well as top-down structural reforms intended to reduce perverse incentives for doing so.
1 Introduction
ML is increasingly applied to scientific problems, including PDE solving, but reproducibility concerns raise questions about whether reported successes are reliable. This paper examines weak baselines and reporting biases in ML-based fluid-related PDE research.
- Reproducibility issues can undermine the credibility and authority of scientific fields, often involving analytical pitfalls and biases toward positive results.
- Reproducibility concerns in ML-based science include data leakage, poor data quality, weak baselines, and insufficient external validation.These pitfalls result in overoptimistic assessments of ML performance.
- PDEs model many physical systems, and ML-for-PDE research addresses forward problems, inverse problems, and reduced-order models.
- ML-based PDE solvers aim to produce sufficiently accurate solutions faster than standard numerical methods.Faster solvers could support optimization, inverse problems, uncertainty quantification, and simulation applications.
- ML-based PDE solvers face concerns about downstream computational cost, generalization, numerical stability, and chaotic-system prediction.Training and data-generation costs are often excluded from speed comparisons, while PINNs can be orders of magnitude slower than standard methods.
2 Weak baselines
The paper evaluates whether ML-for-PDE studies use fair numerical baselines, focusing on accuracy-matched and efficient methods. It finds that weak-baseline comparisons are common among articles claiming ML advantages.
- A fair comparison requires evaluating ML-based solvers against standard numerical methods without underestimating either method’s performance.
- Comparing a highly accurate training-data solver with a less accurate ML solver can exaggerate the ML method’s runtime advantage.The standard solver should be adjusted to comparable accuracy before runtime is compared.
- State-of-the-art numerical methods can be orders of magnitude faster than inefficient alternatives, so baselines should use efficient methods appropriate to the PDE.
- The systematic review covers articles using ML to solve fluid-related PDEs and comparing speed or a speed proxy with standard numerical methods.The review excludes PINNs because standard methods are already known to outperform them for forward problems.
- The review found 82 qualifying articles, including 76 claiming to outperform standard numerical methods.
- The identified weak-baseline problems included inefficient methods, mismatched dimensionality, slow implementations, and inappropriate time-stepping choices.
- 60/76 (79%) articles claiming superiority received a weak-baseline rating, while 14/76 (18.4%) were judged to use strong baselines.Another 2/76 (2.6%) may have used weak baselines and require caution.
3 Reporting biases
Reporting biases appear widespread in ML-for-PDE research: positive results are emphasized, while negative results are often omitted or selectively presented. Together with weak baselines, these biases produce overoptimistic conclusions about ML solver performance.
- 94.8% (220/232) of abstracts mentioning experimental results reported only positive results, while 0% (0/232) reported only negative results.The authors describe this as unusually high compared with other scientific fields and investigate whether negative results are underreported.
- Of 82 reviewed articles, 76 (93%) claimed to outperform standard numerical methods, whereas only 4 (5%) claimed to underperform.The authors interpret this imbalance as evidence consistent with publication bias or outcome reporting bias.
- The review found selective reporting, outcome switching, non-standard performance reporting, and inconsistent baseline choices across articles.Examples include reporting runtime for only some PDEs or highlighting only the most successful PDE.
- 80% (61/76) of sample A reported efficiency in the abstract, compared with 8% (5/60) of sample B.Across the entire article, only 12% (7/60) of sample B reported efficiency relative to standard solvers.
- Weak baselines lead to overly positive results, while reporting biases underreport negative results, producing overoptimism about ML.Figure 1 depicts the cumulative effects of these two mechanisms on samples A and B.
4 Limitations
The analysis has a narrow empirical scope and does not prove that reporting biases caused negative results to be underreported. Its Figure 1 estimates should therefore be read as illustrating trends rather than precise counterfactual results.
- The systematic review covers only forward problems and PDEs related to fluid mechanics.Its findings may not extend to inverse problems, solid mechanics, quantum mechanics, or high-dimensional PDEs.
- The evidence does not prove that reporting biases cause negative results to be underreported.Other contributing factors remain possible.
- Figure 1's bottom-left result is an estimate, and the bottom-center result also contains some uncertainty.The authors intend the figure to show an overall trend, not precise outcomes under stronger baselines and complete reporting.
5 Discussion
The discussion attributes baseline and reporting problems partly to researcher degrees of freedom, limited expertise and benchmarking, and incentives favoring positive results. It recommends stronger evaluation practices alongside cultural and structural reforms.
- Discussion: Weak baselines appear partly linked to limited ML domain expertise, insufficient numerical-analysis benchmarking, and low awareness of baseline selection difficulty.The authors identify these as factors specific to the subfield, especially for violations of baseline-selection rules.
- Discussion: Articles should compare ML solvers with both standard numerical methods and other ML-based solvers, while justifying baseline choices.The authors also recommend matching accuracy or runtime when comparing with standard solvers and plotting cost versus accuracy.
- Discussion: Researcher degrees of freedom and positive-result incentives appear similar to factors implicated in reproducibility crises elsewhere.Researchers make choices about PDEs, boundary conditions, hyperparameters, metrics, and hypotheses, alongside choices about what to report.
- Discussion: Unchecked overoptimism can cause misunderstanding of ML applicability, misallocation of research priorities, and slower scientific progress.The authors argue that negative results are needed to determine whether and how ML advances knowledge.
- Discussion: The authors recommend bottom-up cultural changes and top-down structural reforms to reduce biased reporting and perverse incentives.Proposed measures include internal reporting controls, reproducibility education, registered reports, and domain-scientist-led challenge problems.
6 Methods
The Analysis uses scientific literature to study issues within scientific literature, making it an example of metascience.
- The Analysis studies issues with scientific literature by using scientific literature as its evidence base.It is therefore an example of metascience.
6.1 Systematic review
The systematic review defines a focused question about weak-baseline comparisons in fluid-related ML-for-PDE research and specifies inclusion, search, evaluation, and fairness criteria. It also recommends reporting practices for making comparisons more computationally and methodologically fair.
- Research question and scope: The review asks how frequently ML-for-PDE studies compare against weak baselines.It covers fluid-related PDEs and comparisons of ML solver speed, computational cost, or supported proxies with standard numerical methods.
- Inclusion criteria: Articles had to compare an ML-based solver with a standard numerical method using quantitative or visually supported evidence.Qualitative efficiency claims counted only when supported by quantitative or visual evidence.
- Outcome definition: The primary outcome was selected from quantitative comparisons in the abstract, conclusion, introduction, or main text, with qualitative comparisons used only when quantitative evidence was absent.When multiple comparisons existed, the reviewers used their best judgment to identify the primary outcome.
- Search process: 82 articles matched the inclusion criteria after searches using author lists, citation chasing, and Google Scholar screening.The authors compiled 258 authors in list A and 358 articles in list B; 82 list-B articles matched the criteria.
- Search limitations: The search may have missed articles with fewer citations or articles that did not cite the selected key articles.The search ended on April 1st, 2023 and included articles available by that date.
- Fair-baseline criteria: Fair comparisons require equal accuracy or equal runtime and an efficient numerical method as the baseline.These are necessary but not sufficient conditions for comparing ML-based PDE solvers with standard numerical methods.
- Recommendations: The authors recommend caution when comparing specialized ML solvers with general-purpose libraries and justification of hardware choices.They also recommend reporting the number of surrogate evaluations needed to offset data-generation and model-training costs.
- Numerical baselines: For elliptic PDEs, the authors identify multigrid solvers as typically state-of-the-art for large problems, with direct solvers more efficient for small problems.They recommend considering efficient implementations such as Eigen for LU decomposition and HYPRE for multigrid methods.
6.2 Random sample of ML-for-PDE articles
The random-sample analysis classifies reported positive and negative experimental results in ML-for-PDE research using a citation-based sample and explicit exclusions. It distinguishes genuine negative results from limitations or criticism of an earlier method.
- Sampling and classification: The sample comprises 400 articles citing Kochkov et al. as of June 18th, 2023, including abstracts mentioning positive and/or negative experimental results.A result required proposing a method and describing its performance with quantitative metrics or qualitative descriptions.
- Negative-result definition: Negative results were counted only when an article discussed negative results associated with its own proposed method.Criticism of method A followed by a method B that addressed those problems was not itself treated as a negative result.
- Exclusions: Comments that a method had limitations or limited scope were not counted as negative results.Three articles made such comments, but they were excluded from the negative-result category.
6.3 Random sample of PINN articles
The PINN analysis samples forward fluid-related PDE studies that report solver accuracy using data generated by standard numerical methods. The resulting sample is used to examine claims about PINN performance and baseline comparisons.
- Sampling criteria: The PINN sample requires forward fluid-related PDE problems, reported solver accuracy, and self-generated data from a standard numerical or general-purpose solver.Inverse or ill-posed problems were excluded from these criteria.
- Search process: The search screened 5,640 articles citing Raissi et al. and ultimately produced a sample of 60 articles.The first 250 relevance-sorted results were screened; 155 articles were considered potentially relevant, and eight were excluded for weak-baseline or optimization-time issues.
- Runtime comparison: PINN solvers are described as generally slower than standard numerical methods, with good accuracy often taking hours to days rather than fractions of a second to minutes.The passage notes a possible exception when meta-learning is used.
- Interpretation: The authors infer that the sampled studies’ authors were aware of the runtime difference because they generated comparison data with standard numerical methods and often focused on improving PINN speed.This inference is tied to the sampling design and the stated runtime limitation.
7 Declarations
The paper declares public availability of its review data and categorizations, reproducibility code, and author contributions. Supplementary information is also identified as available.
- Data availability: The systematic-review author lists, article lists, and random-sample categorizations are publicly available through an OSF record.The record is identified by DOI 10.17605/OSF.IO/GQ5B3.
- Code availability: Code reproducing the results in table 2 is available on GitHub with instructions for running and interpreting it.The repository is named WeakBaselinesMLPDE.
- Author contributions: The authors divide contributions across conceptualization, searching, baseline evaluation, bias analyses, coding, manuscript preparation, supervision, and baseline design.N.M. led the review and analyses, while A.H. designed strong baselines and supervised the research.
- Supplementary information: Supplementary Information is stated to be available.
Supplementary Information
The supplementary examples show that claimed ML speedups depend on fair accuracy matching and strong numerical baselines. Several comparisons are judged unfair because they use mismatched accuracy, inefficient general-purpose solvers, or methods that are not state of the art for the problem.
- Several reported ML speedups are judged unfair because runtime comparisons use different accuracy levels.Examples include comparisons against highly accurate solvers while the ML model is less accurate, or comparisons made without matching error.
- A stronger baseline reversed one reported comparison: for Brandstetter et al., the stronger solver was orders of magnitude faster than the ML-based solver.The original comparison used a slow implementation of WENO5 and a slow spectral solver.
- The supplementary cases also include reported accelerations such as 11× on CPU and 40× on GPU, but the associated runtime comparison is judged unfair because values are not matched.The v-Net and a-Net comparison against PhiFlow compares runtime without equal loss values, and the loss does not measure accuracy in this setting.
- Some comparisons are considered fair when accuracy or runtime can be compared directly, including the multigrid and learned-turbulence examples.The review accepts equal-accuracy or equal-runtime comparisons and recognizes state-of-the-art baselines where appropriate.
- Some baseline choices are weakened by inefficient general-purpose software or methods that are not state of the art for the target PDE.The review specifically flags OpenFOAM, finite-volume methods for incompressible Navier–Stokes, and multigrid for sufficiently small problems.
- A claimed 100, 682, and 800 speedup against OpenFOAM is treated cautiously because resolution was not reduced to match accuracy and OpenFOAM has high overhead and slow convergence.The review recommends comparing with both finite-volume and discontinuous-Galerkin methods for Navier–Stokes where appropriate.