PaperlayerView exact revision on arXiv

Source-linked AI summary

Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations

Nick McGreivy, Ammar Hakim

arXiv:2407.07218v1math.NAcs.LGphysics.flu-dyn

TL;DR

Whether ML solvers truly outperform standard numerical methods for fluid-related PDEs matters to efforts to obtain accurate solutions faster. This paper systematically reviews that literature and finds that 79% of reviewed articles used weak baselines, alongside evidence of reporting biases.

  • Problem

    Whether ML solvers can produce accurate solutions faster than standard numerical methods for fluid-related PDEs remains an important research question.

    Source: §1 · p. 2

  • Method

    The paper systematically reviews ML studies comparing speed with standard numerical methods for fluid-related PDEs and examines reporting biases using anecdotal and statistical evidence.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    79% of reviewed articles showed common baseline-comparison pitfalls, and the review found evidence of publication bias and outcome reporting bias.

    Source: §1 · p. 2, §3 · p. 6

  • Takeaways & Limitations

    The authors conclude that cultural changes and structural reforms are needed to minimize biased reporting and reduce perverse incentives.

    Source: §5 · p. 9

    The authors have evidence, but not proof, that reporting biases cause negative results to be underreported.

    Source: §4 · p. 7

Abstract

from arXiv · show

One of the most promising applications of machine learning (ML) in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of ML-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. We first perform a systematic review of the ML-for-PDE solving literature. Of articles that use ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) compare to a weak baseline. Second, we find evidence that reporting biases, especially outcome reporting bias and publication bias, are widespread. We conclude that ML-for-PDE solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to underreporting of negative results. To a large extent, these issues appear to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting as well as top-down structural reforms intended to reduce perverse incentives for doing so.

1 Introduction

The introduction argues that weak baselines and reporting biases make the ML-for-PDE literature overoptimistic: its review finds weak baselines in most articles claiming to outperform standard methods.

1 Introduction

  • Across science, reproducibility issues are linked to analytical and statistical pitfalls and positive-result publication bias; ML-based science also faces leakage, poor data quality, weak baselines, and insufficient validation.These pitfalls can produce overoptimistic assessments of ML performance.Source: §1 · p. 1
  • ML-for-PDE research has focused on forward initial-boundary-value problems, inverse problems that infer parameters or missing data, and reduced-order models.PDEs model many physical systems.Source: §1 · p. 2
  • Standard numerical solvers trade accuracy for efficiency, while ML could learn algorithms or surrogates that produce approximate PDE solutions faster for downstream applications.Potential uses include optimization, inverse problems, uncertainty quantification, and simulation codes.Source: §1 · p. 2
  • ML solver speed may not offset data-generation and training costs, while concerns include generalization, stability, chaotic systems, and PINNs’ slow or failed convergence.For inverse problems, PINNs have had some success but appear not to outperform discrete grid-based alternatives.Source: §1 · p. 2
  • 79% of reviewed articles had weak baseline comparisons, and anecdotal and statistical evidence indicates reporting biases underreport negative results, making the literature unreliable for judging ML PDE-solving success.The systematic review covers research using ML to accelerate fluid-related PDE solutions.Source: §1 · p. 2

2 Weak baselines

The section defines fair ML-versus-numerical-solver comparisons and finds that most reviewed claims of ML superiority rely on weak baselines. Stronger-baseline replications often change the comparison substantially.

2 Weak baselines

  • Fair comparisons require equal accuracy or runtime and an efficient numerical baseline, rather than the training-data solver or a less efficient method.A standard solver can trade accuracy for runtime, so comparing runtimes at different accuracies can mislead; the appropriate efficient method depends on the PDE.Source: §2 · pp. 2–3
  • The systematic review identified 82 fluid-related PDE articles comparing ML speed with standard numerical methods, excluding PINN studies; 76 claimed outperformance.The remaining articles included four claiming underperformance and two reporting similar or varied performance.Source: §2 · p. 3
  • Inefficient baselines for elliptic PDEs were identified in eight articles, and inefficient baselines for advection-dominated PDEs in five.Other Rule 2 concerns included inefficient implementations and using implicit timestepping where explicit timestepping would have been faster.Source: §2 · p. 3
  • 60/76 (79%) articles claiming ML outperformance received a weak-baseline rating; only 14/76 (18.4%) were rated strong, and strong-baseline articles tended to report smaller relative improvements.Another 2/76 (2.6%) were flagged as potentially using weak baselines, so their claims warrant caution.Source: Table 1 · p. 4, §2 · p. 4
  • In 7/10 replications, a stronger numerical baseline outperformed the ML solver, and in 9/10 it was at least two orders of magnitude faster than the original baseline.Four of the ten articles violated rule 1, and at least six violated rule 2 through an inefficient method or implementation.Source: §2 · p. 5, Table 2 · p. 6

3 Reporting biases

The literature reports overwhelmingly positive results, and comparisons between articles with positive and negative results indicate that negative efficiency findings are rarely reported. The review also finds evidence consistent with publication bias and outcome reporting bias, which together with weak baselines contribute to overoptimism about ML for PDEs.

3 Reporting biases

  • Reporting biases arise when analysis, reporting, or interpretation is influenced by the nature or direction of results, and include publication bias, spin bias, and outcome reporting biases such as p-hacking and selective reporting.Other outcome reporting forms named in the paper are outcome switching and data-dredging.Source: §3 · p. 6
  • 94.8% (220/232) of sampled articles with experimental results mentioned only positive results, 5.2% (12/232) mentioned both positive and negative results, and 0% (0/232) mentioned only negative results.The authors describe this positive-result share as unusually high compared with other fields and use it to motivate investigation of underreporting.Source: §3 · p. 6
  • Of 82 reviewed articles, 76 (93%) claimed to outperform a standard numerical baseline and only 4 (5%) claimed to underperform, alongside evidence of selective reporting, outcome switching, and publication bias.Examples included reporting runtime for some PDEs but not others, highlighting the most successful PDE, using non-standard performance reporting, and switching to a weaker baseline in the abstract; some methods also performed worse on other PDEs or failed to reproduce under nearly identical conditions.Source: §3 · p. 6
  • 80% (61/76) of sample A articles reported efficiency in the abstract, versus 8% (5/60) in sample B; only 12% (7/60) of sample B reported it in the full article.Sample A articles claimed to outperform standard numerical methods, whereas sample B authors believed their ML-based solvers underperformed relative to those methods; positive results were highlighted far more often than negative ones.Source: §3 · pp. 6–7
  • The combined effects of weak baselines and outcome reporting bias are that weak baselines produce overly positive results and reporting biases suppress negative results, creating overoptimism about ML.Figure 1 depicts the cumulative effects across samples A and B.Source: §3 · p. 7, Figure 1 · p. 7

4 Limitations

The analysis is limited in scope, provides evidence rather than proof that reporting biases underreport negative results, and cautions that Figure 1’s estimates are uncertain.

4 Limitations

  • The review covers only forward fluid-mechanics PDEs, leaving ML for inverse problems and other PDE domains unanalyzed and their reproducibility risks unclear.The excluded domains include solid mechanics, quantum mechanics, and high-dimensional PDEs.Source: §4 · p. 7
  • The authors offer evidence, not proof, that reporting biases underreport negative results, and acknowledge that unknown alternative factors could contribute.Source: §4 · p. 7
  • Figure 1’s bottom-left column is an estimate and its bottom-center column is also uncertain, though less so; the figure illustrates trends, not precise outcomes under stronger baselines and complete reporting.Source: §4 · pp. 7–8

5 Discussion

The discussion attributes weak baselines and biased reporting to field-specific gaps, researcher choices, and incentives favoring positive results. It recommends stronger baseline practices alongside cultural and structural reforms to improve scientific evaluation of ML for PDEs.

5 Discussion

  • Field-specific gaps contributing to weak baselines include limited ML domain expertise, insufficient numerical-analysis benchmarking, and low awareness of baseline difficulty and importance.Numerical analysis often emphasizes theoretical algorithm properties, while relative solver performance depends on PDE parameters and implementation details and can be difficult to predict.Source: §5 · p. 8
  • The authors recommend comparing ML solvers with both standard numerical methods and other ML solvers, while justifying baseline choice and fairness.For standard methods, reduce resolution or iterations to match accuracy or runtime and ideally plot cost versus accuracy; explain why the baseline is efficient or state of the art, compare multiple methods when possible, and consult experts or acknowledge uncertainty.Source: §5 · p. 8
  • Weak baselines and reporting biases are linked to researcher degrees of freedom and incentives that favor positive outcomes over negative results.Researchers make choices about PDEs, settings, metrics, hypotheses, experiments, and which results to report or emphasize; bias arises when analysis or reporting decisions change after results are known. Good intentions and rewards for novelty and positive findings contribute to underreporting negative results.Source: §5 · pp. 8–9
  • Negative results are necessary to determine whether and how ML advances science; unchecked overoptimism can distort applicability, research priorities, and scientific progress.The discussion argues that negative results matter in science regardless of whether they are valued in ML research.Source: §5 · p. 9
  • The authors recommend cultural and structural reforms to reduce biased reporting and perverse incentives, even though these changes may require more work and more negative results.Cultural changes include complete and unbiased reporting, strong baselines, transparent limitations, publication of negative results, and preregistration or registered reports. Structural reforms include registered reports and funding domain scientists to create scientifically valuable, verifiable challenge problems that are unsolved or tedious for standard methods.Source: §5 · p. 9

6 Methods

The analysis treats the scientific literature as a subject of study, making it an example of metascience focused on studying and improving science.

6 Methods

  • The authors study issues in the scientific literature using that literature, framing the analysis as metascience.Metascience uses scientific methods to study and improve science.Source: §6 · p. 9

6.1 Systematic review

The review defines a focused search for fluid-related PDE solvers and evaluates whether their speed or computational-cost comparisons use fair, strong numerical baselines. Its search identified 82 eligible articles, while its criteria and exclusions delimit which comparisons count.

6.1.1 Inclusion criteria

  • Eligible studies had to use ML to approximate a fluid-related PDE solution and compare solver speed or cost with a standard numerical method.Comparisons could use speed proxies such as iteration count or spatial or temporal resolution, but qualitative claims required quantitative or visual support. The review selected the primary outcome using a hierarchy that prioritized quantitative comparisons in the abstract, then conclusion, introduction, and main text.Source: §6.1.1 · p. 10

6.1.2 Exclusion criteria

  • The review excluded non-fluid topics, accuracy-only or unsupported comparisons, duplicate results, and PINN-based methods.PINNs were excluded because the literature was too vast to review comprehensively and the few reported wins were understood to use weak baselines or omit optimization time; standard numerical methods are known to outperform PINNs on forward problems, and Raissi et al. had over 9,000 citations.Source: §6.1.2 · p. 10

6.1.3 Search process

  • Researchers added 258 authors to list A and 358 articles to list B; 82 articles in list B matched the review’s inclusion criteria.The search included author Google Scholar profiles, citation searches for two key articles [72], and keyword searches, which found no new articles.Source: §6.1.3 · pp. 10–11

6.1.4 Criteria for evaluating baselines

  • A fair comparison requires equal accuracy or equal runtime and an efficient numerical baseline.These are necessary, but not sufficient, conditions for comparing an ML-based PDE solver with a standard numerical method.Source: §6.1.4 · p. 11
  • Authors should explain hardware fairness, beware comparisons of specialized ML solvers with general-purpose tools, and report N to account for data-generation and training costs.N is defined only when the ML-based solver is faster than the numerical baseline.Source: §6.1.4 · p. 11
  • The review recommends PDE-appropriate strong baselines, including multigrid for large elliptic problems and higher-order FV, DG, or spectral methods for advection-dominated problems.For Navier–Stokes, it recommends comparing FV, DG, and, when applicable, spectral methods, then choosing the strongest baseline; first-order methods should not be used for fluid-related PDEs.Source: §6.1.4 · pp. 11–12

6.1.5 Details of stronger baselines in Table 2

  • For selected articles, the stronger baselines included RKDG with SSPRK3, Godunov finite-volume, and continuous-Galerkin/LU schemes for specified PDEs and boundary conditions.The implementations covered incompressible Navier–Stokes, advection, Burgers’, Poisson, and wave equations.Source: §6.1.5 · p. 12

6.2 Random sample of ML-for-PDE articles

The authors sampled articles citing Kochkov et al. and classified abstracts for positive and negative experimental results. Of 400 articles, 232 (58%) met the inclusion criteria.

6.2 Random sample of ML-for-PDE articles

  • The sample comprised all 400 articles found via Google Scholar that cited Kochkov et al. as of June 18, 2023.Articles were included if their abstracts mentioned positive and/or negative experimental results.Source: §6.2 · p. 12
  • 232 of 400 sampled articles (58%) met the inclusion criteria.The authors excluded ten articles that mentioned experimental results without indicating whether they were positive or negative.Source: §6.2 · p. 13
  • A negative result required discussion of negative performance for the proposed method, not merely limitations of a previous method or a method’s limited scope.The authors excluded reviews, theses, duplicates, non-English articles, articles unrelated to ML or statistical learning, and articles without experimental results in the abstract.Source: §6.2 · p. 13

6.3 Random sample of PINN articles

This subsection defines and constructs a random sample of PINN articles solving forward fluid-related PDEs, then considers what the sample implies about solver runtime and authors’ awareness of speed limitations.

6.3 Random sample of PINN articles

  • Eligible articles used PINNs for forward fluid-related PDE problems, reported solver accuracy, and generated accuracy data with a standard or general-purpose solver rather than an analytic solution.For articles addressing both forward and inverse problems, a majority of solved problems had to be forward problems; reviews, theses, duplicates, technical reports, and non-English articles were excluded.Source: §6.3 · p. 13
  • The search screened the first 250 of 1,000 Google Scholar results from 5,640 articles citing Raissi et al., using relevance order and a PINN-plus-PDE keyword query.The search was conducted as of June 21, 2023; 155 articles (62%) were flagged for closer review.Source: §6.3 · p. 13
  • 60 articles formed sample B, after excluding eight that claimed to outperform standard methods but used weak baselines or often omitted PINN optimization time.The search added 155 articles (62%) as potentially relevant before close screening.Source: §6.3 · p. 13
  • PINN solvers typically take hours to days for good accuracy, while standard methods take a fraction of a second to minutes, leading the authors to infer that sample authors knew PINNs were less efficient [45] [46].The authors base this inference partly on the sample’s inclusion criterion that articles generate their own accuracy data using a standard solver, and on many articles’ focus on speeding up PINNs; meta-learning may be an exception [179].Source: §6.3 · p. 13

7 Declarations

The authors made the systematic-review materials and code for reproducing Table 2 publicly available and described each author’s contributions.

Data availability.

Code availability.

  • Code and instructions for reproducing and interpreting the Table 2 results are available on GitHub [181].Source: Code availability · p. 14

Authors’ contributions.

  • N.M. led the review, bias analyses, coding, and manuscript, while A.H. designed strong baselines, evaluated rule 2, supported implementation, edited, and supervised.Source: Authors’ contributions · p. 14

Supplementary Information

The supplement audits reported comparisons across articles, assessing accuracy matching and baseline efficiency in speedup and superiority claims. It documents fair comparisons alongside claims affected by unequal accuracy, weak or questionable baselines, and cases where no superiority is claimed.

Article 1: Li et al.

  • A DG replication of the Fourier neural operator study [59] found a 7× speedup on GPU over a laptop CPU, not the claimed three orders of magnitude.The original runtime comparison was not at equal accuracy.Source: Supplementary Information · p. 15

Article 2: Lu et al.

  • On 1D advection, a DG method achieved similar accuracy with an order of magnitude lower runtime than the 100 × 100 finite-difference method used for DeepONet [60].The reported runtimes were 9 × 10−3 seconds for finite differences and 6×10−5 seconds for DG.Source: Supplementary Information · p. 15

Article 4: Kochkov et al.

  • The finite-volume baseline in the Navier–Stokes study was over 80x slower than a pseudo-spectral baseline on GPU; DG was 20-40× faster on CPU.The DG methods also solved the equations at 10 to 11× coarser resolution than the original baseline.Source: Supplementary Information · p. 16

Article 6: Bar-Sinai et al.

  • For 1D Burgers’ equation, DG matched WENO’s 4× to 8× coarser-resolution result, while the ML solver used 2-4× fewer degrees of freedom than DG.The authors judged the original comparison unfair because a stronger baseline improved performance over WENO.Source: Supplementary Information · p. 16

Articles 5, 9, 11, and 16

  • For airfoil-wing 2D compressible Navier–Stokes, CFD-GCD ran in 2.0s versus ground truth at 137s, but RMSE was 5.4×10−2 versus 0.0, so this was not an equal-accuracy comparison.Source: Supplementary Information · pp. 16, 18

Articles 3, 14, 18, and 19

  • For a 256 × 256 square-domain Poisson problem, direct LU took 12 ms versus FEniCS’s almost 20 s, over three orders faster; the article compares runtime at equal accuracy.Source: Supplementary Information · p. 18

Supplementary Information

  • Two audits found primary outcomes foregrounding weaker baselines despite stronger-baseline comparisons elsewhere, prompting a recommendation to report performance against the strong baseline.One case placed the stronger comparison in the appendix; another included it in a table but not the abstract’s primary outcome.Source: Supplementary Information · p. 20
  • Stevens and Colonius fairly found their method outperforms WENO in diffusive cases at equal accuracy, while Illarramendi et al. reported 3.2 times faster than Jacobi at the same accuracy.The review nevertheless notes that the Poisson comparison could use more efficient multigrid or direct-solve baselines.Source: Supplementary Information · p. 22
  • Stachenfeld et al. [87] could fairly compare accuracy with Athena++ at equal accuracy, but the 1000× faster claim was not fair and the baseline’s suitability for weakly compressible turbulence was uncertain.The review notes that DG or spectral methods may be state-of-the-art for weakly compressible turbulence, but says it lacks enough evidence to conclude they would outperform Athena++ on this test.Source: Supplementary Information · p. 23
  • 100, 682, and 800 speedups were reported for Han et al. [88], but resolution was not reduced to match accuracy and OpenFOAM drew an efficiency warning.The review describes OpenFOAM as an inefficient general-purpose tool and advises caution about the comparison.Source: Supplementary Information · p. 23
  • 8,000-fold speedups were claimed against a pseudo-spectral solver, but the ML models had non-zero error while the solver had zero error, and the baseline implementation was unusually slow.The review therefore judged the comparison unfair on both accuracy and baseline grounds.Source: Supplementary Information · p. 24
  • Between 600X and 300X speedups were reported for Chen et al., but surrogate errors were not matched and OpenFOAM may be inefficient.The review recommends comparing runtimes at equal error and cautions that OpenFOAM has high overhead and slow convergence.Source: Supplementary Information · pp. 24–25
  • 35× faster [100] is not an equal-accuracy comparison: at equal resolution, the ML solver’s error was 2.5× higher than FEM’s.The authors caution that the comparison likely overstates the advantage because resolution was used as a proxy for runtime instead of matching accuracy.Source: Supplementary Information · pp. 26–27
  • 840× speedup is qualified because table 5 compares highly accurate PhiFlow runtime with that of a less accurate ML solver.The article also reports significantly smaller runtime than the MP-PDE model, but the authors flag the ground-truth solver comparison as unfair.Source: Supplementary Information · pp. 27–28
  • [101]’s Jacobi baseline was questioned for elliptic PDEs, and [106]’s 1D Poisson multigrid baseline was challenged because direct solves are faster for small problems.The authors consider multigrid or a multigrid preconditioner highly efficient for elliptic PDEs, while noting that direct methods can outperform multigrid on small problems.Source: Supplementary Information · pp. 27–28
  • WENO3-NN was compared with weaker WENO5-JS rather than WENO5-Z, against which both schemes underperformed on 5 or 6 of 6 cases.The authors therefore judged Rule 2 unsatisfied, despite the reported comparison with WENO5-JS.Source: Supplementary Information · p. 29
  • 4 to 5 times faster at comparable accuracy was judged fair, while reports higher accuracy at about the same runtime versus Hybrid-MR.The comparison uses high-order DG and is reported at equal accuracy; the comparison is assessed against the Hybrid-MR scheme.Source: Supplementary Information · pp. 26, 31
  • The Nektar comparison is not at constant accuracy, and Zhang et al. and Yang et al. compare runtime against less accurate neural solvers without matching accuracy.For the airfoil case, Nektar has zero error while DeepONet has positive error; reports 0.02 sec versus 1.23 sec but does not reduce resolution to equalize accuracy.Source: Supplementary Information · p. 32
  • Bezgin et al. [122] compare accuracy at equal resolution, but the first-order Rusanov baseline is described as inefficient for this problem.Its excess numerical diffusion produces a smeared solution.Source: Supplementary Information · p. 32
  • Gopakumar et al. [126] report FNO predictions six orders of magnitude faster than JOREK without matched accuracy; separately, cross-coupling ML-CFD is 1.8× faster than the ground-truth CFD solver but has positive error.Source: Supplementary Information · pp. 33–34
  • The reported method reaches up to 12× speedup over MGPCG at all resolutions at equal accuracy, but direct LU is 35–500× faster than multigrid on sufficiently small Poisson problems.The listed LU times are 0.2, 0.4, 3, and 12 ms on 32×32 through 256×256 grids; multigrid takes 90, 189, 299, and 425 ms on 33×33 through 257×257 grids. Nastorg et al. [125] provide no evidence for their ten-times-faster-than-LU claim.Source: Supplementary Information · pp. 33, 35
  • Shit et al. [127] report a 19.2% speedup at equal error, but the supplement argues that super-time-stepping could be faster than the implicit baseline.The baseline uses timesteps of 0.2 and up to 25 fixed-point iterations; the proposed alternative is an explicit super-time-stepping method.Source: Supplementary Information · pp. 33–34
  • Sun et al. [131] compare accuracy at constant degrees of freedom and use efficient FEM and DG baselines for Poisson’s equation.This is assessed as a fair comparison.Source: Supplementary Information · p. 35
  • Articles,,,, [55], and [56] report parity, no speedup, slower performance, limited potential beyond spectral solvers, or no superiority claim.The supplement says the weak-shock NN scheme in is roughly 10 times slower than WCD; [56] is described as slow to train and evaluate.Source: Supplementary Information · pp. 35–36

Related papers

Loading 2407.07218v1…