Source-linked AI summary
Why So Many Published Sensitivity Analyses Are False. A Systematic Review of Sensitivity Analysis Practices
Andrea Saltelli, Ksenia Aleksankina, William Becker, Pamela Fennell, Federico Ferretti, Niels Holst, Sushan Li, Qiongli Wu
TL;DR
The paper addresses inadequate exploration and limited standardization in published uncertainty and sensitivity analyses by reviewing papers in the literature. It reports substantial flaws in the reviewed analyses and highlights concerns about model-linearity clarity and large-budget models used for significant decisions.
Problem
Published analyses leave a very large fraction of the input space unexplored, while uncertainty and sensitivity analysis lack standardization.
Method
The paper reviews published papers to assess whether their uncertainty and sensitivity analyses adequately address the input space.
Results
65% ((57+125)/280) of the reviewed papers are based on flawed methods, while 37% (57/(280-125)) contain a fundamentally flawed method among the remaining papers.
Takeaways & Limitations
Both uncertainty and sensitivity analysis should generally be performed, with appropriate methods needed as model-input complexity increases.
Takeaways & Limitations
The analysis is constrained by unclear linearity in the models being analysed and raises particular concern for large-budget models used in significant decisions.
Abstract
from arXiv · showhide
Sensitivity analysis (SA) has much to offer for a very large class of applications, such as model selection, calibration, optimization, quality assurance and many others. Sensitivity analysis offers crucial contextual information regarding a prediction by answering the question "Which uncertain input factors are responsible for the uncertainty in the prediction?" SA is distinct from uncertainty analysis (UA), which instead addresses the question "How uncertain is the prediction?" As we discuss in the present paper much confusion exists in the use of these terms. A proper uncertainty analysis of the output of a mathematical model needs to map what the model does when the input factors are left free to vary over their range of existence. A fortiori, this is true of a sensitivity analysis. Despite this, most UA and SA still explore the input space; moving along mono-dimensional corridors which leave the space of variation of the input factors mostly unscathed. We use results from a bibliometric analysis to show that many published SA fail the elementary requirement to properly explore the space of the input factors. The results, while discipline-dependent, point to a worrying lack of standards and of recognized good practices. The misuse of sensitivity analysis in mathematical modelling is at least as serious as the misuse of the p-test in statistical modelling. Mature methods have existed for about two decades to produce a defensible sensitivity analysis. We end by offering a rough guide for proper use of the methods.
1. Background
Sensitivity analysis apportions output uncertainty among model inputs, whereas uncertainty analysis characterizes prediction uncertainty. The paper argues that many published analyses inadequately explore input space and reviews practices across disciplines to identify problems and propose guidance.
- Sensitivity analysis attributes uncertainty in model output to different input factors, while uncertainty analysis characterizes uncertainty in the prediction.
- SA supports factor prioritisation, factor fixing, structural optimisation, model understanding, and quality assurance.It can guide additional information collection about influential parameters or identify inputs that contribute little and may be fixed.
- The paper assesses sensitivity analysis across academic disciplines through a systematic review of highly cited papers and discusses known problems and misinterpretations.
- Global methods should account for multidimensional averaging, model nonlinearity, non-additivity, interaction effects, input distributions, and grouped factors.One-factor-at-a-time methods vary one input while holding others at nominal values, unlike global methods.
- Many published UA and SA leave a very large fraction of the input space unexplored, failing to propagate input uncertainty comprehensively.The paper describes this as a central problem in the literature.
- Global sensitivity analysis can also function as model verification because it may detect model errors or unexpectedly strong dependencies.
- The authors present the paper as a concerned scholarly contribution rather than a statement representing the entire field.
2. The literature review
The review uses a bibliometric search of highly cited, model-related sensitivity-analysis papers to assess practice across disciplines. Its restrictive criteria yield a manageable, field-spanning sample but undercount the literature.
- Review design: The study analyzes highly cited sensitivity-analysis papers to assess methodological rigor across academic disciplines.The authors reason that highly cited articles should represent average good practice within their fields.
- Search criteria: The restrictive query returned around 6000 articles, whereas searching only “sensitivity analysis” returned around 47,000 and was judged too irrelevant.The authors considered the restrictive query an unbiased automatic selection strategy across fields.
- Scope: The sample is considered representative, but its paper count is significantly below the true number of sensitivity-analysis papers.The query excludes relevant papers that do not mention “model” in the title, abstract, or keywords.
- Sampling: Articles were assigned one or more subject identifiers, while fields with fewer than 100 qualifying articles were excluded.The resulting set contained 19 subject areas.
15. MATHS (Maths)
The review covers sensitivity-analysis publications across fields, using a citation-based sample after screening the search results. Its field distribution is reported both by density and by raw article count.
- Field distribution: The review examines sensitivity-analysis occurrence across research fields using article density and raw publication counts.Density is defined as the number of sensitivity-analysis papers divided by the total number in the search period.
- Field distribution: Decision science has the greatest sensitivity-analysis density, alongside model-intensive fields such as earth and environmental science and energy.Environmental science, engineering, and medicine have the greatest raw numbers.
- Sampling: The review selected the top twenty most-cited papers from each field, with some papers appearing in multiple field lists.A total of 324 papers were reviewed, and 280 were retained for analysis after exclusions.
- Screening: Forty-four papers were discarded because they lacked sensitivity or uncertainty analysis or analyzed output dependence on only one factor.The authors explicitly exclude one-factor dependence as not constituting sensitivity analysis.
3. Results
The review finds widespread confusion between uncertainty and sensitivity analysis and substantial reliance on one-factor-at-a-time methods. Global methods are often necessary because most reviewed models are nonlinear or unclear in linearity.
- Model focus versus application focus: Most reviewed papers focus on applications rather than introducing sensitivity-analysis methodology.Model-focused papers use sensitivity analysis to investigate a model, whereas method-focused papers use a model as a case study.
- Sensitivity-analysis methods: Of 35 methodological papers, 24 advocate global methods, while a small but significant fraction still advises statistically incorrect OAT methods.The review also reports a marked preference for variance-based sensitivity measures and active research on moment-independent methods.
- Model linearity: Only 8% of cases were definitely linear, whereas over half included clearly nonlinear models and the remainder were unclear.For linear models, OAT or derivative-based approaches would be adequate; nonlinear models require sampling derivatives across different input-space locations.
- Sensitivity-analysis methods: Global methods are essential for methodologically sound sensitivity analysis in the large majority of reviewed cases.Global methods account for simultaneous variation of other factors and can capture interaction effects in nonlinear, non-additive models.
- Uncertainty analysis: Twenty-four of 280 papers contained only uncertainty analysis, indicating clear conflation of UA and SA.In Pharmacology and Toxicology, four reviewed papers had sensitivity analysis, compared with ten having uncertainty analysis.
- Uncertainty analysis: About three-quarters of papers had no uncertainty analysis or an unclear UA methodology, although about three-quarters of observed UAs were global.The authors attribute the first result partly to targeting sensitivity-analysis papers.
- Sensitivity analysis: Only 41% of sensitivity analyses used global methods, while 34% used OAT methods and 25% had unclear or absent method types.The authors describe the OAT share among highly cited papers as deficient.
- Sensitivity analysis: Global-method prevalence varies widely across disciplines, exceeding 70% in Immunology and Microbiology but remaining about 10% in Pharmacology and Toxicology and 20% in Business, Management and Accounting.Earth and Environmental Science also show low global-sensitivity-analysis rates despite reliance on large computer models.
Conclusions
The review finds that published UA/SA practices are frequently methodologically flawed, especially when analyses fail to explore the joint input space. It recommends paired, exploratory UA and SA as routine components of model quality assurance.
- Findings: The analysis identified 125 papers with unclear method or model linearity and 57 papers using one-factor-at-a-time UA or SA.This lack of clarity complicates estimating the proportion of inadequate or inappropriate methods.
- Findings: 65% ((57+125)/280) of reviewed papers were judged based on flawed methods when unclear linearity was treated as a methodological flaw.Among papers with clear method and model linearity, 37% (57/(280-125)) contained a fundamentally flawed UA/SA approach; even the most generous interpretation found over 20% (57/280) inadequate.
- Findings: One-at-a-time methods fail to explore the input-factor space properly, systematically under-estimating uncertainty and wrongly estimating sensitivity.The review links this failure to elementary problems of experimental design.
- Interpretation: The authors note that their highly cited sample may represent average or better-than-average practice, making the results potentially optimistic.The sample therefore cannot necessarily be treated as a complete picture of all published practice.
- Recommendations: Despite disciplinary differences, all fields would benefit from good practices combining uncertainty and sensitivity analysis.The authors recommend exploring the input-factor space through experimental design, Monte Carlo, or other approaches, then assessing relative factor importance visually or quantitatively.
- Recommendations: UA should precede SA because factor importance has different relevance depending on whether the model output has small or large variance.The authors therefore regard sensitivity analysis without uncertainty analysis as illogical.
- Recommendations: The preferred sensitivity methods are exploratory, model-free, able to capture interactions, and able to treat groups of factors.The authors describe carefully performed UA followed by SA as important for model quality assurance and model-based inference.
- Implications: The review describes inaccurate sensitivity analyses as a serious modelling-quality problem and calls for urgent action on model quality assurance.It compares the concern to the misuse of p-values in data analysis, while grounding its conclusion in the poor quality of many published sensitivity analyses.