PaperlayerView exact revision on arXiv

Source-linked AI summary

Prior knowledge elicitation: The past, present, and future

Petrus Mikkola, Osvaldo A. Martin, Suyog Chandramouli, Marcelo Hartmann, Oriol Abril Pla, Owen Thomas, Henri Pesonen, Jukka Corander, Aki Vehtari, Samuel Kaski, Paul-Christian Bürkner, Arto Klami

arXiv:2112.01380v2stat.ME

TL;DR

Prior elicitation can translate domain knowledge into priors, but practical tools remain insufficient for routine use in Bayesian workflows. The paper organizes the problem and existing methods into a framework, then calls for coordinated research, including usable software and stronger demonstrations of value.

  • Problem

    Prior elicitation tools are not routinely used in practical Bayesian workflows, and existing solutions are often insufficient for practitioners’ needs.

    Source: §1 · p. 2, §2.2 · p. 6

  • Method

    The paper organizes prior elicitation into dimensions covering the model, elicitation algorithm, expert, and interaction, and reviews literature using that framework.

    Source: §2.3 · p. 8

  • Results

    The paper concludes that practical tools remain far off and that coordinated research involving multiple disciplines is needed to change this situation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend model-agnostic methods, efficient strategies compatible with iterative modelling, easier multivariate elicitation, and application examples demonstrating value.

    Source: §4 · p. 22

    Many multivariate methods are difficult to apply beyond textbook examples and do not scale well to high-dimensional parameter spaces.

    Source: §2.4 · p. 13

Abstract

from arXiv · show

Specification of the prior distribution for a Bayesian model is a central part of the Bayesian workflow for data analysis, but it is often difficult even for statistical experts. In principle, prior elicitation transforms domain knowledge of various kinds into well-defined prior distributions, and offers a solution to the prior specification problem. In practice, however, we are still fairly far from having usable prior elicitation tools that could significantly influence the way we build probabilistic models in academia and industry. We lack elicitation methods that integrate well into the Bayesian workflow and perform elicitation efficiently in terms of costs of time and effort. We even lack a comprehensive theoretical framework for understanding different facets of the prior elicitation problem. Why are we not widely using prior elicitation? We analyse the state of the art by identifying a range of key aspects of prior knowledge elicitation, from properties of the modelling task and the nature of the priors to the form of interaction with the expert. The existing prior elicitation literature is reviewed and categorized in these terms. This allows recognizing under-studied directions in prior elicitation research, finally leading to a proposal of several new avenues to improve prior elicitation methodology.

1 Introduction

Prior elicitation aims to turn domain experts’ knowledge into model priors, but its methods are not routinely used in Bayesian workflows. The paper examines why, organizes the problem’s dimensions and literature, and proposes research directions for practical tools and evaluation.

1 Introduction

  • Elicitation is one part of the Bayesian workflow: it seeks to convert domain knowledge into priors that help models capture essential properties of the system under study.The paper also presents faster, easier model specification and reduced required statistical expertise as goals for good tools.Source: §1 · p. 2
  • Despite a long history and available methods, prior elicitation tools remain far from routine use, and practitioners often specify priors ad hoc.The paper notes that existing interactive tools serve only a tiny fraction of probabilistic model users.Source: §1 · p. 2
  • Technical, practical, and societal barriers to uptake are interconnected, so addressing them requires coordinated effort across complementary expertise.The paper names limited model scope, difficulty for non-statistical experts, weak software integration, and insufficiently visible demonstrations of value.Source: §1 · p. 3
  • The paper frames prior elicitation around seven dimensions, then reviews methods and offers practical guidelines for how these characteristics shape desirable methods.The dimensions cover the prior, model and method, elicitation space, interpretation of expert input, computation, interaction, and expert capability.Source: §1 · p. 3
  • The authors propose future research directions and identify easy-to-use software integrated with open probabilistic programming platforms as necessary for practical impact.They also call for considerably extended user evaluation to verify methods’ practical value.Source: §1 · pp. 3–4

2 Prior Elicitation

This section examines prior elicitation as the conversion of domain experts’ knowledge into Bayesian priors and considers technical, practical, and societal barriers to its adoption. It uses a seven-dimension framework to organize the problem and research directions, including uneven coverage across univariate, observable-space, and multivariate methods.

2.1 What is prior elicitation?

  • Prior elicitation structures the gathering of experts’ subjective knowledge and its computational transformation into prior distributions; whether it should depend on the likelihood remains debated.The authors distinguish elicitation from other ways to specify informative priors.Source: §2.1 · pp. 4–5

2.2 Why isn’t the use of prior elicitation widespread (yet)?

  • The authors attribute limited adoption to intertwined gaps in general methods, workflow-ready tools, evaluation resources, and high-profile demonstrations of value.They identify technical, practical, and societal reasons, and argue that concrete success stories could attract broader interest and resources.Source: §2.2 · pp. 6–7

2.2 Why isn’t the use of prior elicitation widespread (yet)

  • Many algorithms address specific models or prior forms, tools remain isolated from common modeling ecosystems, and real-workflow evaluation requires costly studies with expert users.Evaluating complex, iterative modeling may require substantial time and large user groups, and is feasible only after software has matured.Source: §2.2 · pp. 6–7

2.3 Prior elicitation hypercube

  • The proposed hypercube organizes prior elicitation along seven dimensions covering prior properties, model family, elicitation space and model, computation, and expert assumptions and interaction.The framework is intended to structure the literature review and reveal understudied directions.Source: §2.3 · p. 8
  • Most reviewed methods are model-specific; observable-space elicitation can help when parameters lack meaning, while iterative predictive methods require fast computation.The authors encourage methods applicable across a wider range of models.Source: §2.3 · pp. 9–11
  • Assessment format and one-shot versus iterative interaction matter because tasks should elicit thoughtful, auditable answers without psychological bias, while experts may struggle to express knowledge probabilistically.The interaction framework also considers how multiple experts discuss and combine their judgments.Source: §2.3 · pp. 11–12
  • When probabilistic judgments are unreliable, hypothetical-sample tasks can substitute; vague knowledge calls for validation with seed variables, while even expert judgments remain vulnerable to heuristics and biases (Tversky and Kahneman, 1974; Hogarth, 1975).Hypothetical samples are described by Casement and Kahle (2018), and seed variables by Cooke (1991).Source: §2.3 · p. 12

2.4 Overview on past literature

  • Univariate parametric elicitation is extensively studied, often querying summaries such as quantiles; methods vary in summary, order, and framing, with bias-reduction protocols that not all methods properly follow.Source: §2.4 · p. 12
  • Elicitation in observable space is abundant but almost always model-specific, with substantial work on generalized linear models and Beta and Dirichlet priors.Source: §2.4 · p. 12
  • Separate lines address scoring rules that encourage careful probability assessments (Murphy and Winkler, 1970), Gaussian-process supra-Bayesian nonparametric elicitation (Oakley and O’Hagan, 2007), and active learning to economize costly expert effort (Cohn et al., 1994).Source: §2.4 · pp. 12–13
  • Multivariate methods often fail to scale beyond textbook examples because feasible yet informative assessment tasks and algorithms for forming joint priors from elicited data remain unresolved.Copula-based elicitation requires cognitively challenging dependency assessments; a Gaussian copula needs dim(θ)(dim(θ) + 1)/2 covariance elements (Garthwaite et al., 2005, Sec. 2.3; Clemen and Reilly, 1999). Gaussian-process elicitation has been demonstrated only for two parameters (Moala and O’Hagan, 2010), while predictive methods face growing query burdens and, in some cases, troublesome covariate-independence assumptions (Kadane et al., 1980; Garthwaite and Dickey, 1988; Bedrick et al., 1996).Source: §2.4 · p. 13

3 Where Should We Be Going?

This section proposes avenues to address technical, practical, and societal challenges in prior elicitation and increase its adoption. It highlights software that combines accessible visual interaction with modular, interoperable components configurable across elicitation spaces, assessment tasks, and models.

3.1 Bayesian treatment of the expert in prior elicitation

  • A unified Bayesian treatment of the expert can combine parameter- and observable-space inputs, with active query selection and AI assistance supporting sample-efficient elicitation.The approach models expert input as data and updates beliefs about the expert’s knowledge using Bayesian inference.Source: §3.1 · pp. 14–15

3.2 Bayesian modelling workflow

  • Prior elicitation belongs within an iterative Bayesian workflow, occurring when domain knowledge and limited data make it worthwhile and extending to choices among models.The workflow should assess whether elicited information is useful and examine sensitivity to prior choice, including possible prior-data conflicts.Source: §3.2 · pp. 16–17

3.3 Developing better priors

  • Priors with fewer, intuitive hyperparameters can ease expert understanding and computation; globally joint priors spanning most or all parameters could make elicitation more practical.Existing globally joint shrinkage priors have not been studied for elicitation and remain limited in generalization beyond linear regression.Source: §3.3 · pp. 18–19

3.4 Evaluating prior elicitation

  • Evaluation needs broader comparative studies, standardized validation paradigms, and more reliable, model-agnostic metrics, since current comparisons are sparse and common proxies may be misleading.Faithfulness can be tested against a known ground truth, while evaluation should also consider how elicited priors affect predictions and inference sensitivity.Source: §3.4 · pp. 19–20

3.5 Software for Prior Elicitation

  • Prior-elicitation software should interoperate with existing Bayesian tools, automate tedious or error-prone computation, and help users make informed decisions while retaining control.The absence of general software integrating with probabilistic programming languages and other tools hinders adoption.Source: §3 · pp. 13–14, §3.5 · p. 21
  • Software should be open-source and intuitive, with visual input and validation, interactive and switchable displays, and user-defined transformations to support elicitation.Sarma and Kay (2020) report that visualizations can prompt different elicitation strategies and that participants often combine strategies; Kay et al. (2016) describe quantile dotplots as an alternative amid difficulties interpreting probability densities.Source: §3.5 · p. 21
  • Language-agnostic interoperability and modular design built on existing libraries could reduce duplicated effort and computational costs while enabling community-maintained software components.The passage names ggdist, Bayesplot, and ArviZ as libraries to build on.Source: §3.5 · p. 21
  • Modular tools should let users switch between parameter and observable spaces, choose assessment tasks, and elicit priors independently of model specification to support Bayesian workflow.The appropriate elicitation space remains an open research question, and model changes can otherwise preclude prior elicitation.Source: §3.5 · pp. 21–22

4 Conclusion

The paper calls for coordinated work on workflow-oriented elicitation methods, integrated open software, practical evaluation, and applications that demonstrate value. It hypothesizes that addressing these four priorities will speed future development through infrastructure, public interest, and funding.

4 Conclusion

  • Prioritize methods that are model- and prior-agnostic, efficient for modellers, compatible with iterative model-building, and better suited to multivariate priors.Simpler-to-elicit hierarchical priors are one example of a multivariate-prior strategy.Source: §4 · p. 22
  • Develop modular, open-source software that integrates with modelling workflows and reuses existing libraries for non-algorithm-specific components where possible.The proposed software should let new elicitation algorithms be taken up and evaluated quickly in concrete modelling cases.Source: §4 · p. 22
  • Build cost-efficient evaluation techniques and a testbed for comparing elicitation methods across situations, alongside ways to measure efficiency in real use.The testbed should support feasible experimentation costs.Source: §4 · p. 22
  • Use societal-interest applications to demonstrate value where subjective prior knowledge is clearly useful and elicitation improves model value, cost, or robustness.The paper also points to repeated or parallel prior-specification settings as examples for cost reduction or improved robustness.Source: §4 · p. 22
  • The authors hypothesize that these four priorities together could provide infrastructure, public interest, and funding to accelerate future development.They say concrete directions for the first two priorities are outlined in Section 3.Source: §4 · p. 23

Current Main Lines of Research

Prior elicitation research spans many disciplines and application areas, and differing perspectives and terminology complicate systematic review. The review combined forward searches with backward reference searching, which found most of the relevant references.

Current Main Lines of Research

  • Prior elicitation research spans statistics, psychology, economics, decision sciences, machine learning, and many application areas, with varied terminology complicating systematic review.The passages note terms such as “expert judgement,” “expert knowledge elicitation,” and “probability encoding” in decision sciences.Source: Current Main Lines of Research · pp. 33–34
  • Forward searches used three Google Scholar queries in March 2021, yielding 1,315 results screened for published, available, methodologically pertinent work.The review included exceptional cases that did not meet all three criteria.Source: Current Main Lines of Research · p. 34
  • Backward reference searching found most relevant references, while the forward-search strings reached only a small fraction of prior elicitation research.Source: Current Main Lines of Research · p. 34

S1 Elicitation in parameter space

This section reviews elicitation in parameter space, where experts provide information about model parameters and methods turn that information into prior distributions. It covers univariate and multivariate techniques, prior-specific methods, and scoring rules.

S1.1 Univariate prior

  • Univariate elicitation often asks experts for summaries such as quantiles, moments, or bounds, which can identify a prior distribution when well chosen.The section describes quantile-based variable-interval methods and probability-based fixed-interval methods; one experiment found slight but consistent superiority for the variable-interval method across several dimensions.Source: S1.1 Univariate prior · p. 35
  • Fixed-interval methods fit a parametric prior to elicited probabilities, while histogram methods assign probabilities to bins and can yield nonparametric or fitted priors.The variable-interval method elicits quartiles, fits a parametric distribution, and presents it back for possible revision; the histogram method can use roulette chips to allocate probabilities.Source: S1.1 Univariate prior · pp. 35–36

S1.2 Multivariate prior

  • Multivariate elicitation can transform parameters to subjectively independent coordinates, elicit conditional distributions, or combine marginal priors with dependence information.Correlations can be difficult to assess; Gaussian-copula approaches decompose the joint prior into marginals and a copula, while other methods elicit joint probabilities or isoprobability contours.Source: S1.2 Multivariate prior · pp. 36–37

S1.3 Beta and Dirichlet prior

  • Beta-prior methods elicit a location and a dispersion measure or credible interval, then use these assessments to determine the distribution’s parameters.The described methods differ in whether they ask for the mode, median, mean, or interval first.Source: S1.3 Beta and Dirichlet prior · p. 38
  • Dirichlet elicitation from beta marginals requires reconciliation because arbitrary marginal beta distributions may not form a consistent Dirichlet prior.Consistency requires the beta parameters to sum to the same value across marginals and their expectations to sum to one.Source: S1.3 Beta and Dirichlet prior · p. 38

S1.4 Scoring rules

  • Scoring-rule elicitation frames prior assessment as a gamble, with rewards designed to make the expert’s reported distribution reflect their subjective probabilities.Scoring-rule elicitation assumes risk neutrality in the example, and choosing a rule that reliably reflects an expert’s risk preference is difficult; such preferences should also be elicited.Source: S1.4 Scoring rules · pp. 39–40
  • Scoring-rule systems often require experts to report a full distribution at once, while ELI offers an interactive graphical approach and showed improved calibration and accuracy in a study of 192 subjects.Other scoring approaches have been studied that elicit quantiles rather than a full density.Source: S1.4 Scoring rules · p. 40

S2 Elicitation in observable space

Elicitation in observable space asks experts about model outputs and uses those assessments to construct priors for model parameters. The section reviews its rationale, regression methods, and approaches based on prior predictive distributions.

S2 Elicitation in observable space

  • The review found one model-agnostic method eliciting in observable space, compared with 27 model-specific methods in its table.Observable-space elicitation can indirectly assess correlations that experts may not naturally consider between regression coefficients.Source: S2 Elicitation in observable space · p. 41
  • Observable-space elicitation can be difficult because randomness arises both from parameters and from observations, and complex models may yield similar predictions from different priors.Highly correlated parameters can make unique prior distributions difficult to recover from elicited patterns in observables.Source: S2 Elicitation in observable space · p. 41
  • For regression coefficients, elicitation procedures can use conditional percentiles of responses to estimate a positive-definite covariance matrix, while some alternatives do not use expert knowledge about response-predictor relationships.The coefficients are modeled as normal with mean vector b and covariance matrix σ2R; a structured sequential query procedure is one approach to ensuring R is positive-definite.Source: S2 Elicitation in observable space · p. 42
  • The conditional means prior derives a prior on regression coefficients from expert-assessed mean responses at selected covariate combinations.Given the GLM link function and covariate matrix, the method obtains b by transforming the elicited means.Source: S2 Elicitation in observable space · p. 42
  • When an elicited observable distribution and the likelihood are specified, recovering the parameter prior requires solving a first-kind Fredholm integral equation that is ill-posed.The review notes that additional regularity assumptions may be needed, including smoothness constraints, or a parametric prior may reduce the task to estimating hyperparameters.Source: S2 Elicitation in observable space · p. 43
  • Observable-space methods also use simulations and predictive judgments, including comparisons with reference values and judgments about whether simulated outcomes are realistic.One approach trains a Gaussian-process classifier on binary realism judgments to recover a prior over realistic parameters.Source: S2 Elicitation in observable space · p. 44

S3 Nonparametric priors

This section contrasts parametric and nonparametric priors for representing elicited knowledge. Nonparametric approaches can accommodate a wider range of elicitation data, while particular methods retain important scope limits.

S3 Nonparametric priors

  • Parametric elicitation constrains the expert’s knowledge distribution to a chosen family, whereas nonparametric priors can represent a wider range of elicitation data structures.The review characterizes nonparametric priors as offering greater flexibility for finding a prior that satisfactorily reflects elicited knowledge.Source: S3 Nonparametric priors · pp. 44–45
  • A supra-Bayesian approach models the expert’s density as a Gaussian process and uses elicited summaries, such as expectations and percentiles, to infer the process posterior.The GP’s mean and covariance function provide a way to incorporate assumptions about the expert’s knowledge distribution.Source: S3 Nonparametric priors · p. 45
  • Later work addressed possible underestimation of uncertainty in the tails and situations where the analyst’s method assumes certainty about the expert’s true density.The proposed modification used a Student’s t distribution for the prior mean of the expert’s knowledge distribution.Source: S3 Nonparametric priors · p. 45
  • A multivariate GP elicitation study found that marginal probabilities supplemented by a few joint probabilities could produce good results, but demonstrated the method only for two parameters.In a numerical experiment, the nonparametric GP model adapted to a bimodal target distribution.Source: S3 Nonparametric priors · p. 45
  • Nonparametric alternatives discussed include histograms built from relative odds and Gaussian kernel density estimates for locations in two-dimensional visual scenes..Source: S3 Nonparametric priors · p. 46

S4 Model-specific methods

The authors focus on model-agnostic methods elsewhere in the article, while noting that model-specific methods can still be useful for widely used model families such as GLMs.

S4 Model-specific methods

  • The review does not cover model-specific elicitation methods in detail, prioritizing model-agnostic methods discussed elsewhere in the article.The named examples of model-agnostic methods include work by Gelfand et al. (1995), Gosling (2005), Oakley and O’Hagan (2007, 2019), and Hartmann et al. (2020).Source: S4 Model-specific methods · p. 46
  • Despite their limited model family, generalized linear models are widely used across fields, so useful elicitation tools for GLMs may support practical modeling tasks.The section points to Table 1 as a starting point for exploring model-specific methods by model, prior, and elicitation space.Source: S4 Model-specific methods · p. 46

S5 Active elicitation

Active elicitation uses iterative expert queries to make better use of limited elicitation time. The section reviews query strategies, convergence questions, and the role of modelling expert behaviour.

S5 Active elicitation

  • A limited expert-query budget motivates active elicitation, which selects new queries to make the most of the expert’s time.It applies active-learning principles by querying the expert rather than collecting new data.Source: S5 Active elicitation · p. 46
  • Prior-elicitation strategies for sparse linear regression include maximizing expected information gain in outcome prediction and selecting queries through a linear bandit bound on feature-relevance uncertainty.Sundin et al. (2018) considered gain for a single personalized-medicine prediction rather than a whole training dataset.Source: S5 Active elicitation · pp. 46, 48
  • Active elicitation is iterative, but the authors report no known convergence guarantees for prior-elicitation tasks, unlike cited clustering-query settings.Examples of clustering guarantees are attributed to Balcan and Blum (2008), Awasthi et al. (2014), Ashtiani et al. (2016), and Kane et al. (2017).Source: S5 Active elicitation · p. 48
  • Modelling expert behaviour can improve performance: Daee et al. (2018) reported a significant gain in feature-relevance elicitation for sparse linear regression.The expert is modelled to anticipate input and account for biases; some cited studies also showed data during elicitation or used elicitation data as auxiliary data.Source: S5 Active elicitation · p. 48

S6 Multiple experts

The section briefly surveys elicitation with multiple experts, pointing readers to further references and describing protocols and two approaches to combining expert knowledge.

S6 Multiple experts

  • The article treats elicitation from groups of experts only briefly, directing readers to reviews and a comparison study for fuller coverage.It points to O’Hagan (2019, Sec. 3), Garthwaite et al. (2005, Sec. 5), Jenkinson (2005, Sec. 7), and Williams et al. (2021).Source: S6 Multiple experts · p. 48
  • Four protocols are identified for coordinating multi-expert elicitation: Sheffield, Cooke, Delphi, and IDEA.The cited overviews include EFSA (2014), Dias et al. (2018), and O’Hagan (2019).Source: S6 Multiple experts · p. 48
  • Aggregation combines separately elicited expert distributions through a pooling rule, or fits a distribution to group consensus judgments.Supra-Bayesian pooling is given as a popular mathematical pooling rule.Source: S6 Multiple experts · p. 49

S7 Heuristics and biases

The section notes cognitive biases in expert probability judgments and calls for elicitation methods to address them. It points readers to relevant reviews and discussion.

S7 Heuristics and biases

  • Expert probabilistic judgments are subject to cognitive biases and heuristics, including anchoring, availability, range-frequency, and overconfidence.O’Hagan (2019) highlights these four; the passage also notes controversy about the literature’s treatment of framing.Source: S7 Heuristics and biases · p. 49
  • Every prior-elicitation method should address systematic cognitive biases; minimizing them has been a leading principle in designing the cited elicitation protocols.The passage cites O’Hagan (2019) on protocol design and points to O’Hagan (2019, Sec. 2), Garthwaite et al. (2005, Sec. 2), Kynn (2008), Gigerenzer and Todd (1999), and Burgman (2016) for entry points.Source: S7 Heuristics and biases · p. 49
Loading 2112.01380v2…