Source-linked AI summary
Toward a principled Bayesian workflow in cognitive science
Daniel J. Schad, Michael Betancourt, Shravan Vasishth
TL;DR
Bayesian analyses in cognitive science require models that reflect both data structure and domain expertise, but choosing and checking such models remains difficult. This paper presents a principled workflow using reading-time data and shows that the observed data provided little evidence for the experimental effect of interest.
Problem
Choosing priors and determining whether a model adequately reflects the data and generating process remain difficult in Bayesian cognitive-science analyses.
Method
The paper demonstrates a principled Bayesian workflow using prior and posterior predictive checks, computational faithfulness, model sensitivity, and domain-informed priors.
Results
The observed data did not provide strong evidence for the experimental effect of interest.
Takeaways & Limitations
Principled workflow checks can help cognitive scientists assess whether Bayesian models capture relevant data structure and support robust scientific answers.
Takeaways & Limitations
Posterior contraction can be low for some simulated datasets, and large posterior predictive discrepancies may reflect extreme observations.
Abstract
from arXiv · showhide
Experiments in research on memory, language, and in other areas of cognitive science are increasingly being analyzed using Bayesian methods. This has been facilitated by the development of probabilistic programming languages such as Stan, and easily accessible front-end packages such as brms. The utility of Bayesian methods, however, ultimately depends on the relevance of the Bayesian model, in particular whether or not it accurately captures the structure of the data and the data analyst's domain expertise. Even with powerful software, the analyst is responsible for verifying the utility of their model. To demonstrate this point, we introduce a principled Bayesian workflow (Betancourt, 2018) to cognitive science. Using a concrete working example, we describe basic questions one should ask about the model: prior predictive checks, computational faithfulness, model sensitivity, and posterior predictive checks. The running example for demonstrating the workflow is data on reading times with a linguistic manipulation of object versus subject relative clause sentences. This principled Bayesian workflow also demonstrates how to use domain knowledge to inform prior distributions. It provides guidelines and checks for valid data analysis, avoiding overfitting complex models to noise, and capturing relevant data structure in a probabilistic model. Given the increasing use of Bayesian methods, we aim to discuss how these methods can be properly employed to obtain robust answers to scientific questions. All data and code accompanying this paper are available from https://osf.io/b2vx9/.
1 University of Potsdam, Germany · 2 Tilburg University, Netherlands · 3 Symplectomorphic, New York, USA
The supplied material identifies the paper as “Toward a principled Bayesian workflow in cognitive science” and situates it within prior work, a conference presentation, and previously published experimental data. It also highlights workflow and Bayesian data-analysis concepts as keywords.
- 3 Symplectomorphic, New York, USA: The principled Bayesian workflow was previously developed and documented by Betancourt (2018).
- 3 Symplectomorphic, New York, USA: Betancourt’s earlier work used a different, non-cognitive running example.
- 3 Symplectomorphic, New York, USA: The example experimental data analyzed in the manuscript were previously published by Gibson & Wu (2013).
- 3 Symplectomorphic, New York, USA: The listed keywords include Workflow, prior predictive checks, posterior predictive checks, model building, and Bayesian data analysis.
- 3 Symplectomorphic, New York, USA: The paper is titled “Toward a principled Bayesian workflow in cognitive science.”
Model building
Model building follows a principled Bayesian workflow that expands models iteratively while checking computational faithfulness and model adequacy. Prior and posterior predictive checks use domain knowledge to identify implausible assumptions and assess whether the model captures observed data structure.
- Model building: Iterative model expansion aims to approach an aspirational model while retaining a principled workflow for model building.Computation enables fitting and interpreting models that otherwise could not be validly estimated.
- Prior predictive checks: More informative priors produced expectations that were more reasonable and imposed more modest assumptions, with predicted effects smaller than 500 ms.The revised prior simulations reduced the frequency of extremely large reading times, variances, and effect sizes.
- Computational faithfulness: Computational checks found no divergent samples, while R̂ was very close to 1 in all models, indicating no convergence problems.Simulation-based calibration ranks were uniformly distributed, supporting accurate posterior estimation even for more diffuse priors.
- Posterior predictive checks: Posterior predictive checks found that the maximal model captured the data’s summary statistics well, although the lognormal approximation retained systematic deviations.Predictions captured observed means, residual standard deviations, effects, maximal effects, and random-effects standard deviations.