PaperlayerView exact revision on arXiv

Source-linked AI summary

Statistical methods for linguistic research: Foundational Ideas - Part II

Bruno Nicenboim, Shravan Vasishth

arXiv:1602.00245v1stat.APstat.ME

TL;DR

Bayesian data analysis is not yet standard in linguistics, and frequentist quantities are often given incorrect Bayesian interpretations. This paper offers a practical review centered on Bayesian linear mixed models, covering estimation, Bayes factors, cross-validation, and flexible model specification, while illustrating both interpretive benefits and prior sensitivity.

  • Problem

    Bayesian data analysis is not yet part of linguistics’ standard statistics curriculum, while p-values and confidence intervals are often misinterpreted as probabilities about hypotheses or parameters.

  • Method

    The paper provides a non-technical, practically oriented review using Bayesian linear mixed-model analysis of psycholinguistic data, including parameter estimation, Bayes factors, cross-validation, and flexible model specification.

  • Results

    The review demonstrates how Bayesian analyses summarize effects with posterior estimates and credible intervals, test hypotheses with Bayes factors, compare models using cross-validation, and interpret weak evidence without equating failed detection with no effect.

  • Takeaways & Limitations

    Bayesian methods can complement linguists’ existing frequentist toolkit by providing hypothesis-relevant uncertainty statements, evidence for null hypotheses under specified conditions, and flexible model specification.

  • Takeaways & Limitations

    Bayes factors are always affected by the prior, so sensitivity analysis is advisable when reporting them.

Abstract

from arXiv · show

We provide an introductory review of Bayesian data analytical methods, with a focus on applications for linguistics, psychology, psycholinguistics, and cognitive science. The empirically oriented researcher will benefit from making Bayesian methods part of their statistical toolkit due to the many advantages of this framework, among them easier interpretation of results relative to research hypotheses, and flexible model specification. We present an informal introduction to the foundational ideas behind Bayesian data analysis, using, as an example, a linear mixed models analysis of data from a typical psycholinguistics experiment. We discuss hypothesis testing using the Bayes factor, and model selection using cross-validation. We close with some examples illustrating the flexibility of model specification in the Bayesian framework. Suggestions for further reading are also provided.

1 Introduction

Bayesian data analysis is presented as a practically accessible framework that complements frequentist methods and addresses common interpretive problems. The review focuses on tools, foundations, and applications relevant to linguistics and related fields.

  • Frequentist methods remain important in linguistics, but Bayesian data analysis offers an additional framework with substantial practical value.
  • Researchers often interpret p-values as probabilities that null hypotheses are true, although this interpretation is incorrect.A p-value of 0.06 can therefore seem misleadingly similar to 0.04 when treated this way.
  • Bayesian methods have become easier to use as computing power and probabilistic programming languages have advanced.
  • The paper provides a non-technical, practically oriented review of Bayesian data-analysis tools, centered on linear mixed models for linguistics.It also mentions extensions and supports readers seeking to understand papers using Bayesian statistical inference.
  • The review introduces Bayesian foundations, hypothesis evaluation with Bayesian linear mixed models, applications of standard and less-standard models, and further reading.

2 Why bother to learn Bayesian data analysis?

The paper argues that Bayesian methods deserve greater attention in linguistics because they directly address hypothesis plausibility and support flexible hierarchical modeling. These advantages are especially relevant to repeated-measures research.

  • Bayesian methods directly quantify how plausible a hypothesis is given the data by representing uncertainty about parameters of interest.
  • Bayesian methods make hierarchical models more flexible to specify than frequentist methods.
  • Hierarchical models account for between-group and within-group variance while pooling information through shrinkage.
  • These models can reduce overfitting and preserve information about group-level variability in repeated-measures designs.
  • Bayesian methods can fit maximal random-effects structures that frequentist models may fail to converge on or estimate unrealistically.

3 Bayesian data analysis: An informal introduction

Bayesian analysis estimates plausible parameter values by combining prior information with the likelihood through Bayes’ theorem. Unlike significance testing, it emphasizes posterior uncertainty rather than only null-hypothesis rejection.

  • Bayesian analysis estimates a parameter and its uncertainty from the data together with prior information about plausible values.
  • The posterior distribution of a parameter is expressed in terms of its prior distribution and the likelihood.
  • Bayesian inference focuses on plausible parameter values and evidence for hypotheses rather than dichotomous null-hypothesis decisions.
  • When analytical derivation is unavailable, numerical methods such as MCMC can approximate posterior distributions.
  • The likelihood gives the joint probability of observed data for different parameter values.
  • Figure 1 illustrates posterior distributions produced by different likelihoods and priors for binomial data.

4 An example of statistical inference using Bayesian methods

The paper demonstrates Bayesian inference with a linear mixed model for a two-condition psycholinguistic reading-time experiment. The example emphasizes prior specification, posterior computation, and improved handling of random-effects correlations.

  • The Gibson and Wu example compares reading times for subject and object relative clauses using a Bayesian linear mixed model.
  • The Bayesian model parallels familiar lme4 syntax while adding priors and MCMC settings such as chains, iterations, and cores.
  • Table 1 compares estimates from the frequentist maximal LMM with estimates from its analogous Bayesian LMM.
  • The latency characterization used in the example is not necessarily the best characterization of latencies.
  • Bayesian priors on random-effects correlations yield near-zero posterior mean correlations with wide uncertainty when data are insufficient.With more data, the posterior compromises between prior and likelihood, although uncertainty may remain high.

5 Prior specification

Bayesian models require priors for each parameter, and prior specification should reflect plausible values while being checked for sensitivity and scale. The paper also highlights that Bayesian and frequentist estimates can be comparable while random-effect correlations may differ substantially.

  • 5 Prior specification: Priors express initial knowledge about possible parameter values and should be specified for every parameter in a Bayesian model.Flat priors are possible but may place too much probability mass outside reasonable posterior values.
  • 5 Prior specification: Weakly informative priors generally have little effect on posteriors in typical psycholinguistic experiments, but sensitivity analysis is recommended.Informative priors can incorporate results from previous experiments, meta-analyses, or expert judgments.
  • 5 Prior specification: Frequentist and Bayesian estimates are comparable for coefficients and variance components, but correlations between varying intercepts and slopes differ substantially.The table reports medians and median absolute differences for Bayesian fixed effects, while means and standard errors could also be used.
  • 5 Prior specification: Explicit priors are preferable because software defaults may not suit the specific data, and their interpretation depends on the dependent variable’s scale.A Normal(0,10) intercept prior is extremely vague on a log scale but implies implausible millisecond values without log transformation.
  • 5 Prior specification: The example begins with Normal(0,1) as the prior for the subject-versus-object relative-clause effect, corresponding to 68% certainty that the difference is below 1300 ms.The authors then vary the experimental-condition prior to illustrate prior effects and note that random-effect covariance priors can regularize correlations when data are sparse.

6 The posterior and statistical inference

Bayesian inference summarizes the posterior distribution of a parameter and uses probabilities, point estimates, and credible intervals to describe plausible values. In the running example, the posterior assigns probability 0.89 to a negative effect and 0.67 to an effect below −0.02.

  • 6 The posterior and statistical inference: A posterior distribution represents the relative plausibility of each parameter value conditional on the data, priors, and model.Software such as rstanarm typically provides posterior samples for inference.
  • 6 The posterior and statistical inference: For an approximately symmetric normal posterior, the mean and median are nearly identical and differ only in the fourth decimal digit.Posterior samples can be summarized with point estimates such as the mean, median, or sometimes the maximum a posteriori estimate.
  • 6 The posterior and statistical inference: 0.89 is the posterior probability that the difference between object and subject relative clauses is less than zero in Gibson and Wu’s data.This probability directly addresses the predicted negative effect.
  • 6 The posterior and statistical inference: 0.67 is the posterior probability that the effect is less than −0.02, corresponding to an advantage for object relatives of approximately 20 ms.The threshold −0.02 expresses a substantively meaningful advantage rather than merely testing whether the difference differs from zero.
  • 6.1 The 95% credible interval: A credible interval contains a specified amount of posterior probability and can be interpreted as the probability that the model’s true parameter value lies within its bounds.The paper distinguishes percentile intervals, which allocate equal tail mass, from highest posterior density intervals, which are the narrowest intervals containing the specified mass.
  • 6.1 The 95% credible interval: Figure 2 displays the posterior distribution of the subject–object relative-clause difference under a Normal(0,1) prior.Figure 3 instead displays posterior probabilities below zero and below −0.02.

6.2 Investigating the effect of prior specification on posteriors

Prior specification can leave posteriors stable when priors are weakly informative, but overly constrained priors can dominate the posterior. The Gibson and Wu analysis illustrates reporting posterior estimates, credible intervals, and prior sensitivity.

  • Prior sensitivity: Weakly informative priors generally have little effect on posteriors in typical psycholinguistic experiments.The review recommends sensitivity analysis to evaluate how different priors affect the posterior.
  • Using prior information: Bayesian priors can formally incorporate information from earlier studies, meta-analyses, or expert judgments into a new analysis.The review presents this as an advantage over relying only on binary significant/non-significant classifications.
  • Prior sensitivity: Overly constrained priors can dominate the posterior, making posterior estimates largely determined by the prior rather than the data.The review contrasts reasonable weakly informative priors with tightly constrained alternatives in the Gibson and Wu analysis.
  • Reporting posterior results: Bayesian reporting includes the posterior mean effect estimate, its credible interval, and the probability that the effect is negative.For the Gibson and Wu analysis, the reported effect was ˆβ = −0.04 with 95% CrI = [−0.1, 0.02] and P(ˆβ < 0) = 0.89.
  • Reporting posterior results: The Gibson and Wu analysis found an effect with the predicted negative sign, but the evidence for negativity was not very strong.The posterior probability of a negative effect was 0.89, while the credible interval included zero.

6.5 Hypothesis testing using the Bayes factor

The Bayes factor compares hypotheses through the ratio of their marginal likelihoods, quantifying which model makes the observed data more likely. It can support either the null or alternative hypothesis, but its estimation and prior sensitivity pose challenges.

  • Definition and interpretation: The Bayes factor is the ratio of the marginal likelihoods of two models corresponding to hypotheses H0 and H1.BF01 = p(D|H0)/p(D|H1), where p(D|M) is the probability of the data given model M.
  • Definition and interpretation: BF01 values below one favor H1, whereas values above one provide evidence for H0.Unlike significance testing, the Bayes factor can quantify evidence in favor of the null hypothesis.
  • Computing marginal likelihoods: A model’s marginal likelihood averages likelihoods over parameter values weighted by their prior probabilities.The coin-toss example illustrates this calculation for a parameter with a prior distribution over possible values.
  • Prior sensitivity: Bayes factors can be sensitive to prior specification, even when posteriors are not strongly affected by weakly informative priors.Broad priors can lower marginal likelihoods and thereby influence the Bayes factor.
  • Practical challenges: Marginal likelihoods can be difficult to estimate by sampling when samples are moderate or models are relatively complicated.The review notes available tools for some t-tests and repeated-measures ANOVA designs despite this challenge.

6.6 An example: Computing Bayes factor in the Gibson and Wu data

The review illustrates Bayes-factor and cross-validation approaches using the Gibson and Wu data, emphasizing how prior choices shape inference and how predictive model comparison has scope limits. Bayesian inference supports graded evidence, but neither method automatically establishes the null or identifies the best theoretical model.

  • Bayes factors: The Savage–Dickey method computes a Bayes factor for nested models by dividing posterior density by prior density at the parameter value of interest.For the example, the relevant value is zero for the experimental condition effect; shared nuisance parameters do not affect the calculation.
  • Bayes factors: 15.67 times more evidence favored the null than other values in the Gibson and Wu example, but broad priors may have penalized the alternative hypothesis.The alternative assigns prior mass to values considered too unlikely, whereas the null concentrates its mass at zero.
  • Bayes factors: Tighter, more realistic weakly informative priors decreased evidence for H0, without providing sufficient evidence to accept H1.Table 4 compares the numerator and denominator of the Bayes factor under different prior beliefs about the object–subject relative-clause effect.
  • Bayes factors: Bayes factors are prior-sensitive, so reported Bayes-factor conclusions should include sensitivity analyses.The review also notes that researchers may disagree about appropriate prior distributions for theoretically predicted effect sizes.
  • Cross-validation: Cross-validation compares models by expected predictive performance on unseen observations, using validation subsets separate from training data.The review covers LOO-CV, k-fold-CV, PSIS-LOO, and information criteria such as WAIC; k-fold-CV reduces computation by fitting fewer models.
  • Cross-validation: Predictive accuracy can be nearly indistinguishable from noise for small experimental effects, making the null model nearly as good as a predictor model unless effects or samples are large.This difficulty is especially relevant for nested hierarchical models such as linear mixed models, even though small changes may remain theoretically informative.
  • Inference: The Gibson and Wu analysis provides some weak evidence that object relative clauses are easier than subject relative clauses in Chinese, but not evidence for no effect.Interpreting the effect depends partly on whether its magnitude is theoretically meaningful relative to comparable experiments; ROPE offers one way to assess practical equivalence.

7 Examples of applications of Bayesian Methods

The review highlights Bayesian applications that extend beyond standard linear mixed models, including flexible random-effects structures, non-linear response-time and evidence-accumulation models, cognitive modeling, and informative priors for scarce data.

  • Linear mixed models: Bayesian methods can fit full random-effects structures that frequentist models may fail to converge on or may estimate with excessive random-effect correlations.Weakly informative LKJ priors can regularize correlation matrices in these models.
  • Response-time models: Bayesian modeling supports shifted log-normal hierarchical models for reaction times, which are bounded on the left and highly right-skewed.Log or reciprocal transformations retain an assumed shift of 0 ms, whereas shifted models represent a nonzero distributional shift.
  • Cognitive modeling: Bayesian methods are used both to model cognitive processes that assume a Bayesian mind and to model those processes without that assumption.Examples include Bayesian or noisy-channel approaches to parsing and word learning, alongside other cognitive-process models.
  • Cognitive and response models: Evidence-accumulation models integrate accuracy and reaction times in two-choice tasks rather than treating them as independent outcomes.Bayesian hierarchical extensions can account for within- and between-subject and between-item variability.
  • Scarce data: Informative priors can support statistically valid quantitative research when data are scarce but prior knowledge about effects exists.The review connects this advantage to SNAP judgments in syntax and semantics and to studies involving impaired participants.

8 Concluding remarks

The review argues that Bayesian methods offer direct hypothesis-relevant uncertainty and flexible hierarchical modeling, while acknowledging that Bayesian analysis requires substantially more decisions and effort than standard frequentist fitting.

  • Costs: Bayesian analysis requires defining priors, conducting sensitivity analyses, and deciding how to interpret results.This overhead remains even when convenient packages such as rstanarm are used.
  • Costs: For large samples with weakly informative priors, Bayesian and corresponding lme4 fixed-effect estimates can be quite similar or identical.The review contrasts this convergence in estimates with the greater time and effort required for Bayesian analysis.

9 Further reading

The review recommends introductory and advanced resources covering Bayesian methods, linear mixed models, and Bayesian cognitive modeling.

  • Introductory resources: McElreath (2015) and Kruschke (2015) are recommended as first introductions to Bayesian methods.Lynch (2007) is also recommended, though it assumes some calculus.
  • Advanced and applied resources: Gelman et al. (2014a) is recommended for advanced Bayesian treatment, while Gelman and Hill (2007) covers linear mixed models from frequentist and Bayesian perspectives.Lee and Wagenmakers (2014) is suggested for an accessible introduction to Bayesian cognitive modeling.
Loading 1602.00245v1…