Source-linked AI summary
Bayesian Item Response Modeling in R with brms and Stan
Paul-Christian Bürkner
TL;DR
IRT software often restricts users to prespecified model classes, while Bayesian implementations remain limited. This paper presents a brms-and-Stan framework for flexible Bayesian IRT models, including nonlinear and distributional specifications, custom response distributions, and joint response-time models. The framework supports consistent fitting and post-processing across binary, ordinal, and response-time examples, but full Bayesian inference can be computationally costly and may be too slow for real-time applications.
Problem
Existing IRT packages tend to implement prespecified model classes, and Bayesian IRT methods remain comparatively limited.
Method
The paper uses brms with Stan to specify and fit flexible Bayesian IRT models with multilevel formulas, nonlinear parameters, custom distributions, and hierarchical Stan code.
Results
The framework specifies, estimates, and post-processes binary, ordinal, and response-time IRT models consistently without switching packages.
Takeaways & Limitations
The framework provides a common Bayesian workflow for multiple IRT model classes and response types.
Takeaways & Limitations
Full Bayesian inference can be computationally costly and may be too slow for real-time applications such as adaptive testing.
Abstract
from arXiv · showhide
Item Response Theory (IRT) is widely applied in the human sciences to model persons' responses on a set of items measuring one or more latent constructs. While several R packages have been developed that implement IRT models, they tend to be restricted to respective prespecified classes of models. Further, most implementations are frequentist while the availability of Bayesian methods remains comparably limited. We demonstrate how to use the R package brms together with the probabilistic programming language Stan to specify and fit a wide range of Bayesian IRT models using flexible and intuitive multilevel formula syntax. Further, item and person parameters can be related in both a linear or non-linear manner. Various distributions for categorical, ordinal, and continuous responses are supported. Users may even define their own custom response distribution for use in the presented framework. Common IRT model classes that can be specified natively in the presented framework include 1PL and 2PL logistic models optionally also containing guessing parameters, graded response and partial credit ordinal models, as well as drift diffusion models of response times coupled with binary decisions. Posterior distributions of item and person parameters can be conveniently extracted and post-processed. Model fit can be evaluated and compared using Bayes factors and efficient cross-validation procedures.
1. Introduction
The paper presents a flexible Bayesian IRT framework using brms and Stan to address limitations of prespecified and predominantly frequentist IRT software. It extends model specification across response distributions, nonlinear structures, and distributional parameters.
- IRT models responses to items measuring latent constructs and provides a formal statistical basis for modern psychological measurement.
- Existing IRT software packages support particular model classes and related post-processing methods rather than a single broadly flexible framework.
- The proposed framework combines brms and Stan for Bayesian IRT model specification, estimation, and post-processing.
- brms supports over 40 built-in response distributions, including binary, categorical, ordinal, count, response-time, and proportion models.
- The framework permits custom response distributions, nonlinear multilevel formulas, arbitrary person or item covariates, multilevel structure, and prior distributions.
- The paper develops the framework conceptually, implements it in software, and demonstrates its practical use through hands-on examples.
2. Model description
The paper formulates IRT models by combining response distributions with functions linking person and item parameters to distributional parameters. This framework covers binary, categorical, ordinal, count, compositional, response-time, and joint decision-time models, while supporting Bayesian priors and variable selection.
- IRT models predict each observation’s response through all parameters of a chosen response distribution, using long-format data for multilevel formula modeling.
- Bernoulli responses support 1PL, 2PL, and 3PL logistic IRT models, with the success probability constrained to [0, 1].
- Ordinal responses can use cumulative or adjacent-category models, yielding graded response, partial credit, and generalized partial credit models.
- Count, compositional, and response-time data can be modeled with Poisson, beta or Dirichlet, and flexible three-parameter distributions, respectively.
- Joint drift diffusion models combine binary decisions with response times and represent drift, evidence accumulation, and non-decision processing within a cognitive process model.
- The functions linking person and item parameters to distributional parameters determine parameter meaning, model complexity, and valid range restrictions.
- Bayesian priors incorporate prior knowledge, while variable-selection techniques can address the large number of potential differential-item-functioning variables.
3. Model specification in brms
brms specifies Bayesian IRT models through response families, multilevel formulas, and priors. Its syntax supports partial pooling, covariates, correlated and nonlinear parameters, distributional regression, custom families, and non-centered hierarchical parameterization.
- IRT models in brms are specified mainly through the family, formula, and prior arguments.
- Response families define distributions and link functions, with links chosen to respect parameter constraints such as positivity for standard deviations.
- Users can select built-in response families or define custom families when the desired distribution is not available.
- Standard formulas represent person and item effects, covariates, interactions, varying slopes, and partial pooling in long-format data.
- Shared identifiers allow person or item parameters across different distributional formulas to be modeled as correlated.
- Nonlinear formulas let arbitrary functions depend on estimable parameters with person- and item-level predictors, and nl = TRUE makes the formula literal.
- Prior specification is essential, but example priors are not practical recommendations and should reflect the model and subject-matter knowledge.
- Generated Stan code uses non-centered parameterization for group-level coefficients, scaling independent standard-normal parameters by hierarchical hyperparameters.
4. Parameter estimation and post-processing
brms estimates models through Stan’s adaptive Hamiltonian Monte Carlo and provides tools for posterior summaries, prediction checks, and model comparison. These tools support extracting parameter distributions and evaluating fit with cross-validation or Bayes factors.
- Stan estimates brms models using adaptive Hamiltonian Monte Carlo, which produces less-autocorrelated posterior samples than random-walk Metropolis methods.
- HMC provides higher-quality samples but requires gradient calculations and more time per sample, while Stan automates critical tuning parameters.
- brms provides numerical and graphical summaries of central parameters, including posterior distributions of person parameters.
- Posterior predictive sampling enables predictions for existing or new data and comparisons between observed and model-predicted responses through posterior-predictive checks.
- Cross-validation measures and marginal likelihoods support model comparison, selection, and averaging.
- brms implements marginal-likelihood and Bayes-factor workflows through bridge_sampler, bayes_factor, and post_prob.
5. Examples
The examples show how Bayesian IRT models in brms handle binary, covariate, guessing, and ordinal-response analyses while exposing posterior uncertainty and model-comparison tools. Across these examples, model complexity is evaluated against its empirical payoff, and retaining ordinal information can improve parameter precision.
- Binary Models: Person and item parameters vary substantially in the 1PL model, while equal item discriminations create blocks of persons with identical total endorsements.The 1PL assumption makes persons endorsing the same number of items receive the same estimate, regardless of which items they endorsed.
- Binary Models: 2PL discrimination estimates are roughly 1.2–1.5, but the model changes little for this dataset: person-parameter correlation is r = 1 and ΔLOOIC = 5.97 (SE = 4.78).The 2PL item easiness pattern remains similar to the 1PL, with somewhat less spread.
- Modeling Covariates: The covariate examples translate logit effects into probabilities and show higher agreement with increased trait anger, plus a gender interaction in actual versus desired verbal aggression.Women and men report wanting to be verbally aggressive at roughly similar probabilities, whereas men report actually being verbally aggressive more often.
- Modeling Covariates: The binary analysis estimates a negative uniform DIF effect, with b = -0.94 (95% CI = [-1.22, -0.66]), indicating women report cursing and scolding less often than men.The analysis concerns DIF independent of the specific latent-trait value.
- Ordinal Models: Treating the VerbAgg responses as ordinal rather than binary preserves the yes, perhaps, and no categories, improves precision especially for high person parameters, and leaves the simpler shifted-threshold model preferred by ΔLOOIC = -4.7 (SE = 14.1).The GRM and binary Anger coefficients are similar, but their predicted response probabilities differ because the GRM models three categories.
6. Comparison of Packages
The comparison positions brms as a flexible, fully Bayesian general-purpose framework for IRT, while specialized packages can be simpler for standard models. This flexibility brings broader response-distribution and parameter-modeling support, but also greater setup, computational, and usability costs.
- brms capabilities: brms combines arbitrary-complexity non-linear multilevel formulas, broad response-distribution support, custom distributions, and predictions for multiple distributional parameters.It can also combine multiple response variables into joint multivariate models so person and item parameters inform one another across responses.
- brms capabilities: brms supports distributions for binary, categorical, ordinal, response-time, count, and proportion data, covering a wider range than the compared packages.The framework uses the same general-purpose algorithms across these response distributions.
- Estimation and post-processing: brms is fully Bayesian, whereas most compared implementations rely on point estimation; it also automates posterior visualization, prediction, and model comparison.TAM and mirt allow priors but still estimate by optimization, while sirt provides MCMC sampling only for selected models.
- Trade-offs: The value of full Bayesian inference depends on the model, data, and inferential goal because its additional computational costs may outweigh its benefits in some settings.For simple models with abundant data, Bayesian and maximum-likelihood estimates may differ little; real-time adaptive testing may also require faster inference.
- Trade-offs: Dedicated IRT software is often more straightforward for standard 1PL or 2PL models and may provide more directly applicable post-processing for common use cases.Users interested only in standard models may obtain reliable solutions faster with specialized software.
- Trade-offs: General-purpose frameworks require more initial familiarization, and multilevel syntax makes constraining or fixing parameters less intuitive than in dedicated IRT packages or lavaan.These costs accompany greater modeling flexibility and consistency in specification and post-processing.
7. Conclusion
The paper presents a general Bayesian IRT framework in R via brms and Stan for consistently specifying, estimating, and post-processing diverse IRT models. Its flexibility supports complex relationships, but sensible inference still requires theoretical reasoning and subject-matter knowledge.
- The framework supports consistent specification, estimation, and post-processing of a wide range of IRT models without switching between packages.Its demonstrated applications include binary, ordinal, and response-time data.
- Advanced brms formulas enable complex non-linear relationships between person and item parameters and observed responses.
- Successful model estimation does not guarantee a theoretically sensible model or valid inference about the effects under study.Model comparison and selection tools can guide decisions but cannot replace theoretical reasoning and subject-matter knowledge.
- Bayesian modeling quantifies uncertainty probabilistically, allowing decisions to average over uncertainty rather than rely on premature binary decisions from p-values or confidence intervals.