Source-linked AI summary
BAT - The Bayesian Analysis Toolkit
Allen Caldwell, Daniel Kollar, Kevin Kroeninger
TL;DR
The paper addresses the need for a general data-analysis toolkit that supports Bayesian inference across model assessment and parameter estimation. BAT uses Markov Chain Monte Carlo to sample the full posterior probability density, enabling parameter estimation, uncertainty propagation, limit setting, and goodness-of-fit evaluation. Its examples show useful model discrimination through p-values, while the p-value interpretation has explicit validity and prior-sensitivity limitations.
Problem
The toolkit targets general data analysis involving model comparison, parameter estimation, uncertainty propagation, and experimental-condition modeling.
Method
BAT applies Bayes' theorem with Markov Chain Monte Carlo to sample the full posterior probability density and derive analysis quantities.
Results
p-values of 0.3 · 10^-3, < 3 · 10^-6, 0.563, and 0.551 for models I–IV clearly distinguish models I–II from models III–IV, but not models III–IV from each other.
Takeaways & Limitations
BAT provides posterior-based parameter estimates, correlations, uncertainty propagation without approximations, limits, and a practical goodness-of-fit criterion.
Takeaways & Limitations
The p-value interpretation is strictly valid only when parameters are not adjusted using the data being evaluated, and Bayesian model testing can be sensitive to prior choices.
Abstract
from arXiv · showhide
We describe the development of a new toolkit for data analysis. The analysis package is based on Bayes' Theorem, and is realized with the use of Markov Chain Monte Carlo. This gives access to the full posterior probability distribution. Parameter estimation, limit setting and uncertainty propagation are implemented in a straightforward manner. A goodness-of-fit criterion is presented which is intuitive and of great practical use.
1. Introduction
The paper frames data analysis as comparing model predictions with data to assess model validity or constrain parameters. BAT formalizes Bayesian knowledge updating while accounting for experimental conditions and user-defined inputs.
- Data analysis compares model predictions with data to assess model validity or constrain parameters within the model.
- 1.1. Modeling: Experimental effects may require modifying predictions or correcting data, making accurate experimental modeling necessary for reliable conclusions.
- 1.1. Modeling: The model frequency distribution g(y|λ, M) describes the relative frequency of result y given model M and parameters λ.
- 1.1. Modeling: Nuisance parameters ν represent additional experimental parameters, assumptions, and information that constrain model parameters.
- 1.2. Formulation of the Learning Rule: BAT updates prior knowledge using Bayes' rule, but result precision and accuracy depend primarily on the quality of user-defined likelihoods and priors.
2. Parameter Estimation
BAT performs parameter estimation with a user-defined likelihood and prior, producing a normalized posterior density that preserves parameter correlations. This posterior supports marginal summaries, intervals, correlations, and uncertainty propagation without approximation.
- Posterior output: The evaluation returns a normalized parameter probability density that includes all correlations and supports best-fit values and probability intervals.
- Model specification: BAT requires users to define the likelihood and prior probabilities for each model and application.The likelihood incorporates theory input and experimental-condition modeling, while common histogram and curve fits require minimal effort.
- Marginal summaries: Marginalization produces distributions for selected parameters, whose modes generally differ from the maximum of the full posterior density.The mode of a marginalized posterior can be evaluated for one or multiple parameters, with the remaining parameters marginalized.
- Interval definitions: Central intervals allocate equal probability to both tails, whereas smallest intervals retain a specified probability within the highest-density region.The central interval is defined by tail fraction α, while the smallest set satisfies P(⃗λ|⃗D, M) > Pmin.
- Reported quantities: Standard posterior summaries include means, modes, rms values, pairwise correlation coefficients, and central or narrowest 68% and 90% probability intervals.The toolkit also defines medians, central intervals, and smallest probability-containing sets for individual parameters or parameter vectors.
- Uncertainty propagation: Full posterior access propagates uncertainty to functions of parameters without the approximations required by standard techniques.For y = mx + b, the resulting distribution can provide quantities such as the central 68% interval at any x.
3. Model Validity
BAT defines a Bayesian goodness-of-fit p-value from the likelihood of observed data at the posterior mode, and uses it for model validity and comparison. Model comparison can also use posterior model probabilities, though this requires a complete model set and prior specifications.
- Definition of p-value: BAT defines a p-value from the likelihood of the observed data under the model parameters at the full-pdf mode.The quantity is intended to evaluate model validity and is analogous to the familiar χ2 probability.
- Definition of p-value: The p-value is the cumulative probability for obtaining a likelihood lower than the observed one under the specified model and parameters.Under valid modeling, it has a flat distribution between 0 and 1; poor model representation produces small values.
- Definition of p-value: Using fitted parameters biases the extracted p-value upward, with the bias depending on data volume, parameter count, and priors.The strict flat-distribution argument applies when parameters are not adjusted using the data being evaluated.
- Model Comparisons and Discovery Criteria: Models can be compared through p-values, favoring the largest p-value while treating models with reasonable p-values, such as values above 0.1, as good fits.The simplest model can be selected directly when applying Occam’s razor.
- Model Comparisons and Discovery Criteria: Absolute posterior model probabilities provide degrees-of-belief but require a complete model set, denominator integration, and priors that may strongly affect results.The toolkit describes discovery criteria using probabilities for explicitly specified hypotheses.
4. Setting limits on Parameters
BAT sets parameter limits by integrating the posterior probability density, but prior sensitivity can make posterior-based limits problematic for unproven models. A likelihood-ratio alternative expresses relative disfavor rather than a probability of exclusion.
- Setting limits on Parameters: A 90% upper limit on λ_i is obtained by integrating the posterior pdf until the cumulative probability reaches the specified level.The upper limit is found by solving for λ_upper.
- Setting limits on Parameters: Posterior-based limits become problematic when the prior is highly uncertain and the posterior depends strongly on the chosen prior.This issue can arise when assessing parameter ranges for new, unproven models.
- Setting limits on Parameters: A likelihood ratio can instead identify parameter values with R < R_cut as disfavored, with 0.1 offered as one possible cutoff.Small R indicates low degree-of-belief, but does not assign a probability that those parameter values are ruled out.
5. Markov Chain Monte Carlo
BAT uses Markov Chain Monte Carlo to sample the posterior distribution, including complicated densities without analytic forms. The Metropolis algorithm generates proposals and accepts or rejects them to produce a chain with the desired limiting distribution.
- Markov Chain Monte Carlo: BAT determines the posterior pdf with Markov Chain Monte Carlo and reviews the Markov-chain algorithm used to realize it.The resulting chain provides posterior samples for analysis.
- Markov Chains: A Markov chain is a sequence whose next state depends only on the current state and whose limiting distribution is defined under suitable conditions.Recurrence, irreducibility, and aperiodicity establish ergodicity.
- Metropolis Algorithm: The Metropolis algorithm proposes a new state from a symmetric proposal function and accepts it when a uniform random draw is below the calculated acceptance quantity.Otherwise, the chain retains its current state.
- Metropolis Algorithm: With a reasonable proposal function, the algorithm produces states distributed according to a desired distribution, including complicated densities without analytic forms.The target distribution only needs to be calculable.
6. Implementation
BAT is a C++ library designed to formulate Bayesian models and perform numerical analysis operations, with interfaces to common scientific software. Its implementation emphasizes MCMC while supporting estimation, integration, model validation, and varied outputs.
- Implementation: BAT provides a flexible framework for formulating models and reliable numerical operations such as optimization, marginalization, and integration.The framework was designed to address both requirements of the Bayesian analysis workflow.
- Implementation: The C++ library supplies model and numerical-operation classes, interfaces to ROOT, Minuit, and CUBA, and multiple output formats.It can be linked into standalone programs or loaded into ROOT.
- Model Formulation: Users define model terms as class methods, while predefined classes simplify histogram and function fits with Poissonian or Gaussian uncertainties.Simple applications require minimal programming effort.
- Numerical Operations: MCMC receives particular emphasis among the implemented or planned algorithms for optimization, marginalization, and integration.A pre-run is used to assess convergence and identify reasonable sampling parameters.
- MCMC Implementation: The pre-run uses parallel chains with random starting points and evaluates acceptance efficiency separately for each parameter and chain.The convergence procedure compares within-chain and estimated target variances, requiring (r − 1) < 0.1 by default for all parameters simultaneously.
- MCMC Implementation: The default sampling run uses 100,000 iterations to update modes, marginalized-distribution histograms, and applicable model-function distributions.Users can perform additional operations through the interface during sampling.
- Analysis Operations: BAT supports extracting posterior modes through Minuit, numerical integration for model probabilities, and p-value estimation from generated ensemble data sets.More complicated likelihoods require an external generator for ensemble data.
- Outputs: Default outputs include model summaries, comparable ROOT trees, stored Markov chains, and one- and two-dimensional distribution histograms.Stored chains retain points, indices, and posterior densities for offline analysis.
7. Examples
BAT examples show how Bayesian posterior outputs support model fitting, uncertainty intervals, and goodness-of-fit assessment for Gaussian and Poisson data. The examples also expose multimodal posteriors and show that model discrimination depends on the data uncertainties.
- Examples: The examples fit Gaussian- and Poisson-fluctuated data with four models: second-order polynomial, constant plus Gaussian, linear plus Gaussian, and quadratic plus Gaussian.The same models and parameter ranges were used for both data sets.
- Gaussian uncertainties: The Gaussian fits all give a reasonable description of the data within their allowed functional forms.BAT displays the fits using global-mode parameters and reports parameter modes, means, and central 68% intervals.
- Gaussian uncertainties: The global-mode function can lie outside the central probability interval, as illustrated by the peak region in model III.The uncertainty band is computed from model y-values generated from Markov-chain output at different x-values.
- Multiple modes: Model III has substantial multimodality: µIII has significant regions near 5 and 17, with different associated Gaussian widths and amplitudes.The global mode can therefore lie outside the central 68% probability interval, while two-dimensional posterior projections reveal several high-probability regions.
- Uncertainty intervals: The narrowest intervals containing 68% probability can be discontinuous, providing an alternative uncertainty definition for multimodal posteriors.BAT also propagates this interval definition to uncertainty bands for fitted function values.
- Model comparison: For Gaussian data, p-values are 0.154, 0.025, 0.479, and 0.667 for models I–IV, respectively, disfavouring model II while supporting models I, III, and IV.The p-value summarizes how well each model represents the data and agrees with visual inspection of the fits.
8. Summary
BAT is a general-purpose Bayesian analysis toolkit based on Markov Chain Monte Carlo, providing standard analysis quantities and access to the full posterior probability density.
- BAT is a general-purpose analysis tool based on a Bayesian learning algorithm and Markov Chain Monte Carlo.
- The package provides best-fit parameters, goodness-of-fit measures, upper limits, and other standard analysis quantities.
- BAT samples model parameters according to the full posterior probability density function.
- Posterior sampling enables detailed investigations of parameter correlations and probability-density evaluation for arbitrary functions of the parameters without approximations.
- The BAT package is coded in C++ and is available with a user manual from the stated project website.