Source-linked AI summary
Simulation-based model selection for dynamical systems in systems and population biology
Tina Toni, Michael P. H. Stumpf
TL;DR
Comparing mechanistic biological models is difficult with sparse, noisy data and intractable likelihoods. The paper develops ABC SMC model selection and demonstrates its applicability across biological models, including influenza and JAK-STAT data.
Problem
Selecting among candidate mechanistic models is difficult when observations are sparse or noisy, parameters are unknown, and likelihoods are unavailable or computationally intractable.
Method
The paper develops a computationally efficient Bayesian model-selection framework using approximate Bayesian computation with sequential Monte Carlo sampling.
Results
ABC SMC supported model selection across chemical reaction, Gibbs random-field, influenza, and JAK-STAT examples, including non-nested ordinary and time-delay differential-equation models.
Takeaways & Limitations
The framework extends Bayesian model selection to efficiently simulated biological systems when exact likelihood evaluation is computationally intractable and data are scarce.
Takeaways & Limitations
Applying the method routinely to models with hundreds or thousands of parameters requires further numerical developments because repeated simulations are computationally costly.
Abstract
from arXiv · showhide
Computer simulations have become an important tool across the biomedical sciences and beyond. For many important problems several different models or hypotheses exist and choosing which one best describes reality or observed data is not straightforward. We therefore require suitable statistical tools that allow us to choose rationally between different mechanistic models of e.g. signal transduction or gene regulation networks. This is particularly challenging in systems biology where only a small number of molecular species can be assayed at any given time and all measurements are subject to measurement uncertainty. Here we develop such a model selection framework based on approximate Bayesian computation and employing sequential Monte Carlo sampling. We show that our approach can be applied across a wide range of biological scenarios, and we illustrate its use on real data describing influenza dynamics and the JAK-STAT signalling pathway. Bayesian model selection strikes a balance between the complexity of the simulation models and their ability to describe observed data. The present approach enables us to employ the whole formal apparatus to any system that can be (efficiently) simulated, even when exact likelihoods are computationally intractable.
1 Introduction
Complex biological models are often simulation-dependent, making rational comparison of competing models difficult when data are sparse, noisy, or likelihoods are intractable. The paper develops an ABC model-selection framework using sequential Monte Carlo to retain Bayesian model-selection machinery and demonstrates it across simulated and real biological systems.
- Simulation-based analysis complicates comparison of candidate biological models because observations are frequently sparse and noisy, while standard model-selection methods require suitable probability models.
- ABC replaces likelihood calculation with simulation-data comparison, accepting parameters when d(D0, D∗) ≤ ϵ and approximating the posterior as ϵ becomes sufficiently small.
- When multiple plausible models exist, Bayesian selection compares their marginal posterior probabilities by integrating likelihoods over parameters, allowing arbitrary non-nested models to be ranked.
- The paper develops a computationally efficient ABC model-selection formalism based on sequential Monte Carlo and applies it to chemical reaction dynamics, Gibbs random fields, influenza spread, and JAK-STAT signalling.
2 ABC for model selection
This section estimates marginal model posteriors by sampling jointly over model indicators and parameters, then marginalizing parameters. It presents ABC SMC model selection, which progressively tightens simulation tolerances to approximate the target posterior more efficiently than ABC rejection.
- Joint-space formulation: Joint-space model selection estimates P(m|D0) by obtaining P(θ,m|D0) and marginalizing over parameters.Particles contain both a model indicator and parameter vector, allowing the same joint posterior approximation to yield marginalized parameter distributions.
- ABC SMC model selection: Compared with ABC rejection, the SMC extension makes both model-selection approaches computationally more efficient, addressing rejection’s prohibitive cost for moderately parameterized models.The paper presents the joint-space ABC SMC approach and refers detailed derivations and discussion to the supplementary material.
- ABC SMC model selection: ABC SMC propagates joint model–parameter particles through decreasing tolerances ϵ1 > … > ϵT until they approximate the target posterior.The algorithm samples from intermediate distributions conditioned on simulated-data distances and uses model and parameter perturbation kernels between populations.
- Algorithm specification: The algorithm requires priors, distance functions, tolerance schedules, and perturbation kernels; examples use uniform priors and adaptive truncated-uniform or Gaussian kernels.Uniform priors make candidate models equally plausible a priori while defining feasible parameter regions, and kernels adapt using parameter ranges from previous populations.
3 Results
ABC SMC selected among stochastic biological models across synthetic, influenza, and JAK-STAT applications. It correctly identified known models, improved computational efficiency over ABC rejection, and provided evidence favoring biologically interpretable alternatives.
- Applications: The results span model selection for autocatalytic and non-autocatalytic reactions, Gibbs random fields, influenza outbreaks, and competing STAT5 shutoff mechanisms.The applications address whether different outbreaks share models and compare alternative mechanistic hypotheses in signalling biology.
- Stochastic reaction kinetics: Applied to synthetic reaction-kinetics data generated from model 2, ABC SMC identified the correct model with high confidence.The datasets contained Y measurements at 20 time points and were simulated with k2 = 30, X0 = 40, and Y0 = 3.
- Gibbs random fields: ABC SMC correctly estimated posterior model distributions for Gibbs random fields and achieved a considerable computational speed-up compared with ABC rejection.The comparison used 1000 simulated datasets across parameter values and sequence length n = 100.
- Influenza dynamics: The two H3N2 outbreaks appeared to share the same epidemiological characteristics, whereas the different influenza strains showed distinct spread dynamics, mainly across the community.For the different strains, the two-parameter model had negligible posterior probability; combining the H3N2 datasets provided evidence favoring model (2) over model (1).
4 Discussion
The ABC SMC model-selection methodology is broadly applicable to simulated biological systems, including comparisons of ODE and time-delay models. Applying it routinely to very large models remains computationally costly, motivating numerical and parallelization improvements.
- Methodological scope: ABC SMC provides a useful, widely applicable model-selection method for scarce experimental data and systems with limited measurements.The framework applies across simulation and modelling approaches, including qualitative modelling.
- Methodological scope: The procedure can compare the explanatory power of ODE and time-delay differential-equation models and extends beyond dynamical systems when efficient simulation is available.Its scope is described as limited only by the availability of efficient simulation approaches.
- Limitations and future work: Routine use for complex systems, computational, and population-biology models with hundreds or thousands of parameters requires further numerical development because repeated simulations are costly.The authors identify parallelization of SMC-based ABC methods as a key route to greater computational efficiency.
5 Conclusion
The conclusion emphasizes the urgent need for statistically sound inferential methods that assess the relative performance and reliability of competing models in biomedical simulation studies.
- 5 Conclusion: Statistically sound model-selection procedures are urgently needed to differentiate among models as simulation-based biomedical research expands.These methods should assess models’ relative performance and reliability.
Supplementary material A: Derivation of ABC SMC model selection algorithms
The supplement derives ABC SMC model-selection algorithms from sequential importance sampling and ABC intermediate distributions. It presents joint model–parameter selection as the practical algorithm, while also deriving naive sequential and marginal-likelihood alternatives that are computationally expensive or require special normalization handling.
- ABC SMC building blocks: ABC SMC constructs intermediate distributions from prior-weighted simulated datasets accepted when their distance from observations is below tolerance ϵ_t, with proposals initialized from the prior and then perturbed.The intermediate distributions use datasets generated conditional on fixed parameters, while later proposals exploit previous particles through perturbation kernels.
- II) ABC SMC model selection on the joint space: Algorithm II performs ABC SMC model selection jointly over model indicators and parameters, using perturbation kernels and sequential importance weights; it is the practical algorithm used in the main paper and examples.Algorithms I and III are described as computationally too expensive and impractical to use.
- I) Naive ABC SMC model selection: Algorithm I updates model probabilities sequentially but repeatedly samples each model’s parameters from its prior, making parameter integration computationally expensive.The supplement motivates joint-space sampling as a way to exploit information about parameters contained in earlier particles.
- III) ABC SMC approximation of the marginal likelihood: The marginal-likelihood approach estimates P(D0|m) from ABC acceptance rates or ABC SMC normalization constants, then converts these estimates into posterior model probabilities.For ABC SMC, the usual normalized output cannot be used directly because the target distribution is unnormalized; the derivation instead tracks intermediate marginal likelihoods.