PaperlayerView exact revision on arXiv

Source-linked AI summary

Likelihood based observability analysis and confidence intervals for predictions of dynamic models

Clemens Kreutz, Andreas Raue, Jens Timmer

arXiv:1107.0013v1physics.data-an

TL;DR

Complex parameter-confidence regions in mechanistic dynamic models make prediction intervals difficult to obtain, while high-dimensional parameter sampling may be infeasible. The paper profiles predictions directly through constrained likelihood optimization and extends the approach to validation intervals. Its examples demonstrate prediction intervals, data-based observability analysis, and experimental planning, with point-wise intervals for individual prediction conditions.

  • Problem

    Complex parameter-confidence regions in dynamic mechanistic models make parameter-space sampling difficult or infeasible for generating prediction intervals and assessing observability.

    Source: §IV · p. 6

  • Method

    The prediction profile likelihood samples prediction values directly and uses constraint maximum likelihood to assess their agreement with the data.

    Source: §III · p. 4

  • Results

    The approach is demonstrated on two-reaction and MAP kinase models, including prediction intervals, a non-observability setting, and experimental planning.

    Source: §IV · p. 6

  • Takeaways & Limitations

    Prediction intervals support data-based observability analysis, while narrow intervals can inform model validation and large intervals can identify informative measurements for parameter calibration.

    Source: §III · p. 5

    The intervals are point-wise; a common confidence region for statistically dependent predictions requires multidimensional profiling, which is beyond the article's scope.

    Source: §III · p. 5

Abstract

from arXiv · show

Mechanistic dynamic models of biochemical networks such as Ordinary Differential Equations (ODEs) contain unknown parameters like the reaction rate constants and the initial concentrations of the compounds. The large number of parameters as well as their nonlinear impact on the model responses hamper the determination of confidence regions for parameter estimates. At the same time, classical approaches translating the uncertainty of the parameters into confidence intervals for model predictions are hardly feasible. In this article it is shown that a so-called prediction profile likelihood yields reliable confidence intervals for model predictions, despite arbitrarily complex and high-dimensional shapes of the confidence regions for the estimated parameters. Prediction confidence intervals of the dynamic states allow a data-based observability analysis. The approach renders the issue of sampling a high-dimensional parameter space into evaluating one-dimensional prediction spaces. The method is also applicable if there are non-identifiable parameters yielding to some insufficiently specified model predictions that can be interpreted as non-observability. Moreover, a validation profile likelihood is introduced that should be applied when noisy validation experiments are to be interpreted. The properties and applicability of the prediction and validation profile likelihood approaches are demonstrated by two examples, a small and instructive ODE model describing two consecutive reactions, and a realistic ODE model for the MAP kinase signal transduction pathway. The presented general approach constitutes a concept for observability analysis and for generating reliable confidence intervals of model predictions, not only, but especially suitable for mathematical models of biological systems.

I. INTRODUCTION

For nonlinear mechanistic models, complex parameter-confidence regions make prediction intervals difficult to calculate, and high-dimensional parameter-space sampling can be infeasible. The paper presents prediction profile likelihood for prediction uncertainty and data-based observability, and extends the approach to validation experiments.

I. INTRODUCTION

  • Nonlinear model responses make complex parameter-confidence regions difficult to translate into prediction intervals, while dense sampling becomes infeasible for high-dimensional models.The difficulty is especially relevant when models have tens to hundreds of parameters.Source: §I · pp. 1–2
  • MCMC and bootstrap approximations can be numerically demanding and rough, making prediction-interval coverage hard to control and non-identifiable parameters difficult to handle.The cited approaches may also yield results that are questionable and difficult to interpret.Source: §I · p. 2
  • Prediction profile likelihood fixes a predicted value, optimizes parameters under that constraint, and thresholds the resulting likelihood to obtain confidence intervals without requiring identifiable parameters.The threshold controls the confidence level and its agreement with the experiments.Source: §I · p. 2
  • The approach is applied to ODE models for data-based observability analysis and extended to confidence intervals for validation experiments.Appropriate assessment of uncertainty or ambiguity is important across model discrimination, calibration, prediction, and validation.Source: §I · pp. 1–2

A. Small illustration model

The two-consecutive-reaction example uses measurements of C(t) to assess predictions for all three states. Prediction intervals are bounded when the measurements provide sufficient information, while a design that misses steady-state information yields non-observability for A(t) and B(t).

A. Small illustration model

  • The illustrative system comprises two consecutive reactions with rates θ1 = 0.05 and θ2 = 0.1, initial A(0) = 1, and B(0) = C(0) = 0.The simulated measurements assume Gaussian observational noise with σ = 0.1.Source: A. Small illustration model · p. 2
  • The prediction and validation profiles for A(t = 10) are thresholded at 68%, 90%, and 99% confidence levels; validation intervals are wider because they include validation measurement error.The illustrated measurements assume Gaussian noise with σ = 0.1, corresponding to an approximately 10% signal-to-noise ratio.Source: A. Small illustration model · p. 2, Figure 1 · p. 3
  • With C(t) measured at t = 0, 10, . . ., 100, prediction confidence intervals for A(t) and B(t) are bounded and can be interpreted as observability; intervals for measured C(t) are much smaller.Source: A. Small illustration model · pp. 2–3
  • For the small example, asymptotic-threshold intervals can be slightly conservative, while an improved threshold gives slightly smaller intervals with correct coverage.Coverage is assessed by comparing the confidence level with the frequency that the true value lies inside the prediction interval.Source: A. Small illustration model · p. 3
  • When measurements sample only C(t)'s transient increase and provide insufficient steady-state information, A(t) and B(t) become non-observable.The altered design measures at t = 0, 2, . . ., 20 and limits information about the total compound amounts.Source: A. Small illustration model · pp. 3–4
  • In that altered design, A(t) has unbounded prediction intervals, and B(t) has unbounded intervals for t > 0 despite known B(0) = 0.The figure illustrates these unbounded intervals as well as the corresponding flat prediction profiles.Source: Figure 1 · p. 3, A. Small illustration model · p. 4

B. MAP kinase signaling model

The MAP kinase example applies prediction and validation intervals to an eight-state signaling model. The intervals distinguish well-specified dynamics from states and times with substantial uncertainty, including a large interval for Erk** when negative feedback activates.

B. MAP kinase signaling model

  • The eight-state MAP kinase model has 14 estimated parameters, with known initial concentrations and identical parameters assumed for paired Mek and Erk phosphorylation and dephosphorylation steps.Measurements are assumed for phosphorylated forms, with additive Gaussian noise of σ = 10 nM.Source: B. MAP kinase signaling model · p. 4
  • Prediction intervals are below 40 nM for Raf and Raf* and narrow for unphosphorylated Mek and Erk, but Mek* intervals nearly span zero to 100 nM at most time points.The figure displays prediction and validation intervals for all dynamic states.Source: B. MAP kinase signaling model · p. 4
  • Erk** has the largest prediction-interval width, 176 nM after 181 seconds, when negative feedback activates; further measurements at this condition are informative for specifying its dynamics.Source: B. MAP kinase signaling model · p. 4

III. DISCUSSION

The prediction profile likelihood (PPL) explores data-consistent predictions directly, rather than sampling a complex high-dimensional parameter space. Its prediction intervals also support data-based observability analysis and experimental planning, with point-wise coverage and correlated predictions qualifying their interpretation.

III. DISCUSSION

  • The PPL samples prediction space directly and uses constrained maximum-likelihood optimization to find parameters consistent with each prediction, without requiring identifiable parameters.This avoids explicitly sampling the parameter space; predictions with insufficiently specified intervals can indicate non-observability.Source: §III · p. 4
  • The MAP kinase example has its largest absolute-concentration PCI for Erk** at 181 seconds, when negative feedback is activated; Mek* is also poorly observable.Measuring both compounds would be informative for model calibration.Source: Figure 2 · p. 5
  • Exploring a one-dimensional prediction space reduces the high-dimensional sampling problem, and the computational effort is expected to scale slightly super-linearly with parameter count.The scaling claim carries over from parameter profile likelihood because the computations are similar.Source: §III · p. 5
  • Prediction confidence intervals have point-wise coverage, while intervals from one data set at neighboring time points are statistically dependent.A joint confidence region for correlated predictions would require a multidimensional prediction profile likelihood, beyond this article's scope.Source: §III · p. 5
  • Narrow prediction intervals identify conditions already specified accurately by the data, making them candidates for model validation rather than additional parameter calibration.Large intervals instead indicate weakly specified conditions that may be informative for calibration experiments.Source: §III · p. 5
  • Prediction targets can include functions such as concentration ratios, integrals, or peak positions, not only dynamic-state values.These functions can also be considered for observability and prediction or validation confidence intervals.Source: §III · p. 6

IV. SUMMARY

The paper presents prediction profile likelihood as a way to calculate data-consistent predictions and their confidence intervals without sampling a high-dimensional parameter space. Examples show its coverage behavior, use for observability analysis, and applicability to experimental planning.

IV. SUMMARY

  • Prediction profile likelihood directly calculates predictions consistent with existing measurements and reduces high-dimensional parameter sampling to scanning a one-dimensional prediction space.It requires a working parameter-optimization procedure and can calculate intervals for predictions and validation outcomes.Source: §IV · p. 6
  • The method is presented as a tool for prediction assessment, observability analysis, and experimental design.Its scope includes parameter spaces of arbitrary dimension, subject to successful maximum-likelihood optimization.Source: §IV · p. 6
  • The approach is demonstrated on a two-reaction system and a MAP kinase model, including prediction and validation intervals for all dynamic states using phosphorylated-protein measurements.The small-system example shows desired coverage properties and unbounded prediction intervals in a setting inducing non-observability.Source: §IV · p. 6

A. The prediction profile likelihood

The prediction profile likelihood evaluates how well each fixed prediction agrees with the data by optimizing model parameters under a prediction constraint. A likelihood threshold then determines the prediction confidence interval, with stated coverage under the specified assumptions.

A. The prediction profile likelihood

  • For additive Gaussian noise with known variance, maximum-likelihood fitting is equivalent to minimizing the residual sum of squares for the model response.The response follows integration of the differential-equation system and mapping to observable quantities.Source: A. The prediction profile likelihood · p. 6
  • The likelihood-based threshold relies on the stated chi-square approximation conditions; if they fail, the decrease in residual sum of squares must be generated empirically.The passages identify linear parameters or an appropriate transformation as conditions supporting the approximation.Source: A. The prediction profile likelihood · pp. 6–7
  • A prediction may be any model function of compounds and parameters, including an observable that differs from the measured observables.Examples include ratios, integrals, and peak positions.Source: A. The prediction profile likelihood · p. 7
  • The desired coverage property is that the true model response lies inside the prediction confidence interval with probability α across realizations of the experimental data.The stated probability concerns a single prediction condition.Source: A. The prediction profile likelihood · p. 7
  • The PPL at a fixed prediction is obtained by maximizing likelihood over parameters constrained to produce that prediction, and a significance threshold defines the confidence interval.Asymptotic thresholds use chi-square quantiles; Monte Carlo can generate a threshold when the asymptotic assumption is violated.Source: A. The prediction profile likelihood · p. 7

B. The validation profile likelihood

The validation profile likelihood extends prediction-profile methods to noisy validation measurements, for which a true-value prediction interval is not the appropriate test. It gives a validation confidence interval calibrated for the measurement noise.

B. The validation profile likelihood

  • Noisy validation measurements are expected to fall outside prediction confidence intervals more often than the true value, so those intervals must be generalized for validation.The validation confidence interval is intended for deciding whether to reject a model using a noisy measurement.Source: B. The validation profile likelihood · p. 7
  • The validation confidence interval targets α coverage for a validation data point with mean equal to the model response and variance SD².A noisy validation measurement outside this interval allows model rejection under the stated measurement error SD.Source: B. The validation profile likelihood · p. 7
  • The validation profile likelihood relaxes the prediction constraint by penalizing mismatch with the validation measurement alongside mismatch with the experimental data.The model prediction need not equal the observed validation point.Source: B. The validation profile likelihood · pp. 7–8
  • Prediction confidence intervals are likelihood-ratio non-rejection regions and, for a given coverage α, are described as the smallest unbiased intervals for predictions.Their calculation evaluates a one-dimensional prediction space rather than sampling a high-dimensional parameter space.Source: B. The validation profile likelihood · p. 7
  • For SD > 0, the validation profile likelihood can be calculated by ordinary least-squares minimization and then rescaled to obtain the prediction confidence interval for the true value.The validation profile likelihood is defined as a maximized joint log-likelihood viewed as a function of the validation value.Source: B. The validation profile likelihood · pp. 7–8

1. Re-parametrization

Re-parametrization connects prediction confidence intervals to parameter profile likelihoods by making the desired prediction a parameter. For realistic ODEs, the paper uses an implicit, constraint-based approach instead of an explicit transformation.

1. Re-parametrization

  • A parameter confidence interval is a special case of a prediction confidence interval, so the prediction profile likelihood generalizes the parameter profile likelihood.When the prediction is a parameter estimate, their profile likelihoods coincide.Source: §1 · p. 9
  • When a prediction can be isolated as a new parameter, its profile likelihood gives a confidence interval for that prediction.The transformed model leaves the desired prediction as a single parameter, with the other parameter effects orthogonal to its effect.Source: §1 · p. 9
  • Because explicit re-parametrization is unavailable for most realistic ODE models, constrained nonlinear optimization implicitly treats prediction z itself as the new parameter.The optimization maximizes over model parameters subject to the model response equaling the considered prediction value.Source: §1 · p. 9
  • If several parameter sets satisfy the prediction constraint, maximizing the log-likelihood resolves the ambiguity or makes it irrelevant to the calculation.Only the maximized log-likelihood value enters the calculation.Source: §1 · p. 9
  • Monte Carlo noise realizations yield a distribution of prediction-profile-likelihood differences whose α quantile can replace the asymptotic χ² threshold.The empirical threshold is used for prediction confidence intervals.Source: Figure 3 · p. 9
  • For predictions of A(t) and B(t), the asymptotic thresholds are slightly too large, making the resulting confidence intervals conservative.The comparison covers nine prediction scenarios and multiple confidence levels.Source: Figure 4 · p. 9

2. Profile likelihood threshold

This section presents Monte Carlo-adjusted profile-likelihood thresholds and illustrates prediction and validation confidence intervals across example models. The intervals cover observability, validation uncertainty, informative experimental designs, prior-dependent predictions, and extrapolation.

2. Profile likelihood threshold

  • When no transformation makes a nonlinear parameter locally linear, non-monotone profile likelihoods can make the standard threshold conservative and confidence intervals too large.For predictions of A(t) and B(t) from measurements of C(t), the asymptotic thresholds are slightly too large and yield conservative intervals.Source: §2 · p. 10

3. Coverage

  • Monte Carlo-adjusted thresholds yield coverage matching the confidence level, while asymptotic thresholds tend to be conservative.Fig. 5 compares nine prediction scenarios across 100 data realizations; the Monte Carlo results show almost perfect agreement with the confidence level.Source: Figure 5 · p. 10, §3 · p. 10

4. Comparison of PCI and VCI

  • As prediction standard error decreases relative to validation-data error, prediction and validation interval boundaries diverge, approaching 1.64 as SE → 0.When SE > SD, the boundaries differ by about 0.38; when SE → 0, their difference approaches the one-sided 5% Gaussian quantile.Source: §4 · p. 11, Figure 6 · p. 11

5. Prior information

  • Prior knowledge can be incorporated through MAP estimation with a penalized likelihood; in the example, omitting the prior widens intervals, especially for unobserved A(t) and B(t).With a log-normal prior on A(0), intervals for all three states are bounded, including the unobserved states A and B.Source: §5 · pp. 11–12

6. Validation profile likelihood for the consecutive reaction model

  • Validation confidence intervals account for validation measurement noise and are larger than prediction intervals; under Gaussian noise, they can extend below zero when the true response is near zero.The comparison uses the same noise realization for A(t), B(t), and C(t).Source: §6 · p. 12

7. Observability of the long-term dynamics

  • When measurements of C capture only the transient, A(t) and B(t) are non-observable; C(t) itself becomes practically non-observable far beyond the measurement sampling interval.Source: §7 · p. 12

8. Characteristics of the MAP kinase model

  • For the MAP kinase model, the analysis covers its strongly nonlinear first 1,000 seconds, a compromise between transient and oscillatory dynamics.The model is used to demonstrate prediction and validation confidence-interval calculations.Source: §8 · p. 12

9. Experimental design conclusions

  • Erk measurements at 300 < t ≤ 1000 provide informative model-validation designs, while Erk** measurements at 100–200 seconds are informative for parameter improvement in absolute units.For relative measurements, prediction intervals instead identify Erk at t < 200 and Mek* and Mek** as informative; validation intervals identify Raf and Raf* as informative.Source: §9 · pp. 12–13

10. Implementation

  • For A(t), B(t), and C(t), the figure compares 90% prediction intervals with and without a log-normal prior on θ3.Dark gray marks the intervals with the prior; light gray marks those when θ3 is estimated without prior information.Source: Figure 7 · p. 15
  • Validation intervals and profile likelihood functions are shown for the unobserved states A(t), B(t), and C(t), with cubic piecewise interpolation along time.The caption describes the interpolated intervals as a way to plot them along the time axis.Source: Figure 8 · p. 15
  • The extrapolated prediction intervals for C(t) diverge because the design lacks sufficient information about its steady-state level, making that level non-observable.FIG. 9 shows this behavior at times much later than the measurements.Source: Figure 9 · p. 15
  • The MAP kinase analysis covers the first 1000 seconds as a compromise between transient and oscillatory dynamics.The model's long-term dynamics show regular oscillations.Source: Figure 10 · p. 16
  • The MAP kinase figures show prediction and validation interval sizes in absolute units and relative to each protein's total concentration.Prediction intervals are in the upper row and validation intervals in the lower row.Source: Figure 11 · p. 16
Loading 1107.0013v1…