PaperlayerView exact revision on arXiv

Source-linked AI summary

Prediction Uncertainty Controlled Computer Model Calibration

Ryan Heejoon Byun, Dayeon Yoon, Won Chang

arXiv:2610.31163v1stat.MLcs.LGstat.APstat.COstat.ME

TL;DR

Nominal posterior regions from KOH calibration and amortized SBI do not ensure reliable predictive coverage of simulator outputs. The paper combines discrepancy-robust flow-matching posterior estimation with conformal risk control, achieving substantially sharper bands in synthetic and Malawi applications while targeting expected miscoverage under exchangeability.

  • Problem

    KOH and amortized SBI do not directly control simulator-output predictive miscoverage, which can depart from nominal levels when parameters are weakly identified or discrepancy is misspecified.

    Source: §1 · pp. 2–3

  • Method

    The framework trains a flow-matching posterior on outputs with varied Gaussian-process discrepancies, then uses CRC to select simulator-output sets by predictive loss on synthetic calibration data.

    Source: §1 · p. 3, §3.1 · p. 10, §3.4 · p. 13

  • Results

    In the synthetic benchmark, CRC mean pointwise interval lengths were 0.0363 and 0.0426 versus the MCMC mean of 0.2073, and both applications had sharper, near-nominal bands than KOH–MCMC.

    Source: §4.3 · p. 20, §6 · p. 26

  • Takeaways & Limitations

    The framework targets predictive miscoverage rather than treating simulator-output bands as a byproduct of posterior sampling, with amortization making CRC’s repeated inference practical.

    Source: §6 · p. 26

    The finite-sample guarantee relies on exchangeability under a user-specified GP discrepancy family that cannot be verified from one observation, and control is marginal over time, not pointwise or conditional.

    Source: §6 · pp. 26–27

Abstract

from arXiv · show

Computer-model calibration is usually undertaken in service of prediction, yet neither Bayesian Kennedy-O'Hagan (KOH) calibration nor amortized simulation-based inference is built to control the predictive error of the calibrated simulator. When parameters are weakly identified or the data-model discrepancy is misspecified, the simulator-output bands induced by a nominal posterior level can be far from nominal. We propose a two-layer framework that targets predictive miscoverage directly. First, a conditional flow-matching model learns an amortized posterior for the calibration parameters, trained on simulator outputs perturbed with Gaussian-process discrepancies of varying structure. Second, conformal risk control arranges posterior draws for the target observation into nested candidate sets and selects the smallest one whose fixed-rank pointwise bands attain a prescribed time-averaged miscoverage on a synthetic calibration sample. Under exchangeability of calibration and target observations, the bands control expected miscoverage in finite samples. In a synthetic benchmark, the proposed bands are roughly 20% the width of KOH-MCMC bands with miscoverage near the 10% target. In calibrating an age-structured SEIR model to the 2010 Malawi measles outbreak, they are again sharper and near-nominal, and the weeks they fail to cover coincide with the asymmetric epidemic decline, exposing structural inadequacy of the mechanistic model.

1 Introduction

Computer-model calibration serves prediction, but nominal posterior regions from KOH or amortized SBI do not ensure reliable simulator-output uncertainty. The paper combines flow-matching SBI with conformal risk control (CRC) to target predictive miscoverage directly, using discrepancy-perturbed training and exchangeable synthetic calibration data.

1 Introduction

  • Nominal KOH and SBI posterior regions do not guarantee nominal simulator-output coverage, especially with weak identification, discrepancy, or inverse-map error.KOH–MCMC also requires costly, observation-specific MCMC runs.Source: §1 · pp. 2–3
  • The framework combines amortized flow-matching posterior estimation with CRC, which selects parameter sets by evaluating predictive miscoverage after simulator propagation.Under exchangeability between calibration examples and a future target, the resulting intervals control expected miscoverage in finite samples.Source: §1 · pp. 3–4
  • Amortization makes the repeated conditional posterior approximations required by CRC practical, whereas standard KOH would require separate MCMC runs for calibration observations.The trained flow can provide posterior approximations for hundreds of synthetic calibration observations with a few hundred ODE integrations.Source: §1 · pp. 3–4
  • FMPE is trained on simulator outputs perturbed by heterogeneous Gaussian-process discrepancies, and CRC calibration examples use the corresponding discrepancy mechanism.The approach can also be applied to SBI methods other than FMPE.Source: §1 · p. 4

2 Background

The background contrasts posterior-based calibration, which does not directly control predictive miscoverage, with CRC’s expected-loss control under exchangeability. It also introduces flow matching as the conditional posterior-estimation approach used to make repeated inference practical.

2.1 Kennedy–O’Hagan (KOH) Calibration and Simulation-Based Inference

  • KOH calibration can suffer θ–discrepancy nonidentifiability without careful priors or design; ignoring or misspecifying discrepancy can bias physical-parameter inference, and a sufficiently long MCMC chain must be rerun for each new observation.For the Malawi outbreak, the nominal 90% pointwise KOH–MCMC band had 3.45% empirical miscoverage and average pointwise width of 3,526.89 cases, evaluated only for weeks with at least 300 observed cases.Source: §2.1 · p. 5, Figure 1 · p. 7
  • A nominal posterior or SBI probability region does not guarantee nominal predictive coverage; the Malawi SBI band missed no evaluated weeks but averaged 6,078.54 cases in width.The figure evaluates miscoverage only for weeks with at least 300 observed cases.Source: §2.1 · pp. 5–6, Figure 1 · p. 7

2.2 Conformal Risk Control

  • CRC chooses an ordered threshold using calibration losses to control expected loss at target α under exchangeability, with a finite-sample adjustment to the target level.In this setting, the loss can measure empirical miscoverage of the prediction sets.Source: §2.2 · pp. 6, 8

2.3 Flow Matching

  • Flow matching learns a time-dependent vector field that transports a simple reference distribution toward the target; conditioning it on observed covariates enables approximate posterior sampling.The described Gaussian path begins with standard Gaussian noise and concentrates around a target sample; training uses a tractable conditional-flow-matching objective, with sampling by integrating the learned ODE.Source: §2.3 · pp. 8–9

3 Prediction Uncertainty Controlled Calibration

The framework combines amortized flow-matching inference with conformal risk control (CRC) to calibrate nested parameter sets against predictive miscoverage. Its finite-sample guarantee depends on exchangeability between synthetic calibration observations and the target under the specified discrepancy mechanism.

3.1 Miscoverage Risk

  • CRC chooses the smallest threshold whose fixed-rank pointwise bands satisfy an empirical time-averaged miscoverage constraint.The candidate sets are nested, and increasing their size lowers fixed-rank miscoverage loss.Source: §3.1 · pp. 10–11
  • Two candidate-set constructions retain posterior draws by truncating the base distribution's radius or thresholding conditional posterior density.The posterior-based set retains the top ⌈λN⌉ draws ranked by conditional log-density.Source: §3.1 · p. 11

3.2 Selection of the trimming parameter.

  • The trimming parameter is tuned on separate simulated tuning and validation data, selecting the sharpest intervals that meet the target validation risk.The selected parameter is frozen before generating a fresh final CRC calibration sample.Source: §3.2 · p. 12

3.3 Deep Approximate Distribution with Flow Matching

  • A conditional flow-matching model estimates the posterior of calibration parameters from observed outputs and auxiliary summaries.Its vector field is conditioned on the time embedding, sequence representation, and auxiliary summary.Source: §3.3 · pp. 12–13

3.4 Handling Data-Model Discrepancy

  • Training on simulator outputs perturbed by Gaussian-process discrepancies with varying hyperparameters is intended to make the inverse map robust to discrepancy.The discrepancy exposure is designed to help identify informative features for recovering physical parameters while accounting for mismatch.Source: §3.4 · p. 13

3.4.1 Training Data Generation

  • Training data pair each sampled parameter configuration with each sampled GP discrepancy realization to create contaminated simulator trajectories.The noise pool uses a zero-mean GP; the training set also includes auxiliary outputs and parameters.Source: §3.4.1 · p. 14

3.4.2 Calibration Data Generation for Conformal Risk Control

  • Synthetic CRC calibration observations use new GP draws modulated by a state-dependent envelope, and the finite-sample guarantee assumes joint exchangeability with the target.The envelope is intended to reproduce anticipated state-dependent variability; calibration discrepancies are generated separately from the training noise pool.Source: §3.4.2 · pp. 14–15

4 Simulation Study

The simulation study evaluates CRC-calibrated prediction bands against a KOH–MCMC baseline using repeated synthetic observations. Across experiments, the CRC bands are near the target miscoverage and substantially shorter than the MCMC bands, while the representative draw illustrates the sharpness trade-off.

4 Simulation Study

  • The study uses a closed-form synthetic model and holds the ground-truth parameter fixed across 200 held-out experiments with independently regenerated discrepancies.CRC thresholds are estimated once on a separate calibration dataset and frozen for held-out evaluation.Source: §4 · p. 15, §4.2 · p. 18

4.1 Synthetic Model Outputs and Observational Data

  • Training uses 150,000 contaminated trajectories, while the target and CRC calibration observations share the same envelope-modulated GP discrepancy mechanism.This matches their discrepancy mechanisms but does not itself establish joint exchangeability, which remains a working assumption.Source: §4.1 · pp. 16–17

4.3 Results

  • On a representative synthetic observation, KOH–MCMC intervals were roughly 7× longer and had roughly twice the RMSE of the two tighter CRC bands.The MCMC band covered the observation entirely on this draw.Source: §4.3 · p. 18, Figure 3 · p. 19
  • Mean pointwise interval lengths were 0.0363 (radius) and 0.0426 (posterior), about 20% of the KOH–MCMC mean of 0.2073, while mean miscoverage was 0.1222 and 0.0721, respectively.Across 200 experiments, radius-based and posterior-based mean RMSEs were 0.0089 and 0.0085, versus 0.0222 for KOH–MCMC.Source: §4.3 · pp. 18, 20, Figure 4 · p. 20
  • Fixed 90% retention without CRC produced zero miscoverage but intervals about 1.8 times as long as the corresponding CRC bands on the representative draw.The ablation indicates the near-target, tight-band behavior is driven by CRC calibration rather than the set constructions alone.Source: §4.3 · p. 20

5 Application to a Measles SEIR Model

The paper applies conformal risk control (CRC) to an age-structured SEIR calibration for the 2010 Malawi measles outbreak. The proposed bands have empirical miscoverage above the 10% target but are narrower than the KOH–MCMC band, and their misses identify where the model fails to reproduce the epidemic decline.

5 Application to a Measles SEIR Model

  • The application calibrates an age-structured SEIR model to weekly case counts from the 2010 Malawi measles outbreak and uses observed proportions across three coarse age groups as an exact auxiliary summary.The model has 21 age classes and represents 84 coupled ODEs.Source: §5 · p. 21, §5.1 · pp. 21–22
  • The real-data application tests the framework in a setting with an expensive 84-ODE forward model, non-identifiable posterior directions, and an unknown data-generating mechanism.The discrepancy assumption underlying exchangeability can therefore only be assessed indirectly.Source: §5 · p. 21

5.1 Measles Outbreak Data and Age-Structured SEIR Model

  • The simulator uses a warm-up duration t0 to account for the unknown time from epidemic onset to the first reporting week, then produces weekly case counts and age-group proportions.The six calibration parameters are β, N0, fE, ash, art, and t0.Source: §5.1 · pp. 22–23

5.2 Discrepancy, Synthetic Data, and Implementation

  • The training data perturb simulator outputs with heterogeneous Gaussian-process discrepancies, while the CRC calibration sample uses the corresponding discrepancy mechanism and assumes the real-world discrepancy is a draw from that family.Under this working assumption, the synthetic calibration sample and target observation are exchangeable.Source: §5.2 · p. 23

5.3 Results

  • Both CRC parameter sets have empirical miscoverage 0.1379 versus the target α = 0.1, with average interval lengths of 2540.26 and 3157.48 cases; KOH–MCMC has miscoverage 0.0345 and length 3526.89 cases.The three methods have comparable RMSE values.Source: Table 2 · p. 24, §5.3 · p. 24
  • CRC misses cluster around the peak and late tail, where the asymmetric outbreak decline differs from the approximately symmetric, bell-shaped SEIR trajectories, regardless of parameter choice.The authors describe these failures as a diagnostic signal of structural inadequacy.Source: §5.3 · pp. 24–25, Figure 6 · p. 25

6 Discussion

The framework combines amortized flow-matching posterior approximation with conformal risk control to target predictive miscoverage. Across synthetic and Malawi measles applications, it attains near-nominal coverage with sharper bands than KOH–MCMC, subject to exchangeability and other scope limits.

6 Discussion

  • In both synthetic and Malawi measles applications, the proposed parameter sets attained near-nominal coverage with substantially sharper bands, while KOH–MCMC overcovered.The framework targets predictive miscoverage directly and narrows parameter sets only as much as the calibration data demand.Source: §6 · p. 26
  • Amortized posterior queries make conformal risk control feasible, and clustered time-point coverage failures can localize structural inadequacy in the mechanistic model.The calibration and tuning samples require on the order of n+ntune+nv conditional posterior approximations per target observation.Source: §6 · p. 26
  • The finite-sample guarantee depends on exchangeability between the synthetic calibration set and real observation, requiring the true discrepancy to lie in the user-specified GP family.That condition cannot be verified from a single observation.Source: §6 · pp. 26–27
  • The amortized posterior depends on the training simulation pool’s size and coverage, with higher-dimensional parameter spaces potentially requiring larger training budgets.The paper also suggests diagnostics for gross exchangeability violations and extensions to localized conformal control and broader discrepancy families.Source: §6 · p. 27
  • The framework controls marginal miscoverage averaged over time, not pointwise or conditional miscoverage at particular time points.Source: §6 · p. 27

S1 Model Details

The model estimates a conditional posterior by learning a time-dependent vector field from calibration parameters, auxiliary covariates, and observed time series. Flow matching trains this field along a probability path and uses its integration for conditional sampling.

S1.1 Flow-Matching Architecture

  • A conditional vector-field network maps latent state x, auxiliary covariates A, time series Z, and integration time t to a vector field for posterior parameter estimation.Its context combines time and covariate information with a sequence representation of Z.Source: S1.1 Flow-Matching Architecture · p. 33
  • The sequence encoder represents Z with overlapping width-7 patches at stride 2, a four-layer Transformer, and learned-query attention pooling.It also appends the log-transformed maximum and normalized time index of the maximum as geometric features.Source: S1.1 Flow-Matching Architecture · pp. 34–35
  • The vector-field backbone uses four residual blocks with hidden dimension 256, adaptive layer normalization, and Swish-gated linear units.Zero-initialized context projections and output heads make blocks initially act as identities and set the initial vector field to zero, improving early-training numerical stability.Source: S1.1 Flow-Matching Architecture · p. 35

S1.2 Flow Matching Details

  • Flow matching learns a continuous-time transport from a reference distribution to a target distribution by regressing a time-dependent vector field along a conditional probability path.The path starts at the reference distribution and ends at a Gaussian centered on the target.Source: S1.2 Flow Matching Details · pp. 36–37
  • Training minimizes a Monte Carlo estimate of the conditional flow-matching objective using sampled target-condition pairs, times, and noise, without solving an ODE during training.After training, approximate samples are obtained by solving the learned flow; guided flows interpolate unconditional and conditional fields to increase adherence to the condition.Source: S1.2 Flow Matching Details · p. 37

S2 WAIFW Matrix for the Age-Structured SEIR Model

The age-structured SEIR model uses a structured WAIFW matrix whose contact entries depend only on the separation between age classes. Its off-diagonal decay represents lower contact rates between more widely separated age groups.

S2 WAIFW Matrix for the Age-Structured SEIR Model

S3 MCMC Specifications and Diagnostics

The KOH–MCMC baselines use Gaussian-process discrepancy likelihoods and four NUTS chains in both analyses. The appendix reports overlapping chain densities and no visually apparent chain-specific drift for the synthetic and measles data analyses.

S3 MCMC Specifications and Diagnostics

  • Both KOH–MCMC analyses used four independent NUTS chains with maximum treedepth 10 and adapt_delta 0.85, and bounded Stan parameters induced uniform priors over their supports.The time-series likelihood used a Gaussian process centered on the forward-model trajectory, with range, partial-sill, and nugget hyperparameters.Source: S3 MCMC Specifications and Diagnostics · pp. 38–39

S3.1 Simulation Study

  • For the synthetic study, the Gaussian density-shaped forward trajectory was evaluated at 53 equally spaced values on [−2, 2].Source: S3.1 Simulation Study · p. 39
  • The synthetic analysis retained 280,000 post-warmup draws from four 100,000-iteration chains, whose density curves largely overlapped without visually apparent chain-specific drift.Each chain used 30,000 warmup iterations; the diagnostics cover three forward-model parameters and three GP covariance hyperparameters.Source: S3.1 Simulation Study · p. 39, Figure S1 · p. 40

S3.2 Real-Data Application

  • The measles analysis combined a Gaussian-process time-series likelihood with a second likelihood component for observed age proportions, conditionally independent given model and GP parameters.The deterministic age-structured SEIR trajectory replaced the synthetic mean trajectory.Source: S3.2 Real-Data Application · p. 41
  • The measles analysis retained 28,000 post-warmup draws from four chains, with substantial overlap among parameter-density curves and no obvious long-term chain-specific drift.Each chain ran 10,000 iterations, including 3,000 warmup iterations; Figure S2 shows six SEIR parameters and three GP hyperparameters.Source: S3.2 Real-Data Application · p. 41, Figure S2 · p. 42

S4 Ablation Study for Parameter-Set Selection

In both simulated- and real-data analyses, fixed 90% retention by either of two non-conformal rules yielded zero empirical miscoverage and similar RMSE and interval length. These results are empirical diagnostics; unlike CRC, fixed retention does not adapt to a prescribed risk level or provide its finite-sample guarantee.

S4 Ablation Study for Parameter-Set Selection

  • The ablation compares retaining the top 90% of draws by smallest latent-sample Euclidean norm or largest posterior density.The remaining pipeline components are unchanged, isolating replacement of the CRC-selected set with a fixed 90% set.Source: S4 Ablation Study for Parameter-Set… · p. 43
  • Zero empirical miscoverage was attained by both fixed-90% filtering rules in the simulated and real-data analyses, with similar RMSE and average interval length.Table S1 defines RMSE against the observed trajectory and average interval length across time.Source: S4 Ablation Study for Parameter-Set… · p. 43, Table S1 · p. 43
  • Similar empirical miscoverage across the two heuristics does not establish risk control: fixed retention neither targets a prescribed level nor provides CRC’s finite-sample guarantee.The ablation’s miscoverage values are empirical diagnostics, not conformal risk-controlled guarantees.Source: S4 Ablation Study for Parameter-Set… · pp. 43–44
  • Figures S3 and S4 show the simulated and real-data ablations, respectively, with the median output, a 90% fixed-rank prediction interval, and the observation.Each panel retains 90% of candidate draws.Source: Figure S3 · p. 45, Figure S4 · p. 46

Related papers

Loading 2610.31163v1…