Evaluating Type-1 Diabetes Digital Twins by Decision Quality

• T1D treatment requires patient-specific basal, meal, and correction insulin settings, while a digital twin simulates glucose under candidate treatments to support their ranking.Source: §1 · pp. 1–2
• The proposed protocol compares each twin’s ranking of candidate treatments with the patient’s true ranking, using Spearman correlation and regret alongside prediction accuracy.Source: §1 · p. 2
• The benchmark covers 30 UVA/Padova virtual patients and 20 counterfactual treatments, evaluating three published methods and three baselines.Source: §1 · p. 2
• Prediction accuracy and decision quality disagree: a baseline that never reads the patient’s CGM has the second-highest mean Spearman, while a black-box neural baseline ranks treatments at chance despite nearly twice-better glucose prediction.Source: §1 · p. 2

• Published T1D digital twins are validated by how closely predictions match held-out glucose measurements, while a glucose-specific metric weights clinically consequential pointwise errors but still assesses one trajectory.Source: §2 · p. 2
• Decision-focused learning weights prediction errors by their downstream decision effects, and Amad and van der Schaar apply that argument to digital twins.Source: §2 · pp. 2–3
• Unlike work that changes training, this paper holds each method’s published training procedure fixed and changes only evaluation, allowing existing T1D twins to be assessed without retraining.Source: §2 · p. 3

3.1 Setup

• The patient’s true ranking orders candidate treatments by the reward assigned to noise-free glucose trajectories produced under the same patient conditions.Source: §3.1 · p. 3

3.2 Digital Twins

• Each twin is fitted once from noisy CGM recorded under the baseline treatment, then predicts noise-free glucose for candidate treatments; the definition accommodates mechanistic and black-box models.Source: §3.2 · p. 3

3.3 Metrics

• Spearman measures agreement between predicted and true treatment rankings, while regret evaluates the recommended top treatment; the protocol selects Spearman as its primary metric.Source: §3.3 · p. 3, Table 1 · p. 3

3.4 Statistical Analysis

• Methods are ranked per patient after averaging test days, then compared using an omnibus Friedman test and Holm-corrected pairwise Wilcoxon tests.Source: §3.4 · pp. 3–4
• Table 2 identifies whether each benchmark design choice comes from a simglucose default, a published protocol, or the authors’ own design.Source: Table 2 · p. 4

4 Benchmark Design

• Each method follows its original implementation within the benchmark protocol, with design choices based on simglucose defaults or published protocols unless identified as the authors’ own.The benchmark is intended to evaluate methods consistently with their original implementations.Source: §4 · p. 4

4.2 Candidate Treatments and Risk Kernel

• Candidate treatments scale basal and bolus parameters relative to each patient’s baseline, while correction insulin is computed from noise-free IG to make the true ranking independent of sensor seed.The correction term is upward-only and applies when IG exceeds 150 mg/dL; each treatment begins after five baseline-treatment hours so all enter the scored window from the same state.Source: §4.2 · pp. 4–5
• The reward is the negative sum of an asymmetric Magni-risk kernel over the scored window, with glucose clipped to 40–400 mg/dL before scoring.The kernel is near zero in the euglycemic range and grows in both directions; unlike RMSE and MARD, it does not weight each mg/dL of error equally across glucose levels.Source: §4.2 · p. 5

4.3 Scenario and Data

• Training uses up to 30 independent identification days under the training treatment, and evaluation uses 10 additional days that switch from baseline to the candidate treatment at 5am.Meals are independently drawn and cached across treatments and methods, preventing differences between treatments from being confounded by different meals.Source: §4.3 · p. 5

4.4 Methods

• The benchmark evaluates three published DT methods and three proposed baselines, preserving each method’s prescribed training-data requirement.MCMC, SBI, and UKF use a single 22-hour window, while Linear and Neural receive all 30 training days and Prior uses no training data.Source: §4.4 · pp. 5–6
• The published methods comprise MCMC-based ReplayBG, SBI using neural posterior estimation, and a UKF adapted to estimate glucose with the simplified ODE.SBI is trained afresh per patient, so this benchmark evaluates its estimator rather than cross-patient amortization.Source: §4.4 · p. 6
• The proposed baselines include matched linear and neural response models, plus a Prior variant that removes CGM-based identification while retaining patient basal insulin and body weight.Linear and Neural share lagged input features, training days, and covariates, differing in model complexity.Source: §4.4 · p. 6

5 Results

• Pairwise tests show metric-family disagreements: MCMC–SBI and SBI–Prior separate on RMSE but not Spearman, while Neural–Prior reverse across both.UKF–Neural separate on Spearman but not RMSE; Neural–Prior separate on both in opposite directions.Source: §5 · pp. 6, 8
• 63.6 mg/dL RMSE yet 0.766 Spearman: Prior ranks treatments near the top despite having the cohort’s lowest prediction accuracy.Its Spearman is not distinguishable from SBI’s 0.728 (p = 0.3931).Source: §5 · p. 7, Table 3 · p. 7
• 0.049 Spearman: Neural ranks treatments at chance despite RMSE being statistically indistinguishable from UKF’s (p = 0.9515).Neural is 30.65 mg/dL better than Prior on RMSE but 0.717 worse on Spearman (p < 10^-4 for both).Source: §5 · p. 7
• Every reported comparison holds across all 30 leave-one-patient-out repetitions; omitting an age group preserves three key comparisons but changes some finer orderings.The preserved age-group comparisons are MCMC–SBI, SBI–Prior, and Neural–Prior.Source: §5 · p. 8

6 Discussion

• Decision-quality metrics penalize prediction errors only when they change the selected treatment or ranking, whereas RMSE and MARD charge for trajectory error at every sample.Prior’s 0.766 Spearman despite being the least accurate predictor illustrates that much of its error preserves reward ordering.Source: §6 · p. 8
• The benchmark is not clinical validation; its intended benefit is lowering the chance of deploying a DT that predicts accurately but recommends the wrong insulin dose.The authors explicitly caution against interpreting a strong benchmark score as clinical validation.Source: §6 · p. 8

6.1 Limitations

• Results cover one candidate set, one risk kernel, and a 24-hour horizon; the dissociation under time-in-range objectives or longer horizons is unknown.Candidates vary a single insulin-dose axis at a time, unlike some clinical adjustments that jointly change basal and prandial doses.Source: §6.1 · p. 8

7 Conclusion

• On 30 virtual patients, prediction accuracy and decision quality disagree on which of the six methods are better.Selecting a DT by prediction accuracy alone can favor one that recommends treatments at chance, as the Neural and Prior baselines show.Source: §7 · p. 8
• A DT selected by prediction accuracy alone can recommend treatments at chance, as the Neural and Prior baselines show.Source: §7 · p. 8
• Decision quality should be reported alongside RMSE and MARD rather than assumed to follow from them.Source: §7 · p. 8

A.1 Scenario

• Table 5 summarizes the RandomScenario implemented in simglucose.Source: §A.1 · p. 11, Table 5 · p. 11
• The table's scope is the RandomScenario; the supplied passage does not report its individual probability values.Source: §A.1 · p. 11, Table 5 · p. 11
• The scenario table uses p to denote the probability of a given meal or snack occurring.Source: Table 5 · p. 11

A.2 Existing assets

• The simulator, 30 virtual subjects, CGM noise model, and RandomScenario come from simglucose v0.2.11.Source: §A.2 · p. 11
• ReplayBG, its prior and MCMC procedure, Hoang et al.'s NPE setup [13], and Hajizadeh et al.'s unscented Kalman filter are re-implemented from published descriptions.Source: §A.2 · p. 11
• The cited papers are credited as specifications for those re-implementations.Source: §A.2 · p. 11

B Compute Resources

• Table 6 reports median per-patient costs for fitting and scoring all 20 candidate treatments on one core across 30 subjects.Source: Table 6 · p. 11, B Compute Resources · p. 11
• MCMC ran for 8 h 30 m of wall clock and 249 core-hours, versus 861 s and 5.5 core-hours for the five fast methods.Generating the shared trajectory cache required a further 1182 s and 9.9 core-hours.Source: B Compute Resources · p. 11

C Statistical Tests

• The omnibus Friedman tests reject for every metric, and Table 7 gives Holm-corrected pairwise Wilcoxon tests on within-patient means for RMSE, MARD, Spearman, and Regret Norm.The tests use 30 patients.Source: C Statistical Tests · p. 11, Table 7 · p. 12