PaperlayerView exact revision on arXiv

Source-linked AI summary

What Next-Event Accuracy Cannot See: Closed-Loop Evaluation of Emergency Department Trajectory Simulators

Zhen Xuen Brandon Low

arXiv:2609.31635v1cs.LGcs.AI

TL;DR

Clinical trajectory simulators are often assessed on observed histories, which do not test behavior when models condition on their own generated events. EDSim-Bench evaluates held-out ED visit rollouts across multiple criteria and finds that similar next-event accuracy can conceal very different closed-loop behavior. All-position supervision was associated with more stable rollouts, but did not resolve every evaluation criterion.

  • Problem

    Generative EHR models are mainly evaluated on observed histories, next-event prediction, or downstream outcomes, leaving their behavior while conditioning on their own outputs insufficiently tested.

    Source: §1 · p. 2

  • Method

    EDSim-Bench generates held-out visit continuations from valid prefixes and scores termination, event composition, timing, conditional fidelity, and occupancy against a train-only order-3 reference.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    Next-event accuracy differed by only 0.001 across neural architectures while rollout behavior varied; all-position Transformer termination was 0.977–0.983 across three seeds.

    Source: §1 · p. 2, §6.1 · p. 6

  • Takeaways & Limitations

    Clinical trajectory models should be evaluated through closed-loop rollout across multiple seeds and criteria before predictive performance is treated as evidence of fidelity.

    Source: §8 · p. 8

    All-position supervision did not resolve every criterion, and model rankings changed under order-sensitive and occupancy criteria.

    Source: §7 · p. 8

Abstract

from arXiv · show

Clinical trajectory models are usually evaluated by next-event accuracy on observed histories. Simulation is different: models must condition on their own generated events, allowing errors to compound. Although this problem is well known in sequence modelling, it has not been systematically quantified for clinical trajectory simulators. We developed EDSim-Bench to evaluate this failure mode using 425,028 MIMIC-IV-ED stays, with external replication on MC-MED, and release the evaluation protocol and scoring code. Starting from held-out visit prefixes, models generate the remainder of each visit and are evaluated on termination, event composition, timing, conditional fidelity, and occupancy forecasting, with a train-only order-3 n-gram as a reference baseline. Despite next-event accuracies within 0.001, three neural architectures behaved very differently under rollout. Across seeds, one Transformer recipe ranged from 0.43 to 0.96 in termination score and from 4.2- to 137-fold the divergence of the n-gram; no prefix-trained neural model approached the n-gram on termination or event composition. Inference-time interventions improved termination but did not jointly recover composition and timing. Supervising every eligible sequence position rather than only the final prefix position was associated with one to two orders of magnitude lower divergence across Transformer, GRU, and LSTM models, with the pattern persisting under model scaling, temporal shift, and external-site evaluation. Nevertheless, even the best model generated visits approximately half as long as observed, and model rankings reversed on occupancy forecasting, a downstream quantity relevant to bed management. These results show that next-event accuracy is insufficient to evaluate clinical trajectory simulators and motivate closed-loop evaluation across seeds, rollout criteria, and downstream tasks.

1 Introduction

Clinical trajectory models are commonly assessed on observed histories, although simulation feeds generated events back into the model and can compound errors. EDSim-Bench evaluates this closed-loop behavior and finds that similar next-event accuracy can conceal markedly different rollouts.

1 Introduction

  • EDSim-Bench evaluates held-out visit continuations on termination, event composition, timing, conditional fidelity, and occupancy, using a train-only order-3 n-gram as a fixed reference.Generation begins from valid visit prefixes and continues until END or a predefined event limit.Source: §1 · p. 2
  • Across architectures and seeds, next-event accuracy differed by only 0.001 while closed-loop event distributions and termination varied substantially.Some models drifted as generated events accumulated; others were miscalibrated from the first generated step.Source: §1 · p. 2
  • Inference-time interventions could improve termination without consistently restoring both event composition and timing.The interventions included process constraints, alternative sampling, increased capacity, and count-model blending.Source: §1 · p. 2
  • All-position supervision was associated with more stable rollouts across Transformer, GRU, and LSTM models, persisting across model scale, temporal shift, and MC-MED replication.The loss was applied at every eligible sequence position rather than only at the final position of sampled prefixes.Source: §1 · p. 2
  • Clinical trajectory models should be evaluated as simulators across multiple seeds and closed-loop criteria, rather than by next-event accuracy alone.The benchmark also measures occupancy under the same protocol.Source: §1 · p. 2

2 Related work

Work outside clinical data has examined generation from model-produced histories, while generative EHR evaluation remains largely focused on observed histories or aggregate population properties. This paper situates its clinical benchmark alongside those approaches and distinguishes its rollout protocol from other simulation methods.

2 Related work

3 The benchmark

The benchmark combines internal and external ED cohorts with sequence models and a common closed-loop protocol. It scores generated trajectories across termination, composition, timing, conditional behavior, and occupancy-related measures.

3.1 Data and task

  • 425,028 internal ED stays from 205,466 patients yield 63,264 test stays and 1,132,806 valid prefixes; external replication uses 105,378 eligible MC-MED stays.The internal cohort combines MIMICEL and MIMIC-IV-ED with subject-level 70/15/15 splits.Source: §3.1 · p. 3

3.2 Models and training setups

  • The benchmark compares Transformer, GRU, and LSTM models with train-only process baselines, including order-2 and order-3 n-grams and a semi-Markov model.Standard neural models train on one observed prefix per example with loss at its final position; the causal Transformer applies supervision at every eligible position.Source: §3.2 · p. 3

3.3 Closed-loop evaluation

  • Each valid prefix generates one continuation until END or a 128-event cap, with sampled events fed back into the history.Under vanilla decoding, events are sampled at temperature 1 with END as an ordinary token; regression-head models feed back the median predicted time increment.Source: §3.3 · p. 3, §3.1 · p. 3

3.4 Evaluation metrics

  • Event composition uses pooled suffix-event JSD normalized to the train-only order-3 reference, while timing uses duration ratio alongside remaining-LOS error.The duration ratio compares mean generated with observed remaining duration; additional analyses address conditional behavior, occupancy, and other criteria.Source: §3.4 · pp. 3–4

4 Open-loop and closed-loop performance

Neural models with nearly identical observed-history accuracy behave differently during rollout, with large variation across models and seeds. Their performance also depends on launch context, while the order-3 count model provides a demanding reference.

4.1 Next-event accuracy versus rollout behaviour

  • 0.442 next-event accuracy for the Transformer and GRU and 0.443 for the LSTM contrasted with END termination of 43.4%, 59.9%, and 56.8%, respectively.Their raw JSD ranged from 0.073 to 0.310, compared with 0.0023 for the train-only order-3 reference.Source: §4.1 · p. 4
  • Neural models reached discharge in only 61–69% of rollouts versus approximately 94% for the order-3 reference, and 7-token models generated only 7–8% of observed remaining duration.With the 7-token vocabulary, neural models nevertheless had lower remaining-LOS error than the count reference; duration ratios also shifted when measured on shared launch points.Source: §4.1 · p. 4

4.2 Variation across training seeds

  • Across seven Transformer seeds, termination ranged from 0.43 to 0.96 and raw JSD from 0.010 to 0.314, or 4.2–137× the reference.GRU runs ranged from 40 to 53× the reference and LSTM runs from 36 to 40×; one checkpoint could therefore misrepresent a training setup.Source: §4.2 · pp. 4–5

4.3 The count-model reference and launch context

  • The order-3 n-gram terminated in 0.9998 of rollouts with mean duration ratio 1.14; every prefix-trained neural run had at least fourfold its event-distribution divergence.All-position models had termination near 0.98 or higher across seeds, while Transformer termination was approximately 0.69–0.74 from triage-anchored prefixes versus about 0.42 from other starts, which made up 94.5% of the test set.Source: §4.3 · p. 5, §4.2 · p. 5

5 Failure modes under rollout

Autoregressive rollouts exposed distinct failure modes: some models drifted after starting near observed next-event distributions, while the 32-token Transformer was already miscalibrated at its first generated step. Inference-time interventions often improved termination but did not jointly restore event composition and timing.

5 Failure modes under rollout

  • With 7-token events, more than half of GRU and LSTM rollouts repeated an event for at least ten steps; the 32-token Transformer instead concentrated on fewer event types.The Transformer’s unique-token ratio was 0.28 versus 0.47 for the order-3 n-gram, and one triage-related event rose from 0.3% observed to 23.8% generated.Source: §5 · p. 5

5.1 First-step error and compounding drift

  • Recurrent models began near observed next-event distributions and diverged as generated events entered their histories, whereas all seven 32-token Transformer seeds had greater first-step JSD than the three GRU runs.The Transformer’s first-step distribution was already miscalibrated before generated events were fed back.Source: §5.1 · p. 5
  • Mean inter-event intervals in some settings contracted from approximately 6 minutes near rollout start to around 0.5 minutes by step 10.The deterioration therefore affected event timing as well as event composition.Source: §5.1 · p. 5
  • The causal Transformer stayed near 0.002–0.004 JSD over the first ten steps, while the decoder-only model rose to approximately 0.010–0.013 by around step 10.Both all-position models largely removed the standard Transformer’s elevated first-step error, but low first-step error did not guarantee later calibration.Source: §5.1 · p. 5

5.2 Inference-time interventions

  • Process masking raised 32-token GRU termination from 0.599 to 0.865 but lowered its duration ratio from 0.67 to 0.26; repetition penalties and greedy decoding also had substantial trade-offs.A 7-token Transformer repetition penalty raised termination from 0.905 to 0.985 and reduced raw JSD from 0.091 to 0.026, but trajectories were only about 4% as long as observed remaining visits.Source: §5.2 · p. 6
  • Inference-time interventions improved termination but none simultaneously approached the order-3 reference on event composition and timing.Increasing model capacity or blending predictions with the count model reduced divergence without consistently restoring temporal calibration.Source: §5.2 · p. 6

6 Training supervision and rollout stability

Supervision at every eligible sequence position was associated with more stable rollouts across Transformer, GRU, and LSTM models, although timing remained imperfect. The association persisted across controls, scaling, temporal shift, and external-site evaluation.

6.1 Prefix-only and all-position supervision

  • All-position training raised causal Transformer next-event accuracy from approximately 0.44 to 0.57, while the comparison used three to seven seeds per configuration.The main comparisons used the same 32-token data and base model configuration; prefix-only and all-position training differed in batch size, addressed by a batch-matched control.Source: §6.1 · p. 6
  • The causal all-position Transformer terminated in 0.977–0.983 of rollouts and achieved raw JSD of 0.0015–0.0039, or 0.7–1.7×F, versus 4.2–137×F and termination of 0.43–0.96 for the standard Transformer.The all-position decoder-only model nearly always terminated but remained at 4.1–4.8×F.Source: §6.1 · p. 6
  • The decoder generated approximately 0.88–0.96 of observed remaining duration, whereas the continuous-time causal Transformer generated only about half.Improved event composition therefore did not extend equally to timing.Source: §6.1 · p. 6

6.2 Architecture and training controls

  • The supervision association persisted when batch size was matched and across recurrent models: causal all-position training reached 1.6–3.9×F at batch size 1024, while GRU and LSTM divergence fell to 1.6–6.0×F and 1.1–2.8×F.Causal masking alone was insufficient, and scheduled sampling worsened divergence and termination relative to a matched 7-token Transformer baseline.Source: §6.2 · pp. 6–7
  • Across three architecture families, all-position supervision was associated with one to two orders of magnitude lower divergence, more reliable termination, and less seed variation.All-position recurrent models and the continuous-time Transformer still generated only approximately 0.32–0.53 of observed remaining duration.Source: §6.2 · pp. 6–7, Figure 1 · p. 7
  • At approximately eight times model size, the causal all-position Transformer reached 1.6×F with termination 0.976 versus 26×F and 0.751 for the same-size prefix-trained model; external MC-MED results also favored causal all-position runs.Under temporal shift, causal-model JSD increased approximately eightfold, similar to the order-3 reference; on MC-MED, three causal all-position runs achieved termination of 0.945–0.979 and divergence of 1.0–2.5×F.Source: §6.2 · p. 7

7 Discussion

Observed-history prediction and trajectory simulation gave different impressions of model behavior: rollouts varied in termination, event composition, and seed sensitivity. Models also failed differently, and inference-time fixes did not consistently restore composition and timing.

7 Discussion

  • Some models drifted as generated events entered their histories, while others were already miscalibrated at the first generated step.Evaluation on observed histories alone does not establish whether a model will remain well behaved as a simulator.Source: §7 · pp. 7–8
  • Inference-time interventions could improve termination without consistently restoring composition and timing, while all-position supervision was associated with lower divergence, better termination, and reduced seed variation across three architectures.Model rankings nevertheless changed under order-sensitive and occupancy criteria.Source: §7 · p. 8
  • The evaluated vocabularies describe ED care processes rather than diagnoses or clinical state, and MC-MED differs from MIMIC in vocabulary and visit length.Absolute performance was therefore compared within, rather than across, sites.Source: §7 · p. 8

8 Conclusion

Nearly identical next-event accuracy concealed markedly different closed-loop behavior, and all-position supervision improved rollout stability without solving every criterion. The paper concludes that clinical trajectory models should be evaluated through closed-loop rollout across multiple seeds and criteria.

8 Conclusion

  • Nearly identical next-event accuracy concealed markedly different closed-loop behavior, while all-position supervision improved stability but did not solve every criterion.The paper recommends closed-loop evaluation across multiple seeds and criteria before predictive performance is treated as evidence of trajectory fidelity.Source: §8 · p. 8

A Open-loop results

On observed histories, the standard architectures were nearly indistinguishable in next-event accuracy, while comparisons involving all-position models require caution because their evaluation samples and decoder inputs differ. Five temporal point-process models were omitted from closed-loop results because their generation path could not be certified under the shared rollout implementation.

A Open-loop results

  • With the 32-token vocabulary, standard-model next-event accuracy was 0.442–0.443, versus 0.594–0.605 with the 7-token vocabulary; these open-loop results are reported in Table 2.The standard-model rows use the full test split, while the all-position and decoder-only rows use a 30k stratified sample.Source: A Open-loop results · p. 10, Table 2 · p. 10
  • The decoder-only model reports 0.650 accuracy and the causal all-position Transformer 0.567, but the decoder is evaluated after teacher-forcing an observed time-bin token.The decoder-only event accuracy is therefore not strictly comparable with continuous-time models; the primary open-loop/closed-loop dissociation is already present among the standard architectures.Source: A Open-loop results · p. 10, Table 2 · p. 10
  • Five temporal point processes were omitted from closed-loop results because their separate generation path could not be certified under the shared rollout and evaluation implementation.A comparable test of whether all-position supervision benefits intensity-based models requires a validated rollout policy for them.Source: A Open-loop results · p. 10

B Full closed-loop results

Matched-launch comparisons qualify duration estimates from each model’s own terminated rollouts, while the decoding ablations show that interventions can trade off termination, event fidelity, and timing. The scheduled-sampling control did not reproduce the all-position result, and two intended GRU evaluations were excluded after a sampler-configuration error.

B Full closed-loop results

  • Duration ratios are defined only for terminated rollouts, creating selection because models may terminate on different launch points; matched comparisons change ratios by at most 0.01 in the 7-token setting.In the 32-token setting, the recurrent models move further below 1 after matching, while the count reference remains comparatively stable.Source: B Full closed-loop results · p. 10, Table 3 · p. 11
  • On matched 32-token launch points, the standard Transformer’s duration ratio rises from 0.97 on its own terminated rollouts to 1.18, showing that its near-calibrated aggregate partly reflects which visits it terminates.Table 3 contrasts each model’s own terminated rollouts with launch points where all four models terminate; unmatched subsets alone do not establish the direction of temporal bias.Source: B Full closed-loop results · p. 10, Table 3 · p. 11
  • Table 4’s process-masked decoding can improve termination while worsening temporal calibration, and remaining-LOS MAE can favor substantially miscalibrated duration, particularly with 7 tokens.The full closed-loop results include vanilla and process-masked decoding; the reported rows are single-run results, not architecture-level seed averages.Source: B Full closed-loop results · p. 10, Table 4 · p. 11

C Decoding ablations

  • The decoding sweep varied one knob at a time, but two intended 32-token GRU evaluations were omitted because the requested sampler configurations had not been applied correctly.Learning-rate tuning, process masking of the tuned checkpoint, and count-model blending were tested only for the Transformer.Source: C Decoding ablations · p. 11
  • With scheduled sampling, the 7-token Transformer reached 71×F divergence and 0.824 termination, versus 55×F and 0.889 for the matched non-scheduled baseline.This single configuration does not rule out other schedules, but partial self-conditioning alone did not reproduce the all-position result.Source: C Decoding ablations · p. 11

D Scale ablation

Scaling a standard Transformer improved 32-token closed-loop composition and termination but did not bring it close to the count reference. Effects were smaller for 7-token composition, and greater capacity did not guarantee better temporal calibration; decoding ablations also retained timing or baseline gaps.

D Scale ablation

  • With 32 tokens, increasing the standard Transformer from 0.42M to 3.25M parameters lowered JSD from 0.310 to 0.059 (136 to 26×F) and raised termination from 0.434 to 0.751, but it remained far from the order-3 count reference.The larger model’s event-distribution JSD was approximately at the level of the much smaller GRU.Source: D Scale ablation · p. 11, Table 7 · p. 12
  • With 7 tokens, the same capacity increase changed event-distribution divergence from 56 to 49×F; on matched 32-token launch points, duration ratio shifted from 1.18 for the smaller Transformer to 0.41 for the larger one.The latter comparison shows that increasing capacity did not guarantee better temporal calibration.Source: D Scale ablation · pp. 11–12, Table 7 · p. 12
  • For the 7-token Transformer decoding sweep, repetition penalty 1.75 gave the best tested termination and JSD but generated duration of only 4% of the observed remaining stay.No tested decoding rule brought the duration ratio close to 1.00.Source: Table 5 · p. 12
  • In the 32-token decoding sweep, every tested GRU intervention that increased termination also shortened simulated visits, while Transformer count-model blending improved termination and JSD without matching the pure count reference.Source: Table 6 · p. 12

E Per-seed results

Appendix analyses assess how closed-loop results vary across seeds, launch contexts, rollout caps, and evaluation choices, while distinguishing composition fidelity from occupancy forecasting and interpreting subgroup patterns and suffix-divergence values cautiously.

E Per-seed results

  • 1.3 ± 0.5×F was the all-position causal Transformer’s mean divergence, versus 42.0 ± 47.6×F for the standard Transformer across seeds.The standard Transformer’s mean of 42.0×F and median of 14.4×F illustrate sensitivity to initialization; all-position GRU, LSTM, and decoder means were 3.3 ± 2.3×F, 1.8 ± 0.9×F, and 4.4 ± 0.4×F, respectively.Source: E Per-seed results · p. 13, Table 8 · p. 13

F Higher-order count models

  • With a 182-token vocabulary, order-3 and order-7 JSDs were 0.00269 and 0.00041; under temporal shift from 2008–2013 training to 2017–2019 visits, their advantage contracted from approximately 5.7× to 1.4×.The 182-token order-7 table occupied 8.2 GB, approximately 4,300 times the size of the 1.9 MB causal all-position Transformer checkpoint.Source: F Higher-order count models · p. 14

G Sensitivity to event order

  • One to two orders of magnitude higher divergence for standard models persisted under unigram, bigram, and trigram scoring, while the causal all-position Transformer rose from 0.7×F on unigrams to 1.4×F on trigrams.The results distinguish marginal event composition from local sequential structure.Source: G Sensitivity to event order · pp. 15–16, Table 12 · p. 16

H Sensitivity to rollout weighting

  • 136 to 361×F: the standard Transformer’s divergence increased when each rollout received equal weight, so long capped trajectories alone did not explain its poor composition score.The causal all-position Transformer shifted from below 1×F under pooled-token weighting to 2.3×F under equal rollout weighting.Source: H Sensitivity to rollout weighting · p. 16, Table 13 · p. 16

I Train-context coverage

  • Through order 3, exact train-context coverage was nearly complete for both event vocabularies, even as distinct contexts rose rapidly with event granularity and history length.Coverage alone does not determine count-model fidelity because backoff can use shorter observed histories when a higher-order context is unavailable.Source: I Train-context coverage · p. 17, Table 14 · p. 17

J Rollout-harness audit

  • Six rollout and metric inconsistencies were corrected and regression-tested; the audit underscores that shared stopping rules, prefix boundaries, and cross-producer metric tests are part of reproducible simulator evaluation.The corrections included generation-history and timing fixes, suffix-only remaining-LOS and reached-discharge metrics, and a common decoder stopping rule.Source: J Rollout-harness audit · pp. 17–18

K Downstream length-of-stay prediction

  • 0.72–0.78 internal AUROC and 0.70–0.78 on MC-MED: trajectory-derived delays predicted top-decile total LOS but did not outperform elapsed time alone at any landmark.Discharge-readiness AUROC reached 0.804 internally but fell to 0.564 on MC-MED; recalibration improved calibration, not discrimination, and the analyses were exploratory rather than evidence of clinical utility.Source: K Downstream length-of-stay prediction · p. 18

L Occupancy forecasting

  • 0.7×F composition divergence did not translate to better occupancy forecasts: the causal all-position Transformer had 11.7-point MAE, while the decoder had 4.2×F divergence and 2.0-point MAE.Across seven configurations, non-termination was more strongly associated with occupancy error (ρ = +0.93) than ×F was (ρ = +0.63).Source: L Occupancy forecasting · p. 19, Table 16 · p. 19
  • Occupancy forecasting illustrates that intrinsic composition fidelity and task-specific utility are distinct evaluation axes.Non-terminating rollouts are handled using an explicitly conservative convention.Source: L Occupancy forecasting · p. 19

M Sensitivity to launch context

  • Transformer termination was at least 0.994 from triage-anchored prefixes versus 0.899 from non-triage prefixes, while recurrent-model termination showed the opposite pattern.For the recurrent models, termination was around 0.41–0.54 from triage-anchored prefixes and approximately 0.69 from other launch points. For the 7-token Transformer, termination fell from 0.957 in the shortest trace-length quartile to 0.857 in the longest, though trace length also reflects complexity and case mix.Source: M Sensitivity to launch context · p. 19, Table 17 · p. 19

N Performance by recorded demographic group

  • Among recorded race categories, the Black group had the lowest duration ratio and reached-discharge rate, while the Asian group had the highest duration ratio across the three standard architectures.Transformer duration ratios ranged from 0.927 in the Black group to 1.130 in the Asian group, but these descriptive comparisons are unadjusted and do not establish a systematic fairness disparity. Differences by recorded sex were smaller: termination varied by less than 0.025 and duration ratio by less than 0.018.Source: N Performance by recorded demographic… · p. 20

O Real-versus-real JSD reference

  • Two independent real suffix samples differed by 4.5 × 10−6 with the 7-token vocabulary and 1.0 × 10−5 with the 32-token vocabulary, while vanilla Transformer divergence was more than four orders of magnitude larger.This real-versus-real comparison calibrates the metric and indicates that ordinary sample variability alone does not explain the Transformer’s large JSD; it does not require valid stochastic simulators to match the exact observed marginal.Source: O Real-versus-real JSD reference · pp. 20–21, Table 19 · p. 21

P Sensitivity to the rollout cap

  • Doubling the generation cap from 128 to 256 events raised 7-token GRU termination from 0.680 to 0.815 and LSTM termination from 0.683 to 0.801.The recurrent models still terminated substantially less often than the order-3 count reference, while 32-token Transformer termination changed only from 0.434 to 0.447; termination is cap-dependent and should be compared under a common stopping rule.Source: P Sensitivity to the rollout cap · p. 21, Table 20 · p. 21

Q Repetition and per-step error

Per-step analyses distinguish initial miscalibration from drift during self-conditioning, while intervention and external-site results show that termination and event composition do not necessarily improve together.

Q Repetition and per-step error

  • All seven standard Transformer seeds had higher first-step JSD than all three GRU seeds: 0.016–0.156 versus 0.003–0.006.The Transformer median was 0.046, while recurrent models began closer to the observed first-step distribution and accumulated larger errors later.Source: Q Repetition and per-step error · p. 21
  • By approximately step 10, causal all-position Transformer JSD remained around 0.002–0.004, whereas tokenized-time decoder JSD rose to approximately 0.010–0.013.The decoder’s remaining closed-loop gap is consistent with error accumulating under self-conditioning rather than elevated error at the first generated event.Source: Q Repetition and per-step error · pp. 21–22
  • Nearly identical open-loop accuracy corresponded to substantially different closed-loop event-distribution divergence across the three standard architectures.Figure 2 contrasts next-event accuracy with suffix event-distribution JSD; open and filled markers denote 7- and 32-token vocabularies, respectively.Source: Figure 2 · p. 22
  • Interventions improved termination and sometimes reduced event-distribution divergence, but none jointly reached near-complete termination and the order-3 reference’s composition.The figure compares interventions in the termination–suffix-JSD plane; temporal calibration is evaluated separately in Table 4.Source: Figure 4 · p. 23

R External-site replication

  • Capped non-terminating standard-model trajectories produced suffix JSD values that were not length-matched to terminating count and all-position model trajectories.The authors caution that these JSD values should be interpreted together with termination rather than as standalone comparisons.Source: R External-site replication · p. 24
  • 0.000 termination and reached-discharge rates under vanilla decoding for the standard Transformer and GRU on MC-MED coexisted with observed-prefix accuracy of 0.689 and 0.719, respectively.Process masking’s apparent 0.011 termination rate came only from prefixes where discharge had already occurred; no other masked rollout terminated.Source: R External-site replication · pp. 23–24
  • 0.945–0.979 of MC-MED rollouts terminated for causal all-position Transformer runs, with event-distribution divergence of 1.0–2.5×F relative to the refitted order-3 count reference.The authors report this as replication of the qualitative training-setup association; dataset size, vocabulary, and visit-length differences preclude direct comparison of absolute MIMIC and MC-MED performance.Source: R External-site replication · p. 24
Loading 2609.31635v1…