INTERVenE: Temporal-Abstraction-Interval Based Transformers for Short-Horizon Medical Event Prediction

Introduction

• In diabetic ICU admissions, the task uses the first 48 hours to predict six complications during hours 48–336, amid irregular, sparse observations and dynamic dysglycemia.The clinical significance of observations can depend on duration, contextual trends, and missingness.Source: Introduction · p. 1
• INTERVenE-Enc reached support-weighted AUPRC of 0.672, +0.041 over the strongest neural competitor with non-overlapping 95% confidence intervals, across 57,078 MIMIC-IV admissions.The models were benchmarked against GRU-D, supervised and self-supervised STraTS (Tipirneni and Reddy 2022), and KarmaLego+TPF.Source: Introduction · p. 1
• INTERVenE-Enc uses sparse transcoders to recover clinically meaningful latent drivers, with explanations resolving to named clinical concepts by vocabulary construction.The attribution pipeline is gradient-decoupled.Source: Introduction · pp. 1–2
• KBTA converts observations into interval-based, named clinical-concept tokens, serving as the interpretability substrate rather than a predictive prerequisite.An input-representation ablation supports this distinction.Source: Introduction · pp. 1–2
• INTERVenE-Enc predicts risk and time-to-event in a single pass, while INTERVenE-Ar autoregressively generates abstraction trajectories with per-step risk curves.The encoder is bidirectional; the decoder’s token-level readouts show evolving risk trends.Source: Introduction · p. 1

Temporal abstraction and interval mining

• KBTA converts point observations into named interval states, trends, events, and contexts; KarmaLego mines interval patterns and TPF summarizes them as patient vectors.KBTA’s context-sensitive abstractions draw on declarative medical knowledge; KarmaLego is described by Moskovitch and Shahar 2015, and TPF by Sheetrit et al. 2019.Source: Related Work · p. 2

Deep learning for irregular clinical sequences

• GRU-D models missingness and elapsed time, STraTS attends over value-time-variable triplets, and prior Transformers use visit summaries, concept relations, or time-gap embeddings.INTERVenE instead tokenizes explicit abstracted intervals and combines KBTA semantics with continuous Time2Vec embeddings (Kazemi et al. 2019). The cited approaches are GRU-D, STraTS (Tipirneni and Reddy 2022), and Transformers (Li et al. 2020).Source: Related Work · p. 2

Explainability in sequential clinical models

• INTERVenE’s KBTA-native decision pathways and sparse-transcoder features resolve to named clinical concepts, addressing when and in what context predictions arise.Post-hoc methods such as SHAP and LIME only partially address interpretability when applied to raw measurement streams; sparse-transcoder feature extraction is cited as Dunefsky, Chlenski, and Nanda 2024.Source: Related Work · p. 2

Cohort and Prediction Task

• 57,078 adult diabetes-related MIMIC-IV admissions are split by patient into 39,954 training, 8,562 validation, and 8,562 test admissions, with all methods sharing labels, context, and observation window.GRU-D and STraTS use raw event streams, while KarmaLego and INTERVenE use Mediator-derived KBTA intervals.Source: Cohort and Prediction Task · p. 2
• The task predicts six binary complications during hours 48–336 after admission, excluding complications in the initial 48-hour observation window from positive labels.AUPRC is primary because outcomes are imbalanced; AUROC and best F1 are secondary, and continuous length of stay is an auxiliary regression target.Source: Cohort and Prediction Task · p. 2

Shared Representation and Backbone

• INTERVenE maps raw records to diabetes-specific KBTA intervals, tokenizes boundaries with semantically shared hierarchical concepts, and combines Time2Vec with patient-context conditioning in a backbone shared across variants.Temporal rotary position embeddings are also applied within the attention layers.Source: Shared Representation and Backbone · pp. 2–3

INTERVenE-Ar (Autoregressive Decoder)

• INTERVenE-Ar uses causal attention to generate full abstraction trajectories and per-step risk, with legality-masked soft temporal targets, interval-pair corruption, and risk-head alignment for free-running inference.At inference, the decoder starts from the first 48 hours and reads risk at every generated step.Source: INTERVenE-Ar (Autoregressive Decoder) · pp. 3–4

INTERVenE-Enc (Bidirectional Encoder)

• INTERVenE-Enc uses bidirectional attention and jointly predicts outcome risk and time-to-event after atomic interval masking with temporal-position auxiliaries.Its risk and time heads use per-outcome attention pooling; the time loss is computed over positive patients only, with RELEASE serving as a length-of-stay predictor.Source: INTERVenE-Enc (Bidirectional Encoder) · p. 4

Interpretability via Sparse Transcoders

• Sparse transcoders decompose encoder MLP computations into named-concept attributions without changing the deployed forward pass; these gradient-based scores reflect local sensitivity, not causal necessity.Because inputs are KBTA abstractions, each token attribution resolves to a named clinical state, trend, or event.Source: Interpretability via Sparse Transcoders · pp. 4–5

Baselines

• Baselines span recurrent, raw-triplet Transformer, and KBTA pattern-mining approaches, trained and selected under a shared protocol; only KarmaLego+TPF matches INTERVenE's KBTA input representation.The comparison with KarmaLego+TPF isolates Transformer sequence modeling against symbolic pattern mining on the same representation.Source: Baselines · p. 5

Overall performance

• A support-weighted AUPRC of 0.672 gives INTERVenE-Enc the strongest overall performance, +0.041 over ss-STraTS and +0.085 over KarmaLego, with non-overlapping 95% CIs.It also achieves the best AUROCw, Best-F1w, and LoS MAE. INTERVenE-Ar is scored only after a full autoregressive rollout, a substantially harder generative protocol; its strongest performance is on dysglycemia, while DEATH remains challenging.Source: Results · p. 5

Per-outcome performance

• INTERVenE-Enc has the highest AUPRC on five of six outcomes, including KIDNEY (0.790 vs. 0.726) and DEATH (0.605 vs. 0.563), and ties STraTS on the rarest outcome.The rarest outcome is SEVERE HYPOGLYCEMIA (2.7% prevalence; overlapping 95% CIs).Source: Results · pp. 5–6, Table 2 · p. 6

Time-to-event prediction

• The auxiliary time head reduces MAE by 7–14 h versus the median-time baseline across all outcomes.Time MAEs are reported only for INTERVenE variants; INTERVenE-Ar’s event time is taken from the peak predicted probability along the generated trajectory, matched to the nearest ground-truth event.Source: Results · p. 6

Calibration

• INTERVenE-Enc has support-weighted ECE 0.016 before scaling and 0.013 after; scaling improves INTERVenE-Ar’s prevalent-outcome calibration, including DEATH from 0.050 to 0.011.Temperature scaling is fitted on the validation split and applied unchanged to the held-out test set; INTERVenE-Ar’s peak-detector readout has higher ECE (≈4–12%).Source: Results · p. 6

Interpretability

• DEATH attributions across 300 test patients highlight severe hypoalbuminemia, amino-transferase elevation, and increasing bicarbonate as positive risk drivers, with recovery-associated concepts contributing negatively.Attributions are computed directly from the deployed model and expressed through the KBTA vocabulary as named clinical concepts; similar clinically coherent patterns appear across all six outcomes.Source: Figure 2 · p. 6, Results · p. 6

Discussion

• INTERVenE-Enc consistently outperforms GRU-D, STraTS, and KarmaLego, while replacing KBTA with σ-based discretization preserves essentially the same AUPRC and AUROC.The discussion attributes predictive gains chiefly to the Transformer architecture operating on persistent interval representations, rather than to a particular discretization.Source: Discussion · pp. 6–7
• Collapsing KBTA intervals to 1-second durations drops weighted AUPRC from 0.672 to 0.593 and increases LoS MAE by 7.36 h.The discussion identifies persistence of symbolic information over time as an important predictive signal; KBTA’s ontology-derived states, trends, and events also let attributions resolve into clinically meaningful explanations.Source: Discussion · p. 7
• DEATH AUPRC is 0.605 vs. ss-STraTS 0.563, and length-of-stay is 44.4 h vs. ∼48 h; representation helps most for severe dysglycemia defined by threshold-crossing states and durations.For DEATH, the discussion suggests combinations of abstracted states—including hypoalbuminemia, organ-stress trends, and acid-base shifts—signal physiological deterioration, consistent with the clinically coherent drivers in Figure 2.Source: Discussion · p. 7
• INTERVenE-Enc provides low-latency single-pass prediction, whereas INTERVenE-Ar generates complete abstraction trajectories with per-step risk curves localizing when and after which events risk rises.On one RTX A5000 at batch size 16, the encoder processes 3.3 ms per patient (302 patients/s); the autoregressive model takes 255 ms per patient (3.9 patients/s), an approximately 77× latency difference, and CPU inference is impractical.Source: Discussion · p. 7

Representation limitations

• Concepts without TAK abstraction rules are absent, so adapting INTERVenE to new diseases, populations, or settings requires a labor-intensive, expert-built knowledge base.The diabetes TAK base used here has 159 abstraction definitions and a vocabulary of 506 named clinical concepts.Source: Limitations · p. 7
• Rule-based abstraction can collapse clinically distinct values, but σ-based discretization achieves comparable AUPRC and AUROC, suggesting the architecture can use alternative structured inputs.For example, glucose measurements of 250 mg/dL and 600 mg/dL both map to the severehyperglycemia state.Source: Limitations · p. 7

Study limitations

• Evaluation uses a single MIMIC-IV cohort; external validation, subgroup fairness across demographics and care settings, and attribution robustness across initializations remain future directions.Source: Limitations · p. 7
• The study reports patient-level bootstrap confidence intervals rather than multi-seed variance because training takes 8–12 hours for INTERVenE-Enc and over 24 hours for INTERVenE-Ar on one RTX A5000.Source: Limitations · p. 7

Conclusion

• INTERVenE is a Transformer family for temporal EHR modeling over knowledge-based temporal abstractions.The conclusion presents the family as a neuro-symbolic framework for combining predictive performance with clinically meaningful interpretability.Source: Conclusion · p. 7
• INTERVenE-Enc achieves the strongest predictive performance on the MIMIC-IV diabetic cohort while providing faithful, concept-level explanations.The explanations use gradient-decoupled sparse-transcoder attribution.Source: Conclusion · p. 7
• INTERVenE-Ar generates autoregressive abstraction trajectories and per-step risk curves for inspecting how clinical evidence accumulates during an admission.Source: Conclusion · p. 7

• Complete source code for INTERVenE, all baselines, and the MIMIC-IV cohort pipeline is included in supplementary material, with a public repository planned upon acceptance.Source: Data and Code Availability · p. 7
• Both INTERVenE variants have approximately 1.8M parameters and train on a single NVIDIA RTX A5000 (24 GB VRAM) or smaller.Source: Data and Code Availability · p. 7
• The full pipeline requires no multi-GPU resources and fits within 25 GB on disk; the KarmaLego pipeline runs CPU-only.Source: Data and Code Availability · p. 7

• Intermediate design stages report only metrics tracked during their search, leaving unevaluated metrics marked “–”.Source: Architectural Ablation Trail · p. 8

Decoder ablations

• The decoder design search found that the patient-pool auxiliary gave the largest discrimination gain, while Phase-3 fine-tuning dominated component-drop ablations.The ttt auxiliary improved trajectory-end anticipation, the ttt inference gate restored realistic trajectory lengths, and RoPE, CBM, soft-kernel loss, and damping each contributed measurably. DD-p0.33 weighting gave the best compromise between over- and under-generation while preserving predictive performance.Source: Decoder ablations · pp. 8–9

Encoder ablations

• Weighted AUPRC falls to 0.593 when intervals collapse into 1-second events, while standard-deviation bins preserve AUPRC/AUROC but add 3.2 h LoS error.Phase-3 backbone learning is essential; CBM masking provides the only consistent improvement over the stripped baseline, inverse-prevalence weighting degrades performance, and focal loss has negligible effect. Reducing the time-loss weight improves discrimination, hierarchical MLM masking adds no measurable benefit, and capacity peaks at deployed M-128; even M-64 outperforms ss-STraTS.Source: Encoder ablations · p. 9, INTERVenE-Ar · p. 10

Calibration

• Calibration leaves locked checkpoints unchanged, optimizing only the test-time logit-to-probability mapping.Per-outcome temperatures are fit on validation data and applied unchanged to the held-out test split; monotonic scaling does not affect rank-based metrics.Source: Calibration · p. 9, INTERVenE-Enc · p. 9

INTERVenE-Enc

• INTERVenE-Enc temperatures are near unity (0.87–1.04), and scaling modestly improves support-weighted ECE from 0.016 to 0.013.The fitted temperatures are provided alongside the released checkpoint as an optional refinement.Source: INTERVenE-Enc · p. 9, Table 6 · p. 10

INTERVenE-Ar

• INTERVenE-Ar temperatures span 0.65–1.47, and scaling reduces DEATH ECE from 0.050 to 0.011, while rare hypoglycemia remains difficult to calibrate.Aggregate ECE is omitted because calibration uses the trajectory-derived peak-probability readout rather than a native discriminative output.Source: INTERVenE-Ar · p. 10, Table 7 · p. 10

Autoregressive Risk Trajectories

• INTERVenE-Ar evaluates its risk head at every generation step, producing time-resolved curves that can express multiple risk peaks within an admission.Peak width depends on the Phase-2 soft-kernel (τ = 12 h for DEATH), so absolute risk values should be interpreted relative to τ.Source: Autoregressive Risk Trajectories · p. 10, Figure 5 · p. 13

Implementation Details

• During autoregressive training and generation, deterministic legality masks exclude invalid interval continuations and enforce meal-token order.Outcome labels for complications in hours 48–336 enter only in Phase 3; Phase-2 sequence pretraining sees future tokens, while inference is seeded only with the 0–48 h input window.Source: Implementation Details · pp. 11–12

Sparse-Transcoder Interpretability

• Sparse-transcoder attribution decomposes deployed-model risk logits into token contributions, with clinically coherent drivers expressed as named KBTA concepts.Transcoders reconstruct MLP outputs with per-layer R2 of 0.97–0.995 and are used only for attribution; the explanations represent local sensitivity rather than feature necessity.Source: Sparse-Transcoder Interpretability · pp. 12–13, Figure 6 · p. 14