PaperlayerView exact revision on arXiv

Source-linked AI summary

INTERVenE: Temporal-Abstraction-Interval Based Transformers for Short-Horizon Medical Event Prediction

Shahar Oded, Yuval Shahar

arXiv:2608.29901v1cs.LGcs.AI

TL;DR

ICU prediction models must handle sparse, irregular EHR data while preserving clinical meaning and interpretability. INTERVenE uses named KBTA interval concepts in complementary Transformer architectures, with INTERVenE-Enc achieving the strongest benchmark performance, including weighted AUPRC 0.672.

  • Problem

    ICU complication prediction must model sparse, irregular EHR streams while preserving temporal context and clinically interpretable representations.

  • Method

    INTERVenE transforms observations into KBTA intervals and uses autoregressive or bidirectional Transformers for trajectory forecasting, risk prediction, and time-to-event prediction.

  • Results

    INTERVenE-Enc consistently outperforms GRU-D, STraTS, and KarmaLego, achieving weighted AUPRC 0.672 while preserving performance across structured discretizations.

  • Takeaways & Limitations

    The results support combining competitive prediction with concept-level explanations and temporal risk inspection within one neuro-symbolic framework.

  • Takeaways & Limitations

    Extending INTERVenE to new diseases, populations, or settings requires labor-intensive construction of domain-specific temporal-abstraction knowledge bases.

Abstract

from arXiv · show

Electronic Health Record (EHR) prediction models in the intensive care unit must learn from sparse and irregular measurements while preserving the clinical meaning of time and supporting transparent decision-making. We present INTERVenE, a family of Transformer architectures whose input is an interval-based, knowledge-based temporal abstraction (KBTA), a token stream of named clinical concepts (states, trends, events, contexts) drawn from a curated medical ontology, rather than an unnamed bin index or a raw measurement triplet. This naming layer is what we ask KBTA to do: it makes the model's per-token attributions resolve to clinical concepts by construction. INTERVenE offers two complementary variants: an auto-regressive decoder that generates future abstraction trajectories with a per-step risk readout (localizing \emph{when} and \emph{after which events} risk rises), and a bidirectional encoder for single-pass joint risk and time-to-event prediction. Evaluated on 57,078 MIMIC-IV admissions against GRU-D, STraTS, and KarmaLego, INTERVenE-Enc reaches a support-weighted AUPRC$_w$ of 0.672, improving by 0.041 over the strongest neural baseline with non-overlapping 95\% bootstrap CIs, while also taking the best AUROC$_w$ (0.901) and length-of-stay MAE (44.4\,h). INTERVenE-Ar (AUROC$_w$ $0.854$, AUPRC$_w$ $0.587$ under the same evaluation contract - a strictly harder generative readout) provides a complementary token-level risk trajectory. An input-representation ablation confirms the lift transfers across structured discretizations, positioning KBTA-based intervals as the interpretable substrate that makes per-token attributions resolve to meaningful clinical concepts within the deployed model.

Introduction

INTERVenE represents irregular ICU EHR data as named KBTA intervals to support prediction over clinically meaningful temporal abstractions. It evaluates complementary Transformer variants for diabetic ICU complication forecasting while making model explanations resolve to named clinical concepts.

  • Irregular, sparse, heterogeneous ICU records make short-horizon complication prediction difficult because interpretation depends on duration, trends, context, and missingness.
  • The study converts raw observations into KBTA intervals and tests whether this representation can outperform neural and symbolic alternatives while retaining interpretable intermediate representations.
  • The task observes the first 48 hours of diabetic ICU admissions and predicts six complications during hours 48–336.The cohort is drawn from MIMIC-IV admissions with dynamic inpatient dysglycemia rather than diabetes treated as a static comorbidity.
  • INTERVenE-Enc performs single-pass joint risk and time-to-event prediction, whereas INTERVenE-Ar autoregressively forecasts KBTA trajectories with per-step risk readouts.
  • KBTA supplies a named-concept Transformer vocabulary, while sparse-transcoder attribution and autoregressive decoding support explanations and risk trajectories grounded in clinical concepts.The central thesis distinguishes KBTA as the interpretability substrate from INTERVenE as the predictive framework operating over it.

Related Work

Prior work spans knowledge-based temporal abstraction for clinically meaningful intervals, deep models for irregular EHR sequences, and post-hoc explainability methods. INTERVenE builds on these directions by using named KBTA concepts to support clinically grounded interpretation.

  • Temporal abstraction and interval mining: KBTA converts raw point observations into interval-level states, trends, events, and contextual periods using potentially context-sensitive medical knowledge.Its contextual rules can depend on factors such as age, gender, prior medication administration, meals, or other abstractions.
  • Deep learning for irregular clinical sequences: GRU-D models missingness and elapsed time through learned decays, whereas STraTS uses set-based attention over value-time-variable triplets.These approaches address irregular EHR sequences without relying on the same abstraction vocabulary described for KBTA.
  • Explainability in sequential clinical models: Post-hoc methods such as SHAP and LIME only partially address the need to identify which variables influenced a prediction, when, and in what context.The passage frames temporal and contextual attribution as a central challenge for sequential clinical models.
  • Explainability in sequential clinical models: INTERVenE operates on a KBTA vocabulary so decision pathways and sparse-transcoder features resolve to named clinical concepts rather than raw numerical artifacts.This representation makes the model’s interpretive features correspond to clinically meaningful concepts.

Cohort and Prediction Task

The study uses 57,078 adult diabetes-related MIMIC-IV admissions with patient-level train/validation/test splits. It predicts six complications during hours 48–336 after admission, excluding events observed in the initial 48 hours, with length of stay as an auxiliary target.

  • Cohort: 57,078 adult diabetes-related admissions from MIMIC-IV form the cohort, selected for dense physiological measurements, a mature KBTA ontology, and six clinically meaningful complications.All methods use the same patient split.
  • Cohort: 39,954 training, 8,562 validation, and 8,562 held-out test admissions are assigned using patient-level splits.The split is shared across all methods.
  • Prediction Task: Hours 48–336 after admission define the prediction window, while complications occurring during the initial 48-hour observation window are excluded from positive labels.This ensures models predict future events rather than already observed complications.
  • Prediction Task: Six binary outcomes are predicted: death, kidney complication, hyperglycemia, severe hyperglycemia, hypoglycemia, and severe hypoglycemia.A continuous length-of-stay target serves as an auxiliary regression target.

Methods

INTERVenE converts raw clinical records into semantically named KBTA intervals, embeds their hierarchical concepts with elapsed-time information, and processes them through shared Transformer components with variant-specific masking and objectives. Its attribution method preserves deployed-model logits while decomposing risk into named clinical concepts, and its comparisons use matched evaluation choices across modeling paradigms.

  • Temporal abstraction: Raw temporal records are mapped to KBTA intervals that identify clinically meaningful states, trends, contexts, and events using a diabetes-specific knowledge base.The Mediator temporal abstraction engine performs this mapping.
  • Token representation: Each interval becomes START and END tokens with four hierarchical embeddings, progressing from raw concept to abstraction, value, and positional role.Shared layers enforce semantic similarity across related concepts and interval boundaries.
  • Temporal and patient encoding: Elapsed time is encoded with Time2Vec, concatenated with the four concept embeddings, projected to dimension d, and supplemented by temporal RoPE and AdaLN-Zero patient conditioning.Static attributes such as age and sex provide the adaptive normalization context.
  • Model variants: INTERVenE-Ar uses causal attention to generate abstraction trajectories and read risk at every step, whereas INTERVenE-Enc uses bidirectional attention for its corresponding objectives.Both variants share the same embedder, temporal-RoPE attention, AdaLN-Zero conditioning, and SwiGLU backbone.
  • Interpretability: Attributions retain the deployed model’s true risk logits while using a sparse named-concept basis for gradient decomposition, so each contribution resolves to a clinical state, trend, or event.These gradients represent local model sensitivity rather than causal necessity.
  • Benchmark design: GRU-D and STraTS operate on raw event streams, while the KarmaLego comparison uses the same KBTA representation to isolate Transformer sequence modeling from symbolic pattern mining.The stated evaluation design keeps modeling differences separate from data and tuning choices.

Results

INTERVenE-Enc achieves the strongest overall discriminative performance, with leading support-weighted classification metrics and improved event-time prediction. Across outcomes, calibration and attribution analyses further show reliable probability estimates and clinically coherent risk drivers.

  • Overall performance: 0.672 support-weighted AUPRC is INTERVenE-Enc’s strongest overall result, improving by +0.041 over ss-STraTS and +0.085 over KarmaLego with non-overlapping 95% CIs.It also achieves the best AUROCw, Best-F1w, and LoS MAE, making it the strongest single-pass discriminative model.
  • Per-outcome results: INTERVenE-Enc achieves the highest AUPRC on five of six outcomes, with largest gains for KIDNEY (0.790 vs. 0.726) and DEATH (0.605 vs. 0.563).It ties STraTS on the rarest outcome, SEVERE HYPOGLYCEMIA, where prevalence is 2.7% and 95% CIs overlap.
  • Time prediction: 7–14 h lower MAE than the median-time baseline is achieved by the auxiliary time head across all outcomes.Time MAEs are reported only for INTERVenE variants because baselines lack per-outcome time heads; INTERVenE-Ar uses the peak predicted probability along its generated trajectory.
  • Calibration: 0.016 support-weighted ECE is INTERVenE-Enc’s emitted calibration error, improving only slightly to 0.013 after temperature scaling.INTERVenE-Ar has higher calibration error under its peak-detector readout, while scaling improves prevalent outcomes such as DEATH from 0.050 to 0.011.
  • Interpretability: Severe hypoalbuminemia dominates positive DEATH-risk attribution, followed by elevated amino-transferase and increasing bicarbonate.Negative attribution highlights recovery-associated concepts, and coherent patterns are observed across all six outcomes.

Discussion

The discussion attributes INTERVenE’s gains chiefly to its Transformer architecture operating on persistent interval representations, while emphasizing that interval persistence remains essential. It also presents INTERVenE-Enc and INTERVenE-Ar as complementary variants, balancing production accuracy with trajectory-level interpretability.

  • Architecture and representation: INTERVenE-Enc consistently outperforms GRU-D, STraTS, and KarmaLego, while ablations show comparable gains after replacing KBTA with simple σ-based discretization.This suggests predictive performance is driven primarily by the neural architecture rather than a particular interval discretization.
  • Architecture and representation: 0.672 →0.593 weighted AUPRC and +7.36 h LoS MAE after collapsing KBTA intervals to 1 second demonstrate that persistent symbolic intervals remain essential.The discussion distinguishes persistence over time from the specific discretization scheme.
  • Clinical abstraction: 0.605 vs. 0.563 AUPRC for DEATH and 44.4 h vs. ∼48 h length-of-stay show gains beyond outcomes defined by a single KBTA threshold.For these outcomes, the model appears to recognize combinations of abstracted states and trends.
  • Complementary variants: INTERVenE-Enc targets production prediction with strongest single-pass accuracy and low-latency inference, whereas INTERVenE-Ar generates abstraction trajectories with per-step risk curves for interpretability.The autoregressive variant localizes when and after which events risk increases.

Limitations

INTERVenE depends on expert-authored KBTA knowledge, which limits covered concepts and requires new domain-specific abstraction bases for new settings. The study is also limited by coarse abstraction granularity, single-cohort evaluation, absent subgroup and external validation, and costly training that restricts multi-seed variance analysis.

  • Representation dependence: KBTA excludes concepts lacking explicit abstraction rules, so extending INTERVenE to new diseases, populations, or settings requires constructing a domain-specific TAK base.The process is expert-driven rather than a simple retraining step.
  • Representation dependence: 159 TAK abstraction definitions produce a vocabulary of 506 named clinical concepts in the diabetes knowledge base.These figures characterize the knowledge base used in the study.
  • Granularity: Rule-based abstractions can collapse clinically distinct values, such as 250 mg/dL and 600 mg/dL glucose, into the same severehyperglycemia state.A σ-based discretization ablation achieves comparable AUPRC and AUROC, indicating operation over alternative structured discretizations.
  • Evaluation and computation: External validation, subgroup fairness, and attribution robustness remain unaddressed because evaluation used a single MIMIC-IV cohort and patient-level bootstrap intervals.Training takes approximately 8–12 hours for INTERVenE-Enc and more than 24 hours for INTERVenE-Ar on a single RTX A5000, limiting multi-seed variance analysis.

Conclusion

INTERVenE is a Transformer family for temporal EHR modeling over knowledge-based temporal abstractions. Its encoder provides concept-level explanations, while its autoregressive variant supports temporal inspection through abstraction trajectories and per-step risk curves.

  • INTERVenE is a Transformer family for temporal EHR modeling over knowledge-based temporal abstractions.
  • INTERVenE-Enc achieves the strongest predictive performance on the MIMIC-IV diabetic cohort with faithful, concept-level explanations through gradient-decoupled sparse-transcoder attribution.
  • INTERVenE-Ar generates autoregressive abstraction trajectories and per-step risk curves, enabling temporal inspection of how clinical evidence accumulates over an admission.

Data and Code Availability

The supplementary material includes complete source code and the MIMIC-IV cohort pipeline, with a public repository planned upon acceptance. INTERVenE trains on modest single-GPU resources, while KarmaLego runs CPU-only and the full pipeline fits within 25 GB on disk.

  • Data and Code Availability: Complete source code for INTERVenE, all baselines, and the MIMIC-IV cohort pipeline is included in the supplementary material.A public repository will be released upon acceptance.
  • Data and Code Availability: ∼1.8M parameters characterize both INTERVenE variants, which train on a single NVIDIA RTX A5000 with 24 GB VRAM or smaller.The full pipeline requires no multi-GPU resources.
  • Data and Code Availability: CPU-only execution is sufficient for the KarmaLego pipeline, and the full pipeline fits within 25 GB on disk.These requirements avoid multi-GPU infrastructure.

Appendix Clinical Thresholds for Target Complications · Architectural Ablation Trail

The appendix documents an iterative architectural design search leading to the main-paper endpoints. Early choices were explored on a 10k-admission subset and final recipes were confirmed on the full training cohort, with intermediate stages reporting only tracked metrics.

  • Architectural Ablation Trail: The main-paper architectures are the endpoints of an iterative design search documented in the appendix.
  • Architectural Ablation Trail: Early experiments used a 10k-admission design subset to explore architectural choices.
  • Architectural Ablation Trail: The final recipes were subsequently confirmed on the full training cohort.
  • Architectural Ablation Trail: Intermediate design stages report only metrics tracked during their corresponding search.
  • Architectural Ablation Trail: Metrics not evaluated at an intermediate stage are represented by “–”.
  • Architectural Ablation Trail: The ablation trail therefore separates exploratory architectural development from confirmation of the final recipes.

Decoder ablations … Sparse-Transcoder Interpretability

Ablations identify the training components and capacity choices that shape INTERVenE’s decoder and encoder, while calibration and attribution analyses characterize deployment behavior and interpretability. Sparse-transcoder attribution preserves the deployed computation and yields clinically coherent, named KBTA-concept drivers.

  • Decoder ablations: The decoder trail found the ttt auxiliary head improved trajectory-end anticipation, the patient-pool auxiliary gave the largest discrimination gain, and the ttt gate restored realistic trajectory lengths.Phase-3 fine-tuning was the dominant component-drop contributor; RoPE, CBM, soft-kernel loss, and damping also contributed measurably.
  • Encoder ablations: The encoder trail found that reducing the time-loss weight improved discrimination, hierarchical MLM masking added no measurable benefit, and CBM masking at p=0.15 was the only consistent full-cohort improvement.Phase-3 backbone learning was essential, while inverse-prevalence weighting degraded performance and focal loss had negligible effect.
  • Encoder ablations: M-128 provided the best accuracy–efficiency trade-off, while wider or more heavily headed models added little predictive benefit and smaller models underfit.Even M-64 outperformed ss-STraTS, indicating the headline gains were driven primarily by architecture rather than capacity.
  • Calibration: INTERVenE-Enc temperature scaling modestly improved support-weighted ECE from 0.016 →0.013, with fitted temperatures close to unity at Tk ∈[0.87, 1.04].Scaling was performed without retraining: validation-fitted per-outcome temperatures were applied unchanged to the held-out test split.
  • INTERVenE-Ar: INTERVenE-Ar required stronger calibration adjustment, with Tk ∈[0.65, 1.47] and DEATH ECE improving from 0.050 →0.011 after scaling.Rare hypoglycemia outcomes remained the most difficult to calibrate.
  • Autoregressive Risk Trajectories: The autoregressive variant emits a per-step risk curve that can express multiple risk peaks within an admission, with peak width governed by the Phase-2 soft-kernel τ = 12 h for DEATH.Absolute risk values should be interpreted relative to τ rather than as binary decisions.
  • Implementation Details: Both variants use three-phase training with fixed-seed single-GPU execution, AdamW optimization, gradient clipping, scheduled learning rates, and patient-level 2,000-resample bootstrap uncertainty estimates.INTERVenE-Ar generation is bounded by a 2,000-token budget and 336-hour temporal limit, with early stopping on terminal tokens.
  • Sparse-Transcoder Interpretability: Sparse transcoders reconstructed block MLP outputs with per-layer R2 = 0.97–0.995 and approximately 5–19% active features, while gradient-decoupled hooks preserved the deployed forward computation.Across six outcomes, signed gradient×activation attribution produced clinically coherent, entirely named KBTA-concept drivers averaged over 300 test patients.
Loading 2608.29901v1…