PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis

Hamza Shafiq, Jialu Tang, Ting Dang, Jun Hu, Aaqib Saeed

arXiv:2610.13510v1cs.AIcs.CL

TL;DR

Clinical prognosis requires integrating heterogeneous patient evidence, while existing multimodal systems may omit longitudinal ECG information. PRISM combines numerical feature encoding, ECG history aggregation, and population retrieval in a multimodal language model. It reports the strongest MDS-ED figures across four task groups.

  • Problem

    Clinical prognosis requires combining multiple information sources, but existing multimodal systems encode only index-visit ECGs and forfeit signal embedded in longitudinal change.

    Source: §1 · pp. 1–2

  • Method

    PRISM combines per-feature numerical encoding, recency-weighted longitudinal ECG aggregation, and population retrieval of similar-patient representations in a multimodal language model.

    Source: §1 · p. 2

  • Results

    PRISM reports the strongest MDS-ED figures across four task groups: AUROCs of 86.24% for Diagnosis, 93.12% for Deterioration, 91.79% for ICU admission, and 92.82% for Mortality.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings motivate evaluating the complete evidence pipeline when adapting multimodal models to new clinical settings.

    Source: §5 · p. 17

    The full-serialization comparison differs in sequence length, batch size, and optimization conditions, so it is a practical comparison rather than an isolated encoding test.

    Source: §4.6 · p. 15

Abstract

from arXiv · show

Clinical prognosis requires integrating structured measurements, physiological waveforms, longitudinal history, and population context. We introduce PRISM (Population-Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model that combines per-feature numerical tokenization, recency-weighted aggregation of longitudinal ECGs, and a dual-modality population memory bank. Retrieval operates in representation space: the model receives mean-pooled embeddings of similar training-cohort encounters, while no individual retrieved record enters the prompt. On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality. These results use a 96-hour pre-arrival window unavailable to reported baselines. With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED. A matched-input compact fusion model using the same five evidence streams and frozen encoders reaches 0.7652 AUROC, versus 0.9092 for PRISM. Replacing numerical EHR encoding with text serialization reduces AUROC by 3.66 points; removing pre-arrival history reduces it by 1.97 points; and zeroing retrieval reduces it by 0.99 points. Genuine retrieval outperforms random-neighbor retrieval by 0.97 macro-average points, although a paired task-level sign test does not establish a uniform advantage (p=0.067). On the external MC-MED cohort, supervised fine-tuning reaches 0.8611 AUROC, compared with 0.8227 for locally trained gradient boosting and 0.7841 for source-only PRISM. These results support structure-aware multimodal integration and supervised external adaptation.

1 Introduction

Clinical prognosis requires combining heterogeneous evidence, yet existing multimodal systems have structural limitations in representing measurements, longitudinal ECG change, and population context. PRISM combines dedicated pathways for these sources and reports the strongest figures across all four MDS-ED task groups.

1 Introduction

  • Existing systems leave predictive signal unrecovered through numerical serialization, omission of longitudinal ECG change, and lack of retrieval from similar-patient representations.These limitations concern numerical geometry, temporal history, and population context, respectively.Source: §1 · p. 2
  • PRISM combines per-feature numerical EHR tokens, recency-weighted prior-ECG pooling, and dual-modality retrieval of mean-pooled encounter embeddings without retrieved records entering the prompt.The components feed soft tokens to a shared language-model framework.Source: §1 · pp. 2–3
  • PRISM attains the strongest reported figures across 1,443 tasks: 86.24% AUROC for Diagnosis, 93.12% for Deterioration, 91.79% for ICU admission, and 92.82% for Mortality.The benchmark uses raw 12-lead ECG waveforms and 470 structured clinical variables.Source: §1 · p. 3
  • Replacing numerical EHR encoding with full text serialization reduces AUROC by 3.66 points; removing pre-arrival history, current ECG, or longitudinal history reduces it by 1.97, 1.61, or 1.22 points, respectively, while zeroing retrieval reduces it by 0.99 points.The serialization comparison reports its largest differences on diagnosis tasks that depend strongly on continuous-valued laboratory measurements.Source: §1 · p. 3
  • 14.40 AUROC points separate PRISM from a 5.9M-parameter fusion model using the same five input streams and frozen encoders.The comparison concerns complete model configurations under the stated controls.Source: §1 · p. 3
  • On 122 encounters where history pooling is operative, learned recency weighting is +0.67 AUROC points over the best fixed decay, +1.10 over uniform averaging, and up to +2.33 over single-acquisition selection.The aggregator pools prior-visit ECG embeddings under exponential recency weighting.Source: §1 · p. 3

2 Related Work

Prior multimodal clinical language models commonly serialize structured measurements and use only index-visit ECGs, while clinical retrieval often draws on knowledge sources. This section situates PRISM’s numerical encoding, longitudinal waveform aggregation, and patient-population retrieval against those approaches.

2 Related Work

  • Prior multimodal clinical LLMs commonly serialize structured measurements, while FT-Transformer maps each feature to an individual learned token for numerical encoding.PRISM integrates this encoder into the multimodal pipeline for 470 clinical features.Source: §2 · p. 3
  • Figure 1 shows five independently encoded streams projected as soft tokens to a shared decoder; the memory bank passes mean-pooled neighbor embeddings, not outcome labels.The temporal history aggregator pools prior ECG embeddings under exponential recency weighting.Source: Figure 1 · p. 4
  • Existing multimodal LLMs use index-visit ECGs, while longitudinal EHR models capture discrete events over longer periods, leaving short-horizon waveform history unaddressed.PRISM pools prior-visit ECG embeddings with exponential recency weighting.Source: §2 · p. 4
  • Clinical RAG commonly retrieves guidelines, ontologies, or literature, whereas population retrieval supplies encounter embeddings as context for the language model.PRISM indexes learned encounter representations in a dual-modality memory bank.Source: §2 · p. 4

3 Method

PRISM maps clinical encounters to task-specific outcome probabilities through five modality streams and a shared language-model decoder. Its method combines structured EHR encoding, prior-ECG history pooling, population retrieval, and soft-token fusion.

3.1 Problem Formulation and Overview

  • Five encoding streams feed projected soft tokens to a shared language-model decoder to predict probabilities across 1,443 binary outcome tasks.The streams represent current ECG, structured EHR, longitudinal ECG history, retrieved EHR context, and retrieved ECG context.Source: §3.1 · p. 5, §3.6 · p. 7

3.2 Structured EHR Encoding

  • Each clinical measurement becomes an individual learned token, while 96-hour vital-sign histories add seven per-channel statistics to the EHR input.The benchmark feature window ends 90 minutes after ED arrival; supplementary vital-sign and prior-ECG history comes from the 96 hours before arrival.Source: §3.2 · p. 5

3.4 Temporal History Aggregation

  • Prior ECG embeddings receive normalized exponential recency weights, while patients without prior acquisitions receive a separate learned no-history embedding.The learned decay parameter yields a 67.3-hour half-life; the pooling rule applies to the 1.9% of encounters with two or more prior acquisitions.Source: §3.4 · pp. 5–6

3.5 Population Memory Bank

  • Independent EHR and ECG indices retrieve and mean-pool training-cohort embeddings; no neighbor labels, features, identifiers, or text enter the prompt.Evaluation queries are patient-disjoint, so same-patient retrieval is zero at evaluation; rank-0 removal can miss a near-duplicate in 0.0073% of queries.Source: §3.5 · pp. 6–7

3.6 Multimodal Fusion and Language Model Adaptation

  • Five modality streams are each projected to eight soft tokens; four or five prompt placeholders are replaced in place, interleaving 32 or 40 modality tokens and enabling cross-modal reasoning in MedGemma-4B.Stage 2 applies LoRA with rank 16, scaling factor 32, and dropout 0.05 to specified matrices in every self-attention layer.Source: §3.6 · p. 7

3.7 Training and Inference

  • Stage 1 trains encoders and projectors with the language model frozen; Stage 2 freezes those parameters and adapts LoRA, retrieval projectors, and the classification head.At inference, PRISM reports generative Yes-token probabilities and discriminative probabilities from a linear head over mean-pooled decoder states.Source: §3.7 · p. 8
  • Both training stages combine answer-span language-model loss with a weighted binary-classification loss on mean-pooled decoder states.The binary classification loss weight is α=0.1.Source: §3.7 · p. 8

4 Results

PRISM reports the strongest MDS-ED figures across all four task groups, with its full-input results using 96-hour pre-arrival history unavailable to reported baselines. External evaluation and component analyses further distinguish supervised adaptation, matched-input model comparisons, and the contributions of different evidence streams.

4.1 Overall Benchmark Performance

  • 86.24% on Diagnosis, 93.12% on Deterioration, 91.79% on ICU admission, and 92.82% on Mortality are PRISM’s strongest reported task-group figures.The full-input configuration includes 96-hour pre-arrival history; with benchmark-window inputs alone, PRISM reaches 0.8895 AUROC versus 0.8937 for UniPACT and 0.8889 for MDS-ED.Source: §4.1 · pp. 8–9, Table 1 · p. 9

4.2 External Evaluation on MC-MED

  • 0.8611 overall AUROC after supervised fine-tuning on MC-MED exceeds 0.8227 for locally trained gradient boosting and 0.7841 for source-only PRISM.Replacing the source bank alone changes AUROC from 0.7841 to 0.7858; the results support supervised adaptation rather than bank replacement alone resolving institutional shift.Source: §4.2 · p. 10

4.3 Task-Level Analysis

  • 1,145 of 1,443 tasks (79.3%) meet the AUROC confidence-bound criterion for PRISM, versus 883 for UniPACT and 623 for MDS-ED.Gains vary by task: severe hypoxemia reaches 0.8285 versus 0.6980 for MDS-ED, while musculoskeletal and blood and immune diagnoses show the largest chapter-level improvements.Source: §4.3 · p. 11, Figure 2 · p. 11

4.4 Population Memory Bank Analysis

  • The illustrated dual-modality neighbors share the test encounter’s discharge label and positive outcome, but this selected case does not establish cohort-level concordance.The model receives pooled retrieval vectors rather than individual encounters; the shown similarities are measured within each index and are not directly comparable across modalities.Source: §4.4 · p. 12, Figure 3 · p. 12

4.5 Comparison with Compact Non-LLM Models

  • 14.40 AUROC points separate PRISM from the 5.9M-parameter compact control matched on all five input streams.The comparison concerns complete configurations and does not isolate the language model itself; model class, pretrained capacity, parameter count, and optimization also differ.Source: §4.5 · pp. 12, 14, Table 4 · p. 13

4.6 Ablation Study

  • 3.66 AUROC points separate numerical soft-token encoding from full text serialization, while genuine retrieval reaches 0.9092 versus 0.8993 with zeroed retrieval and 0.8995 with random neighbors.The serialization comparison also differs in sequence length, batch size, and optimization conditions, so it is a practical comparison rather than a pure causal isolation of numerical representation.Source: §4.6 · pp. 14–15

4.7 Control Summary

  • Random-neighbor retrieval beats zeroed retrieval by only 0.02 points, below the 0.32-point full-model seed standard deviation; sensitivity results are descriptive, not causal evidence.The learned decay rule and k=3 have the highest point estimates among their tested alternatives, but the pooling comparison affects only a small subgroup and k was selected on held-out performance.Source: §4.7 · p. 16

4.8 Additional Metrics and Subgroup Analysis

  • AUROC declines from 0.9277 in patients under 40 to 0.8684 in those aged 85 and over, while most pairwise subgroup interval comparisons overlap.Calibration under temporal or cross-site distribution shift was not evaluated; the subgroup analyses are descriptive rather than evidence of demographic equivalence or disparity.Source: §4.8 · pp. 16–17

5 Discussion and Conclusion

The discussion weighs PRISM’s performance against the evidence available and how it is integrated, and recommends evaluating these design choices separately. It also identifies limits to deployment and interpretation of retrieved evidence.

5 Discussion and Conclusion

  • Matched-window inputs bring PRISM close to prior multimodal systems, while matched-input compact-model comparisons favor PRISM; on MC-MED, supervised fine-tuning contributes more than memory-bank replacement alone.The discussion frames observation window, representation, and local adaptation as distinct design choices.Source: §5 · p. 17
  • 0.97 AUROC points separate genuine from random retrieval on MDS-ED, but contrasts vary across cohorts and configurations, supporting evaluation against random and zeroed controls in each intended setting.The other reported gaps are 0.05 points in the temporal evaluation and 3.31 points after MC-MED fine-tuning.Source: §5 · p. 17
  • PRISM retrieves learned training-encounter representations and supplies only mean-pooled embeddings to the language model, while keeping individual records out of the prompt.The underlying neighbors remain enumerable offline for retrospective provenance; how to present that provenance to end users remains open.Source: §5 · p. 17
  • The evaluation is retrospective, covers two institutions, and uses one language-model backbone; broader multi-site and prospective evaluation is needed to characterize robustness to clinical and population differences.The strongest external result uses supervised local adaptation, and temporal baseline comparisons are limited to tested compact fusion architectures.Source: §5 · p. 17

Broader Impact Statement

  • Displaying retrieved cases would expose patient attributes and outcomes and could raise re-identification risk; similarity in learned embeddings does not establish shared causation.The authors consider clinical deployment premature pending subgroup characterization, calibration under distribution shift, and prospective evaluation of how retrieved evidence is framed for users.Source: Broader Impact Statement · pp. 17–18

A Training Hyperparameters

PRISM is trained in two stages, with distinct trainable parameters in each stage.

A Training Hyperparameters

  • Stage 1 jointly trains the EHR FT-Transformer, temporal history aggregator, and three primary soft-token projectors while MedGemma-4B and D-BETA remain frozen.The trainable Stage 1 parameters include the aggregator’s learnable decay scalar λ.Source: A Training Hyperparameters · p. 21
  • Stage 2 freezes all Stage 1 parameters and trains LoRA adapters, two retrieval projectors initialized from scratch, and the classification head loaded from the Stage 1 checkpoint.Source: A Training Hyperparameters · p. 21

B Soft-Token Projector Specifications

PRISM maps modality embeddings into language-model soft tokens through shared projectors and builds its population memory bank from separate EHR and ECG indices.

B Soft-Token Projector Specifications

  • Each modality embedding becomes eight soft tokens through a two-layer projector with layer normalization, GELU activation, and final linear expansion.All five projectors share this architecture, differing only in input dimensionality and training stage.Source: B Soft-Token Projector Specifications · p. 21, Table 9 · p. 22
  • The EHR and ECG retrieval projectors are initialized from scratch and trained during Stage 2 alongside the LoRA adapters.The three primary projectors for current ECG, EHR, and ECG history are trained during Stage 1.Source: B Soft-Token Projector Specifications · p. 21

C Population Memory Bank Configuration

  • The memory bank uses two independent nearest-neighbor indices over 116,186 training-fold encounters, with cosine similarity on ℓ2-normalized EHR and ECG embeddings.Assignments are precomputed as row-aligned lookup tables; top-k embeddings from each index are independently mean-pooled into context vectors before projection.Source: C Population Memory Bank Configuration · p. 21, Table 8 · p. 22

D Per-Task AUROC: Deterioration, ICU, and Mortality

PRISM maintains high AUROC across non-diagnosis tasks and diagnosis chapters, with performance varying by task and clinical domain.

E Diagnosis AUROC by ICD-10 Chapter

  • Mean diagnosis AUROC exceeds 0.87 in 11 of 23 ICD-10 chapters and 0.80 in every chapter; obstetric, infectious, and circulatory chapters score 0.98, 0.92, and 0.89, respectively.Performance is comparatively lower in chapters dominated by symptom-based or musculoskeletal diagnoses.Source: E Diagnosis AUROC by ICD-10 Chapter · pp. 22–23

D Per-Task AUROC: Deterioration, ICU, and Mortality

  • AUROC exceeds 0.95 on five non-diagnosis tasks and 0.90 on 11 of 15; performance decreases as the mortality horizon extends.The reported trend is consistent with a greater contribution from chronic disease burden at longer horizons relative to the acute ED-window state.Source: D Per-Task AUROC: Deterioration, ICU,… · p. 22, Table 11 · p. 23

F AUROC Distribution Across Diagnosis Tasks

Most diagnosis tasks exceed AUROC 0.80, while a small lower-performing tail consists of symptom-based or psychiatric diagnoses.

F AUROC Distribution Across Diagnosis Tasks

G Longitudinal ECG History: Availability and Aggregation

The learned recency-weighted rule is evaluated against alternative ways to pool prior ECGs, but it can affect inputs for only a small subset of encounters.

G Longitudinal ECG History: Availability and Aggregation

  • Only 122 test encounters (1.9%) have two or more prior ECG acquisitions, so only these encounters receive different inputs under alternative aggregation rules.With no prior acquisition, a no-history embedding is used; with exactly one, the normalized weight is 1 regardless of λ.Source: G Longitudinal ECG History:… · p. 25, Table 14 · p. 25
  • The derived τ/|λ| quantity should not be interpreted as a timescale recovered from data because |λ| remains within 1.1% of its initialization.Table 15 reports the converged decay parameter and derived quantities.Source: G Longitudinal ECG History:… · p. 25, Table 15 · p. 25
  • The full-fold range is only 0.09 AUROC points, while the 2.33-point effect is confined to 1.9% of encounters and is below the 0.32-point seed standard deviation.Table 16 compares learned, fixed-decay, and uniform rules across the full test fold and the 122 encounters with at least two prior acquisitions.Source: G Longitudinal ECG History:… · p. 25

H Cohort and Dataset Details

PRISM is evaluated on MDS-ED using standardized patient-level partitions and a retained test set shaped by ECG availability and the benchmark sampling protocol.

H Cohort and Dataset Details

  • The benchmark requires each encounter to have a raw 12-lead ECG, at least one structured laboratory measurement, and a complete ICD-10-CM discharge record; its D=470 feature vector uses data from a 90-minute window beginning at ED arrival.PRISM additionally consumes a retrospective 96-hour window ending at ED arrival, giving it history unavailable to the MDS-ED and UniPACT baselines; this window precedes the prediction cutoff.Source: H Cohort and Dataset Details · p. 26
  • The retained evaluation contains 262,158 encounter–task records after excluding 83 encounters with unavailable ECGs, and is not necessarily exactly balanced within each task.The test-set builder first sampled an equal number of negatives and positives per task; records from encounters without usable waveforms were then removed.Source: H Cohort and Dataset Details · p. 26
  • Aggregate AUROC uses a macro-of-macro average, giving each of the four task groups equal weight after averaging tasks within each group.This avoids allowing the 1,428 diagnosis tasks to dominate the fifteen deterioration, ICU, and mortality endpoints.Source: H Cohort and Dataset Details · p. 26
  • The patient-disjoint folds prevent validation or test queries from retrieving encounters from the same patient, although this guarantee does not extend within the training fold.The population memory bank is built only from training-fold encounters.Source: H Cohort and Dataset Details · p. 26

I Performance by Patient Subgroup

The section reports subgroup performance and details how benchmark records are transformed into PRISM prompts, retrieved context, and phenotype examples.

I Performance by Patient Subgroup

  • Twenty of 21 pairwise subgroup interval comparisons overlap; the exception is the 40–54 versus 85+ age bands, and no sex or ethnicity comparison separates.Interval overlap is descriptive, not a formal test of equal performance; subgroup AUROC is associated with event rate, but this does not establish causation.Source: I Performance by Patient Subgroup · pp. 27–28

J Clinical Prompt Design and Representative Examples

  • PRISM prompts use four modality tokens when no prior ECG exists and five when an aggregated longitudinal ECG representation is available.A representative positive 24-hour mortality case has no prior ECG history in the 96-hour window.Source: J Clinical Prompt Design and… · p. 28, §J.1 · p. 28

K Training Data Construction and Prompt Generation

K.2 Population Statistics and Z-Score Normalization

  • EHR features are flagged as critical when outside clinical reference intervals or when their population z-score magnitude exceeds 2.0.Severity is further classified as elevated or low within 30% of the nearest bound, and critically elevated or low beyond 30%.Source: §K.2 · p. 29

K.3 Language Model Prompt

  • The prompt combines modality tokens, a role and task description, and a patient summary containing demographics, vital-sign history, EHR abnormalities, and prior ECG timing.The modality placeholders are replaced by soft-token matrices at forward time.Source: §K.3 · pp. 30–31

K.4 Memory Bank Lookup Integration

  • After Stage 1, encounter embeddings populate separate EHR and ECG nearest-neighbor indices, and precomputed neighbor rows are used to reconstruct retrieved embeddings during Stage 2.Neighbor lookup is performed before Stage 2 training and evaluation.Source: §K.4 · p. 31

K.5 Train, Validation, and Test Handling

  • Validation and test queries retrieve only from the training-fold index, keeping held-out outcome information out of the retrieved population context.Source: §K.5 · p. 31

L Population Memory Bank: Retrieval Across Clinical Phenotypes

  • Four selected test encounters illustrate different local retrieval neighborhoods; their neighbor prevalences describe index geometry, not cohort-level outcome concordance or the prediction mechanism.Each case uses 50 nearest EHR neighbors, and the examples are selected to span distinct phenotypes.Source: L Population Memory Bank: Retrieval… · p. 31

M External and Temporal Evaluation Protocols

The external and temporal evaluations define separate cohorts, eligible task sets, and prediction procedures. They use patient-level bootstrap intervals, with single-run PRISM intervals not capturing training-seed variability.

M External and Temporal Evaluation Protocols

  • 40,121 MC-MED visits from 30,642 patients form the external cohort; its 35,818-visit local memory bank contains no test patients.Training and validation patients are disjoint from the test partition.Source: M External and Temporal Evaluation… · p. 31
  • The MC-MED input repeats its available lead-II waveform across 12 ECG channels, and structured-data mapping covers 43 of 45 laboratory analytes.Fine-tuning uses 49,120 stratified prompt records for two epochs.Source: M External and Temporal Evaluation… · pp. 31–32
  • Figure 4 reports outcome prevalence among each test encounter’s 50 nearest EHR-index neighbors, with the index patient’s ground-truth label marking clinical status.MC-MED retrieval controls perturb retrieval at inference without retraining, with evaluation records held fixed; their intervals are reported separately rather than for paired differences.Source: Figure 4 · p. 32, M External and Temporal Evaluation… · p. 33
  • The temporal evaluation includes 841 eligible tasks and the MC-MED evaluation 306; their group averages cover different task sets and are not direct estimates of performance change.Eligibility requires at least five positive and five negative evaluation records.Source: M External and Temporal Evaluation… · pp. 32–33
  • The MC-MED evaluation contains 306 tasks spanning 297 diagnosis tasks, treatment and severity outcomes, ICU admission, and mortality horizons through 28 days.The temporal evaluation’s outcome groups include 831 diagnosis tasks and mortality horizons through 365 days.Source: M External and Temporal Evaluation… · pp. 32–33
  • PRISM uses softmax over Yes/No answer-token logits for its generative pathway, while the discriminative pathway applies a classification head to the mean-pooled prompt representation.Overall AUROC is the unweighted mean of four task-group means; per-task AUROC uses average-rank tie handling.Source: M External and Temporal Evaluation… · p. 33
  • 95% percentile confidence intervals use 1,000 patient-level cluster-bootstrap resamples, but single-run PRISM estimates do not capture training-seed variability.Compared runs share resamples, enabling paired contrasts.Source: M External and Temporal Evaluation… · p. 33

N Temporal Generalization

PRISM is evaluated on later-era encounters under a temporal split and compared with compact fusion baselines on identical evaluation records. It achieves the highest overall point estimate, while retrieval-control comparisons show setting-dependent effects.

N Temporal Generalization

  • In the temporal split, training uses encounters certainly assigned to 2016 or earlier and testing uses encounters certainly assigned to 2017 or later; no test patient appears in training, validation, or bank sets.The memory bank contains the union of training and validation, totaling 71,086 encounters from 45,580 patients.Source: N Temporal Generalization · pp. 33–34
  • 0.8587 overall AUROC for PRISM exceeds 0.7669 for the compact fusion MLP and 0.7541 for fusion with retrieval on the temporal evaluation.PRISM also exceeds both compact fusion baselines in each task group; these comparisons assess complete configurations, not the language-model backbone alone.Source: Table 23 · p. 34, N Temporal Generalization · p. 34
  • The later-era evaluation retains sampled test records without resampling, so tasks are not exactly balanced; 841 tasks meet the primary eligibility threshold.A broader sensitivity criterion yields defined AUROC for 1,270 tasks.Source: N Temporal Generalization · p. 34
  • Zeroing retrieval lowers PRISM’s overall AUROC by 0.58 points, while genuine retrieval exceeds random neighbors by 0.05 points in this temporal setting.The reported point-estimate rankings do not establish statistical significance.Source: N Temporal Generalization · p. 34, Table 23 · p. 34
Loading 2610.13510v1…