PaperlayerView exact revision on arXiv

Source-linked AI summary

Prediction Is Not Detection: Evaluating Pre-Recognition Claims in Longitudinal Clinical AI

Jing Yang, Long R. Jiao, Xiujun Cai, Zongjiu Zhang

arXiv:2609.25852v1cs.AI

TL;DR

Event-based evaluations can mistake recognition-mediated care-process signals and recognition-dependent endpoints for evidence of detection before clinical recognition. The paper defines an interval-censored transition and proposes independent as-of adjudication plus a declared recognition proxy; it shows that better event prediction and event-relative lead time do not by themselves establish pre-recognition detection.

Abstract

from arXiv · show

Clinically useful early detection requires validated pre-recognition lead time. Yet event-based evaluations of longitudinal clinical AI can treat recognition-mediated care-process signals as shortcuts and recognition-dependent endpoints as reference standards, inflating apparent performance and lead time while undermining cross-center transport. Such results may serve prognosis without establishing detection before recognition. We define an interval-censored pre-recognition transition, an independent as-of reference standard, and a prespecified recognition proxy to make the claim testable.

Prediction is not detection

Recorded-event prediction can show apparent lead time without showing that a model detected a transition before clinical recognition. Prognosis and pre-recognition detection are distinct claims and require distinct reference standards.

Prediction is not detection

  • Across 77,582 hospitalizations, the Epic Sepsis Model had AUROC 0.62 when all pre-criteria predictions counted, but AUROC 0.47 after predictions following the first treatment-plan indicator were excluded.The excluded indicators were antibiotics, intravenous fluids, a blood culture or a lactate measurement.Source: Prediction is not detection · p. 1
  • Care-process signals can precede a recorded endpoint while following clinical recognition, so counting that interval as lead time misattributes post-recognition time.Such signals may carry genuine clinical information and legitimately improve event prediction.Source: Prediction is not detection · p. 1
  • Prediction of a downstream outcome does not establish detection of an earlier patient-state transition; the two claims need distinct reference standards.The evaluation problem is not predicting recorded outcomes, but treating that prediction as evidence of earlier detection.Source: Prediction is not detection · pp. 1–2

Why the distinction disappears in the record

The EHR records patient-state evolution and clinical response through the same process, making event time a mixture of physiology, recognition, workflow and documentation. A pre-recognition evaluation must separate these clocks and address both feature-side and target-side validity.

Why the distinction disappears in the record

  • The recorded endpoint combines physiological change, clinical suspicion, recognition-triggered workflow and documentation, while care-process features may signal recognition before that endpoint.Consequently, apparent lead time can reflect either an emerging patient-state transition or the healthcare system’s response to it.Source: Why the distinction disappears in the… · p. 2
  • The proposed evidence sequence uses blinded as-of adjudication, model scoring against the adjudicated transition, and temporal evaluation against each declared recognition proxy.The tests assess reference-target validity, model-target validity, and temporal validity, respectively.Source: Why the distinction disappears in the… · pp. 3–4
  • Recognition-mediated features can be available at query time yet fail to establish pre-recognition detection, while a recognition-dependent endpoint is not an independent reference standard.These are distinct from temporal leakage, in which unavailable information enters the model; endpoint changes after treatment create a separate causal problem.Source: Why the distinction disappears in the… · p. 4
  • Care-process-driven models may become most confident after clinicians begin investigating deterioration, when they contribute least to decision support, and varying institutional practices threaten transportability.In major surgery, outcomes after complications depend substantially on timely recognition and rescue.Source: Why the distinction disappears in the… · pp. 4–5

The pre-recognition transition

The pre-recognition transition defines a clinically meaningful departure from a prespecified reference course independently of later recognition and outcome. It then compares an interval-censored onset of prospective recognizability with a declared observable recognition proxy.

The pre-recognition transition

  • A prospectively recognizable transition is a clinically meaningful, reproducibly adjudicable departure from a prospectively specified reference course, identifiable from as-of evidence.The target is specified without reference to recognition so recognition can be compared with it rather than built into it.Source: The pre-recognition transition · p. 5
  • Recognition is operationalized through a prespecified observable proxy, and conclusions should be tested against alternative proxies.The declared proxy sets the recognition time used to evaluate the transition’s timing.Source: The pre-recognition transition · pp. 5, 7, 8
  • The reference course is specified independently of subsequent recognition and outcome, and adjudicators judge departure from that baseline without using the future outcome.The reference course may be normative, historical, model-based or protocol-based, with its uncertainty represented.Source: The pre-recognition transition · p. 6
  • Only information visible by query time may be used by the model and reference standard, so result visibility lags and measurement kinetics bound how early detection is possible.For example, creatinine can lag a fall in filtration by hours to days.Source: The pre-recognition transition · p. 6
  • On a prespecified grid, blinded adjudicators assess as-of evidence; the first qualifying and last negative times bound onset, leaving timing uncertain if recognition falls within that interval.The onset is interval-censored, and the empirical ordering is established beyond doubt only when the interval’s upper bound precedes the proxy time.Source: The pre-recognition transition · pp. 6–7
  • The detector estimates whether prospective recognizability has already emerged from the supplied information, rather than whether an event will later be recorded.The target is the departure itself, not a latent biological onset; different detector formulations may estimate the same adjudicated target.Source: The pre-recognition transition · pp. 5, 7
  • A flag-to-proxy interval is validated pre-recognition lead time only when the flag’s position relative to the interval-censored transition is demonstrated, not assumed.Recognition latency is interval-valued, and no lead time is attributed across the interval of uncertain recognizability.Source: The pre-recognition transition · p. 7

Why this is not early warning

Early-warning systems generally predict downstream events earlier, whereas pre-recognition detection evaluates an independently defined transition before clinical recognition. A long alert-to-event interval alone cannot establish the latter.

Why this is not early warning

  • A long alert-to-event interval does not establish pre-recognition detection when the recorded event lies downstream of recognition.Early-warning scores generally retain a downstream event as the target and ask whether it can be predicted earlier.Source: Why this is not early warning · p. 8
  • Change-point, onset-estimation, informative-observation and latent-state methods may provide useful detectors or temporal representations, but none supplies the required independent as-of reference standard by itself.The key methodological question is whether the reference target is independent of clinical response.Source: Why this is not early warning · p. 8

What would count as evidence

A pre-recognition claim requires three ordered validity checks: independent reference-target construction, model evaluation against that target, and robust temporal ordering. The proposed criteria distinguish anticipatory prediction from validated lead time and define practical boundaries for evaluation.

What would count as evidence

  • Three sequential tests assess independent reference-target validity, model–target validity, and temporal ordering under alternative recognition proxies.Failure at any stage voids the subsequent claim.Source: What would count as evidence · p. 9
  • Model–target validity evaluates discrimination and calibration against adjudicated recognizability, alongside false-transition burden per patient-time.Self-limiting transitions count as transitions, and burden is measured at the operating point used.Source: What would count as evidence · p. 10
  • Temporal conclusions should be repeated under at least one earlier and one later recognition proxy.The declared proxy determines the recognition time used in the analysis.Source: What would count as evidence · pp. 10–11
  • Flags at or before the lower interval bound are anticipatory, flags within the interval are timing-uncertain, and only flags after its upper bound but before the proxy yield conservative validated lead time.The flag-to-proxy interval and the proportion flagged before the proxy are also reported.Source: What would count as evidence · p. 10
  • Evaluation presupposes that the model did not influence care, requiring a design such as a silent-mode period after deployment.Once deployed, recognition depends on the model flag.Source: What would count as evidence · p. 10
  • Because adjudication is costly, it may be concentrated in a prespecified enriched sample if sampling fractions are retained and prevalence-dependent estimates are weighted accordingly.The weighted estimates include calibration and absolute burden.Source: What would count as evidence · p. 10
  • The reference course, query grid, recognizability criterion, and aggregation rule should be prespecified, with uncertainty represented and the transition interval-censored.The checklist also calls for reporting raw recognizability judgments and adjudicator agreement.Source: What would count as evidence · p. 10
  • Records should be frozen as of each query time, distinguishing acquisition, result, and visibility times and reporting visibility lag.The checklist specifies using what was visible at each query time rather than reconstructing records from final values.Source: What would count as evidence · pp. 10–11

adjudication (Test 1)

The checklist specifies how adjudication and model evaluation should be conducted, then separates temporal ordering from care-influenced evaluation and optional downstream use.

adjudication (Test 1)

  • Adjudication criteria, aggregation rules, and persistence requirements should be prespecified, with inter-adjudicator agreement reported.Where feasible, no adjudicator reviews more than one truncation per patient; non-monotone judgments are reported rather than forced into an interval.Source: adjudication (Test 1) · p. 11
  • Test 2 assesses discrimination and calibration against adjudicated recognizability and reports false-transition burden per patient-time at the operating point used.Self-limiting transitions count as transitions, and enriched-sample estimates retain sampling fractions and weights.Source: adjudication (Test 1) · p. 11
  • Test 3 declares a recognition proxy, reports the proportion flagged before it and the flag-to-proxy interval, and repeats analysis under earlier and later alternatives.The checklist distinguishes anticipatory flags, timing-uncertain flags, and the conservative unambiguous case for validated lead time.Source: adjudication (Test 1) · p. 11
  • Evaluation should state that the model did not influence care or describe the design used to preserve a comparison without its flag.This is an explicit checklist item for temporal evaluation.Source: adjudication (Test 1) · p. 11
  • If a system acts on its signal, report its gating rule, the response when evidence is insufficient, and unnecessary-review burden alongside benefit.These are optional downstream-use reporting items.Source: adjudication (Test 1) · p. 11

Downstream implications

A valid reference target establishes when a signal may be called pre-recognition, but does not prescribe the clinical response. Actions should reflect evidence strength, and downstream decision policies remain separate research questions.

Downstream implications

  • Output should be proportionate to evidence, ranging from silent monitoring under weak evidence to escalation when confidence is high.When evidence is insufficient, observation may be more useful than treatment.Source: Downstream implications · p. 11
  • A credible departure may prompt structured re-observation, with unnecessary-review burden reported alongside benefit.After colorectal resection, a C-reactive protein trajectory may warrant earlier imaging.Source: Downstream implications · p. 11
  • Calibrated transition probability can inform value-of-information and active-acquisition policies, while gating and deployment policies remain downstream research questions.Source: Downstream implications · p. 11

Target validity is orthogonal to model capacity

The proposed transition target is intended to apply across model architectures. Greater capacity cannot correct a mis-specified reference target, so the early-detection claim must be tested independently of model class.

Target validity is orthogonal to model capacity

Conclusion

The conclusion distinguishes detection before clinical recognition from prediction before a recorded event. It frames pre-recognition detection as testable using an interval-censored recognizability estimand, an as-of information boundary, and a declared recognition proxy.

Conclusion

  • Performance against a downstream recorded outcome cannot by itself validate detection of an earlier patient-state transition.The conclusion treats prognosis and pre-recognition detection as distinct claims requiring distinct reference standards.Source: Conclusion · p. 12
  • The pre-recognition transition separates prospective recognizability, clinical recognition, and event recording with an interval-censored estimand and declared proxy.Source: Conclusion · p. 12
  • A declared recognition proxy makes the pre-recognition detection claim falsifiable.Source: Conclusion · p. 12

Supplementary Note

This supplementary note examines how clinical assessments and care-process signals inherited by a model can improve event prediction without establishing detection before recognition. It sets out evidence and conditions for distinguishing inherited predictive value from independently established pre-recognition timing and source attribution, including how recording delay and signal reliability affect apparent performance and lead time.

Formal Arguments for Pre-Recognition Detection

  • An alarm after an earlier clinical assessment but before the disease event can have positive event-relative lead time without adding time before that judgment.Investigations, consultation, and closer monitoring may record an assessment formed before their documentation.Source: Formal Arguments for Pre-Recognition… · p. 15

S1 Existing clinical information can reach the model through recorded actions

  • Recorded actions carry target information beyond remaining inputs when the assessment discriminates the target and the actions express that concern; nonrandom actions alone are insufficient.The relay condition is a local mechanism assumption, and information already contained in other inputs can leave no incremental information in action traces.Source: S1 Existing clinical information can… · pp. 17–18

S2 The model can gain predictive information by inheriting an earlier assessment

  • The expected KL divergence between predictions with and without action traces equals the optimal log-risk reduction, so proper-score gains can reflect inherited clinical judgment.This concerns the declared target and proper scores; it does not imply that every training algorithm or discrimination metric improves.Source: S2 The model can gain predictive… · p. 19
  • A population-risk condition can force a predictor to depend on action traces, but under the relay condition its output adds no target information beyond the existing assessment.Functional dependence on an input does not by itself identify that input as originating in clinical concern.Source: S2 The model can gain predictive… · pp. 19–20
  • An inherited assessment can provide genuine disease information and improve a prediction score, but that gain does not establish that model recognition preceded the concern conveyed by its inputs.For a future endpoint, the relay condition also needs examination when actions change the outcome or carry patient information beyond the modeled inputs.Source: S2 The model can gain predictive… · p. 20

S3 Information about a later event does not by itself identify the current transition

  • The observable law of inputs and later events need not identify current-state probabilities when event–state bridge coefficients are unknown; known unequal coefficients suffice under the stated bridge.Even exact event posteriors cannot distinguish the constructed mechanisms; additional state labels or defensible structural information are needed. Consistency alone also does not turn observed treatment mixtures into untreated or intervention-specific risk, and subsequent treatment can alter the future outcome.Source: S3 Information about a later event… · pp. 20–21

S4 Better event prediction can coexist with an alarm after recognition

  • Perfect event discrimination (AUROC = 1) and arbitrarily long event-relative lead time can coexist with an alarm after the recognition proxy.In the construction, a model reads an action expressing earlier concern; the counterexample does not require temporal or availability leakage, and perfect concern is illustrative rather than a claim about ordinary clinical accuracy.Source: S4 Better event prediction can coexist… · pp. 21–22

S5 Dependence on the inherited signal can also make performance context dependent

  • In an analytic construction, accuracy and Brier risk favor the more reliable clinical signal, while absolute calibration discrepancy is equal and the lower-reliability group’s Brier risk exceeds 0.25.These are analytic values under stated assumptions, not observed clinical effect sizes or an ordering of all metrics for arbitrary models.Source: S5 Dependence on the inherited signal… · p. 24
  • Changing group composition can change environment-level performance while marginal action frequency stays fixed, because the action’s predictive meaning changes.Monitoring action frequency alone would miss the change in the target probability conditional on the action.Source: S5 Dependence on the inherited signal… · p. 24
  • Posterior disagreement on positive target probability yields strictly positive excess risk, possibly infinite for log loss, relative to the target optimum.Clinician composition or how concern is translated into actions can change the posterior; attributing their contribution at an institution requires data on those mechanisms.Source: S5 Dependence on the inherited signal… · p. 25
  • With separate optimization under the stated channel ordering, the higher-information signal has no-higher optimal risk, but this does not order arbitrary frozen predictors or overall clinician accuracy.The log-loss difference is exact under the conditional-independence relation; the construction is a Blackwell randomization.Source: S5 Dependence on the inherited signal… · p. 25
  • Conditional on a valid as-of interval, an alarm strictly before its upper bound and after the lower bound establishes the conservative ordering: transition, alarm, then recognition proxy.An alarm at or before the lower bound is anticipatory, one inside the interval is timing-uncertain, and the strict upper-bound case is sufficient but not necessary; the algebra does not validate the interval or proxy.Source: S5 Dependence on the inherited signal… · pp. 25–26
  • Observed action dependence cannot identify whether concern generated an action or a routine state-triggered process generated the same action, even for a transparent predictor.The two mechanisms produce the same observed-variable distribution, but only one has a concern-mediated pathway; distinguishing them requires source evidence.Source: S5 Dependence on the inherited signal… · p. 26
  • Pre-recognition evaluation requires a contemporaneous state reference, discrimination, calibration, false-positive burden, valid time bounds, and a clinically justified recognition proxy.An isolated alarm in the correct interval does not establish detection performance, and excluding future information alone does not resolve whether concern shaped collected observations.Source: S5 Dependence on the inherited signal… · pp. 25–26
  • Under the stated relay, a real score gain can add no information beyond the assessment it transmits, while positive event lead can coexist with an alarm after recognition.The note therefore treats operational timing and attribution to existing judgment as distinct evidentiary requirements.Source: S5 Dependence on the inherited signal… · p. 26
Loading 2609.25852v1…