PaperlayerView exact revision on arXiv

Source-linked AI summary

Certified AI Triage of ICU Alarms

Mohammed Sameer Syed, Rozhin Yasaei

arXiv:2609.12365v1cs.LG

TL;DR

Most VT alarms are false, but suppressing a genuine alarm can delay recognition of a dangerous arrhythmia, and a threshold alone cannot defer uncertain cases. The paper uses retain/suppress/defer policies with finite-sample risk control. Its results show that the correction’s cost grows with the candidate grid, while adding reliability to the policy grid does not improve the certified frontier.

  • Problem

    A classifier threshold trades sensitivity against specificity but does not identify when evidence is too incomplete or degraded to act on, despite the danger of suppressing a true VT alarm.

    Source: §1 · p. 1

  • Method

    The paper selects retain, suppress, or defer policies from a predeclared grid and applies Learn-then-Test with a Bonferroni-adjusted exact upper bound to control unsafe suppression under i.i.d. event sampling.

    Source: §1 · p. 1, §4.2 · p. 4, §4.3 · p. 4

  • Results

    At α = 0.05, selecting the grid on a separate partition raises certified fold-runs from 1 of 15 to 8, while P-only matches or exceeds RC at every certified budget.

    Source: §1 · p. 2, Descriptive Frontiers · p. 10

  • Takeaways & Limitations

    Because correction charges for every candidate, grid size is a budgeted design choice: a finer grid can certify fewer policies.

    Source: §1 · p. 2

    The exact bound assumes independent suppressed outcomes, but events within waveform records share conditions, and patient-level bounds are unavailable without patient identifiers.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

In the VTaC benchmark 71% of ventricular-tachycardia alarms are false, but silencing a real one can delay recognition of a dangerous arrhythmia. We reframe alarm reduction as three-way triage (retain, suppress, or defer) and bound the decision this analysis treats as harmful: among suppressed alarms, the fraction that were genuine stays below a user-set budget with 95% confidence, under i.i.d. event sampling. Alarms sharing a waveform record are dependent, so the clustered analysis is a sensitivity check. On the official split a 5% budget certifies in all three seeds, suppressing 74.8% of false alarms while silencing 1.5% of genuine ones, at AUROC 0.953 and Challenge Score 83.33, numerically comparable to the strongest of the eleven published systems. Our central finding measures what multiplicity costs: the correction charges for every candidate, so a finer grid can certify strictly less. Under held-out calibration the 885-cell grid we declared certifies 1 of 15 fold-runs, while choosing the grid on a separate selection partition certifies 8. We project the calibration volume each budget needs, making an uncertifiable budget a design parameter. Finally, adding a learned reliability dimension to the policy grid did not sharpen the certified frontier.

1. Introduction

The paper treats unsafe VT-alarm suppression as the endpoint to control in a retain/suppress/defer policy. It presents a finite-sample safety guarantee and examines the costs and limits of certification.

1. Introduction

  • The proposed guarantee bounds harmful suppression under i.i.d. event sampling, with record-clustered bootstrap analysis probing dependence that this assumption ignores.The authors state that no published VTaC system reports a finite-sample bound on this harmful error.Source: §1 · p. 1
  • At α = 0.05, choosing the policy grid on a selection partition raises certified fold-runs from 1 of 15 to 8.The correction charges for each candidate, so a finer grid can certify less.Source: §1 · p. 2
  • On the official split, the policy layer leaves base scores untouched and reports metrics comparable to the strongest published baseline.Source: §1 · p. 2
  • When a budget cannot be certified, the analysis distinguishes apparent sample or model obstacles and projects the labelled alarms needed.Source: §1 · p. 2
  • Gating suppression on learned reliability did not improve the certified frontier on clean or degraded cohorts or across the grid surface.Source: §1 · p. 2

2. Related Work

Prior work addresses false arrhythmia alarms through multimodal analysis and quality-aware waveform methods, while selective prediction and risk-control methods provide related guarantees. This paper situates its full-grid, action-specific setting among those approaches.

2. Related Work

  • The PhysioNet/CinC 2015 challenge established multimodal false-alarm analysis, while VTaC offers a larger, adjudicated benchmark with fixed record-level splits and several baseline types.The paper adopts VTaC’s real-time input and scoring protocol, with deviations specified in Appendix C.Source: §2 · p. 2
  • QualityFM self-distills across paired high- and low-quality signals and reports VT false-alarm detection among transfer tasks, but the paper does not compare encoders.The authors note that a stronger backbone might help model-limited cases, while nearly all failures they report are sample-limited.Source: §2 · p. 2
  • Learn-then-Test treats each candidate as a hypothesis test, and this work applies it to a two-dimensional policy family with correction across the full grid.Related approaches include selective-risk bounds and selective conformal risk control, which bounds prediction-set loss conditional on acceptance rather than one harmful action among acted-on events.Source: §2 · p. 2
  • Modality dropout and fusion designed for missing modalities motivate masking, since 6% of VTaC events have no usable pulsatile channel.Source: §2 · p. 2

3. Data and Evaluation Protocol

The study uses VTaC events with variable modality availability and adjudicated labels, evaluated in development and official-split arms. The protocol keeps records disjoint across roles and describes that separation as record-safe, not patient-safe.

3. Data and Evaluation Protocol

  • VTaC v1.0 contains 5,037 VT alarms from three manufacturers and three US hospitals, with 1,441 true alarms (28.6%) and 3,596 false alarms.Events span 2,260 waveform records, and the release exposes no patient identifier.Source: §3 · pp. 2–3
  • The development arm uses five record-disjoint outer folds and three seeds, while the official arm preserves the published split and tests on 482 events across 226 records.The official test set has 137 true alarms; calibration partitions come from official train, not test.Source: §3 · p. 3
  • The official arm follows a 10-second, four-channel protocol with zero imputation and per-segment z-normalization, scoring with the Challenge Score at the validation-maximizing threshold.The development arm instead uses a 60-second pre-alarm window with two ECG slots and one pulsatile slot.Source: §3 · p. 3
  • Unavailable slots are zero-filled and flagged, but some header-listed channels fail the usable-sample test; all events remain in evaluation.The development cache has 304 events with no usable pulsatile slot, versus 292 classified that way by header alone.Source: §3 · p. 3
  • Although the release contains six minutes of samples, all study windows are measured backward from alarm onset, so no input is affected by the text discrepancy about window timing.Source: §3 · p. 3

4. Method

Sentry encodes available multimodal signals and maps its scores to retain, suppress, or defer policies. Learn-then-Test certifies candidate policies against a risk budget, with full-grid multiplicity, calibration size, and record dependence shaping the analysis.

4.1. Sentry: availability-aware multimodal encoder

  • Sentry shares a trunk across modality slots, masks unavailable tokens, and uses local cross-modal attention before pooling slot outputs into fused evidence.The official-split arm uses stride 8 rather than development stride 40 so local attention does not degenerate to global attention on its shorter window.Source: §4.1 · p. 3
  • The reliability head combines fused features, availability, per-slot quality, and shift agreement; training adds intervention-based reconstruction, identification, and evidence-quality losses without VT labels.The reliability target is an observable consequence of the applied intervention.Source: §4.1 · p. 3

4.2. Triage policy family

  • RC suppresses low-p, high-r events unless retention applies, while P-only fixes τrel = 0; their candidate grids contain 885 and 59 policies, respectively.The retain threshold is selected afterward to retain at least 95% of true alarms on the policy-selection partition.Source: §4.2 · pp. 3–4

4.3. Certification by Learn-then-Test

  • At 95% confidence, Learn-then-Test admits a candidate only when its Bonferroni-adjusted exact Clopper–Pearson upper bound on suppressed true alarms is at most α.Among admitted candidates, it selects the one with the most false-alarm suppressions; if none is admitted, the run contributes a suppress-nothing policy.Source: §4.3 · p. 4
  • Fixed-sequence testing lacks power here because the Clopper–Pearson acceptance region depends on suppression-set size as well as unsafe counts, so it is not a prefix of thresholds.An ascending walk halts at its first step in 15 of 18 runs at α = 0.05 because the smallest threshold suppresses zero events and yields an upper bound of 1.0.Source: §4.3 · pp. 4–5
  • The primary calibration mode pools selection and certification partitions; a held-out variant uses disjoint partitions but halves calibration data and is stricter than required.The pooled samples contain 1,555 development events and 1,661 official-arm events.Source: §4.3 · p. 5
  • Suppressed outcomes are not exactly independent on VTaC because events from a record share conditions, so the record-clustered bootstrap is a sensitivity analysis, not a second certificate.The bootstrap is approximate, cannot resolve the corrected tail with 2,000 resamples, and no patient-level bound is computable without patient identifiers.Source: §4.3 · p. 5
  • A budget is sample-limited when point-estimate risk meets it but the bound does not, and model-limited when observed risk exceeds it.The analysis estimates the smallest calibration size admitting a grid cell in every run from frozen evaluation predictions.Source: §4.3 · p. 5

5. Results

The results report official-split prediction performance and evaluate certified alarm suppression, including how policy-grid size and subgroup guarantees constrain what can be certified.

5.1. Base prediction

  • AUROC 0.953, AUPRC 0.875, Score 83.33, and F1 0.799 on the official split; FCN lies inside clustered intervals for AUROC, Score, and F1.The supported comparison is parity, not superiority; no per-event predictions permit a paired test.Source: §5.1 · p. 5, Table 1 · p. 5

5.2. Certified triage

  • On the 482-event official test set, P-only suppresses 265, 259, and 256 alarms across seeds, including 2, 3, and 1 genuine VT alarms; held-out certification succeeds for 2 of 3 seeds.At α = 0.05 in development, 105 of 4,323 true alarms are silenced; at α = 0.10, 690 are silenced.Source: §5.2 · p. 6

5.3. Reliability gating does not help

  • At every certified budget, P-only matches or exceeds RC in folds certified and false alarms suppressed; the reliability head detects availability differences but does not improve suppression.Baseline wander lowers r by 0.317 while moving p by +0.003.Source: §5.3 · p. 6

5.4. What Learn-then-Test buys

  • At α ≥ 0.05, Learn-then-Test alone keeps realised risk inside the advertised budget in every admitted run; Holm changes no verdict across 432 run-cells.Selecting by observed calibration risk exceeds the budget on held-out data in two thirds of runs at α = 0.05.Source: §5.4 · pp. 6–7

5.5. The grid is an expensive hyper-parameter

  • At α = 0.05, the held-out 885-cell grid certifies 1 of 15 runs versus 7 of 15 for 30 cells; matched-size one-dimensional grids dominate two-dimensional grids, replicated officially.No grid on the development ladder certifies α ≤ 0.02.Source: §5.5 · p. 7

5.5.1. Choosing the grid on the selection partition

  • Choosing the grid on a separate selection partition certifies 8 of 15 development runs versus 1 of 15 for the fixed 885-cell grid, while silencing 1.4% of genuine alarms.For 885 candidates, projections estimate about 800 calibration events for α = 0.10 and 4,000 for α = 0.05, compared with 1,555 available; these are not future-sample guarantees.Source: §5.5.1 · pp. 7–8

5.6. One subgroup pays for the marginal guarantee

  • Among suppressed alarms, the 6% without a usable pulsatile channel had 2.6 times the modality-complete risk at α = 0.05 (16/213 versus 180/6,133); the marginal guarantee does not cover subgroups.The reliability head detects this stratum without improving the frontier.Source: §5.6 · p. 8

6. Discussion

The discussion treats the reliability result as unresolved and emphasizes limits on the evidence and guarantee: evaluation is retrospective, and the bound is marginal and assumption-dependent.

6. Discussion

  • The reliability head did not improve the frontier; possible explanations include rare detected conditions, learning interventions rather than naturally occurring errors, or grid multiplicity.Distinguishing these explanations requires a naturally shifted cohort or a reliability target tied to prediction error.Source: §6 · p. 8
  • The evaluation uses no external cohort, has only three official-split seeds, and is record-disjoint rather than patient-disjoint.The release excludes alarms with uncertain or rejected labels.Source: §6 · p. 8
  • The bound is marginal and assumes i.i.d. events; the clustered bootstrap is approximate, and nothing here is validated for bedside use.The retain threshold carries no guarantee, and defer denotes need for review rather than an alert.Source: §6 · p. 8

7. Conclusion

Three-way triage bounds harmful suppression while exposing its cost in labelled alarms and policy-grid size. On VTaC, P-only matches or exceeds RC at every certified budget, while the uncorrected frontier shows how much the selection rule can change suppression.

7. Conclusion

  • At a 5% budget, the bound holds in every official seed while silencing 1.5% of genuine alarms; reliability gating does not sharpen the frontier.The correction charges for every policy considered, so a grid fine enough to contain a useful policy can be too fine to certify any.Source: §7 · p. 8

Descriptive Frontiers

  • P-only matches or exceeds RC at every certified budget on the certified frontier.The figure compares false-alarm suppression with realised risk on the evaluation partition.Source: Descriptive Frontiers · p. 10
  • At α = 0.05, uncorrected selection reaches 82.7% suppression against 37.7–57.3% for corrected policies, placing the difference in the selection rule.The uncorrected frontier selects on the calibration sample's observed risk rather than a corrected bound.Source: Descriptive Frontiers · p. 10

Appendix B. Architecture and Training Details

The appendices document the model, split protocols, benchmark comparisons, and reliability-head experiments. The official-split model is at parity with the strongest published baseline, and the implementation and evaluation details specify the scope of that comparison.

Appendix B. Architecture and Training Details

  • The trunk uses a shared strided stem and eight depthwise-separable residual blocks, then pools tokens before slot embeddings, adapters, and position projections.The token counts are 375 for the 60-second development window and 312 for the 10-second official window.Source: Appendix B · pp. 10–11
  • Training uses modality dropout and three auxiliary losses, with loss weights 0.25, 0.15, and 0.50; the development and official models have 14.83 M and 14.89 M parameters.Optimization uses AdamW, effective batch 32, gradient clipping at 1.0, and early stopping on checkpoint-selection AUPRC.Source: Appendix B · p. 11

Appendix C. Partition and Protocol Details

  • The benchmark intervals use BCa bootstrap with record jackknife acceleration and 20,000 record resamples; percentile intervals leave the reported comparisons unchanged.No endpoint shifts by more than 0.006 on AUROC or 0.7 on Challenge Score under the alternative interval variant.Source: Appendix C · p. 12
  • The development protocol uses record-disjoint training, checkpoint-selection, policy-selection, certification, and evaluation partitions, with 893, 181, 359, 368, and 459 records in fold 0.The official arm uses a record-level partition and a record-disjoint calibration cut from official training data.Source: Appendix C · p. 12
  • The official test split has sensitivity 0.971 against specificity 0.817 at the validation-selected threshold, with mean threshold 0.115 across seeds.The text attributes this operating point to the Challenge Score's five-fold penalty on missed true alarms.Source: Appendix C · p. 12
  • The benchmark scoring implementation reproduces the published rule-based row at Challenge Score 67.32 and F1 0.655.AUROC and AUPRC agree with scikit-learn to 10^-9 on 400 randomized trials.Source: Appendix C · p. 12
  • Training ran on one NVIDIA A100 (40 GB), and best checkpoints were selected between epochs 7 and 31 in development and 4 and 14 in the official arm.The architecture also runs on Apple-silicon MPS for development.Source: Appendix C · p. 12

Appendix D. Benchmark Comparison in Full

  • FCN lies inside the model's 95% record-clustered interval on AUROC, Score, and F1, supporting parity rather than superiority on the official split.Only CNN+CL among the other ten systems has a Score inside that interval; the other nine fall below it.Source: Appendix D · p. 12

Appendix F. Grid Surface

The grid surface shows that certification does not improve monotonically with candidate count, and that a two-dimensional reliability grid does not outperform a comparable one-dimensional grid. The reported ladder omits two configurations, so its properties are limited to the measured grids.

Appendix F. Grid Surface

  • The reliability evidence is available for 4,000 of 15,111 real predictions, and per-run correlation is unstable rather than consistently absent.The intervention panel reports mean changes in reliability across affected events, with standard deviations across 15 runs.Source: Appendix F · p. 13
  • For adaptive selection at α = 0.10, operational false-alarm suppression is 56.3%, versus 84.4% among admitted runs and 86.9% among selected policies regardless of certification.Operational averages count uncertified runs as zero; 10 of 15 runs certify.Source: Appendix F · p. 14
  • Across the 26 measured configurations, certification counts are non-monotone in grid size, and no grid certifies α ≤ 0.02 on the 767-event selection partition.Two declared configurations, 5 × 3 and 20 × 3, are absent from the sweep.Source: Appendix F · pp. 12, 14
  • The ladder crosses probability-grid sizes np ∈ {3, 5, 8, 12, 20, 30, 59} with reliability levels nr ∈ {1, 3, 5, 15}.The declared ladder has 28 configurations, but the measured surface contains 26 because two entries were overwritten by candidate-count keys.Source: Appendix F · pp. 12, 14
  • On the official arm at α = 0.05, certification peaks at 12 candidates and every two-dimensional grid is matched or beaten by a comparable one-dimensional grid.The pattern recurs on a different model and disjoint sample.Source: Appendix F · p. 14

Appendix G. Does the Grid Rule Read Held-Out Data?

The held-out grid rule chooses a grid using the selection partition, then tests whether that choice changes when labels in selection, certification, or evaluation are permuted. Calibration-volume projections state the sample size needed to admit a grid cell across fold-runs.

Appendix G. Does the Grid Rule Read Held-Out Data?

  • The held-out rule selects grids using the selection partition alone; choosing the grid on the same pooled sample as certification would require correcting over the 944-cell union.That pooled procedure is not reported as a result because its smaller correction would be invalid.Source: Appendix G · p. 14
  • Permuting evaluation or certification labels leaves every chosen grid unchanged, while permuting selection labels changes choices, showing the rule responds to the data it reads.Seven of 15 chosen grids change under each of five selection-label permutations.Source: Appendix G · pp. 14–15

Appendix H. Calibration Volume Required

  • Table 8 reports the smallest calibration sample at which a grid cell is admitted at the corrected level in every fold-run, including budgets marked unreachable at tested sizes or never.The projections make the labelled-data volume needed for a budget explicit.Source: Appendix H · p. 15, Table 8 · p. 15

Appendix I. Step-Down Correction and the Confidence Level

Holm widens the admissible set without changing feasibility verdicts or materially improving suppression, while lowering confidence modestly helps marginal certification. Comparisons across selection procedures and modality strata qualify how those findings should be read.

Appendix I. Step-Down Correction and the Confidence Level

  • Across 432 run-cells, Holm changes no feasibility verdict and improves mean false-alarm suppression by at most 0.87 percentage points.It widens the admissible set in 108 cells and changes the selected policy in 51, but never reaches a materially better policy.Source: Appendix I · pp. 15–16
  • Lowering family-wise confidence from 95% to 90% adds one to two fold-runs in five of 24 arm-by-budget cells, without changing grid rankings.At α=0.05, the held-out development arm gains 2 of 15 runs for RC and 1 of 15 for P-only; the official arm gains its third seed.Source: Appendix I · p. 16

Appendix J. Selection Procedures at Every Budget

  • At α=0.05, selection procedures order by suppression as naive threshold highest and Learn-then-Test lowest, but the ordering reverses at other budgets.Split conformal controls a different functional, so its reversals are not like-for-like risk comparisons.Source: Appendix J · p. 16, Table 9 · p. 16

Appendix K. Realised Risk by Modality Stratum

  • The modality-stratum risk direction is consistent across all six seed-by-budget cells, while strata 100 and 101, holding under 1% of events, are omitted.Table 10 reports realised risk for certified P-only policies by modality availability.Source: Appendix K · pp. 16–17, Table 10 · p. 17

Appendix L. Where the Admissible Set Sits

The appendix examines why fixed-sequence testing has little reach on the tested grid and records corrections made during the audit. Its findings distinguish the grid-floor constraint from results that would require unrun tests.

Appendix L. Where the Admissible Set Sits

  • In 15 of 18 runs, the smallest threshold suppresses zero events, forcing a Clopper–Pearson upper bound of 1.0 and stopping an ascending walk under any correction.The accepted regions were computed at corrected level δ/59, so only the low-end stopping result transfers to fixed-sequence testing.Source: Appendix L · p. 17
  • At the corrected level, non-empty accepted sets are contiguous in 17 runs, begin at the smallest threshold in only one, and reach the largest in none.An earlier ascending fixed-sequence implementation certified 0 of 45 fold–budget combinations; a coarser grid floor is identified as the relevant remedy.Source: Appendix L · pp. 17–18, Table 11 · p. 17

Appendix M. Correction Record

  • The correction record says the audited figures, report claims, split handling, and table bodies were corrected, while the engineered-feature comparator remains unreported.Its descriptors remain unstandardized, and absolute-amplitude features could act as a device shortcut.Source: Appendix M · p. 18
Loading 2609.12365v1…