PaperlayerView exact revision on arXiv

Source-linked AI summary

Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI

Quang Bui, Shlok Jaiswal, Samuel Paik-Heintz, Kevin Zhou, Kaushik Madapati, Krittaphas Chaisutyakorn, Noah Dane Hebdon, Dimitrios Proios, Sebastián Andrés Cajas Ordóñez, Kacper Dobek, Boya Zhang, Aly Dhedhi, Ahram Han, Kushul Reddy Palakala, Rahul Gorijavolu, Jacques Kpodonu, Leo Anthony Celi

arXiv:2608.01462v1cs.AI

TL;DR

Multimodal clinical models need reusable evaluations that identify which missing modality causes errors and whether those errors are monitorable or silent. This paper introduces a model-agnostic failure-analysis framework and shows robust recovery of planted modality structure, while echo dominance and failure detectability diverge in validation.

  • Problem

    Multimodal models are usually evaluated with all modalities present, leaving modality-specific responsibility and loud-versus-silent failures under missing inputs insufficiently characterized.

  • Method

    The paper provides a reusable, model-agnostic framework that analyzes per-example failures, modality complementarity, attribution, and loud-versus-silent dropout using deployment-observable signals.

  • Results

    Across 12 seeds, the harness recovered planted modality attribution, while echo dominance and loud-versus-silent rates were comparable, and the framework scaled to three modalities.

  • Takeaways & Limitations

    Modality value and failure detectability need separate measurement, and the evaluation can transfer across mask-aware multimodal models rather than being tied to one checkpoint.

  • Takeaways & Limitations

    The real-data conclusions come from a single US academic center, so degradation and fairness patterns may not transfer across institutions or acquisition hardware.

Abstract

from arXiv · show

Multimodal clinical models are usually judged on accuracy with every modality present, but deployment removes modalities; an echocardiogram is often unavailable where an ECG is routine. Two questions then matter beyond the size of the accuracy loss: which modality was responsible, and whether the model fails loudly or silently once that modality is dropped. The distinction is per-example and modality-level, and is separate from post-hoc feature attribution (e.g. SHAP). Models are replaced often; the evaluation that answers these questions is reused. We present a model-agnostic modality-failure framework: given N modality embeddings, any mask-aware probe, and labels, it returns a per-example failure taxonomy, a per-modality complementarity matrix that attributes error to modalities, and a loud-vs-silent dropout profile separating monitorable failures from those that pass unflagged far from the decision boundary, using only deployment-observable signals. We release it as a small, unit-tested harness and validate it against planted ground truth. Across seeds it recovers that planted modality dominance and complementary subset, reports per-modality loud-vs-silent rates, and scales to a three-modality complementarity matrix; because the planted structure is known by construction, this validates recovery of per-example attribution rather than clinical performance. We then instantiate the framework on frozen EchoJEPA and HuBERT-ECG embeddings for LVEF and the EF <= 40% HFrEF gate over a paired MIMIC-IV cohort, where on the held-out test split (n = 245) dropping echo nearly doubles error. The narrow echo-to-ECG overlap that bounds cohort size is itself a deployment finding for cardiac foundation models. All of our work can be found at https://github.com/criticaldata/PRIMED-AI.

1 Introduction

The paper introduces a reusable, model-agnostic framework for analyzing which missing modality causes failure and whether dropout failures are loud or silent. It validates the framework on planted ground truth and applies frozen EchoJEPA and HuBERT-ECG embeddings to narrowly scoped LVEF and HFrEF tasks in MIMIC-IV.

  • Motivation: Deployment often lacks modalities such as echocardiography even when ECG is routinely available, so full-modality accuracy does not reveal modality-specific failure.The evaluation is intended to remain useful as foundation models are replaced across generations.
  • Framework: The model-agnostic framework takes N modality embeddings, a mask-aware probe, and labels, returning per-example failure taxonomy, modality complementarity and attribution, and loud-vs-silent dropout profiles.Its dropout profile uses only deployment-observable signals and is designed to transfer across model generations.
  • Framework: A small reusable harness supports any N modalities and mask-aware probe, with validation against data containing planted ground truth.The framework is implemented as a reusable evaluation rather than another fusion model.
  • Clinical instantiation: The real-data instantiation uses frozen EchoJEPA and HuBERT-ECG embeddings without fine-tuning for LVEF regression and the EF ≤40% HFrEF gate over MIMIC-IV.The scope is deliberately limited to LVEF and EF ≤40%; valvular disease and HFpEF are deferred for label reasons.
  • Validation: Across 12 seeds, the harness recovers planted modality dominance and a complementary subset, then scales to an N-modality complementarity analysis.Because the structure is planted by construction, this tests recovery of modality-level failure structure.

2 Related Work and Positioning

The study positions itself against cardiac foundation-model and missing-modality work by retaining frozen multimodal backbones and analyzing deployment-time modality loss. It introduces reusable, per-modality failure attribution and loud-versus-silent degradation analysis rather than aggregate ablations or ECG-only distillation.

  • Related cardiac foundation models: EchoJEPA, HuBERT-ECG, and EchoingECG represent related cardiac foundation-model approaches spanning echocardiography, 12-lead ECG, and echo-informed ECG representations.EchoingECG predicts cardiac function from ECG alone using probabilistic cross-modal embeddings and an echo–language teacher.
  • Positioning against EchoingECG: Unlike EchoingECG, the study freezes both backbones, learns only lightweight probes, retains and fuses both modalities, and evaluates missing-modality inference.EchoingECG uses echo during training to improve an ECG-only model, whereas this study examines what happens when one modality is unavailable at inference.
  • Deployment and fairness positioning: The study also prespecifies a demographic fairness audit over full per-example predictions, making inference-time degradation a primary result.This deployment-focused analysis is model-agnostic and asks how a multimodal model fails when a modality is gone.
  • Missing-modality failure analysis: Missing-modality robustness is usually summarized by aggregate accuracy loss, whereas this framework attributes failures per modality and separates loud from silent degradation.The framework is presented as a reusable instrument rather than a one-off ablation.

3 A Framework for Modality-Failure Analysis

The framework provides model-agnostic, per-example modality-failure analysis for any mask-aware predictor, reporting failure taxonomy, modality attribution and complementarity, and loud-versus-silent dropout. It uses deployment-observable predictions to distinguish clinically actionable gate failures from errors that remain unflagged.

  • Framework inputs: Given modalities, labels, and predictions for any present subset, the framework requires only a mask-aware predictor and applies across probes, fusion architectures, and backbones.Predictions are generated with other modalities masked at inference.
  • Failure taxonomy: Each example is classified as correct, imprecise, or critical using HFrEF gate correctness and the regression-error threshold δ.Critical errors flip the HFrEF gate; large regression errors that do not flip it remain imprecise.
  • Complementarity and attribution: For modality m, leave-one-out attribution uses ∆m=MAE(f(M\{m}))−MAE(f(M)), while solo and pairwise errors distinguish redundant from complementary modalities.Per-example attribution assigns the win to the modality whose removal hurts that example most; analyses can also stratify by sex, age, and race.
  • Loud-vs-silent dropout: Induced critical failures after modality dropout are silent when |f −τ| ≥κ and loud when |f −τ| <κ, separating unflagged errors from outputs near the decision boundary.Silent failures appear unremarkable to a margin monitor, whereas loud failures invite a second read.
  • Implementation: All three analytical views are computed from a single call to the released harness.The implementation is packaged as a reusable harness for the framework’s outputs.

4 Data and Cohort Design

The study instantiates the modality-failure framework using frozen echocardiogram and ECG embeddings for continuous LVEF estimation and the EF ≤40% HFrEF gate. It constructs auditable labels from multiple measurement fields and deterministically pairs each echo with a same-patient ECG within ±24 hours.

  • Cardiac instantiation: The cardiac instantiation uses frozen echocardiogram and ECG embeddings for LVEF estimation and the EF ≤40% HFrEF gate.
  • Data sources: The cohort draws on MIMIC-IV-Echo, MIMIC-IV-ECG, and MIMIC-IV v3.1 for imaging, 12-lead waveforms, and demographics, respectively.
  • Label construction: LVEF labels prioritize direct resting measurements, then biplane, explicitly rest-labeled, and 3D estimates, using ranges only without point estimates.Stress measurements are excluded, and the source field for each value is stored for auditability.
  • Pairing and splits: Each labeled echo study is paired with the nearest same-patient ECG within a symmetric ±24 h window, with deterministic tie-breaking for multiple matches.Configurable alternatives include an asymmetric window allowing an ECG up to 30 days before the echo and 24 hours after.

5 Frozen-Embedding Pipeline

The pipeline uses frozen EchoJEPA-L and HuBERT-ECG embeddings with lightweight probes, then evaluates one complete-data checkpoint under modality-dropout conditions without retraining. It reports task metrics, degradation curves, and a prespecified post-hoc fairness audit on held-out predictions.

  • Frozen embeddings: Frozen EchoJEPA-L echo-video and HuBERT-ECG embeddings are used in inference mode, with checkpoint tags preserving encoder-version traceability.The frozen backbones support portability while lightweight probes are trained.
  • Probe architectures: Cross-attention fusion is the primary missing-modality probe, using bidirectional attention between echo and ECG token sequences before an MLP regression head.Echo-only instead uses attentive pooling; all probes predict continuous LVEF, with EF ≤40% derived from predictions.
  • Dropout evaluation: One complete-data cross-attention checkpoint is evaluated on held-out test data as full, echo-dropped, or ECG-dropped, with the missing branch zeroed at inference.This tests the deployed model’s response to missing inputs rather than performance after retraining unimodal models.
  • Dropout evaluation: MAE and EF ≤40% AUROC are reported for each condition, alongside bootstrap confidence intervals when per-example predictions are available and a degradation curve.The procedure is intended to distinguish graceful from silent failure when an expensive modality is absent.
  • Fairness and reproducibility: The post-hoc fairness audit stratifies the same test predictions by sex, age band, and race, flags strata below a minimum count, and uses fixed seeds and manifests.Configuration-driven artifacts record the git commit and resolution details.

6 Results

Synthetic validation recovered the planted modality structure while clarifying that two-modality attribution is partly degenerate and clinically superior performance is not established. A three-modality test produced non-trivial complementarity results, with echo strongest alone and in every pair containing it.

  • Synthetic validation: The synthetic validation recovered the planted structure, but this circular check confirms harness mechanics rather than clinical superiority.Echo was strongly informative, ECG weakly informative, and an approximately 18% subset required ECG because echo was uninformative.
  • Attribution limits: Two-modality per-example attribution is partly degenerate because dropping one modality equals using the other.Non-trivial attribution requires at least three modalities.
  • Three-modality complementarity: On the three-modality test, echo alone achieved the lowest MAE at 9.7, versus 12.9 for ECG and 13.4 for labs.The test instance included echo, ECG, and simulated labs, with n=225.
  • Three-modality complementarity: Every echo-containing pair achieved approximately 8.3 MAE, outperforming the ECG-and-labs pair at 11.3.The marginal values were echo 3.8, ECG 0.9, and labs 0.7; per-example wins were echo 125, ECG 56, and labs 44.
  • Failure taxonomy: Dropping echo introduced many critical gate-flipping failures, making failure counts clinically actionable beyond MAE.Full-modality inference was mostly correct in the representative seed shown.
  • Evaluation procedure: The taxonomy, complementarity, and loud-vs-silent analyses require a per-example prediction dump, which is compute-cheap at this cohort size.A canonical rerun with patient-level cross-validation is in progress.

7 The Released Harness

The primary deliverable is a small, model-agnostic harness that analyzes modality failures across arbitrary mask-aware probes and returns attribution and dropout diagnostics. It recovers planted structure, while real-cohort per-example analyses and canonical cross-validated reruns remain ongoing.

  • Released components: 51 unit tests were passing at writing, including tests that the harness recovers planted structure.The released framework and harness are identified as the primary deliverable.
  • Reusable harness: The reusable harness accepts N modality embeddings, any mask-aware prediction function, and labels, returning failure taxonomy, complementarity matrix, and loud-vs-silent dropout profile.Its single analyze entry point supports reuse across probes, fusion architectures, and future backbone releases.
  • What is established, and what remains: The harness recovers planted dominance, complementarity, and silent-failure structure end-to-end, validating mechanics rather than substituting for clinical numbers.The real-cohort result currently consists of a first cross-attention run reporting missing-modality degradation in Table 4.
  • What is established, and what remains: Real-prediction failure taxonomy, complementarity, loud-vs-silent profiles, unimodal-probe sweeps, bootstrap intervals, and fairness stratification remain to be exported or completed.A patient-level cross-validation rerun replacing the single held-out split is in progress, and the additional exports are described as compute-cheap at this cohort size.

8 Modality Asymmetry Beyond Inference Time

Modality asymmetry extends beyond inference: echocardiography is harder to acquire and develop than ECG, whose clinical ubiquity is matched by simpler infrastructure. This alignment makes deployment without echo more likely and makes understanding dropout failures important.

  • Development and acquisition asymmetry: Echocardiography is harder than ECG to acquire clinically and to support during development, requiring bespoke temporal joins, windowing, tie-breaking, leakage-safe splits, and heavier DICOM processing.No off-the-shelf echo-to-ECG temporal join exists, while echo’s DICOM volume makes it the heavier modality in development.
  • Development and acquisition asymmetry: ECG is cheap and ubiquitous at the bedside, making it easier to handle across both acquisition and development.The passage contrasts ECG’s bedside availability with echo’s bespoke and heavier infrastructure requirements.
  • Deployment implications: The shared direction of acquisition and infrastructure costs is itself a deployment finding: systems are liable to be deployed without echo, making failure analysis under echo dropout consequential.The paper interprets this correspondence as structural to the path from a published cardiac foundation model to a screening tool, rather than project-specific accident.

9 Limitations

The study’s conclusions are limited by single-center data, frozen backbones, proxy-based monitoring measures, and degenerate two-modality attribution. Demographic-audit constraints and one-clip-per-study echo extraction further limit interpretation and absolute accuracy.

  • Generalizability and model scope: Single-center MIMIC-IV data limit transferability, while frozen backbones make absolute accuracy a lower bound and shift claims toward relative missing-modality degradation.The study does not establish state-of-the-art LVEF estimation or external validity across institutions and acquisition hardware.
  • Metric and attribution caveats: The loud/silent split proxies monitorability with distance from the EF ≤40% threshold, and N=2 attribution is partly degenerate because the winner equals the surviving-modality solo.A calibrated head could sharpen monitorability assessment, while per-example attribution is most informative for N≥3.
  • Audit and extraction limitations: Demographic-audit gaps reflect curated data, small per-stratum samples, and post-hoc prevalence or presentation confounding; one-clip echo extraction may also affect absolute echo-branch accuracy.Small strata can yield wide or undefined AUROC, and observed group gaps may reflect the data rather than the model alone.

10 Discussion

The framework reframes multimodal deployment evaluation around which modality drives performance and whether failures are loud or silent when a modality is unavailable. Because it requires only a mask-aware predictor, the evaluation transfers across echo or ECG backbones and outlives the model it assesses.

  • Deployment evaluation should identify the responsible modality and distinguish loud from silent failures after modality removal.
  • The framework transfers to new echo or ECG backbones without modification because it assumes only a mask-aware predictor.
  • The evaluation outlives the model it assesses, motivating release of a reusable instrument rather than a checkpoint.
Loading 2608.01462v1…