PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-modal triage network: a multimodal deep learning framework for severity-based triage and visual explainability in chest radiographs

Zinah Ghulam, Richa Mittal, Eranga Ukwatta

arXiv:2609.04357v1eess.IVcs.AIcs.CV

TL;DR

CXR triage faces a bottleneck, and existing AI often detects findings without encoding urgency. The CMTN uses asymmetric modality dropout and an ordinal focal-loss strategy for severity triage; it agrees strongly with automatically derived severity labels, but that benchmark agreement does not establish clinical validity.

  • Problem

    CXR worklists can leave critical cases behind routine studies, while existing AI often identifies findings without encoding clinical urgency.

    Source: §1 · p. 2

  • Method

    The CMTN uses asymmetric modality dropout and ordinal focal loss with a distance penalty for severity-tier errors.

    Source: §1 · p. 4, §2.6 · p. 9

  • Results

    QWK = 0.9341 for agreement with the automatically derived severity taxonomy.

    Source: §4 · p. 20

  • Takeaways & Limitations

    Agreement with automatically derived severity labels is not proof of clinical validity.

    Source: §4 · p. 20, §4.1 · p. 24

    A larger, multi-reader prospective audit is needed to assess clinical validity beyond automated-label agreement.

    Source: §4.1 · p. 24

Abstract

from arXiv · show

Purpose: Increased number of chest radiograph (CXR) scans create a triage bottleneck, queueing urgent examinations behind routine ones. Existing AI tools are predominantly unimodal binary classifiers lacking severity awareness, and multimodal systems are rarely benchmarked against expert radiologists. To this end, we developed a multimodal deep learning framework for joint severity triage, pathology detection, and native visual explanation. Approach: We propose the cross-modal triage network (CMTN), fusing a Swin Transformer V2 visual encoder with a PubMedBERT text encoder via gated cross-attention. The CMTN was trained on 34,639 image-text pairs (12,489 patients) from MIMIC-CXR-JPG, optimizing an ordinal focal loss for four-tier severity triage and binary cross-entropy for 14 pathologies. Beyond quantitative benchmarking, attention heatmaps were evaluated against a blinded expert radiologist in a two-phase clinical audit comparing model triage output to expert severity assessment (100 cases) and grading spatial-semantic concordance (116 heatmaps). Results: The CMTN achieved strong ordinal agreement with reference labels (quadratic weighted kappa [QWK] = 0.9341, 95\% CI: 0.9219 to 0.9449) and macro-AUROC of 0.9970 across 14 pathologies, with 34~ms latency, outperforming the state-of-the-art BioViL multimodal baseline (QWK = 0.7679). However, the blinded Phase I clinical audit revealed substantially lower agreement with genuine radiologist judgment (QWK = 0.1399). Phase II found 54.3\% of heatmaps achieved clinically acceptable spatial localization. Conclusions: The CMTN demonstrated an efficient multimodal architecture for CXR triage. The divergence between algorithmic and radiologist agreement demonstrates that benchmark performance against NLP-derived labels is insufficient, highlighting the need for radiologist-labeled ground truth before clinical deployment.

1 Introduction

CXR volume growth and subjective urgency assignment can leave critical cases delayed in saturated FIFO worklists, while existing AI often detects findings without encoding urgency. The CMTN combines image and indication text for ordinal triage, pathology prediction, and native spatial explanations, and the paper directly audits its outputs against radiologist judgment.

1 Introduction

  • Most existing CXR AI systems are binary classifiers that identify findings but do not encode clinical urgency.The introduction also identifies a semantic gap between pixel data and clinical reasoning in unimodal AI.Source: §1 · p. 2
  • Critical cases may remain delayed behind routine examinations when saturated priority worklists revert to FIFO ordering.The introduction calls this delay the triage bottleneck.Source: §1 · p. 2
  • The CMTN jointly processes a radiograph and its clinical indication to predict four-tier ordinal severity, 14 pathology labels, and native spatial explanations.The pathology labels are auxiliary, and the explanations are generated via cross-attention.Source: §1 · p. 3
  • The authors construct a four-tier worklist-prioritization taxonomy by mapping 14 binary CheXpert labels to ordinal severity.The labels are extracted from free-text radiology reports using CheXpert NLP.Source: §1 · p. 3
  • Gated cross-attention couples a Swin Transformer V2 visual backbone with a PubMedBERT text encoder while preserving spatial geometry for native heatmaps.This is presented as an alternative to global embedding concatenation and post-hoc saliency methods.Source: §1 · pp. 3–4
  • The ordinal focal-loss objective penalizes non-adjacent-tier errors, and asymmetric modality dropout is intended to prevent text dominance and encourage visual feature extraction.These are presented as risk-asymmetric optimization and a structural necessity, respectively.Source: §1 · p. 4
  • The paper reports a blinded two-phase clinical audit against genuine radiologist judgment and grades the spatial-semantic concordance of model heatmaps.The audit is contrasted with evaluation solely against automatically derived training labels.Source: §1 · p. 4

2 Materials and Methods

The study uses a patient-level MIMIC-CXR-JPG cohort restricted to adult emergency or intensive-care frontal studies with extractable clinical indications. The retrospective, de-identified study and single-radiologist audit involved no patient contact or intervention.

2.1 Dataset and Cohort Construction

  • Eligible cases were adults imaged in frontal projection during ED or ICU encounters with valid indication text; lateral, oblique, missing-text or missing-label, and low-acuity outpatient studies were excluded.The indication was supplied before interpretation rather than taken from the finalized report, preserving workflow fidelity and avoiding leakage from the radiologist’s diagnostic conclusion.Source: §2.1 · pp. 4–5
  • The cohort comprised 34,639 image-text pairs from 12,489 patients, split by patient into training, validation, and quarantined test partitions with no patient overlap.The test set retained the reported acuity skew: Routine 46.47%, Mild 16.89%, Moderate 11.83%, and Urgent 24.81%.Source: §2.1 · p. 5

2.2 Ethics Statement

  • The work used retrospective, fully de-identified data and a retrospective review by one credentialed radiologist, with no patient contact or intervention.The parent database and derivative were IRB-reviewed and approved, with a waiver of informed consent.Source: §2.2 · p. 6

2.3 Label Engineering

The study derives its reference labels from finalized reports and maps 14 binary findings into a four-tier severity scale, treating automated labels as proxies rather than clinical truth. Images and indication text are standardized for model input.

2.3 Label Engineering

  • CheXpert extracts 14 pathology labels from finalized reports, which are mapped into four ordinal severity tiers and treated as proxies—not equivalents—for independent expert clinical truth.Urgent includes pneumothorax, pneumonia, pulmonary edema, or mispositioned support devices; Moderate includes lung lesion, fracture, or lung opacity; Mild includes pleural effusion, consolidation, atelectasis, enlarged cardiomediastinum, or pleural-other; Routine is the default without acute findings.Source: §2.3 · p. 6

2.5 Network Architecture

The network uses separate visual and clinical-text encoders, then applies gated cross-attention so the indication can retrieve spatially relevant image regions. The fused features drive severity and pathology predictions, while attention weights provide native heatmaps.

2.5 Network Architecture

  • The visual encoder produces 64 spatial patch embeddings, while PubMedBERT supplies a projected [CLS] text representation from biomedical-domain pretraining.SwinV2-Tiny uses hierarchical, locally windowed attention; PubMedBERT was selected for its native biomedical vocabulary.Source: §2.5 · p. 7
  • Gated cross-attention uses the projected clinical indication as Query over 64 Swin Transformer V2 visual patch embeddings as Keys and Values, preserving spatial geometry and enabling native heatmap extraction.The visual stream uses SwinV2-Tiny; the text stream uses PubMedBERT.Source: §2.5 · pp. 7–8, Figure 1 · p. 8
  • The attended visual vector and projected text vector form a 1024-dimensional fused representation feeding a four-tier ordinal severity head and a 14-condition auxiliary pathology head.Native cross-attention weights are additionally extracted as an interpretability heatmap.Source: §2.5 · p. 8, Figure 1 · p. 8

2.6 Training Strategy

Training combined ordinal severity prediction with auxiliary pathology supervision, using cross-modal regularization and validation-only model selection. The objective and tuning criteria were designed to preserve severity ordering and anatomical grounding while keeping the test set quarantined until final evaluation.

2.6 Training Strategy

  • The visual and text backbones began from pretrained weights, were frozen for the first epoch, then unfrozen for fine-tuning at a reduced learning rate.The reduced learning rate was 1.0 × 10^-6.Source: §2.6 · p. 8
  • Training-only modality dropout masked 30% of visual patches and dropped 10% of text to discourage unimodal shortcuts; dropout was disabled for validation and testing.The evaluation setting therefore used full bimodal availability.Source: §2.6 · pp. 8–9
  • The joint objective combined triage loss with binary cross-entropy over 14 pathology targets, weighted by λ = 2.0 to encourage pathology-grounded predictions.The condition-loss branch was described as structural regularization against statistical shortcuts.Source: §2.6 · p. 9
  • The severity branch used ordinal focal loss with a tier-distance weight that penalizes farther-tier errors and encodes clinical risk asymmetry.The weight scales with the absolute distance between true and predicted tiers.Source: §2.6 · p. 9
  • Checkpoint selection and early stopping used the composite validation score QWK + Macro AUROC, with early stopping after patience 7.Training used AdamW with cosine annealing for up to 25 epochs.Source: §2.6 · p. 9
  • Timing and modality dropout together accounted for 71% of validation variance; hyperparameters were selected on validation, and the quarantined test set was accessed once after model locking.A 60-trial Bayesian search and fANOVA were used to assess hyperparameter importance.Source: §2.6 · pp. 9–10

2.7 Baselines and Ablations

The study compared unimodal, concatenation, and cross-attention configurations, including two modality-dropout settings. It assessed ordinal triage, pathology discrimination, and calibration with complementary metrics.

2.7 Baselines and Ablations

  • Five configurations covered vision-only, text-only, concatenation fusion, unregularized cross-attention, and cross-attention with 10% modality dropout.CMTN-MR with moderate dropout was selected following a 0%–25% sensitivity sweep.Source: §2.7 · pp. 10–11

2.8 Evaluation Metrics

  • Evaluation used QWK and macro-F1 for triage, macro-AUROC across 14 pathology targets, and expected calibration error with post-hoc temperature scaling for calibration.QWK penalizes ordinal errors in proportion to squared tier distance.Source: §2.8 · p. 11

2.9 Explainability Framework and Clinical Audit

The explainability framework extracted heatmaps from the model’s cross-attention and evaluated them alongside blinded radiologist assessments. The study also stratified performance by demographic subgroups.

2.9 Explainability Framework and Clinical Audit

  • Cross-attention weights from an 8×8 grid were upsampled to 256×256 heatmaps, reflecting the model’s decision pathway rather than post-hoc Grad-CAM saliency.The cited text characterizes Grad-CAM as prone to spatial instability and not reflective of the model’s actual decision pathway.Source: §2.9 · p. 11
  • In Phase I, an independent radiologist assessed 100 quarantined cases while blinded to model predictions and clinical text; in Phase II, the radiologist graded 116 heatmaps on a five-point spatial-semantic scale.The scale ranged from 5 for a hotspot centered precisely on pathology to 1 for non-concordant attention.Source: §2.9 · p. 11

2.10 Fairness Assessment

  • Performance was stratified by self-reported biological sex, age bracket, and race/ethnicity, tracking QWK and macro-F1 across subpopulations.Source: §2.10 · p. 12

3 Results

The results show strong agreement with NLP-derived test labels and high pathology discrimination, alongside a substantial gap in agreement with blinded radiologist judgment and imperfect heatmap localization. The paper also reports subgroup performance and inference speed.

3.1 Validation Ablation and Hyperparameter Selection

  • On validation, CMTN-LR had a marginally higher raw QWK of 0.9126, but CMTN-MR with 10% dropout provided the best composite-objective balance and was locked as the final candidate.A controlled modality-dropout sweep compared 0%, 10%, and 20% rates.Source: Table 3 · p. 12, §3.1 · p. 12

3.2 Quarantined Test-Set Performance and External Baselines

  • CMTN-MR achieved QWK = 0.9341, outperforming DenseNet-121 (0.4438) and BioViL (0.7679), while text-only (0.9401) and concatenation fusion (0.9423) had higher raw QWK; CMTN-MR alone preserved spatial geometry for native heatmaps.The comparison used the quarantined held-out test cohort.Source: Table 4 · p. 13, §3.2 · pp. 13, 15, 16
  • 91.94% of predictions were exact, while 5.78%, 1.21%, and 1.06% differed by one, two, and three tiers; Moderate cases showed a tendency toward over-triage as Urgent.The class-wise results were strongest for Routine and weakest for Moderate cases.Source: §3.2 · pp. 13–14
  • Macro-AUROC was 0.9970 across 14 pathologies, and temperature scaling reduced ECE from 0.1893 to 0.0618.Edema had the highest reported AUROC (0.9997) and Enlarged Cardiomediastinum the lowest (0.9867).Source: §3.2 · pp. 14–15, Table 7 · p. 15, Figure 3 · p. 16

3.3 Demographic Generalization

  • Performance remained broadly stable across demographic groups: QWK exceeded 0.92 for both sexes and exceeded 0.87 in every reported race/ethnicity stratum.Age-stratum QWK ranged from 0.9084 for ages 18–40 to 0.9351 for those older than 65.Source: §3.3 · p. 17

3.4 Computational Efficiency

  • The deployed model averaged 13.94 ms per study, while the full pipeline including preprocessing and tokenization processed a study in 34 ms.Peak throughput was 1,362.5 scans per second at batch size 32.Source: §3.4 · pp. 17–18

3.5 Phase I: Blinded Clinical Triage Audit

  • On 100 audited cases, CMTN-MR’s agreement with the blinded radiologist was QWK = 0.1399; the radiologist’s agreement with NLP-derived labels was also weak (QWK = 0.1033).The audit recorded 39 over-triage and 29 under-triage events.Source: §3.2 · p. 16, §3.5 · p. 18, Figure 4 · p. 19

3.6 Phase II: Spatial-Semantic Concordance

  • Among 116 expert-graded cross-attention heatmaps, 54.3% achieved clinically acceptable localization (scores 4 or 5), including 44.8% perfect; 26.7% were non-concordant (score 1), predominantly with attention captured by high-contrast medical hardware.Edema (mean 4.33) and pleural effusion (mean 4.03) were localized most reliably.Source: §3.6 · pp. 19–20

4 Discussion

Under identical benchmark conditions, CMTN-MR outperformed the compared architectures on ordinal severity agreement, but its strong agreement with NLP-derived labels diverged from agreement with a blinded radiologist. The discussion also weighs spatially interpretable attention against localization failures and identifies data and audit limitations that motivate future work.

4 Discussion

  • QWK = 0.9341 for CMTN-MR, versus 0.7679 for BioViL and 0.4438 for DenseNet-121 on identically conditioned benchmarks.The authors attribute the advantage to the targeted gated cross-attention mechanism under uniform experimental constraints.Source: §4 · pp. 21–22
  • The text-only baseline reached QWK = 0.9401 while the vision-only baseline reached 0.4492, motivating text masking in 10% of training batches to counter text dominance.The authors describe asymmetric modality dropout as forcing reliance on the X-ray image and anchoring the fusion space to anatomy.Source: §4 · p. 22
  • On the same 100-case audit subset, CMTN-MR agreement was QWK = 0.8807 with automated labels but only QWK = 0.1399 with the blinded radiologist.Its reference-label agreement remained QWK = 0.9341 across the full 3,475-case test set; the authors interpret the gap as a mismatch between reference labels and clinical judgment.Source: §4 · p. 22
  • CMTN-MR’s QWK was statistically indistinguishable from concatenation, but its Macro-F1 was lower; spatially preserving cross-attention enabled heatmaps with 54.3% clinically acceptable localization.The discussion frames this as a trade-off: concatenation destroys spatial geometry, while cross-attention supports native heatmap extraction.Source: §4 · p. 23
  • In 26.7% of Phase II cases, attention focused on high-contrast hardware rather than subtle pathology, although exposed maps can make such distraction visible to clinicians.The authors describe this as both a technical limitation and a potential safety benefit in decision support.Source: §4 · pp. 23–24

4.1 Limitations

  • The findings are limited by single-institution data, severity labels derived from noisy automated NLP, and a single-radiologist 100-case audit with wide confidence intervals crossing zero.The audit also prevented calculation of inter-rater reliability; the authors call for a larger, multi-reader prospective audit to establish a generalizable benchmark.Source: §4.1 · p. 24
  • The architecture uses only a single frontal radiograph and brief clinical text, without broader EHR context such as vital signs, laboratory data, or prior imaging.Source: §4.1 · pp. 24–25

4.2 Future Work

  • Future work includes prospective multi-reader radiologist labeling, structured clinical data and multi-view imaging, plus anatomically constrained attention supervision targeting hardware hijacking.Source: §4.2 · p. 25

5 Conclusion

CMTN combines visual and clinical-text encoders for chest-radiograph triage, but its strong agreement with automated labels did not transfer to genuine radiologist agreement. The conclusion calls for large-scale radiologist-labeled validation before clinical readiness can be established.

5 Conclusion

  • QWK = 0.9341 on the patient-isolated internal test, with low calibration error, stable cross-demographic performance, and real-time inference latency.The authors describe the architecture as computationally and structurally ready for pilot integration studies.Source: §5 · p. 25
  • QWK = 0.1399 in the blinded audit, where benchmark performance did not transfer to genuine agreement with an independent radiologist.Spatially interpretable attention remained inconsistent.Source: §5 · p. 25
  • Internal validation against algorithmically derived labels is insufficient as a proxy for clinical readiness; large-scale radiologist-labeled validation is the stated next step.The authors say this validation is required before clinical deployment.Source: §5 · pp. 25–26

Code, Data, and Materials Availability

The study’s dataset is available to credentialed researchers through PhysioNet, while its source code and evaluation materials are available under the MIT License.

Code, Data, and Materials Availability

Loading 2609.04357v1…