PaperlayerView exact revision on arXiv

Source-linked AI summary

Integrating Unimodal and Vision-Language Representations in Latent Space for Multi-Label Chest X-Ray Classification

Quang-Huy Tran, Duc-Tuan Ngo, Minh-Khoi Nguyen-Bui, Dang-Khoa Bui, Thanh-Trong Tran, Tuan-Khoi Nguyen, Hoang-Anh Ngo

arXiv:2609.09185v1cs.CV

TL;DR

Multi-label chest X-ray classification lacks evidence on combining unimodal and vision–language representations with latent refinement. The paper fuses RAD-DINO and BioViL-T branches for 14 MIMIC-CXR-JPG labels, achieving its best reported performance with latent hybrid fusion while requiring external validation.

  • Problem

    Combining RAD-DINO and BioViL-T representations, refining each source in latent space, and hybrid-fusing them remains underexplored for multi-label chest X-ray classification.

  • Method

    The framework keeps both encoders frozen, refines RAD-DINO, BioViL-T, and their combined representation in separate latent branches, and fuses their normalized outputs.

  • Results

    Latent hybrid fusion achieves a mean AUROC of 0.840 and an mAP of 0.467, outperforming the individual branches, early fusion, and unrefined hybrid fusion.

  • Takeaways & Limitations

    Fusion effectiveness depends on the quality of the representations supplied by the branches, with refined representations enabling consistent improvements over early fusion.

  • Takeaways & Limitations

    The results rely on internal evaluation with MIMIC-CXR-JPG, so generalizability to other healthcare institutions remains unvalidated.

Abstract

from arXiv · show

Multi-label chest X-ray classification has attracted considerable attention in recent years, with the effective use of visual representations and clinical semantic knowledge playing an important role. This study proposes a framework that combines unimodal representations from RAD-DINO with vision--language representations from BioViL-T for the classification of 14 labels in the MIMIC-CXR-JPG dataset. The RAD-DINO and BioViL-T embeddings and their combined representation are refined separately in latent space before being normalized and fused across the three branches. In addition to improving classification performance, the study aims to clarify the role of each embedding source and the degree to which they complement one another. Experiments show that RAD-DINO outperforms BioViL-T when used independently, whereas early fusion further improves the results, indicating that the two embedding sources contain complementary information. The best-performing model achieves a mean AUROC of 0.840 and an mAP of 0.467. Ablation analysis shows that hybrid fusion provides consistent and statistically significant improvements over early fusion when each embedding source is refined in latent space, suggesting that fusion effectiveness depends on the quality of the representation supplied by each branch. However, the study has only been evaluated internally on MIMIC-CXR-JPG; its generalizability to data from other healthcare institutions therefore remains to be validated. The source code is available at: https://anonymous.4open.science/r/mimic-report-C210/.

1. Introduction

Multi-label chest X-ray classification benefits from combining visual and clinical-semantic representations, but their complementarity remains underexplored. This study evaluates RAD-DINO and BioViL-T together in an image-only inference setting.

  • RAD-DINO learns image-only biomedical representations, whereas BioViL-T uses vision–language and temporal supervision from chest X-rays and reports.
  • Combining RAD-DINO and BioViL-T addresses an underexplored question about whether unimodal and vision–language representations provide complementary information for chest X-ray classification.
  • The study evaluates whether combining the two encoders improves over either source alone, whether latent refinement improves fusion, and how the branches contribute to prediction.
  • Experiments use frozen pretrained encoders and images alone at inference time, with evaluation on 14 CheXpert observation labels from MIMIC-CXR-JPG.

2. Related work

Prior chest X-ray work has advanced from CNNs to Transformers, foundation encoders, and latent-space refinement. However, integrating RAD-DINO and BioViL-T representations with latent hybrid fusion remains underexplored.

  • Prior work combines CNNs, Transformers, attention, label relationships, and class-imbalance strategies to address multi-label chest X-ray classification.
  • RAD-DINO learns visual representations from image-only biomedical pretraining, while BioViL-T jointly models image, report, and temporal information for vision–language representations.
  • Latent-space methods compress, refine, and reorganize representations before downstream prediction, including through autoencoders, diffusion, and latent bottlenecks.
  • Combining RAD-DINO with BioViL-T, refining each source in latent space, and performing hybrid fusion remains underexplored for multi-label chest X-ray classification.

3. Methodology

The methodology compares single-source, early-fusion, and hybrid-fusion configurations while refining three representation branches in a shared latent space. Branch predictions are normalized and fused under prespecified statistical analyses.

  • 3. Methodology: Five configurations compare RAD-DINO, BioViL-T, direct early fusion, unrefined hybrid fusion, and latent hybrid fusion.
  • 3. Methodology: Each standardized branch embedding is mapped by a separate autoencoder into a common 256-dimensional latent space and trained with reconstruction loss and Gaussian input noise.
  • 3. Methodology: Each branch combines original and refined representations through Transformer processing and a learned gate before producing label logits.
  • 3. Methodology: Normalized branch logits are fused with nonnegative weights summing to one, using uniform one-third weights in the main configuration.
  • 3. Methodology: Six prespecified paired comparisons test early and hybrid fusion against the individual branches using macro AUROC and mAP, with two-sided significance testing at p < 0.001.
  • 3. Methodology: Optimization variability is assessed across five seeds, while patient-level clustered bootstrap resampling quantifies sampling uncertainty.

4. Data processing

The study aggregates multiple radiographic views into study-level RAD-DINO and BioViL-T embeddings, then uses a chronological MIMIC-CXR-JPG split. The final cohort contains 211,130 studies with complete embedding coverage.

  • 4.1. Study-level multi-view aggregation: Frontal and lateral images are averaged within view groups, and studies containing both groups use a validation-selected 0.6/0.4 frontal-to-lateral weighting.
  • 4.1. Study-level multi-view aggregation: When only one view group is present, its embedding is used directly.
  • 211,130 studies form the final cohort, with 100% coverage for both RAD-DINO and BioViL-T embeddings.
  • The train–validation–test partition preserves chronological order, preventing future studies from training models evaluated on earlier studies.
  • Images are cropped to 559 × 559 and resized to 518 × 518 before frozen encoders produce 768-dimensional RAD-DINO and 128-dimensional BioViL-T embeddings.

5. Experimental results

Hybrid fusion delivers the strongest overall performance, while comparisons and ablations show complementary embeddings, latent refinement, and label-dependent gains. Confirmatory analyses support the main fusion improvements, although external comparisons are only reference-level evidence.

  • 5.1. Overall performance: 0.840 mean AUROC and 0.467 mAP make hybrid fusion the strongest configuration, improving over early fusion by 0.0035 AUROC and 0.0029 mAP.Both gains are significant at p < 0.001 across five experimental seeds.
  • 5.1. Overall performance: All six confirmatory paired comparisons have simultaneous 99.17% bootstrap intervals excluding zero, while per-label significance remains exploratory.The hybrid-versus-early gain has a 99.17% lower bound of 0.0030; per-label power is limited for this smallest effect.
  • 5.2. Comparison with previous methods: 0.840 AUROC, 0.467 mAP, and 0.484 Macro F1 are the highest values among the listed comparison results, though protocols are not fully comparable.The reference comparison includes models evaluated on MIMIC-CXR or MIMIC-CXR-JPG, with differing labels, cohorts, and protocols.
  • 5.3. Per-label results: 0.742–0.928 mean AUROC spans labels, with Support Devices highest and Enlarged Cardiomediastinum lowest; Pleural Other pairs 0.865 AUROC with only 0.104 AP.This illustrates that AUROC does not necessarily reflect positive-class performance under label imbalance.
  • 5.3. Per-label results: 10 of 14 labels outperform MBRANet, with the largest gain on Pneumothorax (+0.147), while Enlarged Cardiomediastinum declines by −0.081.The improvement is broad but not uniform across labels.
  • 5.4. Ablation Study: Ablations attribute the gain primarily to refined representations and complementary embedding sources, not prediction-head choice, added capacity, or more complex fusion rules.Uniform fusion of refined branches performs best, and the prevalence analysis finds stronger support for AUROC than AP on rare labels.

6. Discussion

RAD-DINO provides the strongest standalone representation, while combining it with BioViL-T improves classification because the two sources contain complementary information. Latent refinement and separate branch fusion further improve performance, although evaluation remains limited to MIMIC-CXR-JPG.

  • 6.2. Complementarity between RAD-DINO and BioViL-T: 0.021 AUROC and 0.036 mAP separate BioViL-T from RAD-DINO when each is used independently, but their concatenation significantly improves both metrics.The result indicates that lower standalone performance does not eliminate complementary information in BioViL-T.
  • 6.2. Complementarity between RAD-DINO and BioViL-T: 0.834 linear CKA between refined latents and 0.918 prediction-error correlation show that refinement makes branch representations and predictions more similar.The original embeddings have CKA 0.419, compared with approximately 0.002 for the permutation baseline and 1 for linearly equivalent representations.
  • 6.3. Latent refinement and fusion: Separate latent refinement before late fusion is more effective than fusing unrefined branches, while the complete refinement block—not its autoencoder or denoising components alone—drives the gain.The ablation also finds that removing denoising leaves performance unchanged, whereas bypassing the learned inference-time transformation removes the block-level improvement.
  • 6.4. Practical implications: Frozen encoders reduce downstream training cost and enable reuse across configurations, while inference requires only images despite BioViL-T’s vision–language pretraining.The final system does not require radiology reports or clinical variables at inference time.
  • 6.5. Limitations: Generalizability remains unvalidated because all experiments use MIMIC-CXR-JPG, with additional limitations from imbalanced report-derived labels, five-seed tests, and repeated use of one test set.The authors also note that reducing each study to a global label vector may discard small or localized findings.

7. Conclusion

The study combines frozen RAD-DINO and BioViL-T embeddings through separate latent branches and fusion for 14-label chest X-ray classification. Hybrid fusion achieves the strongest reported performance, but external validation is still needed.

  • 7. Conclusion: Frozen RAD-DINO and BioViL-T encoders are refined in separate latent branches before their outputs and combined representation are fused.The design targets differences in the models’ pretraining information sources rather than relying on a single representation.
  • 7. Conclusion: 0.840 mean AUROC and 0.467 mAP are achieved by latent hybrid fusion on the 14 MIMIC-CXR-JPG labels.RAD-DINO outperforms BioViL-T independently, while combining both embeddings improves over either individual branch and early fusion.
  • 7. Conclusion: Unrefined late fusion performs significantly worse than the main configuration, indicating that fusion effectiveness depends on the quality of each branch’s representation.The conclusion attributes the benefit to representation quality alongside an ensemble effect from fusion.
  • 7. Conclusion: External validation on data from other healthcare institutions is needed to confirm generalizability beyond the internal MIMIC-CXR-JPG evaluation.

Supplementary Material Integrating Unimodal and Vision–Language Representations in

The supplementary material identifies the aligned cohort’s label-distribution context and reports substantial imbalance across observations. Pleural Other is rare, whereas No Finding is comparatively prevalent.

  • A Label Distribution: The supplementary material includes a table describing label distribution in the aligned cohort.
  • A Label Distribution: 1.193% prevalence for Pleural Other and 32.778% for No Finding define the reported range in the aligned cohort.
  • A Label Distribution: The label range indicates substantial imbalance across the aligned cohort.

B Experimental Configuration

The experimental configuration reports shared cohort and reproducibility settings, model-selection hyperparameters, implementation costs, and label-wise fusion behavior. Early fusion is dominant on average, but all branches contribute nonredundant predictive information.

  • B Experimental Configuration: Five random seeds share the same train, validation, and test set sizes, with reproducibility and cohort settings documented separately.
  • B Experimental Configuration: Hybrid fusion trains in approximately 4.6 minutes per seed, or about 0.4 GPU-hours for five seeds, under the stated nominal epoch-count estimate.The estimate is an upper bound without early stopping; frozen encoders are excluded from FLOPs counts.
  • B Experimental Configuration: The configuration uses 512-dimensional predictor hidden states, 256-dimensional latents, and 1024-dimensional feed-forward layers.Training schedules include 12 autoencoder epochs, 10 latent-denoiser epochs, and up to 60 predictor epochs.
  • C.2 Label-Wise Late Fusion Weights: The label-wise analysis finds early fusion has the largest mean weight across all 14 labels, while RAD-DINO and BioViL-T contributions vary by observation.RAD-DINO contributes more to Lung Lesion and Pneumothorax, whereas BioViL-T contributes more to Support Devices, Consolidation, and Cardiomegaly.
  • C.2 Label-Wise Late Fusion Weights: 0.930 is the mean early-fusion weight for No Finding, which depends almost entirely on that branch.Lung Lesion and Support Devices receive substantial additional contributions from individual branches.
  • C.2 Label-Wise Late Fusion Weights: −0.0037 macro AUROC and −0.0062 mAP are lost when the RAD-DINO latent branch is removed, the largest leave-one-branch-out degradation.Removing any single branch degrades both metrics with 95% intervals excluding zero, so no branch is redundant within the ensemble.

C.3 Representation Complementarity Analyses

These analyses show that the RAD-DINO, BioViL-T, and combined branches provide distinct predictive signals, with complementarity concentrated in difficult and rare labels. Fusion benefits from branch diversity, while calibration and aggregation choices produce comparatively small changes.

  • Per-label error structure: Complementarity is strongest for Fracture, Pleural Other, Pneumothorax, and Lung Lesion, whose score correlations are below the macro average and whose labels are mostly rare.The rare-label tier also shows the largest hybrid-over-early AUROC improvement.
  • Decision-level disagreement: The branches disagree on 11.5% of label decisions, leaving 4.9 accuracy points of oracle headroom over the better single branch.Uniform fusion recovers only part of this decision-level headroom.
  • Leave-one-branch-out: Removing any branch degrades both macro AUROC and mAP, with the largest loss occurring when the RAD-DINO latent branch is removed.All six paired-difference intervals exclude zero.
  • Representation complementarity: The three branches’ functional diversity is consistent with an ensemble-effect interpretation of the fusion stage.The leave-one-branch-out results indicate that each branch contributes a unique predictive signal.
  • Calibration: Hybrid fusion is preferred to either single source for calibration, but its differences from early fusion are too small to support a calibration preference.The margin is 0.0004 in Brier score and 0.0001 in ECE relative to early fusion.
  • Aggregation sensitivity: Macro AUROC varies by at most 0.0010 and mAP by at most 0.0014 across frontal–lateral aggregation schemes.The intervals overlap substantially across all four schemes.

D.3 The Latent Denoiser at Inference

At inference, the latent denoiser applies a substantial transformation rather than an identity map. Its denoising objective is not necessary for the observed benefit, but bypassing the learned transformation reduces performance.

  • Latent geometry: At s = 0, the module produces a substantial approximately norm-preserving rotation rather than an approximate identity.Mean cosine similarity is 0.851 ± 0.009, with approximately 32° angular deviation and relative L2 deviation of 0.547 ± 0.015.
  • Autoencoder reconstruction: The latent path is lossy: cosine similarity between the input and reconstructed embedding is 0.770 with relative L2 deviation 0.643.The classifier receives the original embedding and transformed latent as separate tokens, so exact reconstruction is not required.
  • Inference-time role: Bypassing the latent transformation reduces macro AUROC by −0.0013 and mAP by −0.0016.The AUROC difference is significant at p < 0.001, while both differences are significant under the four-test Bonferroni level.
  • Mechanism analysis: Removing the stage-two denoising objective does not remove the block-level benefit, whereas removing the inference-time transformation does.Together, the analyses attribute the benefit to the trained nonlinear map rather than specifically to the denoising objective.

E Supplementary Ablation Analyses

The supplementary ablations separate representation quality from fusion-rule effects and compare heterogeneous fusion with homogeneous-model ensembling. Refined representations dominate the decomposition, while equivalence to ensembling holds only within the study’s declared margin.

  • Factorial decomposition: Refined branches outperform raw branches under both fusion rules and on both metrics, with all four simple effects reaching p ≤ 9.65 × 10^-5.The refined-plus-uniform and raw-plus-learned configurations recover the reported A8 − A9 contrast of +0.0040.
  • Fusion-rule effects: Learned weights improve raw-representation mAP by +0.0051, but the refined-representation gain is only +0.0008 and does not reach the threshold.The corresponding refined-level AUROC change is −0.0006 and also does not reach the threshold.
  • Interaction: The representation-by-fusion interaction is significant on mAP at −0.0043, while the AUROC interaction has the same sign without reaching the threshold.The five-seed design has limited power for interaction testing.
  • Ensemble comparison: Deep ensembling and uniform fusion are equivalent at the declared margin, although they are not identical.The deep ensemble is slightly better on mAP by +0.0008, below the smallest effect treated as meaningful.
  • Scope: This equivalence does not establish that heterogeneous fusion and homogeneous ensembling are interchangeable in general.The supported claim is limited to the present study’s resolution and margin.
Loading 2609.09185v1…