PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniMed-FL: A Robust Multimodal Federated Learning Framework for Clinical Diagnosis

Ayush Debnath, Ruelia Saha, Sudip Misra

arXiv:2609.10364v1cs.LGcs.AI

TL;DR

OmniMed-FL studies multimodal federated learning for five-class clinical classification, addressing the challenge of combining imaging and patient-record context across distributed healthcare sites. Using controlled proxy corpora and federated benchmarks, it finds that multimodal fusion leads the evaluated text and image alternatives, while label skew, communication cost, and synthetic non-patient-level pairing bound interpretation.

  • Problem

    Clinical diagnosis may require imaging and patient-record context, while distributed healthcare data complicates centralized multimodal learning.

  • Method

    OmniMed-FL benchmarks multimodal federated models, fusion strategies, initialization schemes, missing-text imputation rules, and federated baselines on controlled proxy corpora.

  • Results

    Multimodal fusion achieves the best overall score in the evaluated setting, with concatenation reaching F1=0.956 versus F1=0.934 for text and F1=0.664 for vision at α=1 and K=5.

  • Takeaways & Limitations

    Label skew matters more than client count, communication grows linearly with participating clients, and multimodal fusion adds model-state, runtime, and memory costs.

  • Takeaways & Limitations

    The results are descriptive proxy comparisons because the client-count study is sequential, traffic estimates are analytic, and the clinical data do not estimate diagnostic safety, generalization, or deployment readiness.

Abstract

from arXiv · show

Simultaneous assessment of medical imaging and patient records is often required in clinical diagnosis. However, standard machine learning algorithms cannot analyze these data types together. Meanwhile, compliance with HIPAA and GDPR can constrain centralized aggregation of sensitive patient data. This leaves a crucial void of secure fusion of visual and textual context across distant networks. Thus, we present OmniMed-FL, a controlled systems study of multimodal federated learning for five-class clinical condition classification (Normal, Pneumonia, COVID-19, Pleural Effusion, Cardiomegaly). Our proxy corpus pairs 3,000 public chest radiographs with 3,000 class-conditioned synthetic notes, matched by class, not by patient. The framework benchmarks eight fusion strategies, three initializations, four missing-text imputation rules, and matched federated baselines under non-IID Dirichlet partitioning across 3 to 20 hospital clients. As all notes are synthetic and pairing is not patient-level, these are descriptive proxy comparisons, not estimates of diagnostic performance or deployment readiness. Within those limits with clients ($K=5$) and severe skew ($α=0.1$), local-only training achieves a macro-F1 score of 0.297, FedAvg achieves $0.662\pm0.074$, FedProx $0.737\pm0.085$, a matched FedMME-style one-shot ensemble $0.647\pm0.080$, and our SCAFFOLD-AdamW adaptation $0.070\pm0.015$, the 0.075 FedProx-FedAvg gap falling inside the wider of the two two-seed standard deviations. Over a $4\times3$ grid, label skew costs up to 0.27 F1 whereas a near-sevenfold client increase costs at most 0.10, while bidirectional volume grows linearly to 183.5 GiB at $K=20$. Multimodal fusion leads on both corpora, scoring 0.956 against 0.934 for text and 0.664 for images on the synthetic corpus and 0.906 against 0.880 and 0.737 on the radiograph corpus, for $2.3\times$ the model state of text alone.

I. INTRODUCTION

OmniMed-FL addresses the gap between multimodal clinical assessment and federated data locality through a controlled five-class benchmark. It evaluates multimodal fusion, label skew, and federation costs while limiting claims to proxy comparisons.

  • Motivation: Multimodal fusion combines radiographs with clinical notes to address contextual information missed by single-modality systems.Image-only models miss cues such as BNP elevation and viral contact history, while text-only models miss radiological patterns.
  • Motivation: Federated learning keeps patient records at their originating institutions while enabling collaboration through model updates.The framework targets multimodal collaboration without exposing patient data.
  • Contributions: 0.956 fusion score exceeds text-only performance of 0.934 on the synthetic corpus and is reported as the leading result across both corpora.The study also evaluates how label skew, majority-class collapse, fusion cost, and communication cost affect the system.
  • Contributions: The study builds a controlled five-class corpus whose notes mix wording across classes rather than exposing labels through a single keyword.This enables comparison of how much class information models recover from wording alone.
  • Contributions: Federated training is evaluated for local-data cost, including severe label skew and the runtime, memory, and communication associated with each configuration.The contribution frames severe skew as a principal accuracy concern alongside systems costs.

II. RELATED WORK

Prior clinical systems generally provide either centralized multimodal modeling or federated single-modality learning. OmniMed-FL combines visual–textual fusion with federated data locality.

  • Centralized and federated clinical learning: Centralized chest-radiograph systems established strong vision baselines but do not incorporate the clinician’s accompanying note.CheXNet, CheXpert, and later Vision Transformers all pool data centrally.
  • Centralized and federated clinical learning: Clinical federated learning supports multi-institutional collaboration without shared patient records, but the cited deployments remain unimodal.FedAvg, FedProx, and SCAFFOLD address decentralized training and client heterogeneity without adding the second modality.
  • Vision–language models: Centralized biomedical vision–language models jointly model images and text, whereas multimodal federated methods remain newer and use distinct data and budget settings.FedMME uses one-shot voting and P-FIN addresses missing features with calibrated uncertainty.
  • Synthesis: OmniMed-FL embeds text–image fusion inside a federated loop while leaving each record at the hospital holding it.This combines the multimodal modeling direction with federated data locality.

III. SYSTEM DESIGN

The system encodes radiographs and notes separately, combines their representations with one of eight fusion rules, and trains the fused classifier through sample-weighted federated optimization. Anti-collapse regularization and local AdamW updates are applied within each client.

  • Multimodal architecture: Eight fusion rules combine projected DistilBERT note representations with mean-pooled ViT image representations.The rules include three non-attention designs and five attention-based designs.
  • Federated objective: Clients hold disjoint shards, and the sample-weighted empirical risk trains both encoders, projection layers, the fusion operator, and classification head.The server receives trainable tensors rather than client shards.
  • Federated objective: The fused classifier maps a radiograph, its class-paired note, and the class label into a five-class training objective.The prose explicitly identifies x as the radiograph, z as the note, y as the label, and fθ as the fused classifier.
  • Anti-collapse design: The entropy-diversity term penalizes concentrated predictions above normalized entropy 0.7, while a fixed confidence penalty remains active.The class-balanced sampler and entropy-diversity term are the two anti-collapse components tested in ablations.
  • Federated training: Each eligible client performs three local AdamW epochs on class-balanced batches, clips gradients, and returns trainable tensors for sample-weighted aggregation.Operational runs initialize from public DistilBERT and ViT-Base/16 weights with random task heads.

C. Controlled Proxy Corpus

OmniMed-FL uses two balanced proxy corpora with shared synthetic notes but different image sources. The corpus supports controlled internal comparisons, not absolute clinical claims, because image–note pairs are class-matched rather than patient-matched.

  • Corpus construction: Dataset A contains 3,000 procedurally generated X-ray-style images, while Dataset B replaces them with public radiographs and retains the notes.Both datasets are reported separately, with Dataset B containing 600 examples per class.
  • Corpus construction: Dataset B contains 3,000 public radiographs balanced at 600 examples per class and split into 2,400 training and 600 validation examples.The images come from three public collections, leaving image source partly confounded with class.
  • Synthetic notes: The text modality contains no patient EHR text and uses template-generated clinical-style notes assembled from observation, symptom, context, and indicator slots.The notes are synthetic rather than extracted from clinical records.
  • Federated workflow: The architecture separates hospital-side computation from the FedAvg aggregation server, with only model updates crossing the client boundary.Figure 1 presents this workflow at the reference K=5 setting.
  • Synthetic notes: 25% of notes use target-class slots, 35% mix target and non-target slots, and 40% are heavily mixed while retaining a probabilistic target indicator.The design reduces direct label wording but does not create patient-level image–note pairs.
  • Scope and limitations: The corpus supports internal comparison rather than absolute claims because templates may leave shortcuts, pairing removes real discordance, and classes come from different source collections.The split is grouped by neither patient nor source.

D. Experimental Setup

The experimental setup fixes a common optimizer and training protocol, then varies client partitioning and matched multimodal baselines under controlled conditions.

  • Protocol: AdamW uses a 10^-4 learning rate and 0.01 weight decay for three local epochs per round, across eight rounds with batch size 16 in FP32.
  • Partitioning: Dirichlet partitioning allocates each class across clients, with smaller α concentrating client label distributions and α →∞ approaching IID allocation.
  • Partitioning: The split remains fixed across rounds, while shards with fewer than four examples skip local optimization but still contribute the unchanged global state at sample weight.
  • Matched baselines: Matched controls preserve the corpus, shard profile, encoders, optimizer, and ablation settings while comparing FedMME-style voting, P-FIN-style imputation, and FedProx.The FedMME-style baseline uses one upload per client; the P-FIN-style stress test compares four missing-text or aggregation rules.

A. Matched Baselines under Severe Skew

Under severe skew, matched federated baselines separate sharply, while ablations show trade-offs among accuracy, class coverage, fusion choice, initialization, and missing-text handling.

  • Matched baselines: 0.297 macro-F1 is achieved by local-only training, versus 0.662±0.074 for FedAvg and 0.737±0.085 for FedProx under severe skew.The 0.075 FedProx–FedAvg gap falls inside the wider of their two sample standard deviations.
  • Matched baselines: 0.070±0.015 macro-F1 is obtained by the SCAFFOLD–AdamW adaptation, while matched-budget FedMME-style voting reaches 0.647±0.080.The native 100-epoch FedMME-style run reaches 0.609±0.109, so extra local computation does not close the gap to iterative aggregation.
  • Anti-collapse: 0.683±0.089 F1 with minimum diversity 0.90 is produced by the full anti-collapse stack, whereas removing sampler and entropy-diversity components raises F1 to 0.758±0.016 but lowers minimum diversity to 0.70.The stack therefore trades accuracy for broader predicted-class coverage under severe skew.
  • Fusion: 0.759 is the highest mean among eight fusion rules, but its uncertainty exceeds the entire between-rule spread, preventing a reliable ranking.Projected concatenation, used elsewhere as the default, scores 0.659±0.000.
  • Initialization: 0.831±0.001 final F1 is reached by public encoders with random task modules, compared with 0.647±0.022 for all-random initialization.Pooled-start reaches 0.877±0.007 but reuses a checkpoint selected on the reporting split and sits outside the shared budget.
  • Missing text: 0.688±0.025 to 0.706±0.010 spans the four forced-missing-text imputation results, with uncertainty-weighted aggregation tying probabilistic imputation but showing the widest seed swing.The four means differ by at most 0.018, and the uncertainty-weighted rule ranges from 0.770 to 0.643 across seeds.

C. Retrieval-Stage Evaluation for Retrieval-Augmented Generation (RAG)

TF–IDF retrieval produced moderate same-label accuracy and precision despite high lexical similarity, indicating that neighboring synthetic notes can be textually close without sharing labels.

  • 0.540 condition-macro top-1 retrieval accuracy exceeded the 0.2 label prior but remained below the 0.707 mean top-1 cosine similarity.The five condition rows covered all 600 validation queries.
  • 0.468 same-label precision@5 contrasted with 0.707 mean top-1 cosine similarity, indicating lexically close neighbors were not always same-label.

D. Scalability, Communication, and Resource Cost

Label skew has a larger effect on Macro-F1 than client count, while communication volume grows linearly with clients. Multimodal fusion improves radiograph performance but adds model-state, runtime, and memory costs.

  • Scalability: Label skew dominates client count: α=5 cells score 0.849–0.916, α=1 cells 0.818–0.862, and α=0.1 cells 0.643–0.738.Within a skew row, changing K by nearly sevenfold shifts mean F1 by at most 0.10.
  • Resource cost: 41–66 s per round and 4.77–5.92 GiB peak memory characterize the timed block on one shared H100 NVL.The measurements are not scaling curves; sequential simulation and shared-server contention explain the spread.
  • Communication: 27.5/45.9/91.8/183.5 GiB are the nominal bidirectional volumes for K=3/5/10/20, respectively.The plotted volume uses a 615,742,484-byte model state, four bytes per parameter, and eight rounds.
  • Communication: 0.067 F1 is lost when K rises from 3 to 20 at α=5, while traffic increases 6.7×.The study reports communication growing linearly in K while accuracy remains nearly flat.
  • Resource cost: 0.906 multimodal F1 exceeds 0.880 for text and 0.737 for images on Dataset B.Multimodal fusion costs 2.3× the text branch's model state, 3.1× its wall time, and 2.1× its memory.

V. DISCUSSION AND LIMITATIONS

The study’s strongest limitation is sensitivity to non-IID partitioning, especially severe label skew. Its simulated, analytic, and proxy evaluation settings constrain conclusions about deployment, privacy, and real-world clinical performance.

  • Partition sensitivity: At α=0.1, identically configured FedAvg reruns spread by ±0.159 F1, limiting robust aggregation under severe label skew.Down-weighting anomalous updates may also discard genuine minority-class signal, but evaluating that trade-off requires an attacker model not defined here.
  • Evaluation bounds: The client-count study omits concurrency, latency, stragglers, dropout, and heterogeneous accelerators because it uses sequential simulation.Equation (3) and Fig. 9(b) report analytic costs rather than end-to-end traffic.
  • Clinical scope: Synthetic notes, label-level rather than patient-level pairing, and non-patient/source-grouped splits prevent F1 and retrieval scores from estimating diagnostic safety or deployment readiness.Two seeds make the reported spreads descriptive, and larger multi-site data, more seeds, native-method replications, and privacy testing remain future work.

VI. CONCLUSION

OmniMed-FL benchmarks multimodal federated learning for five-class clinical condition classification. Multimodal fusion leads the evaluated modalities, while scalability favors attention to label skew and communication costs; external validation and safeguards remain necessary.

  • Benchmark: 18 model variants are benchmarked for five-class clinical condition classification.At α=1 and K=5, the benchmark compares text, vision, multimodal, initialization, and anti-collapse configurations.
  • Multimodal performance: 0.956 F1 for concatenation exceeds 0.934 for text and 0.664 for vision on the synthetic corpus.The multimodal score improves by 0.022 over text and 0.292 over vision.
  • Scalability: Label skew matters more than client count, communication grows linearly with participating clients, and multimodal fusion adds model-state, runtime, and memory costs.These conclusions summarize the benchmark’s scalability and resource analyses.
  • Scope: External clinical validation and safeguards remain necessary before deployment in telehealth or diagnostic workflows.Future work includes patient-paired multi-institutional radiology data, differential privacy auditing, asynchronous aggregation, and end-to-end communication measurements.
Loading 2609.10364v1…