Source-linked AI summary
FedCoRe: Target-Adaptive Completion for Missing Modalities in Healthcare Federated Learning
Holger R. Roth, Ziyue Xu, Peter Cnudde
TL;DR
Federated multimodal learning must accommodate hospitals with different available modalities rather than assuming complete inputs. FedCoRe learns validation-gated representation or logit corrections from paired examples, restoring 49.0% of the missing-ECG performance gap and 52.8% of the CXR gap in controlled tests.
Problem
Federated multimodal evaluation often assumes complete modalities, although hospitals may provide different modality subsets at deployment.
Method
FedCoRe learns modality-specific task-aware corrections from valid paired examples, aggregates eligible client updates, and uses validation to apply correction or retain unchanged predictions.
Results
49.0% of the missing-ECG gap and 52.8% of the CXR gap were recovered in controlled tests, while validation retained no-completion for unsupported CXR deployment cases.
Takeaways & Limitations
FedCoRe supports missing-modality signal recovery when paired examples and validation evidence support the modality, but otherwise retains the unchanged prediction.
Takeaways & Limitations
The experiments impose controlled modality masks rather than natural missingness or institutional covariate shift, and results remain model-development evidence.
Abstract
from arXiv · showhide
Federated multimodal models often assume every site has every modality, although hospitals differ in access to EHRs, chest radiographs, and ECGs. We study this setting on a MIMIC-derived respiratory deterioration task with simulated FL clients and introduce FedCoRe (Federated Cross-Modal Representation Completion). FedCoRe learns representation- or logit-space corrections rather than generating synthetic ECGs or CXR images. When a client observes a modality that may be missing at deployment, it evaluates the same example with and without that modality to obtain paired supervision. Only clients with such pairs update the completion module, and validation may retain the unchanged prediction. We freeze the trained multimodal predictor during evaluation so that measured differences come only from completion. Hiding ECG reduced AUROC by about 0.085; paired-example FedAvg restored 0.0415 AUROC, or 49.0% of the lost performance. We therefore report two distinct effects: paired-example FedAvg partially recovers the missing-ECG gap, while validation-selected completion is a task-specific classifier-logit correction rather than literal ECG recovery. For CXR, effect-aware completion recovers 52.8% of the loss in a controlled test where CXR is hidden. Paired-example FedAvg transfers part of this effect, but validation keeps the no-completion baseline for deployment cases whose inputs lack CXR. Thus, FedCoRe should be read as a validation-gated completion/correction framework: it can recover missing-modality signal in supported settings, but it should be deployed only when paired examples and validation evidence support that modality.
1 Introduction
Federated multimodal clinical prediction faces a deployment gap because hospitals and patients may lack different modalities. FedCoRe addresses this through validation-gated, task-aware corrections learned from clients with valid modality pairs, without generating synthetic modalities or centralizing raw records.
- Motivation: Complete-modality evaluation can overestimate federated-learning performance because hospitals and patients may have different subsets of EHR, CXR, and ECG.Figure 1 contrasts complete-modality evaluation with heterogeneous federated deployment.
- Approach: FedCoRe defines completion as a learned residual in representation or classifier-logit space rather than synthetic ECG or CXR generation.The framework separates robust predictor training from the completion operator itself.
- Approach: FedCoRe updates each task-aware correction only from clients with valid modality pairs, aggregates updates, and uses validation to apply correction or retain unchanged predictions.This design targets lightweight collaborative correction while keeping raw records decentralized.
- Related Work: Prior multimodal federated-learning approaches address missing modalities through pseudo-modality generation, synthesis, retrieval, ensembles, embedding transfer, and prototype or mask completion.FedAvg is identified as the canonical decentralized optimization algorithm, with Flower and NVIDIA FLARE supporting healthcare FL coordination.
2 Method
FedCoRe learns modality-specific corrections from paired examples, operating in representation or classifier-logit space without requiring every client to supervise every modality. It aggregates only valid updates and uses validation to select completion or preserve unchanged predictions.
- Paired supervision: Paired supervision requires examples containing target modality t alongside at least one other modality, evaluated with and without t.Examples lacking t or containing only t cannot supervise its completion.
- Completion operator: FedCoRe predicts either a hidden-state residual or an additive classifier-logit residual, using the t-present pass as the training reference.The formulation is backbone-agnostic and requires paired t-present and t-removed representations or logits.
- Local objective: The local objective combines task prediction, alignment between completed and t-present logits, and representation regularization.For the main missing-ECG result, the representation-regularization weight is set to λrep = 0 after validation.
- Aggregation by available supervision: Clients contribute completion updates only when they have valid paired sets, with server aggregation weighted by the number of valid pairs.Clients with no valid pairs return no completion weights and receive zero aggregation weight, preventing them from diluting the global operator.
- Validation-based deployment: Validation selects an operator source, completion strength, and optional gate threshold, while the no-completion candidate leaves predictions unchanged if no candidate qualifies.Validation can use missing-target cases and optionally a protected safety view.
3 Task and Federated Setup
The study builds a MIMIC-derived multimodal benchmark linking EHR context, CXR images, and ECG records to predict respiratory deterioration within 48 hours. It simulates heterogeneous federated clients with example- and patient-level missingness under patient-disjoint evaluation splits.
- Cohort and label: The benchmark links structured EHR context, chest radiographs, and diagnostic ECG records from public MIMIC-derived PhysioNet datasets.Sources include MIMIC-IV, MIMIC-CXR, MIMIC-CXR-JPG, and MIMIC-IV-ECG.
- Cohort and label: The binary label is respiratory deterioration within 48 hours after the later ECG–CXR index timestamp.Deterioration comprises future invasive ventilation, ICU transfer, or all-cause in-hospital death.
- FL modality heterogeneity: Federated clients simulate both client-level missingness and patient-mixed missingness, with each example observing a subset of EHR, CXR, and ECG modalities.Patient-disjoint splits are maintained across training, validation, testing, and FL clients; Table 1 defines templates for missing-ECG completion testing.
4 Experiments
The experiments compare completion strategies under frozen multimodal predictors, using validation-only selection and modality-specific missing-input tests. They evaluate AUROC-centered performance changes with complementary deployment and uncertainty metrics.
- Baselines and controls: The study compares no completion, local-only completion, paired-example FedAvg completion, and validation-selected completion.CXR controls additionally include server-selected and model-soup sources, scalar logit recalibration, pseudo-CXR tokens, and fixed-scale completion.
- CXR configuration: Missing-CXR experiments use a frozen Qwen3-VL-8B-Instruct predictor with LoRA rank 8 adapters, pre-index EHR context, and CXR images.The effect-aware gated MLP uses hidden size 4096, dropout 0.25, learning rate 5×10−5, 80 local epochs, and batch size 128.
- Selection protocol: Validation selects completion source and strength using missing-modality validation data, always including α = 0 as the no-completion option.ECG selects α ∈{0, 2, 4, 8, 12} subject to nonnegative gain, then chooses the source with greatest validation gain.
- Evaluation protocol: The frozen predictor remains unchanged across ECG-present and missing-ECG tests, so Table 2 varies completion sources and paired-example patterns rather than predictor inputs or weights.Test metrics are summarized only after validation fixes the completion choice and are never used for selection.
- Metrics: AUROC is the primary endpoint for absolute change among cases missing the target modality, supplemented by relative lift and aggregate deployment AUROC change when available.Main ECG/CXR analyses also report AUPRC and patient-level paired-bootstrap 95% confidence intervals conditioned on the validation-selected operator.
5 Results & Discussion
FedCoRe’s completion effects vary by missing modality and evaluation setting: paired-example methods partially recover ECG and CXR signal, while validation can retain baseline predictions when corrections lack deployment support. Frozen-predictor tests show ECG correction may exceed recovery of the original modality, whereas unsupported CXR completion is not deployed.
- Overall modality results: ECG completion improves 69/70 exploratory rows, with mean +0.059 AUROC and +9.2% relative lift; missing EHR improves 20/20 rows with mean +0.027.Missing-CXR effects depend more strongly on the backbone and completion interface, so these results are not a ranking of clinical importance.
- ECG recovery and correction: 0.0415 AUROC is added by paired-example FedAvg, reaching 0.6195 and recovering 49.0% of the lost missing-ECG performance.The frozen predictor and patients remain unchanged; only the completion operator differs.
- ECG recovery and correction: 0.1455 AUROC is gained by validation-selected ECG completion, reaching 0.7235 and exceeding the ECG-present reference.The result is interpreted as classifier-logit residual correction rather than literal ECG reconstruction; AUPRC remains unstable with 42 positive test examples.
- Additional modality tests: 0.095 mean AUROC is gained by FedAvg in a single missing-ECG scenario, versus +0.116 for validation-selected completion across six seeds.Missing-EHR completion is weaker but non-harmful at +0.014 across three seeds.
- CXR recovery and deployment: 0.087 AUROC is added by effect-aware CXR completion across five seeds, recovering 52.8% of the full-vs-missing gap.Four of five seeds are positive, but seed 23 leaves predictions unchanged and only two per-seed intervals exclude zero.
- CXR recovery and deployment: 0.034 AUROC is gained by paired-example FedAvg in 8/12 remove-CXR rows, but validation selects α = 0 when deployment inputs lack CXR.FedCoRe therefore retains the no-completion baseline for unsupported CXR corrections; server-selected sources yield +0.017 and source-subset operators are negative in 0/12 rows.
6 Conclusion
FedCoRe learns modality-specific completion operators from valid paired examples, excludes uninformative clients, and can preserve unchanged predictions. Completion benefits missing-ECG cases and reveals recoverable missing-CXR signal only in supported remove-modality tests, while broader validation remains future work.
- 6 Conclusion: FedCoRe learns modality-specific operators from valid pairs, excludes uninformative clients from aggregation, and can retain the unchanged prediction.These mechanisms define its validation-gated completion behavior.
- 6 Conclusion: Missing-ECG cases benefit from completion, whereas missing-CXR cases show recoverable signal in remove-CXR tests but not unsupported deployment cases.The conclusion distinguishes supported controlled tests from deployment settings lacking evidence for completion.
- 6 Conclusion: Controlled four-client masks impose modality availability rather than modeling natural missingness or institutional covariate shift.External multi-site validation remains future work.