Source-linked AI summary
Reliability-Aware Probabilistic Fusion for Multimodal Learning with Incomplete Evidence
Patitapaban Palo, Abhisek Tiwari, Mattia Carletti, Anshul Thakur, Xiao Gu, David A. Clifton
TL;DR
Multimodal evidence may be missing or degraded, making availability and trust distinct challenges. The paper uses a reliability-aware Gaussian PoE that tempers available experts and omits missing ones; across two clinical benchmarks, it is competitive with complete inputs and shows its clearest mean-AUROC advantage under missing evidence.
Problem
Multimodal inputs may be entirely missing or present but degraded, raising distinct questions about evidence availability and trustworthiness.
Source: §1 · p. 1Method
A reliability-aware Gaussian PoE separately weights each modality’s evidence and latent variance, omitting experts for missing modalities.
Source: §1 · p. 2Results
Across two clinical benchmarks, the framework is competitive with complete inputs and has its clearest mean-AUROC gain under missing evidence; in HAIM, mean incomplete AUROC rises from 0.8104 (D+C) to 0.8251 (D+C →F).
Source: §6.5 · p. 19, §7 · p. 22Takeaways & Limitations
Missing-aware training supplies much of the robustness, while unseen corruption reveals useful responses and failure cases, leaving dependable quality estimation open.
Source: §9 · p. 23Reliability and expert variance are not separately identifiable from task supervision, so the reliability weight is not a calibrated quality score and can fail under severe degradation.
Source: §8 · p. 23
Abstract
from arXiv · showhide
Multimodal inputs may be missing or degraded, raising distinct questions of evidence availability and trust. We introduce a reliability-aware Gaussian Product-of-Experts (PoE) in which an input-dependent reliability weight tempers each modality's latent precision, while missing experts are omitted. We characterize how reliability and availability affect posterior information and expert influence. Across two clinical multimodal tasks, the full framework remains competitive with complete inputs and achieves its clearest mean-AUROC advantage under modality absence, particularly on HAIM-MIMIC-MM. Ablations show a substantial gain from missing-aware training and an additional HAIM improvement from reliability weighting under matched PoE, missing-aware, and consistency training. Unseen test-time corruptions are used diagnostically and reveal modality-dependent reliability-weight and latent-uncertainty responses, including failure under severe degradation. Thus, reliability weighting provides an explicit evidence-control mechanism, while learning dependable quality signals remains an open challenge.
1 Introduction
Multimodal inputs may be absent or degraded, so models must distinguish evidence availability from trustworthiness. The paper introduces a reliability-aware Gaussian PoE and reports competitive complete-input performance with its clearest mean-AUROC advantage under modality absence.
1 Introduction
- Available evidence raises two distinct questions: whether a modality is present and how much a model should trust it.Inputs may be missing or present but degraded by noise, artifacts, or measurement errors.Source: §1 · p. 1
- Prior incomplete-input methods address missingness through modality dropout, information retrieval, representation learning, or availability-based routing [5] [7].These methods do not necessarily provide an explicit statistical account of how strongly available evidence influences predictions.Source: §1 · p. 1
- Gaussian Product-of-Experts combines modality evidence through expert precision and can support inference from arbitrary modality subsets [9] [10].Precision alone may not reflect trustworthiness when an observed modality is corrupted or misleading [10].Source: §1 · pp. 1–2
- Across five seeds, the framework is competitive with complete inputs and shows its clearest mean-AUROC advantage under modality absence.Corruption analysis finds that learned reliability-weight responses are strongly modality dependent.Source: §1 · p. 2
- The proposed Gaussian PoE separately parameterizes latent variance and an input-dependent reliability weight, while omitting missing experts.The paper characterizes how reliability and modality removal affect expert influence and posterior covariance.Source: §1 · p. 2
2 Related Work
Prior multimodal work addresses incomplete inputs with dropout, information retrieval, representation learning, and routing, while probabilistic fusion combines modality evidence through expert distributions. This paper positions reliability weighting as a separate way to temper available evidence.
Learning with missing modalities
- Incomplete-input methods include random modality dropout, similar-patient information retrieval, shared and modality-specific representations [5], graph-based learning, and availability-specific expert routing [7].These approaches handle missing inputs through reconstruction, representation learning, or routing.Source: §2 · pp. 2–3
Probabilistic fusion, uncertainty, and reliability
- Probabilistic multimodal models combine modality-specific posteriors with products, mixtures, or combinations of experts [9] [10].Gaussian PoE supports inference from different modality subsets, but expert contribution is typically determined by the learned posterior distribution.Source: §2 · p. 3
- The paper separates latent variance, which describes evidence concentration, from an input-dependent reliability weight that tempers evidence entering the PoE.This is an input-dependent tempering of Gaussian experts, not new PoE algebra.Source: §2 · p. 3
3 Reliability-Aware Probabilistic Fusion
The method represents each available modality as probabilistic evidence with separate uncertainty and reliability roles, then fuses only observed experts. It learns reliability through supervised prediction and trains across complete and sampled incomplete modality sets.
3 Reliability-Aware Probabilistic Fusion
- Each available modality is treated as probabilistic evidence with latent uncertainty controlling concentration and reliability weighting controlling its contribution to fusion.The formulation supports arbitrary modality types and observed subsets.Source: §3 · p. 3
3.1 Problem Formulation
- At inference, the model predicts target y from the observed subset O of the possible modalities X while representing uncertainty in available evidence..Source: §3.1 · p. 3
3.2 Probabilistic Modality Evidence
- Each modality is mapped to a common representation using a modality-specific feature extractor, then modeled as a diagonal Gaussian expert.The Gaussian expert supplies a modality-specific posterior for probabilistic fusion.Source: §3.1 · p. 3, §3.2 · p. 3
3.3 Reliability-Aware Product-of-Experts
- Raising each Gaussian expert to its nonnegative reliability weight scales its precision; the resulting experts combine with a Gaussian prior in the PoE.Setting r_m = 1 recovers standard Gaussian PoE, while r_m approaching zero removes that expert’s precision contribution.Source: §3.3 · p. 4
3.4 Learning Reliability
- The reliability head learns an input-dependent evidence weight through supervised fusion, without corruption-ranking supervision.Because fusion depends on the product of reliability and inverse variance, task supervision alone does not identify reliability as calibrated quality.Source: §3.4 · p. 4
3.5 Learning with Missing Modalities
- Training uses complete and sampled incomplete modality sets, forming each posterior only from available experts and using complete predictions as stop-gradient targets for incomplete subsets.The objective includes supervised prediction loss over complete and incomplete sets, with dataset-specific experiments potentially omitting auxiliary terms.Source: §3.5 · p. 5
4 Statistical Properties
The section characterizes how reliability-weighted Gaussian fusion and modality availability shape posterior information and expert influence. Its propositions distinguish omitting an expert from reducing the reliability of an observed expert.
4 Statistical Properties
- The fusion results establish that reliability controls an available expert’s information contribution, while missing experts are omitted from the posterior.The proofs in Section 4.1 establish the propositions stated in the main section.Source: §4 · p. 5, §4.1 · p. 6
4.1.1 Proof of Proposition 1
- The reliability-weighted Product-of-Experts remains Gaussian, with reliability scaling an expert’s information rather than its latent estimate.When every available expert has reliability 1, standard Gaussian PoE is recovered; reliability 0 gives that expert zero information contribution.Source: §4 · p. 5, §4.1.1 · pp. 6–7
4.1.2 Proof of Proposition 2
- Reducing an available expert’s reliability strictly increases posterior variance for each latent coordinate considered.The result follows from the coordinate-wise fused variance for diagonal Gaussian experts.Source: §4 · pp. 5–6, §4.1.2 · pp. 7–8
4.1.3 Proof of Proposition 3
- With retained experts’ reliability fixed, removing experts with nonnegative precision cannot decrease posterior covariance.The proof uses the positive-definite prior precision and reversal of the Loewner order under matrix inversion.Source: §4 · p. 6, §4.1.3 · p. 8
4.1.4 Proof of Proposition 4
- Holding an expert’s precision and reliability fixed, its influence on the fused mean vanishes as its reliability approaches zero.This is a conditional, direct-sensitivity result; it does not include changes in reliability predicted by the model.Source: §4 · p. 6, §4.1.4 · pp. 8–9
5 Experimental Setup
The experiments assess clinical prediction with complete, missing, and corrupted multimodal inputs across two datasets. They compare the proposed model with fusion and incomplete-modality methods using repeated-seed evaluation.
5.1 Datasets and Tasks
- Symile-MIMIC evaluates cardiomegaly classification from CXR, ECG, and laboratory measurements; HAIM-MIMIC-MM evaluates 48-hour mortality from structured data, time series, CXR, and text.HAIM uses patient-level splits and distributed precomputed representations associated with CXR-anchored observations.Source: §5.1 · p. 9
5.2 Comparison Methods
- Comparators include unimodal and fusion baselines, modality dropout, and recent incomplete-modality methods instantiated with dataset-specific encoders and interfaces.The implementations follow the compared methods’ standard methodological training procedures.Source: §5.2 · p. 9
5.3 Missing and Corrupted Modalities
- Evaluation covers every non-empty incomplete modality subset and degrades one observed modality at a time at three severities, using corruption operators absent from training.Comparators retain their standard missing-modality mechanisms and training procedures; component effects are isolated separately.Source: §5.3 · p. 9
5.4 Evaluation Protocol
- Main analyses report AUROC and AUPRC; five-seed comparisons aggregate missingness and corruption conditions within each seed before calculating mean ± sample SD.Brier scores are supplementary probability-error measures and are not interpreted as calibration under class-weighted BCE.Source: §5.4 · p. 9
5.5.1 Model Architectures
- Both datasets use the same probabilistic fusion mechanism, with modality encoders and training settings adapted to their inputs and tasks.The implementations use modality-specific projections and a reliability head; the architecture details differ by dataset.Source: §5.5 · p. 10, §5.5.1 · p. 10
5.5.2 Training Objectives
- Symile uses task, unimodal, and consistency losses, whereas HAIM Full-F uses missing-aware mortality training and complete-to-incomplete consistency without the unimodal auxiliary term.Reliability is learned through fusion and the supervised task objective in both settings.Source: §5.5.2 · p. 10
5.5.3 Missing-Modality Sampling
- Symile samples all seven non-empty modality configurations; HAIM samples missing-modality patterns for supervised training and uses a separate incomplete subset for consistency.HAIM’s consistency branch samples one to three missing modalities and never uses the complete-modality case.Source: §5.5.3 · p. 11
5.5.4 Optimization and Model Selection
- Symile trains for 10 epochs, while HAIM uses early stopping on validation AUROC; final results aggregate five independently trained seeds.Both settings clip gradients to maximum norm 5.Source: §5.5.4 · p. 11
5.6 Corruption Protocols
The protocols distinguish a missing modality, which contributes no expert, from a corrupted modality, which remains observed and may provide unreliable evidence. They apply controlled, modality-specific corruption at three severities and match HAIM perturbations across methods.
5.6 Corruption Protocols
- One modality is corrupted at a time at mild, moderate, or severe intensity, while other modalities remain unchanged.Missingness instead removes that modality's expert.Source: §5.6 · p. 11, §5.6.1 · p. 11
5.6.1 Symile-MIMIC Corruptions
- Symile corruptions are modality-specific: CXR combines smoothing and Gaussian noise, ECG adds noise followed by clipping, and laboratory values are progressively replaced by random values.At severe corruption, laboratory values are fully replaced while missingness indicators remain unchanged.Source: §5.6.1 · p. 12
5.6.2 HAIM-MIMIC-MM Corruptions
- HAIM applies feature-space corruption to its precomputed representations, with masking and additive Gaussian noise controlled by severity.Four modalities and three severities yield 12 corruption conditions per model and seed.Source: §5.6.2 · p. 12
5.6.3 Matched Corruption Across Methods
- HAIM methods receive the same feature mask and Gaussian-noise realization for each corruption condition, reducing variation from different random perturbations.The random corruption seed depends on experimental seed, modality, and severity, not method identity.Source: §5.6.3 · p. 13
5.7 Dataset and Cohort Details
The evaluation uses Symile-MIMIC and HAIM-MIMIC-MM, with dataset-specific cohorts and implementations of comparison methods. Methods share partitions and held-out conditions while retaining their defining fusion and missing-modality mechanisms.
5.7.1 Symile-MIMIC
- Symile-MIMIC evaluates cardiomegaly prediction from CXR, ECG, and laboratory measurements, applying a shared valid-label mask and preserving the released partitions.Admission identifiers align modalities; no admission identifiers overlap across partitions.Source: §5.7.1 · p. 13
5.7.2 HAIM-MIMIC-MM
- HAIM-MIMIC-MM evaluates 48-hour mortality across structured, time-series, CXR, and text modalities in a subject-level partitioned cohort.The cohort has 45,036 observations from 6,486 subjects, including 1,229 positive observations (2.73% prevalence).Source: §5.7.2 · pp. 13–14, Table 6 · p. 14
5.7.3 Comparison of Evaluation Settings
- Symile provides 7 non-empty modality configurations, including 6 incomplete ones; HAIM provides 15, including 14 incomplete ones.The datasets span three modalities for an imaging-related finding and four heterogeneous groups for a patient outcome.Source: §5.7.3 · p. 14
5.8.5 Evaluation Fairness
- Methods use the same cohorts, partitions, and held-out conditions but follow their standard training mechanisms rather than a common imposed objective.The benchmark compares complete learning methods; controlled component comparisons are reported separately.Source: §5.8 · p. 14, §5.8.5 · pp. 15–16, Table 8 · p. 16
5.8.1 DrFuse
- DrFuse preserves shared/private representations, alignment, disentanglement, attention fusion, and auxiliary prediction losses, adapting its input branches to each dataset.Unavailable modalities are masked from HAIM fusion.Source: §5.8.1 · pp. 14–15
5.8.2 M3Care
- M3Care estimates missing representations from neighboring training examples using the modalities that remain observed.HAIM validation and test examples query a frozen training-only memory and are never inserted into it.Source: §5.8.2 · p. 15
5.8.3 MUSE
- MUSE represents available modalities as graph nodes and uses learned message passing with contrastive graph-view regularization.Missing modalities are excluded from the graph rather than encoded as zero-valued inputs.Source: §5.8.3 · p. 15
5.8.4 Flex-MoE
- Flex-MoE uses missingness-aware sparse expert routing, retaining its missing-modality bank, combination-aware routing, and load-balancing objective.The implementations use 16 experts with top-4 routing on Symile and eight experts with top-2 routing on HAIM.Source: §5.8.4 · p. 15
6 Results
Across two clinical multimodal tasks, the framework is competitive with complete inputs and shows its clearest mean-AUROC advantage under modality absence, particularly on HAIM. Corruption and ablation analyses qualify that result and examine evidence weighting.
6.1 Complete-Modality Performance
- With complete modalities, the proposed model is comparable to the strongest methods in AUROC on Symile (0.7583) and HAIM (0.8694).On HAIM, Flex-MoE has higher AUPRC (0.2943 ± 0.0127 versus 0.2427 ± 0.0121).Source: §6.1 · p. 16
6.2 Robustness to Missing Modalities
- Across HAIM's 14 incomplete configurations, mean AUROC is 0.7980 ± 0.0077 for the proposed model, versus 0.7825 ± 0.0096 for Flex-MoE and 0.7747 ± 0.0129 for modality dropout.On Symile, mean AUROC is 0.7014 versus 0.7009 for DrFuse and 0.6934 for Flex-MoE; Table 8 summarizes held-out performance.Source: §6.2 · p. 16
- On HAIM with only one modality available, AUROC is 0.7429 ± 0.0111 for the proposed model, versus 0.7166 ± 0.0064 for Flex-MoE and 0.7092 ± 0.0105 for modality dropout.The proposed model also leads with two and three available modalities; Figure 2 groups performance by available-modality count, and Figure 3 contrasts clean and missing AUROC.Source: §6.2 · pp. 16–17, Figure 2 · p. 17, Figure 3 · p. 17
6.3 Robustness to Corrupted Evidence
- Under corruption, the proposed model averages about 0.759 AUROC on Symile, while HAIM mean corrupted-input AUROC is 0.8558 ± 0.0066 versus 0.8589 ± 0.0115 for Flex-MoE.On HAIM, structured and time-series corruption is more harmful than comparable CXR or text perturbation; Figure 4 summarizes HAIM corrupted-modality AUROC.Source: §6.3 · pp. 17–18, Figure 4 · p. 18
6.4 Reliability and Uncertainty under Corruption
- Corruption responses vary by modality: Symile laboratory reliability falls from 0.366 to 0.107 as latent variance rises from 2.98 to 5.85, while severe ECG corruption raises reliability despite lower AUROC.On HAIM, severe structured corruption raises variance from 1.11 to 1.36 as reliability falls from 0.951 to 0.944; reliability is not a universally calibrated corruption score.Source: §6.4 · pp. 18–19
6.5 Component Ablation
- On HAIM validation data, mean incomplete AUROC rises from 0.7856 for Gaussian PoE to 0.8120 with missing-aware training, then to 0.8251 when reliability is added under matched training.The full model also attains Symile clean and mean-missing AUROC of 0.7835 and 0.7210; Table 9 reports validation ablations.Source: Table 9 · p. 18, §6.5 · p. 19
6.6 Additional Symile-MIMIC Results
- Across Symile's six incomplete conditions, the proposed model has mean AUROC 0.7014 and is strongest when ECG, Labs, or ECG+Labs are absent, but does not dominate every configuration.For corruption, severe ECG degradation is the clearest Symile decline, while laboratory corruption does not produce monotonic AUROC loss; Tables 11 and 12 summarize these results.Source: §6.6.2 · p. 20, §6.6.3 · p. 20, Table 11 · p. 20
6.7.1 Paired Seed-Level Analysis of HAIM Missing-Modality AUROC
- HAIM's paired seed-level AUROC difference versus Flex-MoE is positive in all five seeds, averaging +0.0155, with a descriptive 95% interval of [+0.0022, +0.0288].The reliability analysis also finds modality-dependent responses under unseen corruption; detailed HAIM results aggregate five seeds.Source: §6.7 · pp. 20–21, §6.7.1 · p. 21, §6.8 · p. 21
7 Discussion
The framework’s clearest gain is mean AUROC under missing evidence, while complete-input performance remains competitive. Corruption tests show inconsistent reliability responses, including a severe-degradation failure, so dependable suppression of degraded evidence remains unresolved.
7 Discussion
- The framework’s clearest gain is mean AUROC under missing evidence, with competitive complete-input performance.Missing-aware training provides the largest initial robustness gain, with an additional reliability benefit in the matched five-seed HAIM development ablation.Source: §7 · p. 22
- Under unseen corruption, Symile laboratory reliability weights decrease strongly, while severe ECG corruption produces a failure case.HAIM reliability weights are comparatively stable, and Flex-MoE has higher mean corruption AUROC.Source: §7 · p. 22
- The fusion rule provides explicit evidence control, but learning when to suppress degraded evidence remains unresolved.Source: §7 · p. 22
8 Limitations
The reliability weight is not established as a calibrated quality score, and reliability responses can fail under severe degradation. Evaluation is bounded to two clinical benchmarks and synthetic corruptions, with additional representation and calibration caveats.
8 Limitations
- Task supervision does not identify the reliability weight as a calibrated quality score because it and expert variance jointly determine effective precision.Source: §8 · p. 23
- Reliability-weight responses can fail under severe degradation, and HAIM precomputed representations may not share the CXR anchor’s temporal boundary across every feature component.Source: §8 · p. 23
- Evaluation covers two clinical benchmarks and synthetic corruptions, uses dataset-specific comparison implementations, and lacks output-level calibration analysis of fused latent variance.Source: §8 · p. 23
9 Conclusion
The paper introduces reliability-aware Gaussian PoE fusion with input-dependent evidence weighting and omission of missing experts. Across two clinical benchmarks, it is competitive with complete inputs and has its clearest mean-AUROC advantage under modality absence, while dependable quality estimation remains open.
9 Conclusion
- The framework is competitive with complete inputs across two clinical benchmarks and shows its clearest mean-AUROC advantage under modality absence.Source: §9 · p. 23
- Missing-aware training drives much of the robustness, with an additional reliability benefit in the matched HAIM development ablation.Source: §9 · p. 23
- Unseen corruption reveals useful responses and failure cases, leaving dependable quality estimation open.Source: §9 · p. 23
AI Use Statement
The statement describes GPT-5.6 Sol’s assistance with manuscript preparation and assigns responsibility for scientific content to the authors.
AI Use Statement
- GPT-5.6 Sol assisted with manuscript editing, clarity, organization, and LATEX preparation.Source: AI Use Statement · p. 24
- The authors state that they reviewed and verified the scientific content and take full responsibility for the paper.Source: AI Use Statement · p. 24