Source-linked AI summary
RiVaT-Fuse: Reliability-Calibrated Variational Tensor Fusion for Multimodal Prediction under Modality Uncertainty
Yingfan Xu, Tieming Liu, Ye Liang, Taiping Liu
TL;DR
Multimodal prediction must fuse image and metadata evidence whose reliability varies across samples, while existing fusion methods leave the fused representation semantically under-specified. RiVaT-Fuse estimates a consensus latent state through a reliability-calibrated variational framework with matrix-valued trust and structured interactions. On an image-level benchmark, it achieves the strongest overall predictive profile among direct representation-level baselines and improves perturbation stability.
Problem
Existing multimodal fusion methods typically select an aggregation architecture without defining the fused representation as a reliability-calibrated latent state under sample-varying modality uncertainty.
Method
RiVaT-Fuse defines fusion as sample-wise latent-state estimation using a variational objective that combines modality evidence, structured cross-modal interaction, stability, robustness, and task coupling.
Results
RiVaT-Fuse achieves the strongest overall predictive profile among direct representation-level comparisons and the lowest mean label-flip rate, probability shifts, and binary-task-2 Brier score.
Takeaways & Limitations
Reliability-calibrated latent fusion provides a representation-level alternative to heuristic aggregation while combining predictive strength with improved perturbation stability.
Takeaways & Limitations
The empirical study centers on one primary image–metadata benchmark and targets image-level acquisition prediction rather than patient-disjoint deployment evaluation.
Abstract
from arXiv · showhide
Image-metadata prediction requires fusing heterogeneous evidence whose reliability can vary across samples and latent factors. Existing representation-level fusion methods typically choose an aggregation architecture, such as concatenation, gating, conditional modulation, or attention, without explicitly defining what the fused representation should mean under modality uncertainty. We propose RiVaT-Fuse, a reliability-calibrated variational tensor fusion framework that defines fusion as sample-wise latent-state estimation. Rather than producing a fused vector by direct aggregation, RiVaT-Fuse estimates a consensus latent state through a variational objective that balances image evidence, metadata evidence, structured cross-modal interaction, and stability. The resulting framework replaces scalar modality confidence with matrix-valued trust geometry, decomposes interaction into additive, multiplicative, and relational components, and couples the latent state with conditional robustness and structured multi-task prediction. We provide well-posedness and stability interpretations of the latent solve and instantiate the framework with efficient low-rank-plus-diagonal trust operators. On an image-level image-metadata prediction benchmark, RiVaT-Fuse achieves the strongest overall predictive rank among direct representation-level baselines while improving probability and label stability under perturbation.
1 Introduction
Multimodal fusion must account for sample-varying modality reliability rather than merely combine visual and structured features. RiVaT-Fuse addresses this by estimating a reliability-calibrated latent state with structured interaction, robustness, and task coupling.
- Sample-varying reliability makes multimodal fusion a trust-allocation problem, not merely a feature-combination problem.Images can be degraded by acquisition conditions, while metadata can be incomplete, noisy, or weakly informative for individual predictions.
- Existing fusion layers provide aggregation mechanisms but rarely define the fused representation as a reliability-calibrated consensus latent state.Concatenation, gating, modulation, and attention introduce different interactions without specifying the intended sample-wise latent meaning.
- RiVaT-Fuse defines fusion as sample-wise latent-state estimation through a variational objective balancing modality evidence, cross-modal interaction, and stability.This replaces a hand-designed aggregation rule with a reliability-calibrated optimization problem.
- Evaluation targets image-level multimodal prediction under repeated visual acquisitions, with direct baselines compared under a matched protocol.Patient-disjoint deployment evaluation is treated as a different estimand rather than the primary target.
- The framework combines matrix-valued trust, structured additive–multiplicative–relational interaction, conditional robustness, and structured multi-task coupling.These components are integrated into one framework rather than treating modality trust, robustness, and task sharing separately.
2 Problem Formulation
RiVaT-Fuse represents each image-level acquisition and its associated metadata in a shared latent space for multimodal prediction. Its fusion objective is designed to adapt trust by sample and latent direction while incorporating structured cross-modal evidence and robustness.
- Each prediction instance is an image-level acquisition paired with structured metadata, supporting binary, ordinal, or multiclass task labels.The empirical instantiation uses retinal prediction tasks including ordinal disease-severity grading and binary clinical outcomes.
- Labels from repeated observations or aggregated annotations are mapped back to image-level rows, so the supervised unit remains the acquisition rather than a patient-disjoint deployment unit.This protocol evaluates acquisition-level multimodal inference.
- Image and metadata encoders map their inputs into a shared latent dimension, where fusion estimates h⋆(x) under sample-dependent reliability.The image and metadata embeddings are denoted zI and zM, and the fused representation lies in R^d.
- The formulation requires sample-adaptive and direction-dependent trust, structured cross-modal interaction, and robustness to unreliable evidence.These desiderata distinguish the method from simple feature aggregation.
3 RiVaT-Fuse
RiVaT-Fuse defines multimodal fusion as estimating a reliability-calibrated latent state rather than aggregating features directly. Its trust geometry, structured interactions, robustness objective, task coupling, and stability analysis are organized around this latent solve.
- 3.1 Variational Latent Fusion: RiVaT-Fuse defines fusion as a variational sample-wise latent-state solve that balances modality evidence, cross-modal interaction, and stability.The fused state is obtained by solving for a consensus close to image and metadata evidence while incorporating interaction and regularization.
- 3.1 Variational Latent Fusion: The closed-form latent system combines reliability geometry with modality evidence and cross-modal synergy, enforcing a reliability-weighted compromise for each sample.When the system is positive definite, h⋆(x)=H(x)^−1b(x), and implementation solves H(x)h=b(x) directly.
- 3.2 Reliability State and Matrix-Valued Trust: Matrix-valued trust operators represent dimension-wise and correlated reliability, replacing scalar confidence with an anisotropic latent-space geometry.A low-rank-plus-diagonal construction captures correlated trust directions while avoiding dense precision-matrix costs and instability.
- 3.3 Structured Tensor Interaction: Structured tensor interaction combines additive correction, multiplicative conjunction, and relational metadata-conditioned token selection.These components target distinct cross-modal dependencies, with the relational branch using metadata to guide local image-token evidence selection.
- 3.4 Conditional Robustness and 3.5 Structured Multi-Task Prediction: The same reliability state controls both fusion geometry and robustness pressure, while state-binned conditional DRO protects less reliable samples from local distribution shift.Lower image reliability, higher uncertainty, and lower metadata reliability enlarge the local robust budget; task coupling is added through a graph-Laplacian regularizer.
- Theoretical interpretation: Under a positive-definite latent system, the fused state is unique, and conditioning-based perturbation analysis links latent stability to prediction and label stability.The paper states that Lipschitz task heads transfer latent stability to predictions and that sufficient margins imply label invariance under bounded perturbations.
4 Experiments
The experiments compare RiVaT-Fuse with direct image–metadata fusion baselines under a common image-level protocol, evaluating predictive performance, calibration, perturbation stability, and mechanism-oriented behavior.
- Main Results: RiVaT-Fuse achieves the strongest overall predictive profile, including the best ordinal accuracy and QWK, both binary-task AUCs, binary-task-1 sensitivity, binary-task-2 accuracy, and average rank.The comparison uses ten displayed predictive metrics across one ordinal and two binary tasks.
- Main Results: RiVaT-Fuse provides the best balance across ordinal prediction, binary discrimination, and sensitivity-oriented behavior, while attention-based fusion is weaker in this benchmark.Concat, FiLM, and gated fusion each show narrower strengths or trade-offs.
- Baselines: RiVaT-Fuse combines matrix-valued trust, structured interaction, and conditional robustness within a variational latent fusion framework rather than relying on simple aggregation alone.The baseline set targets representation-level fusion methods operating at the same level as the proposed method.
- Diagnostic Interpretation: RiVaT-Fuse achieves the lowest mean label-flip rate, smallest binary-task probability shifts, and lowest binary-task-2 Brier score while ranking first predictively.Metadata MLP has the lowest mean ECE, but its calibration advantage coincides with very low binary-task-1 sensitivity.
- Dynamics and Qualitative Analysis: The study interprets learned reliability, uncertainty, ambiguity radius, trust operators, and contribution profiles as evidence of dynamically balanced robustness and multimodal interaction.Attention visualizations show metadata context guiding attention over image tokens.
- Diagnostic Interpretation: The evidence is limited to completed direct comparisons, calibration and stability diagnostics, and visualization-based mechanism analysis because no component-ablation experiments are included.Representation-shift comparisons require caution because architectures use different latent-space scales.
5 Related Work
Related work spans multimodal representation fusion, reliability and robustness, multi-task learning, and medical image–metadata prediction. RiVaT-Fuse connects these strands through reliability-aware latent trust and early representation-level fusion.
- Multimodal representation fusion: Multimodal representation fusion includes concatenation, gating, conditional modulation, co-attention, transformers, Perceiver-style architectures, and tensor-based interaction methods.The literature covers early, late, colearning, and hybrid fusion strategies.
- Reliability, robustness, and task structure: Reliability, calibration, distributionally robust learning, group robustness, and multi-task learning provide complementary foundations for handling uncertainty and task structure.RiVaT-Fuse connects these ideas through reliability states, matrix-valued latent trust, state-dependent robustness, and task coupling.
- Medical image–metadata learning: Medical image–metadata learning combines imaging with clinical or tabular variables, while this work focuses its main comparison on representation-level early fusion.The application context includes retinal AI and broader medical AI systems.
6 Conclusion and Limitations
The paper presents RiVaT-Fuse as a principled alternative to heuristic multimodal fusion and evaluates it in a focused image-level setting. Its scope excludes patient-disjoint deployment as the primary estimand and acknowledges additional computation and limited ablation evidence.
- Conclusion: RiVaT-Fuse unifies matrix-valued trust, structured cross-modal interaction, conditional robustness, and task coupling through sample-wise latent-state estimation.The framework is designed for aligned image–metadata prediction protocols.
- Scope and limitations: The empirical study centers on one primary image–metadata benchmark and targets image-level acquisition prediction rather than patient-disjoint deployment evaluation.Patient-disjoint evaluation is identified as a complementary clinical validation question.
- Scope and limitations: The latent solve and structured interaction modules add computation relative to simple fusion, although structured operators and batched solves are used.Broader deployment-oriented extensions are left for future work.
- Supporting materials: The appendices provide derivations, stability analysis, implementation details, protocol transparency, baseline specifications, full numerical results, mechanism interpretation, and visualization materials.These materials cover both reproducibility and supporting analyses.
A Derivations and Stability Analysis
RiVaT-Fuse computes a fused latent state by solving a reliability-weighted quadratic system that combines modality evidence with structured interaction. Positive definiteness guarantees a unique solution, while conditioning controls first-order sensitivity to perturbations.
- Latent solve: The fused state h⋆(x) is obtained by solving H(x)h⋆(x)=b(x), equivalently h⋆(x)=H(x)−1b(x) when H(x) is nonsingular.H(x) encodes reliability geometry, while b(x) combines trusted modality embeddings and tensor interaction.
- Well-posedness: Under uniform positive definiteness, the variational objective is strongly convex and has a unique global minimizer.The condition H(x)⪰µI ensures well-posed latent fusion for every sample.
- Stability: The perturbation bound scales with ∥H−1∥2 and bounded changes in image embeddings, metadata embeddings, and the interaction operator.This links latent-state sensitivity to the conditioning of the reliability system and perturbations in its evidence terms.
- Prediction stability: If a task head is Lt-Lipschitz, output changes are bounded by latent-state changes, and labels remain unchanged when score perturbations stay below half the clean margin.The result provides a sufficient condition for prediction stability rather than an unconditional guarantee.
- Implementation: The implementation uses a latent linear solve with d=256, low-rank-plus-diagonal trust operators of rank 32, and multiplicative interactions of rank 64.The trust operator uses numerical stabilizer ϵ=10−3.
C Dataset, Labels, and Image-Level Protocol
The study targets image-level acquisition prediction, pairing each image with structured metadata and evaluating ordinal and binary tasks under a consistent protocol. Labels and preprocessing are defined without using test data, while baselines isolate modality-specific and representation-level fusion effects.
- Image-level protocol: The target estimand is image-level acquisition prediction, with each image-level observation paired with structured metadata.The same protocol is used for RiVaT-Fuse and direct representation-level baselines.
- Labels and tasks: The retinal instantiation contains one ordinal severity task and two binary decision tasks, with labels constructed using patient-level aggregation when required before returning to image-level rows.Main tables retain generic task names for method-level framing.
- Dataset split: The aligned run contains 3603 training images, 772 validation images, and 773 test images, with patient overlap permitted by the declared image-level protocol.The split is not patient-disjoint because the target is acquisition-level prediction.
- Evaluation controls: Preprocessing statistics, model selection, and binary thresholds use training and validation data only, reserving the test set for final reporting.This includes metadata imputation, standardization, categorical vocabularies, checkpoint selection, and threshold tuning.
- Baselines: The comparison includes metadata-only and image-only baselines alongside Concat, FiLM, gated, and cross-/co-attention representation-level fusion models.These baselines isolate unimodal signal and alternative early-fusion mechanisms.
E Decision-Level and Hybrid Methods
Decision-level and hybrid methods combine predictions or logits after representation learning, so they answer a complementary question rather than serving as direct representation-level baselines. Their performance does not invalidate RiVaT-Fuse’s representation contribution.
- Method distinction: Decision-level late-fusion and hybrid methods combine predictions, logits, or outputs after representation learning rather than forming an early fused representation.They are therefore excluded from the main direct representation-level comparison.
- Interpretation: A decision-level ensemble may outperform a representation-level model by leveraging multiple predictors, but that comparison does not invalidate RiVaT-Fuse’s representation-level contribution.A strong RiVaT-Fuse representation could itself be used within later late-fusion or hybrid systems.
F.1 Full predictive metrics
The empirical evidence combines predictive comparisons with calibration, perturbation-stability, and reliability-stratified diagnostics. The reported claim is deliberately bounded to the complete RiVaT-Fuse framework rather than attributing effects to individual internal modules.
- Claim boundary: The submission does not claim component-removal ablation evidence for trust, interaction, robustness, or task-coupling modules.The completed analyses support the full formulation under the aligned protocol without isolating each module’s quantitative contribution.
- Overall interpretation: RiVaT-Fuse provides a favorable complete-framework alternative to heuristic representation-level fusion, with predictive improvements over direct early-fusion baselines.The supplied results do not assign separate effect sizes to individual internal modules.
- Stability metrics: Probability deltas and label-flip rates are the primary cross-model stability metrics because output quantities are directly comparable across architectures.Representation shifts remain model-specific because latent spaces have different scales.
- Calibration: Calibration uses ECE for ordinal and binary outputs and Brier score for binary outputs, while operating-point behavior is interpreted alongside sensitivity and specificity.The manuscript notes that calibration alone may not ensure suitability for a screening-oriented decision point.
- Calibration interpretation: RiVaT-Fuse is reported to balance discrimination, operating-point sensitivity, and perturbation stability more strongly than the metadata MLP in the discussed comparison.The metadata MLP shows strong ECE but a sensitivity–specificity imbalance on Binary-1.
- Stress-test evidence: Worst-group and reliability-stratified summaries serve as descriptive stress tests for error concentration in difficult or low-reliability strata, not as standalone fairness claims.Visualization artifacts also document optimization, reliability-state, trust, and contribution dynamics.
I.2 Training-history visualizations
The visualization appendix organizes training-history and final diagnostics, probe-panel sequences, representation-space projections, and case studies to inspect model behavior. These visuals are interpreted qualitatively and alongside quantitative metrics, not as standalone evidence of clinical reasoning or component necessity.
- I.2 Training-history visualizations: Training-history and final diagnostic visualization families organize the appendix’s generated artifacts for inspecting optimization, reliability, calibration, perturbation stability, and related behaviors.The supplied captions identify training-history and final diagnostic families, while the appendix describes the broader purposes of these visualization families.
- I.2 Training-history visualizations: Probe-panel sequences track predictions, reliability state, trust summaries, and contribution patterns across epochs through the selected checkpoint.Six tracked probes are shown with panels at multiple epochs and an evolution GIF, providing qualitative temporal diagnostics.
- I.2 Training-history visualizations: PCA and t-SNE compare image, metadata, and fused latent representations using projections colored by ordinal and binary labels.The diagnostics include validation and test projections for zI, zM, and h⋆, but projection geometry is interpreted qualitatively.
- I.2 Training-history visualizations: Case-study panels combine image evidence, metadata context, reliability state, trust summaries, contribution summaries, and task predictions at selected checkpoints.The qualitative cases include stable-correct, corrected-by-fusion, and failure categories to avoid presenting only favorable examples.
- I.2 Training-history visualizations: Attention and relational maps illustrate metadata-conditioned attention allocation but are not standalone evidence of causal or clinical reasoning.These maps are treated as model-behavior visualizations rather than clinical explanations.
L Reproducibility Statement
The reproducibility statement records the fixed image-level evaluation setup, implementation and archive structure, and important reporting boundaries. Reproduction and deployment remain constrained by missing statistical uncertainty reporting, restricted artifacts and data access, and the need for local validation and human oversight.
- L Reproducibility Statement: The aligned comparison uses fixed image-level and training seeds, records split and evaluation metadata, and selects binary thresholds on validation data.The records include image counts, patient counts, thresholds, checkpoint-selection metrics, and final test outputs.
- L Reproducibility Statement: The implementation uses PyTorch and torchvision with a ResNet-style backbone, AdamW, gradient clipping, early stopping, and validation-selected binary thresholds.The archive includes serialized validation/test records, checkpoints, training histories, final diagnostics, and probe-panel sequences.
- L Reproducibility Statement: The archived analysis structure links numerical results, checkpoints, and visual diagnostics to corresponding evaluation records.The package is organized into analysis_records/, checkpoints/, plots_history/, plots_final/, and probe_panels/.
- L Reproducibility Statement: The study reports fixed-seed results without multi-seed error bars or statistical significance tests, and computational-resource details are incomplete.The statement attributes these omissions to computational constraints and notes that worker type, GPU memory, and wall-clock time are not fully specified.
- L Reproducibility Statement: The source package omits executable code, checkpoints, serialized evaluation records, and raw clinical images, while dataset access remains subject to original asset conditions.The manuscript sources and required figures are distributed, but broader artifact release and dataset access remain constrained.
- L Reproducibility Statement: RiVaT-Fuse is a research framework rather than a stand-alone clinical decision system, so deployment requires local validation and human oversight.The statement also identifies distribution-shift, privacy, and clinical-overreliance risks and notes that broader dataset validation would strengthen generality.