Source-linked AI summary
Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study
Hao Dong, Hongzhao Li, Shupan Li, Muhammad Haris Khan, Eleni Chatzi, Olga Fink
TL;DR
MMDG progress is difficult to assess because studies use inconsistent datasets, modalities, protocols, and metrics while often omitting realistic robustness and trustworthiness tests. MMDG-Bench standardizes evaluation across diverse tasks and settings, finding that specialized methods yield only marginal gains, rankings vary by context, and substantial robustness and performance gaps remain.
Problem
Inconsistent MMDG evaluation protocols leave unclear whether reported gains reflect genuine algorithmic progress, especially under missing modalities, corruptions, and trustworthiness requirements.
Method
MMDG-Bench standardizes evaluation across six datasets, three task families, six modality configurations, nine methods, and 95 cross-domain tasks, including robustness and trustworthiness tests.
Results
Under fair evaluation, specialized methods offer only marginal gains over strong baselines, no method consistently dominates, and substantial gaps remain across tasks and deployment conditions.
Takeaways & Limitations
MMDG remains far from solved, and clean-domain benchmark performance does not reliably predict robustness under corruption or missing-modality scenarios.
Takeaways & Limitations
The benchmark covers discriminative and regression tasks and evaluates robustness using only two representative perturbations, excluding broader corruption suites and adversarial attacks.
Abstract
from arXiv · showhide
Despite the growing popularity of Multimodal Domain Generalization (MMDG) for enhancing model robustness, it remains unclear whether reported performance gains reflect genuine algorithmic progress or are artifacts of inconsistent evaluation protocols. Current research is fragmented, with studies varying significantly across datasets, modality configurations, and experimental settings. Furthermore, existing benchmarks focus predominantly on action recognition, often neglecting critical real-world challenges such as input corruptions, missing modalities, and model trustworthiness. This lack of standardization obscures a reliable assessment of the field's advancement. To address this issue, we introduce MMDG-Bench, the first unified and comprehensive benchmark for MMDG, which standardizes evaluation across six datasets spanning three diverse tasks: action recognition, mechanical fault diagnosis, and sentiment analysis. MMDG-Bench encompasses six modality combinations, nine representative methods, and multiple evaluation settings. Beyond standard accuracy, it systematically assesses corruption robustness, missing-modality generalization, misclassification detection, and out-of-distribution detection. With 7, 402 neural networks trained in total across 95 unique cross-domain tasks, MMDG-Bench yields five key findings: (1) under fair comparisons, recent specialized MMDG methods offer only marginal improvements over ERM baseline; (2) no single method consistently outperforms others across datasets or modality combinations; (3) a substantial gap to upper-bound performance persists, indicating that MMDG remains far from solved; (4) trimodal fusion does not consistently outperform the strongest bimodal configurations; and (5) all evaluated methods exhibit significant degradation under corruption and missing-modality scenarios, with some methods further compromising model trustworthiness.
1 Introduction
MMDG aims to improve robustness under domain shifts, but inconsistent evaluations make reported progress difficult to assess reliably. MMDG-Bench addresses this gap with standardized, broad evaluation and finds that apparent progress is limited and context-dependent.
- Multimodal models integrate complementary signals but remain brittle under domain shifts, modality imbalance, unreliable fusion, and missing or corrupted inputs.
- Existing MMDG evaluations vary across datasets, modality configurations, training protocols, and metrics, while often overlooking missing modalities, input corruptions, and model trustworthiness.
- MMDG-Bench standardizes evaluation across six datasets, three tasks, six modality combinations, nine methods, and 95 cross-domain tasks.
- The benchmark evaluates both standard accuracy and deployment-relevant properties, including corruption robustness, missing-modality generalization, misclassification detection, and OOD detection.
- Under fair evaluation, specialized methods provide only marginal gains over ERM, no method consistently dominates, and a substantial gap to Oracle performance remains.
- These findings suggest that MMDG progress may be partly overestimated because of inconsistent evaluation protocols.
2 A Comprehensive Benchmark for Multimodal Domain Generalization
MMDG-Bench formalizes multimodal domain generalization and evaluates representative methods across diverse tasks, modalities, domains, and robustness settings. Its design combines broad benchmark coverage with standardized protocols and extensive hyperparameter evaluation.
- Multimodal Domain Generalization Paradigms: Multi-source MMDG learns from multiple labeled source domains and generalizes to an unseen target domain without target-domain training data.
- Multimodal Domain Generalization Paradigms: Single-source MMDG transfers from one labeled source domain to an unseen target domain without target-domain access during training.
- Multimodal Domain Generalization Paradigms: Corruption robustness measures performance degradation between clean and realistically perturbed target modalities.
- Multimodal Domain Generalization Paradigms: Missing-modality generalization measures performance when modalities available during training are absent at target-domain inference.
- Methods: The benchmark compares nine representative MMDG methods with ERM as a foundational baseline and Oracle as an empirical target-domain performance ceiling.
- Benchmark Scope: MMDG-Bench spans six datasets across action recognition, mechanical fault diagnosis, and sentiment analysis, using six modality configurations that include bimodal and trimodal fusion.
- Evaluation Protocols: Multi-source evaluation uses leave-one-domain-out testing, while single-source evaluation covers all source-target pairs and sentiment analysis reports ACC2, F1, and MAE.
- Hyperparameter Search: Each algorithm-dataset pair receives default settings and 10 random-search trials, with retraining across additional seeds and averaged final performance.
3 Multimodal Domain Generalization Under Fair Comparison
Under standardized evaluation, MMDG results are highly context-dependent: specialized methods usually provide marginal gains, rankings shift across tasks, and trimodal fusion is not reliably superior. A large Oracle gap and severe single-source degradation show that multimodal domain generalization remains unresolved.
- Fair evaluation: Standardized training-domain validation isolates algorithmic contributions from evaluation artifacts.Data splits, batch sizes, optimizers, and model-selection strategies are standardized across methods.
- Cross-task results: No single method consistently dominates across action recognition, fault diagnosis, or sentiment analysis.MOOSA leads HUST Motor with 78.23% mean accuracy versus ERM at 69.90%, whereas sentiment gains over ERM are less than one percentage point.
- Cross-task results: Single-source transfer sharply reduces performance, with severe HUST Motor transfers reaching 1.75%−18.14% accuracy.The results suggest multi-source improvements depend heavily on broader source-domain coverage.
- Modality combinations: Trimodal fusion does not consistently improve generalization across methods or datasets.On HAC, V+A+F outperforms V+F for only five of nine methods; MOOSA declines from 71.29% to 70.17%, while CMRF, MBCD, and GMP gain +1.36%, +0.59%, and +0.92%.
- Upper-bound comparison: The Oracle reaches 92.81% mean accuracy on HAC V+A, nearly 22 percentage points above MOOSA at 70.95%.This persistent gap indicates that MMDG remains an open and challenging problem.
4 Robustness under Corruptions and Missing Modalities
MMDG methods remain vulnerable to realistic input failures: corruptions cause broad accuracy losses, while missing modalities produce strongly asymmetric degradation. Clean-domain rankings also fail to reliably predict deployment robustness.
- Robustness under corruptions: Video corruption causes accuracy drops of 7.97 −12.82 points, substantially exceeding the 0.77 −4.22 point declines under audio corruption.All methods except SimMMDG decline under audio corruption, while video corruption affects every method more severely.
- Robustness under corruptions: Corruption reverses clean-data rankings: MOOSA rises to first place, whereas SimMMDG falls from second to seventh.The most robust methods under defocus blur incorporate modality-balancing or competition-aware objectives.
- Missing modalities: Removing audio causes 0.32 −3.20 point drops, but removing video causes severe 36.50 −43.93 point drops.SimMMDG loses 0.33 points with video-only inference but 41.66 points with audio-only inference.
- Missing modalities: Video is the dominant modality under domain shift, while removing audio can improve performance in most A, C →H transfer cases.The results indicate that auxiliary modalities may introduce instability when multimodal integration is not properly optimized.
5 Trustworthiness: Misclassification and Out-of-Distribution Detection
The benchmark evaluates whether multimodal models can detect incorrect predictions and novel inputs, showing that predictive accuracy and trustworthiness are distinct dimensions. Different methods excel on different trustworthiness measures.
- Evaluation setup: MMDG-Bench evaluates misclassification detection and out-of-distribution detection using standardized metrics on HAC with video and audio inputs.Misclassification detection uses AURC, AUROC, and FPR95; OOD detection uses AUROC and FPR95, with EPIC-Kitchens as the OOD dataset.
- Misclassification detection: SimMMDG achieves the strongest overall misclassification-detection performance, with the best mean AURC and AUROC.Its modality-shared and modality-specific representation decomposition is associated with better-calibrated uncertainty estimates.
- Misclassification detection: MOOSA achieves the best mean FPR95 for misclassification detection, while CMRF ranks last across all misclassification-detection metrics despite competitive classification accuracy.This exposes a disconnect between predictive accuracy and model trustworthiness.
- Out-of-distribution detection: SimMMDG again performs strongest overall for OOD detection, whereas CMRF ranks second in mean OOD AUROC despite ranking last for misclassification detection.MOOSA shows the inverse pattern, leading misclassification FPR95 but ranking lowest in OOD AUROC.
- Cross-metric findings: Misclassification detection and OOD detection are non-redundant capabilities, and high classification accuracy does not guarantee model trustworthiness.Trust-oriented metrics can favor different methods depending on whether the focus is confidence calibration or OOD separation.
6 Conclusion
MMDG-Bench consolidates multimodal domain-generalization evaluation across diverse tasks, datasets, modalities, and methods. Its findings show marginal specialized-method gains, persistent unsolved generalization, and substantial weaknesses beyond clean accuracy.
- 6 Conclusion: MMDG-Bench standardizes evaluation across six datasets, three task families, six modality configurations, and nine representative methods.It covers both multi-source and single-source settings and evaluates clean accuracy alongside deployment-oriented capabilities.
- 6 Conclusion: Under fair evaluation, specialized methods yield only marginal gains over strong baselines, with no method consistently dominating across datasets, modalities, or task families.This conclusion is consistent with the benchmark’s motivation to separate algorithmic progress from protocol differences.
- 6 Conclusion: A substantial gap to the target-trained Oracle remains, confirming that MMDG is far from solved.The benchmark therefore indicates considerable room for improvement in cross-domain generalization.
- 6 Conclusion: Trimodal fusion does not reliably outperform the strongest bimodal configurations, challenging the assumption that adding modalities inherently improves generalization.MMDG’s heterogeneous modalities can exhibit distinct statistical properties and spurious cross-modal correlations under shift.
- 6 Conclusion: All methods remain highly vulnerable to corruptions and missing modalities, and some methods degrade model trustworthiness despite clean-accuracy gains.Evaluating clean cross-domain performance alone is insufficient for assessing real-world deployment capability.
A.3 Domain Generalization Benchmarks
Community benchmarks have reshaped domain-generalization research by standardizing comparisons and exposing persistent gaps on natural distribution shifts. MMDG-Bench extends this benchmarking role to multimodal settings and adds deployment-relevant evaluation axes.
- Prior benchmarks: DomainBed standardized evaluation of 14 algorithms across seven image datasets, revealing that reported gains often reflected inconsistent protocols rather than algorithmic innovation.Its findings helped reshape understanding of progress in unimodal domain generalization.
- Prior benchmarks: WILDS expanded benchmarking to 10 real-world datasets and showed that substantial performance gaps persist under natural distribution shifts.Its examples include satellite imagery and histopathology.
- MMDG-Bench: MMDG-Bench provides a multimodal analogue by standardizing backbones, data splits, hyperparameters, and model selection across nine methods, six modality combinations, and six datasets.It additionally evaluates corruption robustness, missing modalities, and trustworthiness, dimensions absent from prior MMDG evaluations.
B.1 Limitations
MMDG-Bench identifies unresolved weaknesses in current multimodal domain generalization and outlines priorities for developing models that generalize, withstand input failures, and provide trustworthy predictions.
- MMDG-Bench currently covers discriminative and regression tasks but not multimodal retrieval or generative modeling.
- Specialized MMDG methods offer marginal gains over ERM, fail to dominate consistently, and remain substantially below upper-bound performance.
- Trimodal fusion produces inconsistent benefits because dominant modalities can overshadow auxiliary modalities or reduce performance when poorly integrated.
- Current methods degrade substantially under realistic corruptions and missing modalities, motivating explicit resilience objectives beyond clean-training performance.
- Reliable confidence estimation remains difficult: misclassification detection and out-of-distribution detection are non-redundant trustworthiness challenges.
C Introduction of Datasets
MMDG-Bench assembles datasets across action recognition, mechanical fault diagnosis, and sentiment analysis, with modality-specific configurations and standardized sentiment targets.
- Action recognition: MMDG-Bench includes action-recognition datasets with video, audio, and optical-flow modalities.EPIC-Kitchens covers eight actions across three kitchens, while HAC covers seven actions performed by humans, animals, and cartoons.
- Mechanical fault diagnosis: HUST Motor provides synchronized vibration and acoustic signals across six motor health states and four rotational-speed domains.
- Sentiment analysis: The sentiment datasets provide aligned text, acoustic, and visual modalities across English-language, large-scale English, and Chinese-language settings.CMU-MOSI contains 2,199 clips; CMU-MOSEI contains over 23,500 utterances; CH-SIMS contains 2,281 video segments with modality-specific labels.
- Evaluation preparation: Sentiment classification is formulated as binary negative-versus-positive prediction, while regression targets are normalized to [-3, +3].
- Evaluation preparation: The benchmark uses random hyperparameter sweeps with documented defaults, distributions, and search settings.
E Detailed Experimental Results
Detailed experiments report single-source domain generalization results and evaluate robustness under corruptions and missing modalities.
- The experiments cover single-source domain generalization, corruption robustness, and missing-modality settings.
F Compute Resources
MMDG-Bench required substantial GPU computation across many cross-domain tasks and experimental tables covering datasets, modalities, corruptions, and missing modalities.
- Compute resources: 7,402 neural networks were trained across 95 cross-domain tasks using NVIDIA RTX 3090 and RTX 4090 GPUs.
- Experimental coverage: Experimental results span EPIC-Kitchens, HAC, HUST, sentiment datasets, corruption settings, and missing-modality settings.