PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A. Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency

arXiv:2107.07502v2cs.LGcs.AIcs.CLcs.CVcs.MM

TL;DR

Multimodal research lacks broad resources for studying generalization, complexity, and robustness across heterogeneous domains and modalities. MultiBench and MULTIZOO address this gap with a unified benchmark, standardized pipeline, and 20 core multimodal methods; applying methods across research areas improves performance on 9 of 15 datasets. The benchmark also exposes tradeoffs among performance, complexity, and robustness while defining scope for future expansion.

  • Problem

    Multimodal research has limited resources for evaluating generalization across domains and modalities, training and inference complexity, and robustness to noisy or missing modalities.

  • Method

    MultiBench provides a unified benchmark and end-to-end evaluation pipeline, complemented by MULTIZOO's standardized implementations of multimodal methods.

  • Results

    Applying methods from different research areas improves state-of-the-art performance on 9 of 15 datasets, while experiments reveal tradeoffs involving performance, complexity, and robustness.

  • Takeaways & Limitations

    MultiBench unifies disjoint multimodal research efforts and supports more accessible, reproducible analysis of model capabilities and limitations.

  • Takeaways & Limitations

    The current benchmark omits several research areas outside multimodal fusion and remains limited by the models and metrics it supports.

Abstract

from arXiv · show

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. MultiBench provides an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, MultiBench offers a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench introduces impactful challenges for future research, including scalability to large-scale multimodal datasets and robustness to realistic imperfections. To accompany this benchmark, we also provide a standardized implementation of 20 core approaches in multimodal learning. Simply applying methods proposed in different research areas can improve the state-of-the-art performance on 9/15 datasets. Therefore, MultiBench presents a milestone in unifying disjoint efforts in multimodal research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility. MultiBench, our standardized code, and leaderboards are publicly available, will be regularly updated, and welcomes inputs from the community.

1 Introduction

Multimodal research spans important real-world applications but remains limited by understudied domains and by benchmarks that often omit complexity and modality imperfections. MultiBench addresses these gaps with a unified benchmark and reproducible pipeline spanning diverse datasets, modalities, tasks, and research areas.

  • Multimodal problems combine heterogeneous signals and support applications including language, vision, video, healthcare, finance, robotics, and human-computer interaction.
  • Current research underrepresents several domains, modalities, and tasks, while benchmarks often emphasize performance without measuring complexity or robustness to imperfect modalities.
  • MultiBench evaluates generalization across domains and modalities, training and inference complexity, and robustness to noisy or missing modalities.
  • 15 datasets span 10 modalities and 20 prediction tasks across 6 research areas, including understudied areas such as healthcare, finance, and HCI.
  • Its end-to-end pipeline standardizes preprocessing, dataset splits, algorithms, metrics, and cross-validation protocols to support comparison and reproducibility.
  • Applying methods from different research areas improves state-of-the-art performance on 9 of 15 datasets.

2 MULTIBENCH: The MULTISCALE MULTIMODAL BENCHMARK

MultiBench focuses initially on multimodal fusion: combining heterogeneous signals from multiple modalities to make predictions. Its benchmark suite covers varied research areas, modalities, tasks, and evaluation dimensions, including performance, complexity, and robustness.

  • Background: A modality is a single mode in which a signal is expressed or experienced; multimodal data combine multiple heterogeneous signals.
  • Background: The first MultiBench version benchmarks multimodal fusion, where two or more modalities are joined to perform prediction tasks such as classification or regression.
  • Datasets: The 15-dataset suite spans research areas, dataset sizes, input modalities, and prediction tasks, with standardized data loaders and state-of-the-art multimodal models.
  • Research areas: Healthcare coverage includes MIMIC, which combines hourly ICU time-series measurements with demographic variables to support multimodal medical decision-making.
  • Evaluation: Evaluation covers dataset-specific performance metrics, time and space complexity, and robustness to noisy or missing modalities.
  • Toolkit: MULTIZOO complements the benchmark with modular standardized implementations intended to improve accessibility, compositionality, and reproducibility.

3 MULTIZOO: A Zoo of Multimodal Algorithms

MULTIZOO provides a standardized, modular toolkit spanning multimodal fusion, alignment, optimization, and training approaches. Its unified implementations support reproducible comparisons while exposing tradeoffs among performance, complexity, and modality selection.

  • Toolkit: MULTIZOO implements 20 multimodal methods covering data preprocessing, fusion paradigms, optimization objectives, and training procedures.
  • Alignment: Temporal alignment maps information from other modalities to a shared temporal granularity, while WORDALIGN applies this approach when text is one modality.
  • Fusion Paradigms: Early fusion concatenates inputs before modeling, whereas late fusion encodes modalities separately before concatenating their representations for classification.
  • Fusion Paradigms: Tensor fusion captures multimodal interactions but can be expensive because tensor dimensions scale exponentially with the number of modalities; low-rank approximations address this cost.
  • Fusion Paradigms: Multimodal gated units dynamically modify representations using element-wise products with learned attention weights.
  • Optimization Objectives: Optimization methods include CCA for prediction-level alignment, REFNET for feature-level contrastive alignment, and gradient-based methods for balancing modality contributions.
  • Results: Table 2 reports stronger performance on 9 of 15 datasets after applying methods from different research areas.
  • Evaluation: Standardized experiments keep confounding factors constant across methods, enabling cleaner comparisons and reproducible evaluation of performance, complexity, and robustness.

4 Experiments and Discussion

MultiBench standardizes multimodal evaluation across datasets while revealing tradeoffs among performance, complexity, modality count, and robustness. The experiments identify simple fusion as competitive and highlight unresolved challenges in scalable, robust multimodal modeling.

  • Applying methods across research areas improves state-of-the-art performance on 9 of 15 datasets, especially in understudied domains and modalities.
  • Tradeoffs between performance and complexity: Figure 3 visualizes performance–complexity Pareto tradeoffs across all datasets and datasets with more than 6 tested approaches.Circle size represents performance variance, and the dotted blue curve is a quadratic best fit.
  • Methods such as MFAS and CCA support only 2 modalities, while TF and MI do not scale efficiently beyond 2/3 modalities.
  • Tradeoffs between modalities: 2−3× the parameters may be required for further improvement from multimodal models over the best-performing modality.
  • Tradeoffs between performance and complexity: Simple fusion techniques such as LF score highly on both performance and complexity, while MFAS and MULT are slightly better-performing but more complex.LF is easier to adapt to new datasets and domains than MFAS or MULT.
  • Tradeoffs between performance and robustness: Few models achieve both positive relative and effective robustness, because some well-performing methods drop faster after equalizing clean-data accuracy.Performance correlates positively with relative robustness but negatively with effective robustness.

5 Related Work

MultiBench extends multimodal benchmarking beyond commonly studied areas by covering diverse datasets, modalities, and tasks. Its evaluation also examines how performance relates to robustness under imperfect data.

  • MultiBench is presented as the first multimodal benchmark with such a large number of datasets, modalities, and tasks.
  • Figure 4 compares performance with relative and effective robustness, showing that few models achieve both forms of robustness.Circle size represents robustness variance across datasets, and the dotted blue line is a linear best fit.
  • The benchmark goes beyond language, vision, and audio to encourage methods handling tabular data, time-series, sensors, graphs, and sets.

6 Conclusion

MultiBench unifies multimodal research around accessible, reproducible evaluation while acknowledging boundaries in generality, scale, and supported research problems. The authors plan continued expansion through community involvement and additional datasets, models, and metrics.

  • The benchmark’s generality can trade off against modality- and task-specific models that leverage domain knowledge for performance and interpretability.
  • The current release excludes several research areas beyond multimodal fusion and remains limited by its supported models and metrics.The authors identify question answering, retrieval, grounding, and reinforcement learning as areas for expansion.
  • The authors plan to expand datasets and tasks, including captioning, question answering, retrieval, and multimodal reinforcement learning, with community suggestions and feedback.
  • MultiBench unifies previously disjoint multimodal research efforts with an emphasis on ease of use, accessibility, and reproducibility.
  • The conclusion identifies more generalizable, lightweight, and robust multimodal models as future directions enabled by the benchmark’s broad evaluation scope.

A Broader Impact Statement

MULTIBENCH addresses the practical and societal challenges of deploying multimodal models by evaluating complexity, privacy, security, and social bias alongside performance. It standardizes fusion research while supporting reproducibility and future safety-focused extensions.

  • Technical Challenges: Standardized preprocessing, model implementations, evaluation metrics, and public leaderboards reduce duplicated effort and support reproducible comparisons.MULTIBENCH and MULTIZOO cover the pipeline from data loading through evaluation.
  • Privacy and Security: Recorded human behavior data may create privacy risks, so the authors report using consented public data that were anonymized and stripped of personal and protected attributes.The paper also identifies federated learning, differential privacy, and encryption as techniques for keeping device data private during deployment.
  • Social Biases: Imbalanced human-centric datasets can create exposure bias and amplify social biases correlated with prediction targets.The authors identify fairness-performance trade-offs and propose expanding MULTIBENCH to evaluate them.
  • Future Directions: MULTIBENCH is positioned as an accessible, maintainable benchmark for studying multimodal capabilities, limitations, safety, and reliability.The authors plan continual expansion, including fairness and privacy evaluations.
  • Technical Challenges: MULTIBENCH evaluates multimodal fusion across performance, complexity, and robustness, including alignment across modalities and time.The benchmark targets heterogeneous granularities, training and inference costs, and noisy or missing modalities.

C.1 Dataset Selection

MULTIBENCH selects datasets to reflect realistic multimodal data, diverse research areas, and real-world transfer, spanning affective computing, multimedia, and robotics examples. The selected datasets cover varied modalities, tasks, collection processes, and preprocessing choices.

  • Dataset Selection: Dataset selection considers collection methods, input modalities, evaluation tasks, metrics, and train/test splits that reflect real-world applications.Domain experts were consulted across application areas.
  • Dataset Selection: MULTIBENCH prioritizes realistic data collected in the wild so models encounter natural noise patterns and task imperfections.The benchmark seeks settings with potential for transfer toward real-world deployment.
  • Dataset Selection: The benchmark spans research areas chosen through literature surveys and consultation with domain experts to capture promising and challenging applications.Community expansion is planned through proposals for new research areas, datasets, and prediction tasks.
  • Affective Computing: Affective-computing datasets fuse language, video, and audio time series for sentiment, emotion, humor, and sarcasm prediction.Examples include CMU-MOSI, CMU-MOSEI, UR-FUNNY, and MUSTARD.
  • Affective Computing: The affective datasets use modality-specific features including BERT or GloVe language representations, ResNet and facial features, and Librosa or COVAREP audio features.Preprocessing follows methods described for the corresponding datasets.
  • Affective Computing: CMU-MOSI contains 2,199 opinion video clips with sentiment intensity labels ranging from −3 to +3 and annotated visual and audio features.Its access restrictions state that the benchmark authors cannot redistribute the dataset.
  • Other Datasets: The robotics dataset combines proprioception, RGB-D imagery, and force-torque sensing collected under random and heuristic policies.These modalities support multimodal robotic prediction tasks.

C.3 Documentation

MULTIBENCH documents a broad, publicly maintained multimodal benchmark with standardized dataset information, splits, labels, and evaluation resources. Its documentation also records external dependencies, sensitive-data safeguards, potential deployment risks, and dataset-scale variation.

  • The benchmark spans datasets from 1,460 to 306,245 instances, with reported single-GPU training times ranging from a few minutes to roughly two weeks.
  • MULTIBENCH provides raw and processed modality data, labels, and train, validation, and test splits through its data-loading pipeline.
  • MULTIBENCH covers 6 research areas and 15 prediction tasks, including affect recognition, robot manipulation, stock prediction, design interface, action recognition, movie genre prediction, and digit prediction.
  • Robustness tests inject noise and imperfections into otherwise complete datasets to simulate real-world imperfections.
  • Some datasets depend on external resources, while healthcare and financial datasets are publicly available for research and MIMIC is rigorously de-identified.
  • The authors acknowledge privacy, security, and real-world deployment risks for human-centric data and plan continuous hosting, maintenance, and updates.

D.3.3 Robustness Measure

MULTIBENCH evaluates robustness by testing clean-trained models under increasing modality-specific and multimodal imperfections. It combines curve visualizations with relative and effective robustness measures and aggregates results across heterogeneous datasets.

  • Robustness evaluation: Models are trained on clean data and evaluated on test data with increasing modality-specific or multimodal noise levels.
  • Relative robustness: Relative robustness measures performance under imperfections against the simple late-fusion concatenation baseline LF.
  • Quantitative measures: The robustness metrics approximate areas between performance-imperfection curves over imperfection levels σ from 0.0 to 1.0.
  • Effective robustness: Effective robustness measures accuracy-drop rates after accounting for differences in clean-test accuracy using the LF accuracy trend.
  • Interpretation: A robust multimodal model should achieve both high relative robustness and high effective robustness.
  • Cross-dataset aggregation: Min-max normalization to a 0−1 scale before averaging provides a reliable cross-dataset indicator despite differing metric scales.

E.2 Data Preprocessing

MULTIZOO standardizes multimodal preprocessing and fusion methods across alignment, interaction, attention, tensor, and optimization-based approaches. The implementations expose common baselines alongside more expressive or efficient alternatives.

  • Temporal alignment: WORDALIGN averages higher-frequency non-text features within each word’s start-end interval, producing equally long text and aligned non-text sequences.
  • Fusion paradigms: Early fusion concatenates input modalities, whereas late fusion independently encodes modalities before concatenating their representations.
  • Tensor methods: Tensor Fusion captures higher-order interactions, while Low-rank Tensor Fusion approximates tensor products whose dimension grows exponentially with modality count.
  • Model and objective variants: MULTIZOO includes multiplicative-interaction variants, architecture search, and contrastive feature alignment among unimodal and multimodal representations.
  • Attention: Gated attention computes zmm = z1 ⊙ h(z2), using sigmoid-based attention weights from one representation to modulate another.
  • Temporal attention: Crossmodal Transformers let each modality attend to another and concatenate the resulting representations, while extension to three modalities repeats all modality pairs.

F.6 Training Structures

MULTIZOO organizes multimodal models as interchangeable training structures and plugin modules, enabling consistent experimentation across datasets. Its toolkit standardizes metrics, complexity measurement, robustness testing, and evaluation workflows.

  • Architecture: Training structures define the main optimization algorithm, while unimodal models, fusion paradigms, objectives, and heads operate as exchangeable plugins.
  • Training variants: Advanced methods modify supervised training through additional objectives, dynamic modality weighting, or outer-loop architecture search.
  • Standardized experimentation: Fusion-based methods such as EF, LF, TF, LRTF, MI, NL-GATE, and MULT share supervised_learning structures while swapping fusion modules.
  • Evaluation: Evaluation uses dataset-specific regression and classification metrics, including MSE, MAE, accuracy, micro and macro F1-score, and AUPRC.
  • Complexity: MULTIZOO reports training memory using peak Python-process usage and excludes ephemeral networks when counting persistent training parameters.
  • Robustness: Robustness modules add increasing noise levels σ∈[0,1] to clean test sets and evaluate performance across the resulting noisy datasets.
  • Usability: The toolkit loads data, defines models, trains, and evaluates complete multimodal systems in fewer than 10 lines of code.

G Experimental Setup

Experiments use standardized dataset- and domain-specific configurations, with validation-based hyperparameter selection and repeated runs. The setup fixes confounding factors when comparing multimodal methods and reports standardized performance and complexity results.

  • Configuration: Experiments were conducted across datasets using domain- and dataset-specific hyperparameters, encoders, preprocessing, and training configurations.Previously tested models generally reuse hyperparameters from their original papers or public code.
  • Configuration: Hyperparameters were selected using validation-set performance, while small datasets and rapidly overfitting settings motivated constrained model complexity.For ENRICO, a learning rate of 10^-5 and hidden dimension 16 were found sufficient for unimodal encoders.
  • Reproducibility: All experiments were repeated 10 times, and means and standard deviations were computed.
  • Evaluation: Evaluation reports dataset-appropriate metrics, including MSE, MAE, accuracy, micro and macro F1-score, and AUPRC.The benchmark also provides full performance and complexity results for the evaluated domains.

H.1 Affective Computing

Across affective computing, healthcare, robotics, and finance, multimodal methods show domain- and task-dependent performance and robustness. The results also expose trade-offs between predictive strength, complexity, and resilience to noisy modalities.

  • Affective Computing: MULT consistently performs best across the affective-computing datasets, generalizing from sentiment and emotion prediction to humor and sarcasm.However, MULT requires more than 12× the inference time of unimodal models and 3–4× that of several simpler fusion baselines.
  • Affective Computing: Multimodal models are generally only slightly more robust than unimodal models under increasing noise, with performance often dropping quickly toward random.Multimodal models are more robust to noisy video and audio, but can be less robust to noisy language, the strongest modality.
  • Healthcare: On MIMIC, model results vary little overall, while Late Fusion performs competitively and achieves the best performance on the ICD-9 group 7 task.MFAS, despite broader generality elsewhere, struggles on MIMIC.
  • Healthcare: MIMIC robustness declines rapidly with increasing noise, although multimodal models generally remain more robust than unimodal models.Robustness varies substantially across prediction tasks: LRTF is most robust for ICD-9 group 7 but least robust for group 1.
  • Robotics: Robotics tasks contain one especially strong unimodal modality, yet multimodal methods are much more robust to sensor disturbances than unimodal methods.Late Fusion outperforms Sensor Fusion on both VISION&TOUCH tasks.
  • Finance: Finance results show multimodal models can outperform unimodal models, but robustness and performance vary widely across methods and partitions.GRADBLEND begins with the lowest MSE but becomes the least robust as noise increases, while Late Fusion can underperform the best unimodal method.

H.5 HCI

On ENRICO, multimodal inputs can improve prediction, but performance and robustness vary substantially across methods and noise levels. Simple fusion methods often outperform more complex architectures on this small, noisy dataset.

  • 4%: Using multiple modalities boosts ENRICO performance over the best unimodal model.
  • TF, LRTF, and MI do not improve over simple LF on ENRICO.The paper hypothesizes that their larger number of trainable parameters increases overfitting risk on small, noisy datasets.
  • LF and sometimes GRADBLEND are more robust than the best unimodal model, although some multimodal models perform worse.TF is specifically reported as both less robust and lower-performing than the best unimodal method.
  • 5% to 40%: Multimodal performance spans this range at the highest noise levels.

H.6 Multimedia

Multimedia results show substantial variation across datasets and methods. Multimodal approaches can improve both initial performance and robustness, but gains depend on the dataset, modality, and computational cost.

  • 0.5%: MFAS beats the second-best method, MVAE, on AV-MNIST.
  • Contrastive learning with REFNET and CCA improve MM-IMDB performance over architectural baselines without alignment objectives.CCA, originally proposed for affect recognition, also generalizes to the multimedia domain.
  • ∼3×: GRADBLEND can improve KINETICS multimodal performance at the expense of this increase in training time.
  • Multimodal models outperform unimodal models in MM-IMDB robustness, especially against image imperfections.The authors attribute this to reliance on the other modality when one becomes imperfect; the gap is smaller for text imperfections.

H.7 Performance

MULTIBENCH exposes limited cross-domain generalization, strong performance–complexity tradeoffs, and a tension between accuracy and robustness. It also shows that standardization can transfer methods across research areas, while simple fusion remains broadly adaptable.

  • Benefits of standardization: 9 out of 15: Applying methods from different research areas achieves state-of-the-art performance on this many datasets.The gains are especially observed in relatively understudied healthcare, finance, and HCI domains.
  • Generalization across domains and modalities: Many multimodal methods fail to generalize across domains and datasets, often performing best on the settings for which they were designed.MFAS, MCTN, and GRADBLEND are cited as examples of limited transfer, while LF is relatively stable across datasets.
  • Tradeoffs between modalities: 2−3×: Further improvement beyond the best modality may require approximately this many additional parameters.
  • Performance and complexity: Simple fusion methods such as EF and LF score highly on both performance and complexity, compared with slightly better but more complex methods such as MFAS and MULT.The best unimodal model is below the performance–complexity Pareto frontier, while LF lies above it.
  • Performance and complexity: LF is easier to adapt across datasets and domains, whereas MFAS and MULT face architecture-specific adaptation difficulties.MFAS is suited to convolutional-network components, while MULT is suited to multimodal time-series data.
  • Performance and robustness: Few models achieve both relative and effective robustness: higher-performing models tend to retain relative performance but can drop faster after accuracy equalization.MVAE and RMFE show strong performance and relative robustness, whereas MULT, CCA, and MVAE can decline faster in effective robustness.

I.1 Datasets

MultiBench plans to expand its dataset and task coverage across multimodal retrieval, question answering, grounding, interactive reinforcement learning, generalization, uncertainty, fairness, and community-supported research. These extensions build on its modular benchmark and toolkit for standardized evaluation and reproducible experimentation.

  • Planned dataset and task expansion: MultiBench plans to add datasets for cross-modal retrieval, multimodal question answering, grounding, and interactive multimodal reinforcement learning.The proposed additions cover both alignment-focused tasks and embodied settings involving visual, language, auditory, and sensor modalities.
  • Planned dataset and task expansion: Grounding links entities across modalities and provides a finer-grained alternative to retrieval by targeting image subregions.The benchmark currently lacks grounding tasks but plans to add them in language-and-vision settings and extend them to other modalities.
  • Benchmark infrastructure: MULTIZOO’s modular organization supports adding new unimodal encoders and multimodal methods while preserving compatibility with existing components.Its code is partitioned into data processing, model design, optimization objectives, and training structures.
  • Evaluation expansion: Current MultiBench evaluation covers prediction performance, time and space complexity, and robustness to noisy and missing modalities.Future versions are planned to add further evaluation dimensions, including uncertainty estimation, distribution shift, and fairness.
  • Evaluation expansion: The planned fairness extension will address multimodal bias through data annotations, mitigation algorithms, and fairness metrics.The paper notes that relevant fairness definitions remain unclear across multimodal settings.
  • Community and education: MultiBench and MULTIZOO are intended to support reproducible research, future workshops, competitions, and multimodal machine-learning courses.The resources are positioned as starter infrastructure for dataset loading, modeling, evaluation, analysis, and student projects.
Loading 2107.07502v2…