PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A. Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency

arXiv:2107.07502v2cs.LGcs.AIcs.CLcs.CVcs.MM

TL;DR

Multimodal benchmarks have limited coverage of generalization, complexity, and robustness to imperfect modalities. MultiBench provides a unified evaluation framework, and reports that applying methods from other research areas improves state-of-the-art performance on 9 of 15 datasets.

  • Problem

    Existing multimodal benchmarks have limited coverage of understudied domains, modalities, and tasks, and typically emphasize performance without quantifying complexity or robustness to imperfect modalities.

    Source: §1 · p. 2

  • Method

    MultiBench is a unified benchmark assessing generalization, training and inference complexity, and modality robustness across datasets, paired with a standardized pipeline and 20 implemented multimodal approaches.

    Source: §1 · p. 2

  • Results

    9 of 15 datasets see improved state-of-the-art performance when methods from other research areas are applied.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Its broad evaluation highlights future directions in building more generalizable, lightweight, and robust multimodal models.

    Source: §6 · p. 10

    MultiBench currently omits several important areas outside multimodal fusion, including question answering, retrieval, grounding, and reinforcement learning, and is limited by supported models and metrics.

    Source: §6 · p. 10

Abstract

from arXiv · show

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. MultiBench provides an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, MultiBench offers a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench introduces impactful challenges for future research, including scalability to large-scale multimodal datasets and robustness to realistic imperfections. To accompany this benchmark, we also provide a standardized implementation of 20 core approaches in multimodal learning. Simply applying methods proposed in different research areas can improve the state-of-the-art performance on 9/15 datasets. Therefore, MultiBench presents a milestone in unifying disjoint efforts in multimodal research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility. MultiBench, our standardized code, and leaderboards are publicly available, will be regularly updated, and welcomes inputs from the community.

1 Introduction

MultiBench unifies multimodal benchmarking across diverse datasets and tasks, evaluating generalization, training and inference complexity, and robustness to noisy or missing modalities. Its standardized toolkit supports reproducible comparisons, and applying its core methods improves state-of-the-art performance on 9 out of 15 datasets.

1 Introduction

  • Existing benchmarks leave many domains, modalities, and tasks understudied and often omit time and space complexity or robustness to imperfect modalities.The introduction argues that real-world deployment requires balancing performance, robustness, and complexity.Source: §1 · p. 2
  • The benchmark covers 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas, assessing generalization, complexity, and robustness.Its robustness tests include modality-specific and multimodal imperfections that reflect real-world noise.Source: §1 · p. 2, Figure 1 · p. 2, Table 1 · p. 3
  • Domain experts helped adapt datasets for real-world relevance, multimodal learning challenges, and opportunities in algorithm design and evaluation.The benchmark includes understudied areas such as healthcare, finance, and human-computer interaction.Source: §1 · p. 2
  • Applying 20 core multimodal approaches improves state-of-the-art performance on 9 out of 15 datasets through a standardized evaluation pipeline.The pipeline includes data preprocessing, dataset splits, evaluation metrics, and cross-validation protocols.Source: §1 · p. 2
  • The public benchmark and model zoo are intended to support ease of use, accessibility, and reproducibility, with plans for ongoing maintenance and expansion.Proposed expansion includes future workshops, competitions, and support for multimodal learning courses.Source: §1 · p. 3

2 MULTIBENCH: The MULTISCALE MULTIMODAL BENCHMARK

MultiBench centers on multimodal fusion and brings together 15 datasets to study complementary information, scalability, and robustness across domains. Its evaluation protocol covers predictive performance, computational costs, and robustness to noisy or missing data.

2 MULTIBENCH: The MULTISCALE MULTIMODAL BENCHMARK

  • The first MultiBench version focuses on multimodal fusion: combining heterogeneous signals to make predictions, with datasets that probe complementary information, modality scale, and real-world imperfections.The paper distinguishes fusion from multimodal translation, grounding and question answering, and unsupervised or self-supervised representation learning, which it plans to address in future versions.Source: §2 · p. 3

2.1 Datasets

  • The 15-dataset portfolio spans affective computing, healthcare, robotics, finance, HCI, and multimedia, addressing varied multimodal tasks and challenges.Examples include sentiment and emotion prediction, ICU disease and mortality prediction, robot pose and dynamics objectives, correlated-stock price and volatility prediction, app-screen motif categorization, and multimedia classification or action recognition; the finance groups use 18, 63, or 100 input modalities.Source: §2.1 · pp. 3–4

2.2 Evaluation Protocol

  • MultiBench evaluates dataset-appropriate predictive performance, training and inference costs, and robustness to modality-specific and cross-modal imperfections.It records data size, parameter count, training time and memory, and CPU/GPU inference time; robustness tests include modality-specific corruptions and missing modalities or time-series chunks, with performance-imperfection curves and quantitative measures of relative and effective robustness.Source: §2.2 · pp. 4–5

3 MULTIZOO: A Zoo of Multimodal Algorithms

MULTIZOO provides standardized implementations of 20 multimodal methods across preprocessing, fusion, objectives, and training. Applying methods across research areas improves performance on 9 of 15 MULTIBENCH datasets, especially in healthcare, HCI, robotics, and finance.

3 MULTIZOO: A Zoo of Multimodal Algorithms

  • MULTIZOO implements 20 methods spanning data preprocessing, fusion paradigms, optimization objectives, and training procedures.The paper introduces the algorithms using two modalities for notational convenience and refers readers to Appendix E for detailed descriptions and implementations.Source: §3 · p. 5

3.1 Data Preprocessing

  • WORDALIGN aligns remaining modalities to a shared temporal granularity, such as words, when text is one of the modalities [26].The approach assumes that the modalities have a temporal granularity.Source: §3.1 · p. 5

3.2 Fusion Paradigms

  • MULTIZOO includes early fusion of inputs, late fusion of unimodal features, Tensor Fusion, and its efficient low-rank approximation [10] [106].Tensor products capture higher-order cross-modal interactions, but their dimensions scale exponentially with the number of modalities.Source: §3.2 · p. 5
  • Fusion methods also include multiplicative interactions, gated attention, crossmodal Transformers, and architecture search [154] [122] [173].Constrained multiplicative interactions yield matrix, vector, or scalar outputs; MULT implements bidirectional crossmodal attention, while MFAS searches over atomic operations.Source: §3.2 · pp. 5–6

3.3 Optimization Objectives

  • MULTIZOO includes objectives for prediction- and feature-level alignment, input reconstruction, and robustness to noisy or missing modalities [145] [135] [123].CCA maximizes correlation between auxiliary predictions, REFNET uses contrastive alignment, and MFM jointly reconstructs inputs and predicts labels. MCTN uses cycle-consistent translation and reconstruction losses; although trained on multimodal data, it accepts one modality at test time.Source: §3.3 · p. 6

3.4 Training Procedures

  • GRADBLEND weights modalities using their generalization statistics, while RMFE uses functional entropy to balance their contributions [53].These methods address the observation that modalities can overfit and generalize at different rates during direct multimodal training.Source: §3.4 · p. 6
  • Methods from different research areas achieve stronger performance on 9/15 MULTIBENCH datasets, especially in healthcare, HCI, robotics, and finance.The comparison distinguishes each dataset’s best in-domain method from the best out-of-domain method.Source: Table 2 · p. 7

3.5 Putting Everything Together

  • MULTIZOO standardizes and disentangles model, optimization, and training modules, enabling multimodal models to be trained in fewer than 10 lines of code.Its sample workflow loads a MULTIBENCH dataset, defines the model and training procedures, then runs the evaluation protocol.Source: §3.5 · p. 7

4 Experiments and Discussion

MultiBench standardizes multimodal comparisons and reveals gains from cross-area methods alongside persistent tradeoffs in generalization, complexity, and robustness.

Benefits of standardization

  • Using a shared training loop, cross-area methods improve state-of-the-art performance on 9 of 15 MultiBench datasets.The setup varies only the method of interest while keeping other potentially confounding factors constant.Source: §4 · p. 7

Generalization across domains and modalities

  • Multimodal methods often fail to generalize across domains, while modality-agnostic late fusion is broadly applicable but never best on a dataset.MFAS [122], MULT [154], and GRADBLEND perform strongly in their target settings but struggle elsewhere; performance varies substantially across datasets.Source: §4 · p. 7
  • Several methods are modality-limited: MFAS and CCA target two modalities, while TF and MI do not scale efficiently beyond two or three.The paper encourages generalizing these approaches across datasets and modalities.Source: §4 · p. 8

Tradeoffs between modalities

  • Decent performance is possible with the best unimodal modality, while multimodal improvements may require around 2−3× the parameters.The comparison is reported from Table 2.Source: §4 · p. 8

Tradeoffs between performance and complexity

  • Simple fusion scores highly on performance and complexity, while complex methods are only slightly better on average but excel on well-studied datasets.LF is easier to adapt to new domains; adapting methods such as MFAS and MULT to new datasets can be difficult.Source: §4 · p. 8, Figure 3 · p. 8

Tradeoffs between performance and robustness

  • Performance correlates positively with relative robustness but negatively with effective robustness, and few models achieve both forms of robustness.Some well-performing methods, including MULT, CCA, and MVAE, drop off faster after equalizing for clean-test accuracy.Source: §4 · p. 8

5 Related Work

MultiBench is positioned as a broader multimodal benchmark than prior area-specific benchmarks and language–vision-focused methods. It also provides a way to evaluate diverse modalities and tasks while extending analysis of multimodal representations across datasets and models.

Comparisons with related benchmarks

  • MULTIBENCH is presented as the first benchmark spanning many datasets, modalities, and tasks, beyond prior benchmarks focused on single areas [50].It aims to encourage work on understudied modalities such as tabular data, time-series, sensors, graphs, and sets.Source: §5 · pp. 8–9

Standardizing multimodal learning

  • Prior multimodal task-suite models and toolkits were limited to language and vision or tailored to text and images, while MULTIBENCH goes beyond Transformers and text to assess performance, complexity, and robustness.The passage notes that Transformers suit language and image data and that a toolkit incorporated multimodal data on top of text-based Transformers.Source: §5 · p. 9

Analysis of multimodal representations

  • Prior analyses found models relying on unimodal statistics or dominant modalities, but were limited to specific datasets and models; MULTIBENCH enables scalable analysis across modalities, tasks, and models [59].The cited work reports that some models do not learn cross-modal interactions and may ignore less dominant modalities.Source: §5 · p. 9

6 Conclusion · Appendix

MultiBench unifies previously disjoint multimodal research efforts through an accessible, reproducible benchmark, and its broad evaluation scope points toward more generalizable, lightweight, and robust models.

6 Conclusion

  • The benchmark currently focuses on multimodal fusion and has limits in research areas, supported models, and metrics, while leaving room for domain-specific models and interpretability research.The paper notes that defining interpretability beyond image and text remains an open question.Source: §6 · pp. 9–10
  • Planned growth includes community-suggested domains and datasets, workshops and competitions, and expansion into question answering, retrieval, and multimodal reinforcement learning.The benchmark already includes Yummly-28K for paired video and recipe descriptions, Clotho for audio captioning, and the language-guided RTFM environment.Source: §6 · p. 10
  • Planned evaluation additions include distribution-shift and uncertainty tests, fairness and social-bias assessments, and interpretable-learning metrics including EMAP.The proposed EMAP score assesses whether cross-modal interactions improve performance.Source: §6 · p. 10
  • Transfer learning from UR-FUNNY and CMU-MOSEI raised CMU-MOSI performance from 75.2 to 75.8 with the same late-fusion model.The initial experiments found that pre-training on larger datasets in the same domain could help smaller datasets after fine-tuning.Source: §6 · p. 10
  • Multi-dataset multitask learning raised CMU-MOSEI late-fusion performance from 79.2 to 80.9 and transformer performance from 82.1 to 82.9, but lowered CMU-MOSI performance from 75.2 to 70.8.One model trained across four affective-computing datasets used adjustable input embeddings and dataset-specific classification heads.Source: §6 · p. 10
  • MultiBench unifies multimodal research with a focus on ease of use, accessibility, and reproducibility, while highlighting directions for more generalizable, lightweight, and robust models.The authors describe it as a large-scale benchmark spanning research areas, datasets, modalities, tasks, and evaluation metrics.Source: §6 · p. 10

Appendix

  • The appendix is organized around dataset selection and domain details, benchmark distribution and maintenance, performance, complexity, robustness, modeling methods, evaluation, and future directions.Its contents also list documentation, hosting, persistence, and broader outreach.Source: Appendix · pp. 21–22

A Broader Impact Statement … C MULTIBENCH Datasets

MULTIBENCH standardizes multimodal research and evaluates fusion across performance, complexity, and robustness, while identifying privacy and social bias as concerns for responsible deployment. Its broader impact includes enabling study of these issues and future evaluation of fairness and privacy.

A Broader Impact Statement

  • MULTIBENCH reduces repeated preprocessing and code-adaptation work, standardizes core approaches to support reproducibility and sharing, and explicitly tests complexity to encourage lightweight, greener models.The paper notes that adapting publicly released methods to its standardized implementation still required effort.Source: A Broader Impact Statement · p. 24
  • Because multimodal human data can pose privacy risks, the authors stress consent, use only public data released with participant or creator consent, and anonymize data and remove personal and protected attributes.The experiments use these datasets for research purposes.Source: A Broader Impact Statement · p. 24
  • Imbalanced data may expose or amplify social biases, and MULTIBENCH enables large-scale study while planning fairness and privacy evaluations and analysis of trade-offs with performance.The authors identify fairness-performance trade-offs as an important focus for future multimodal research.Source: A Broader Impact Statement · pp. 24–25
  • Dataset safeguards differ by domain: affect and action-recognition models are for scientific study, healthcare data are de-identified and research-only, and finance, robotics, and HCI datasets contain no personal human data.Affective-computing features are non-invertible and cannot identify speakers; multimedia sources are public or licensed, and the paper reports no offensive content in the reviewed videos.Source: A Broader Impact Statement · p. 25

B Background: Multimodal Representation Learning

  • The background frames multimodal representation learning around core technical challenges, and MULTIBENCH evaluates fusion against performance, complexity, and robustness desiderata.The broader benchmark discussion distinguishes multimodal fusion from tasks such as question answering and grounding.Source: B Background: Multimodal… · p. 26, §B.2 · p. 26, §B.1 · p. 26

B.1 Problem Statement

  • Multimodal fusion combines heterogeneous modalities to predict a label; this benchmark version focuses on fusion, where a single modality should also suffice to define the prediction task.Modalities may be static, such as images or tabular data, or temporal, such as language, video, and time series.Source: §B.1 · p. 26

B.2 Technical Challenges

  • Suitable unimodal encoders, higher-order cross-modal interactions, and cross-modal and temporal alignment are key performance challenges in multimodal fusion.Encoders must capture task-relevant information at the right granularity, while models must identify interactions that generalize to unseen data.Source: §B.2 · pp. 26–27
  • Multimodal models face large time and space costs and noisy or missing inputs, motivating lightweight models and robustness tests under realistic imperfections.MULTIBENCH tests these complexity and robustness desiderata alongside performance.Source: §B.2 · pp. 26–27

C.1 Dataset Selection … C.2.6 Multimedia

This section describes MultiBench’s dataset selection, emphasizing realism, research-area and modality diversity, and room for model improvement, and documents benchmark inputs, prediction tasks, preprocessing, and splits across affective computing, healthcare, robotics, finance, HCI, and multimedia. It also covers resources for examining noisy modalities and large-scale learning.

C.1 Dataset Selection

  • Selection prioritizes real-world realism, research-area and modality diversity, and tasks where models remain far from human performance, while inviting community expansion.The selection process incorporates domain-expert input and seeks datasets that challenge machine-learning models and offer potential for real-world transfer.Source: §C.1 · pp. 28–29

C.2.1 Affective Computing

  • CMU-MOSI [181] and CMU-MOSEI, UR-FUNNY, and MUSTARD [24] cover sentiment, emotion, humor, and sarcasm across language, video, and audio.CMU-MOSEI supports sentiment and multiple discrete and continuous emotion tasks, while MUSTARD uses contextualized audiovisual utterances to study sarcasm where modalities may disagree.Source: Figure 6 · p. 29, §C.2.1 · pp. 29, 31

C.2.2 Healthcare

  • MIMIC-III pairs hourly 24-hour medical time series with static patient features for mortality and two ICD-9 prediction tasks.The data are de-identified under HIPAA standards; access requires training and approval, while MultiBench provides scripts to preprocess and split the data.Source: §C.2.2 · pp. 32–33

C.2.3 Robotics

  • Robotics coverage contrasts simulated MUJOCO PUSH, which predicts a pushed object's planar pose, with real-world VISION&TOUCH peg insertion using visual, force, and proprioception data.MUJOCO PUSH combines grayscale images, force/contact information, and end-effector position; VISION&TOUCH also records robot actions during the contact-rich task.Source: §C.2.3 · pp. 33–34
  • VISION&TOUCH pairs proprioception, RGB-D, and force-torque sensing with robot actions to predict next-step binary contact and end-effector position.The data comprises 150 trajectories of 1,000 timesteps; after using next-timestep labels, 49 of 50 timesteps per file remain usable, and validation performance is reported because no test set exists.Source: §C.2.3 · pp. 34–35

C.2.4 Finance

  • Finance benchmarks use correlated stock groups as modalities to predict next-day squared returns from 500 previous returns, with chronological train, validation, and test splits.For MULT, stocks are evenly divided into three groups to keep memory use practical; the splits contain 3,200, 500, and 1,017 continuous days, respectively.Source: Figure 9 · p. 35, §C.2.4 · p. 36

C.2.5 HCI

  • ENRICO combines app screenshots and view hierarchies, represented as wireframes and encoded with pretrained VGG-11, for design-motif classification.The labeled subset contains 1,460 screens across 20 design categories; deterministic splits allocate 65% to training, 15% to validation, and 20% to testing.Source: §C.2.5 · p. 37

C.2.6 Multimedia

  • AV-MNIST pairs spoken-digit audio with MNIST images for 10-class prediction, challenging fusion with reduced visual energy and ESC-50 background noise.The source datasets are FSDD and MNIST; the paper cautions that its preprocessing may differ from other versions, so performance should not be compared directly across papers.Source: §C.2.6 · pp. 37–38
  • MM-IMDB [8] provides plots, posters, and metadata for 25,959 movies, with the task of multilabel classification into 23 genres.The genre-based split contains 15,552 training, 2,608 validation, and 7,799 test movies.Source: §C.2.6 · pp. 38–39
  • KINETICS [80] contains 306,245 clips across 400 human actions; small and full partitions support experiments at different computational scales.KINETICS-S has 2,624 clips and takes roughly two hours to train a basic supervised model on one GPU, while KINETICS-L takes roughly two weeks.Source: §C.2.6 · p. 39

C.3 Documentation … D.3.2 Multimodal Imperfections

This section documents MultiBench’s datasets, research uses and safety limits, distribution and maintenance, and protocols for evaluating performance, complexity, and robustness. It describes modality-specific and cross-modal tests for perturbations, correlated noise, and missingness, including spatial, temporal, and element-level changes tailored to each data type.

C.3 Documentation

  • MultiBench spans 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas, with an automated pipeline for data loading, experimental setup, and evaluation.The benchmark also provides a standardized implementation of 20 core multimodal learning approaches.Source: §C.3 · p. 40
  • Dataset sizes range from 1,460 instances, trainable in minutes on one GPU, to 306,245 instances, requiring roughly two weeks on one GPU.This range is intended to support both researchers with limited computational resources and large-scale studies.Source: §C.3 · p. 41
  • Datasets are samples intended to represent larger populations, include raw data and processed features, and come with train, validation, and test splits aligned with prior work.The authors describe dataset collection, features, labels, and splits in Appendix C.2.Source: §C.3 · pp. 41–42
  • All datasets are provided in full, while robustness tests inject noise and imperfections to simulate real-world conditions.The raw data also supports user-defined feature extraction and robustness tests, except where redistribution restrictions apply.Source: §C.3 · pp. 41, 45
  • Most datasets are self-contained, but MIMIC and finance data cannot be redistributed; users must obtain those raw data through the original sources.MultiBench provides automated download and preprocessing scripts for these restricted datasets once users obtain the data through the required procedures.Source: §C.3 · pp. 42, 45
  • MIMIC is rigorously de-identified under HIPAA, and the authors caution that models trained for prediction should be used only for scientific study, not real-world prediction.MultiBench includes financial and healthcare data that are publicly available for research purposes.Source: §C.3 · p. 43
  • Beyond prediction, MultiBench datasets can support unsupervised learning, modality-relationship interpretation, robustness and alignment research, and tasks such as co-learning, translation, retrieval, and grounding [10].The passage also identifies unsupervised multimodal pre-training as a possible use.Source: §C.3 · p. 45
  • The authors caution that multimodal datasets and models may pose privacy, security, bias, and deployment risks, and should be used only for research, not real-world decisions.Predictions concerning affective states, human actions, health, and financial indicators are identified as particularly sensitive.Source: §C.3 · p. 46

C.4 Benchmark Distribution

  • MultiBench is distributed through public GitHub and a landing page with dataset and algorithm links and a leaderboard, with channels for community feedback and contributions.The authors also plan to publish future updates and target release dates on the webpage.Source: §C.4 · p. 48

C.5 Hosting and Maintenance

  • The authors plan long-term hosting and maintenance while expanding datasets, evaluation, and methods to cover additional tasks, robustness to distribution shift, uncertainty, fairness, social bias, and interpretability.Planned dataset areas include retrieval, question answering, grounding, and reinforcement learning.Source: §C.5 · p. 48

D MULTIBENCH Evaluation Protocol

  • The evaluation protocol uses dataset-specific metrics and summarizes generalization through performance variance across datasets and comparisons of in-domain versus out-domain results.Metrics include MSE and MAE for regression and accuracy, micro and macro F1-score, and AUPRC for classification.Source: D MULTIBENCH Evaluation Protocol · p. 50, §D.1 · p. 50

D.2 Complexity

  • MultiBench records training data size, parameter counts, time, and peak memory, and measures inference time and parameters to assess model complexity.Training-time comparisons use the same CPUs and GPUs across models on each dataset.Source: §D.2 · p. 50

D.3 Robustness to Imperfect Data

  • Because real-world data may contain missing entries, noise, or absent modalities, MultiBench includes standardized robustness tests for imperfect multimodal inputs.The motivation includes potential performance drops when systems trained on acted dialogue encounter less expressive real-world users.Source: §D.3 · p. 50

D.3.1 Modality-specific Imperfections

  • MultiBench tests text with simulated spelling and short-message errors and letter permutations, and images with noise, color changes, flips, crops, and rotations.Language errors include adjacent-key typos and changes such as repeating or omitting letters; image noise includes Gaussian and salt-and-pepper corruption.Source: §D.3.1 · p. 51
  • Image tests include translations, while video receives image-specific tests per frame plus random and structured temporal drops.Structured drops remove m consecutive time steps containing at least one nonzero signal with probability p.Source: §D.3.1 · pp. 51–52
  • Audio tests add Gaussian noise and apply random or structured time-step drops; the suite also describes noise and missing-value tests for time series following Liang et al. [101].Optical flow is treated like time-series data and receives the same robustness tests.Source: §D.3.1 · p. 52
  • Force and proprioception sensors receive more frequent noise or time-step drops, while tabular data uses random element drops or swaps.Set-valued inputs instead use random element dropping or zero-mean Gaussian noise with standard deviation p.Source: §D.3.1 · p. 52

D.3.2 Multimodal Imperfections

  • Multimodal tests model correlated collection imperfections through shared noise or drops, temporal drops across parallel modalities, structured temporal drops, and whole-modality removal.The motivating examples include missing modalities [123] and missing chunks in multimodal time series [101].Source: §D.3.2 · pp. 52–53

D.3.3 Robustness Measure … E.1 Selection of Algorithms in MULTIZOO

MultiBench measures robustness by testing clean-trained models under increasingly noisy test conditions and comparing robustness across models. It also standardizes cross-dataset aggregation and selects MULTIZOO algorithms for breadth, technical coverage, dataset performance, and reproducibility.

D.3.3 Robustness Measure

  • Test robustness by adding increasing noise only to clean-trained models’ test data, creating modality-specific partitions and, where possible, an additional multimodal-imperfection partition.When modalities lack a shared dimension, only the modality-specific partitions are created.Source: §D.3.3 · p. 53
  • A qualitative robustness plot tracks performance from no noise to complete noise; robust models should retain high accuracy or low MSE as imperfections increase.The method evaluates unimodal or multimodal models trained on clean data.Source: §D.3.3 · pp. 53–54
  • Relative robustness compares a model’s performance-imperfection curve with the late-fusion concatenation baseline (LF), using the area between curves across noise levels.A discrete approximation to the integral is computed as imperfection σ increases from 0.0 to 1.0.Source: §D.3.3 · p. 53
  • Effective robustness measures performance above the LF baseline’s accuracy trend, controlling for differences in clean-test accuracy, following Taori et al. [149].It is the area between a model’s curve and the LF curve shifted to match the model’s initial accuracy; higher values indicate performance above that shifted curve across imperfection levels.Source: §D.3.3 · pp. 53–54

D.4 Aggregating Measures Across Datasets and Tasks

  • Min-max normalization maps each dataset’s metrics to 0–1 before averaging, reversing the scale for RMSE/MSE to aggregate results across datasets.This provides an indicator of overall performance despite differences in metric scales and direction.Source: §D.4 · p. 54

E MULTIZOO: A Zoo of Multimodal Algorithms

  • MULTIZOO standardizes multimodal methods to improve accessibility and reproducibility, covering data processing, fusion, optimization objectives, and training procedures.The approaches address alignment granularity, complementary information across aligned signals, and robustness to noisy or missing modalities.Source: E MULTIZOO: A Zoo of Multimodal… · p. 55, Table 3 · p. 55

E.1 Selection of Algorithms in MULTIZOO

  • MULTIZOO algorithms were selected for diversity, technical-challenge coverage, state-of-the-art performance on chosen datasets, and community expansion.The selection spans distinct approaches within categories, including domain-informed architectures, general-purpose Transformers, and architecture search; researchers are encouraged to add methods using a modular structure to support standardized sharing and reproducibility.Source: §E.1 · p. 55

E.2 Data Preprocessing … F.5 Optimization Objectives

These sections describe a standardized toolkit spanning temporal alignment, multimodal fusion, specialized objectives and training procedures, and modular data-to-prediction components. They also specify adaptations for scalability, robustness, and domain-specific tasks.

E.2 Data Preprocessing

  • WORDALIGN [26] aligns higher-frequency non-text features to word intervals by averaging them within each word’s start and end times, yielding equal-length text and non-text sequences.The implementation uses temporal alignments provided by the CMU-MultimodalSDK.Source: §E.2 · pp. 55–56

E.3 Fusion Paradigms

  • MULTIZOO implements early and late fusion, tensor and low-rank tensor fusion, and multiplicative-interaction variants; tensor products capture higher-order interactions but scale exponentially with modalities.The interaction variants include matrix, vector, and scalar forms corresponding to HyperNetworks, FiLM, and sigmoid units.Source: §E.3 · p. 56
  • Gated attention dynamically weights one stream from another, while MULT cross-attends modality pairs and scales quadratically, prompting three-cluster adaptation for financial data with more than 10 modalities; MFAS searches architectures via nested loops [122].MULT becomes intractable as modality count grows, so the financial task groups modalities by data similarity and applies MULT to the three clusters.Source: §E.3 · pp. 56–57

E.4 Optimization Objectives

  • CCA aligns unimodal label predictions through batch correlation, while REFNET aligns unimodal and multimodal representations with cosine-based contrastive loss [135].REFNET’s original loss does not use negative samples; related multi-view contrastive work adds them.Source: §E.4 · p. 57
  • MFM combines input reconstruction and label prediction, while MCTN uses cycle-consistent translation and reconstruction losses so it can predict from one modality when another is noisy or missing [123].MCTN’s test-time robustness follows from requiring only x1 when x2 is noisy or missing.Source: §E.4 · p. 58

E.5 Training Procedures; E.6 Domain-specific Methods

  • GRADBLEND weights modalities using generalization statistics, and RMFE uses functional entropy to balance their contributions to classification [53].Both are included in MULTIZOO’s training procedures.Source: §E.5 · p. 58, §E.6 · p. 58

F Integrating MULTIBENCH and MULTIZOO: A Brief Tutorial; F.1 Reading the Dataset; F.2 Unimodal Models; F.3 Multimodal Models; F.4 Classification Head

  • MULTIBENCH provides dataset loaders, while MULTIZOO standardizes unimodal models, reusable multimodal fusion modules, and classification heads within a modular pipeline.The unimodal models cover modalities including images, audio, text, time series, and sets; users can also add specialized fusion modules.Source: F Integrating MULTIBENCH and MULTIZOO:… · p. 59, §F.1 · p. 59, §F.2 · p. 59

F.5 Optimization Objectives

  • MULTIZOO combines standard classification or regression losses with optional reconstruction, ELBO, alignment, and contrastive objectives as a user-weighted sum.Users specify the weights for prediction and auxiliary objectives as hyperparameters.Source: §F.5 · p. 60

F.6 Training Structures … F.10 Code Snippets

MULTIZOO separates training structures from exchangeable model components, standardizes performance, complexity, and robustness evaluation, and provides concise code for reproducible multimodal experiments.

F.6 Training Structures

  • Training structures define the main algorithm, while unimodal models, fusion, objectives, and heads act as exchangeable plugins; advanced methods use distinct structures for added objectives, dynamic modality weighting, or architecture search.The framework also lets researchers combine fusion paradigms with training-structure advances; methods using different fusion paradigms can share supervised_learning while using different fusion modules.Source: §F.6 · p. 60

F.7 Performance Evaluation

  • Performance evaluation uses dataset-specific metrics, including MSE and MAE for regression and accuracy, micro and macro F1-score, and AUPRC for classification.The metrics use standard PyTorch and scikit-learn implementations.Source: §F.7 · p. 60

F.8 Complexity Evaluation

  • Training memory is measured as peak Python-process memory over training, and parameter counts include persistent training modules but exclude ephemeral networks or modules.Excluded examples include GRADBLEND weight-determination networks and fusion architectures created during MFAS architecture search.Source: §F.8 · p. 61

F.9 Robustness Evaluation

  • Robustness tests use dataset-specific loaders to add unimodal or multimodal imperfections at increasing noise levels σ ∈[0,1] to clean test data, with a dedicated module evaluating and reporting metrics across noise levels.This modular approach requires minimal changes to the regular training and testing workflow.Source: §F.9 · p. 61
  • Robustness protocols visualize performance-imperfection curves and quantify relative robustness by approximating their area and effective robustness against an LF curve matched on clean-test initial accuracy.Both metrics normalize performance across models evaluated on the same dataset.Source: §F.9 · p. 61

F.10 Code Snippets

  • A Python example loads a MULTIBENCH dataset, defines architectures, objectives, and training, then runs evaluation; MULTIZOO trains entire multimodal models in less than 10 lines of code.Standardized modules disentangle the effects of models, optimizations, and training to support accessibility and reproducibility.Source: §F.10 · p. 61

G Experimental Setup … H Experimental Results

The experiments used domain-specific hyperparameter choices and repeated runs, while the results section reports MULT as best on the affective-computing datasets and defines its metric notation.

G Experimental Setup

  • Experiments ran on a server with 4 GTX 980 Ti, 5 Tesla P40, 2 Tesla K40c, 4 TITAN X, 1 Tesla T4, and 1 Tesla V100 GPU, plus 32 Xeon E5-2670 CPUs.Source: G Experimental Setup · p. 62

G.1 Affective Computing

  • Affective-computing hyperparameters were tuned by dataset, selected using validation performance, and evaluated across 10 repeated runs.Learning rates ranged from 0.00001 to 0.001, with 0.0001 as the initial setting; UR-FUNNY and MUSTARD used only target punchline sentences.Source: §G.1 · pp. 62–63

G.2 Healthcare

  • Healthcare experiments followed the MIMIC partition's original unimodal architectures, tuned learning rates from 0.1 to 0.0001, and repeated each experiment 10 times.Epochs were selected based on overfitting, while architecture- or training-specific settings such as GRADBLEND and MFAS followed their original papers.Source: §G.2 · p. 63

G.3 Robotics

  • Robotics experiments reused original-paper settings for MUJOCO PUSH [90] and VISION&TOUCH [91], reporting VISION&TOUCH's best validation performance because it lacked a unique test set.Other hyperparameters were selected by validation performance, previously tested methods reused reported settings, and each experiment was repeated 10 times.Source: §G.3 · p. 63

G.4 Finance

  • Finance experiments tuned embedding dimensions, Transformer/MULT layers and heads, epochs, and batch size on validation performance, keeping several settings small because the dataset overfits quickly.Experiments were repeated 10 times, with means and standard deviations computed.Source: §G.4 · p. 63

G.5 HCI

  • For ENRICO, a learning rate of 10^-5 performed best and hidden dimension 16 sufficed for unimodal encoders; the small dataset overfits as model complexity increases.The learning-rate search spanned 10^-2 to 10^-6, starting from 10^-4 reported in the original paper; experiments were repeated 10 times.Source: §G.5 · p. 63

G.6 Multimedia

  • Multimedia setups used dataset-specific encoders and prior configurations, with validation-selected hyperparameters and 10 repeated runs.AV-MNIST used LeNet encoders; MM-IMDB used MaxoutLinear encoders [8], with early stopping after 7 validation epochs without macro-F1 improvement. KINETICS used ResNet-LSTM for vision and Wang et al.'s architectures, with a 0.0001 learning rate, batch size 16, and 15 epochs for small-dataset experiments.Source: §G.6 · pp. 63–64

H Experimental Results

  • MULT is the best-performing model on all affective-computing datasets, while many out-of-domain methods struggle.Results tables use Acc(c) for accuracy with c classes, AUPRC for area under the precision-recall curve, and MSE for mean squared error; upward and downward arrows mark metrics where higher and lower are better, respectively.Source: Table 13 · p. 71, H Experimental Results · p. 71

H.1 Affective Computing … H.4 Finance

Across affective computing, healthcare, robotics, and finance, multimodal models show domain- and task-dependent performance and robustness: they can outperform unimodal models, but noise, modality count, and dataset characteristics expose important limitations.

H.1 Affective Computing

  • MULT performs best across the affective datasets and generalizes beyond sentiment and emotion to humor and sarcasm, but takes over 12× unimodal inference time.Language is usually strongest for sentiment and emotion, while its advantage over audio and video is smaller for humor and sarcasm. MULT also takes 3–4× the inference time of several simpler fusion baselines; TF, MVAE, and MFM do not generalize well to sarcasm, and MCTN struggles on that task.Source: §H.1 · p. 71
  • Across affective datasets, performance drops rapidly toward random as noise increases, though multimodal models are slightly more robust overall, especially to audio and video noise.Language-only models can be more robust than multimodal models to language noise; GRADBLEND appears robust across datasets, while MCTN stays constant under video or audio imperfections because it uses only language at test time.Source: Figure 12 · p. 73, Figure 13 · p. 73, Figure 14 · p. 73

H.2 Healthcare

  • On MIMIC, results vary little across models; Late Fusion performs best on ICD-9 group 7 and is close to the top models on the other two tasks.Reconstruction-based MVAE and MFM also perform strongly, possibly because the input modalities are low-dimensional; MFAS, despite broader generality elsewhere, struggles on MIMIC.Source: §H.2 · p. 75
  • On MIMIC, noise quickly degrades unimodal and multimodal models, but multimodal models are generally slightly more robust, with robustness varying sharply by task.LRTF is most robust on ICD-9 group 7 but least robust on the similar ICD-9 group 1 task.Source: §H.2 · p. 76, Figure 16 · p. 76

H.3 Robotics

  • In robotics, one modality is exceptionally strong on each task, and Late Fusion outperforms Sensor Fusion on both VISION&TOUCH tasks.The authors note that action is generally treated as a modality without specialized modeling and identify further robotics and reinforcement-learning tasks as future benchmarking needs.Source: §H.3 · p. 77
  • In robotics, multimodal models are more robust to noise than unimodal models on both datasets; Sensor Fusion and Late Fusion appear most robust on VISION&TOUCH.Unimodal performance is especially poor for object-pose prediction; the reported robustness aligns with prior findings that multimodal models can recover from force-sensor disturbances or image occlusions [91].Source: §H.3 · p. 77, Figure 17 · p. 78, Figure 18 · p. 79

H.4 Finance

  • Finance results favor multimodal models over unimodal ones overall, but vary across partitions; Early Fusion works well, while Transformers perform worse than LSTMs.The datasets contain 18, 63, or 100 modalities, requiring TF and MULT to be adapted by clustering modalities into 2–3 groups before fusion; this may partly explain their weaker performance. Early Fusion may benefit from the relatively homogeneous time-series inputs.Source: Table 19 · p. 80, §H.4 · p. 80
  • In finance, MULT and sometimes Early Fusion are more robust than the best unimodal model, but Late Fusion performs worse; GRADBLEND starts with the lowest MSE yet is least robust.Some methods on STOCKS-HEALTH and STOCKS-TECH may be underfitting, as they begin with poor MSE and change little as noise increases.Source: §H.4 · pp. 80, 82, Figure 19 · p. 81

H.5 HCI

On ENRICO, multimodal models can improve prediction over the best unimodal model, but more complex methods do not consistently help. Robustness also varies: late fusion is stable to image noise, while some multimodal models perform worse than the best unimodal model.

H.5 HCI

  • Multimodal models improve ENRICO performance over the best unimodal model by 4%, while TF, LRTF, and MI offer no improvement over simple LF.The authors hypothesize that the more complex models have more trainable parameters and are more prone to overfitting small, noisy datasets.Source: §H.5 · p. 83
  • The reproduction outperformed the ENRICO paper’s reported set-modality result but performed worse for screenshot modality; the paper provides no code or many experimental details.Source: §H.5 · p. 83
  • LF and sometimes GRADBLEND are more robust than the best unimodal model, but TF performs worse and is not robust; LF remains stable under image noise by relying on the set modality.Robustness results are shown for increasing noise levels in Figure 20.Source: §H.5 · p. 83, Figure 20 · p. 84
  • At the highest noise levels, multimodal-model performance ranges from 5% to 40%, indicating high variance in robustness.Source: §H.5 · p. 84

H.6 Multimedia

The multimedia results show dataset-specific differences in model performance, training cost, and robustness. On MM-IMDB, multimodal models are more robust than unimodal models, particularly to image noise, but less so to language noise.

H.6 Multimedia

  • On AV-MNIST, MFAS remains the best method, beating MVAE by 0.5%, while GRADBLEND performs worse than all other multimodal methods.Source: §H.6 · p. 85
  • On MM-IMDB, REFNET contrastive learning and CCA outperform architectural baselines without alignment objectives, and CCA generalizes from affect recognition to multimedia.Source: §H.6 · p. 85
  • On KINETICS-S, the visual unimodal model slightly outperforms late fusion despite using fewer modalities, and GRADBLEND also struggles on this smaller partition.Source: §H.6 · p. 85
  • On KINETICS, simple late fusion cannot beat unimodal methods, while GRADBLEND improves multimodal performance at ~3× training time and struggles on AV-MNIST and KINETICS-S.The authors call for lightweight, effective multimodal models on KINETICS and other MultiBench datasets.Source: §H.6 · pp. 85–86
  • On MM-IMDB, multimodal models outperform unimodal models in initial performance and robustness, especially to image imperfections; the robustness gap is smaller for text noise.Overall, multimodal models are more robust to image than language imperfections, although language is more informative and models tend to overfit to it.Source: §H.6 · pp. 86–87, Figure 21 · p. 87
  • On MM-IMDB, MFM has poor initial performance and robustness; the authors attribute its weakness to difficult reconstruction of high-dimensional image and text inputs.Source: §H.6 · p. 87
  • Across MultiBench, many methods perform strongest on their in-domain datasets and decline out of domain, with high performance variance across datasets.Source: Figure 22 · p. 88

H.7 Performance … I Future Directions

Applying methods across research areas achieves state-of-the-art performance on 9 out of 15 tasks, but multimodal models often generalize unevenly across domains, modalities, and datasets. The analysis highlights tradeoffs among performance, complexity, and robustness, and calls for broader benchmarks and adaptable methods.

H.7 Performance

  • Applying methods from different research areas achieves state-of-the-art performance on 9 out of 15 tasks, especially in less-studied healthcare, finance, and HCI domains.The authors argue that standardizing and unifying multimodal research can help accelerate progress.Source: §H.7 · p. 88, §H.10 · p. 93
  • Many methods generalize poorly across domains and datasets, with performance varying by research area and no one-size-fits-all model, especially for understudied modalities and tasks.MFAS and MCTN fail to transfer to cited datasets outside their design domains; GRADBLEND also performs poorly beyond the video and audio datasets on which it was tested. LF is comparatively stable and generalizable, while some methods are difficult to adapt beyond particular modalities or domains.Source: §H.7 · pp. 88–89, Figure 23 · p. 89
  • Decent performance is possible with the best-performing unimodal modality, while further multimodal improvement may require around 2 −3× the parameters.The authors call for datasets that pose challenges and opportunities not achievable with unimodal methods.Source: §H.7 · p. 89, §H.10 · p. 93

H.8 Complexity

  • Simple fusion methods such as early fusion and late fusion score highly on both performance and complexity, while complex methods can perform slightly better; the best unimodal model falls below the Pareto curve.LF lies above the curve and is described as an appealing choice.Source: Figure 24 · p. 90, §H.8 · p. 90
  • Adapting MFAS and MULT to new datasets is difficult, and training-time complexity does not capture their fast, cheap test-time inference or all real-world resource considerations.MFAS is most suitable for convolutional networks, while MULT is designed for multimodal time-series; the authors note that changing the complexity metric could change the observed tradeoffs.Source: §H.8 · p. 91

H.9 Robustness

  • Performance has a slight positive correlation with relative robustness but a slightly negative correlation with effective robustness, and few models achieve both.MVAE and RMFE show strong performance and robustness, while MULT, CCA, and MVAE can drop off faster after equalizing initial accuracy.Source: Figure 25 · p. 92, Figure 26 · p. 92, §H.9 · p. 93

H.10 Summary of Takeaway Messages

  • The takeaways emphasize lightweight multimodal models that generalize across datasets, since few methods combine strong performance with both relative and effective robustness.The summary also describes simple methods such as LF as surprisingly generalizable, though not the best-performing on any dataset.Source: §H.10 · p. 93

I Future Directions

  • Future work will expand MultiBench datasets and MultiZoo algorithms and broaden holistic evaluation of multimodal models.The authors also plan to maintain and continually expand the benchmark.Source: I Future Directions · p. 94

I.1 Datasets … I.4 Broader Outreach

The authors outline plans to expand MultiBench’s datasets, models, evaluation dimensions, and community support, extending the benchmark toward more modalities, tasks, and real-world concerns.

I.1 Datasets / I.1.1 Fusion

  • Planned fusion datasets include safer multimedia learning, more robotics and HCI tasks, privacy-restricted human-centric data, and multilingual affect recognition.The authors are working with dataset creators to integrate some restricted datasets; the additions aim to support privacy and fairness considerations and languages beyond English.Source: §I.1.1 · p. 94

I.1.2 Retrieval / I.1.3 Question Answering / I.1.4 Grounding

  • MultiBench plans to add cross-modal retrieval, question answering across visual, video, and embodied domains, and fine-grained grounding tasks.Retrieval aligns representations across query and retrieved modalities; question answering aligns words with queried entities, while grounding links entities across modalities and may extend beyond language-to-image tasks.Source: §I.1.2 · p. 94, §I.1.3 · p. 94, §I.1.4 · pp. 94–95

I.1.5 Reinforcement Learning

  • Future MultiBench versions will include multimodal interactive and reinforcement-learning datasets, while encouraging research beyond language and vision.These tasks require actionable representations linking instructions and perceptual inputs to actions and long-term rewards, beyond prediction and modality alignment.Source: §I.1.5 · p. 95

I.2 Models

  • MULTIZOO’s modular code structure is designed to accommodate new unimodal encoders, multimodal methods, objectives, and training structures, with community help planned for method additions.Its modular areas include data processing, model design, optimization objectives, and training structures.Source: §I.2 · p. 95

I.3 Evaluation / I.3.1 Uncertainty Estimates

  • MultiBench currently evaluates prediction performance, time and space complexity, and robustness to noisy or missing modalities, and plans to add uncertainty prediction evaluation.The proposed additions would support evaluation of multimodal models that estimate uncertainty and may abstain or defer predictions.Source: §I.3 · p. 95, §I.3.1 · p. 95

I.3.2 Robustness to Distribution Shifts

  • To benchmark multimodal distribution shifts, the authors plan domain and subpopulation data splits, established shift algorithms in MULTIZOO, and a standardized robustness evaluation pipeline.They note that shifts can affect each modality’s marginal distribution and the joint distribution across modalities; examples include affect recognition across users and robotic manipulation across robots.Source: §I.3.2 · p. 96

I.3.3 Fairness

  • Future fairness support will span human-annotated data, bias-mitigation algorithms, and evaluation metrics for unimodal and multimodal models.The authors note that fairness definitions vary by setting and that multimodal systems may compound biases across modalities; proposed tools include data annotations, algorithms in MULTIZOO, and evaluation in MultiBench.Source: §I.3.3 · pp. 96–97

I.4 Broader Outreach

  • Workshops, courses, and community proposals will use MultiBench and MULTIZOO to support reproducible research and expand supported datasets and models.The authors plan contribution instructions and compatibility tests, with proposals periodically reviewed for inclusion in future benchmark versions.Source: §I.4 · p. 97

Related papers

Loading 2107.07502v2…