PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, Ruslan Salakhutdinov

arXiv:2203.01311v4cs.LGcs.AIcs.CLcs.CVcs.MM

TL;DR

High-modality learning needs to represent diverse modalities without building separate models for each one, but efficient parameter sharing requires measuring which modalities and interactions are similar. The paper estimates modality and interaction heterogeneity through information transfer and uses those measures in HighMMT; across its evaluated tasks, the model improves the performance–efficiency tradeoff and transfers to new modalities and tasks.

  • Problem

    In high-modality learning, adding a separate model for every modality or task is prohibitively expensive, while it remains a challenge to quantify which modalities encode similar information and interactions for parameter sharing.

    Source: §1 · p. 2

  • Method

    The paper measures modality and interaction heterogeneity through transferable information, then uses the measurements to group parameters in HighMMT.

    Source: §1 · p. 2

  • Results

    Across 10 modalities, 15 tasks, and 5 research areas, HighMMT improves the performance–efficiency tradeoff over task-specific and fully shared multimodal models and transfers to new modalities and tasks.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    Transfer-based heterogeneity measures provide a way to prioritize unique modality information or interactions while sharing parameters among similar modalities and pairs.

    Source: §1 · p. 2

    The modality heterogeneity measure is a metric only under certain assumptions; positivity and the triangle inequality may fail in the general case.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots. While there has been an explosion of interest in multimodal learning, these methods are focused on a small set of modalities primarily in language, vision, and audio. In order to accelerate generalization towards diverse and understudied modalities, this paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. Since adding new models for every new modality becomes prohibitively expensive, a critical technical challenge is heterogeneity quantification: how can we measure which modalities encode similar information and interactions in order to permit parameter sharing with previous modalities? This paper proposes two new information theoretic metrics for heterogeneity quantification: (1) modality heterogeneity studies how similar 2 modalities {X1,X2} are by measuring how much information can be transferred from X1 to X2, while (2) interaction heterogeneity studies how similarly pairs of modalities {X1,X2}, {X3,X4} interact by measuring how much information can be transferred from fusing {X1,X2} to {X3,X4}. We show the importance of these 2 proposed metrics as a way to automatically prioritize the fusion of modalities that contain unique information or interactions. The result is a single model, HighMMT, that scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas. Not only does HighMMT outperform prior methods on the tradeoff between performance and efficiency, it also demonstrates a crucial scaling behavior: performance continues to improve with each modality added, and it transfers to entirely new modalities and tasks during fine-tuning.

1 Introduction

High-modality learning must handle diverse inputs without adding separate models for every modality or task. The paper proposes transfer-based measures of modality and interaction heterogeneity and applies them in HighMMT.

1 Introduction

  • As sensing platforms incorporate diverse modalities, adding new models for each modality or task becomes prohibitively expensive, making heterogeneity quantification a central challenge.The motivating setting extends beyond language, vision, and audio to inputs including time-series, proprioception, sets, and tables.Source: §1 · pp. 1–2
  • Modality heterogeneity measures transferable information between two modalities, while interaction heterogeneity measures transferable interaction information between two modality pairs.The measures are intended to quantify which modalities or modality pairs should be processed or fused separately.Source: §1 · p. 2
  • These measures prioritize fusion of modalities with unique information or interactions while supporting parameter sharing among similar modalities and pairs.HighMMT uses the measurements to dynamically group parameters to balance performance and efficiency.Source: §1 · pp. 1–2
  • Across 10 modalities, 15 tasks, and 5 research areas, HighMMT improves the performance–efficiency tradeoff over task-specific and fully shared multimodal models, transfers to new modalities and tasks, and benefits low-resource settings.The paper reports these outcomes for a single model operationalizing its heterogeneity measures.Source: §1 · p. 2

2 High-Modality Multimodal Transformer

HighMMT measures modality and interaction heterogeneity through information transfer, then uses those measurements to group model parameters. Its architecture first shares parameters across modalities and tasks, then fine-tunes parameter groups to account for heterogeneity.

2.1 Measuring Heterogeneity via Modality Information Transfer

  • Modality heterogeneity is estimated by comparing a model transferred from X1 to X2 with a model trained directly on X2, capturing task-relevant feature transfer.The measure depends on the end task rather than only on raw input differences.Source: §2.1 · pp. 2–3
  • The modality distance sums nonnegative transfer difficulty in both directions and is a metric under stated assumptions, though positivity and the triangle inequality may fail generally.In the reported real heterogeneity matrix, the relaxed triangle inequality holds 96% of the time.Source: §2.1 · pp. 3–4
  • Interaction heterogeneity compares transfer from a model pretrained on one modality pair to a target pair against direct training on that target pair.The distance aggregates transfer difficulty over tasks and transfer directions.Source: §2.1 · p. 4
  • Agglomerative hierarchical clustering uses modality and interaction heterogeneity matrices to group similar modalities and modality pairs, with cluster count set by the parameter budget.Source: §2.1 · p. 4
  • Although exact heterogeneity matrices require O(M^2) models, low-rank reconstruction can estimate them from a random sample of O(M) pairs in practice.The paper reports that rank 3 suffices to approximate the full matrix in its experiments.Source: §2.1 · p. 4

2.2 Capturing Heterogeneity and Homogeneity in HighMMT

  • HighMMT first pre-trains a fully shared model across modalities and tasks, then fine-tunes modality and interaction parameter groups according to measured heterogeneity.The grouped parameters are initialized from the shared model, while modality embeddings and task classifiers are also fine-tuned.Source: §2.2 · pp. 5–6
  • HighMMT standardizes modality inputs as padded sequences, adds modality and positional embeddings, and applies a shared Perceiver encoder to produce modality representations.The Perceiver's latent representation has runtime linear in input sequence length and dimension.Source: §2.2 · pp. 5–6
  • A shared crossmodal Transformer processes modality-representation pairs with bidirectional attention, after which task-specific classifiers use the resulting representations.For tasks with more than two modalities, a crossmodal block is applied to each modality pair before representations are concatenated.Source: §2.2 · p. 6

3 Experiments

Experiments use MultiBench data spanning 10 modalities, 15 prediction tasks, and five research areas, with more than 370,000 examples. They evaluate heterogeneity patterns, performance and efficiency, transfer, ablations, and parameter overlap and interference.

3.1 Heterogeneity Measurements and Parameter Groups

  • AV-MNIST modalities transfer poorly beyond their task, whereas the same modality across tasks is generally similar; UR-FUNNY text and MIMIC table transfer broadly.In MOSEI, video and audio are not the most informative modalities, and predictions are dominated by language.Source: §3.1 · p. 7
  • Interaction heterogeneity generally exceeds unimodal heterogeneity; AV-MNIST image+audio transfers poorly, while UR-FUNNY audio+text transfers strongly and recurring modality pairs remain cross-dataset similar.MOSEI crossmodal pairs are quite different from one another, while UR-FUNNY's are more similar.Source: §3.1 · pp. 7–8, Figure 5 · p. 7
  • A rank-3 approximation captures the low-rank heterogeneity matrices, suggesting exhaustive transfer measurement across every modality pair is unnecessary.The observed low-rank structure is attributed to symmetry and approximate triangle inequality.Source: §3.1 · p. 8

3.2 Qualitative Results

  • HighMMT's heterogeneity-aware fine-tuning advances the performance-efficiency Pareto front across datasets, sometimes beats task-specific state of the art, and uses 10× fewer parameters.Figure 6 compares normalized average performance with inverted total parameters; Table 4 reports strong performance and efficiency, sometimes surpassing task-specific state-of-the-art models.Source: Figure 6 · p. 8, §3.2 · pp. 8–9, Table 4 · p. 9
  • Performance-optimized HighMMT matches current state-of-the-art with 8× fewer total parameters; the efficiency-optimized model reaches within 96% of state-of-the-art using 30× fewer.Increasing the number of parameter groups gives a controllable performance-efficiency tradeoff.Source: §3.2 · pp. 8–9, Table 2 · p. 8
  • Across four pretraining and fine-tuning combinations, transfer improves target-task performance; UR-FUNNY gains +2.4% from zero to three pretraining datasets, while disjoint MIMIC gains +0.8%.The reported transfer results are means and standard deviations over 10 runs; gains are larger for the smallest target task and for targets with modality overlap.Source: §3.2 · p. 9, Table 3 · p. 9

3.3 Ablation Studies

  • Removing modality embeddings, unimodal encoders, or crossmodal layers worsens performance; heterogeneity-aware grouping beats alternative sharing schemes, and grouping from scratch overfits smaller datasets.The unimodal encoder is particularly important; alternatives include full sharing, full separation, random groups, and feature-distance grouping.Source: §3.3 · p. 10, Table 5 · p. 10

3.4 Understanding homogeneity and heterogeneity in HighMMT

  • More than 92% of unimodal-encoder neurons participate in at least three of four tasks, versus only 10% of multimodal-layer neurons in three or four tasks.Flipped-label training causes the largest performance drops when applied to shared unimodal encoders; task effects vary, with AV-MNIST more affected than UR-FUNNY.Source: Table 6 · p. 11, §3.4 · pp. 11–12, Table 7 · p. 11

4 Related Work

Prior work includes shared architectures for unimodal inputs and unified models for multimodal tasks. HighMMT addresses gaps in partial parameter sharing and transfer to new modalities.

Unified encoder for unimodal learning

Multimodal multitask and transfer learning

  • Unified multimodal models cover suites such as vision-language tasks, but do not investigate whether heterogeneity calls for partial parameter sharing.Examples include UniT, VLBERT, ViLBERT, VL-T5, VATT, FLAVA, and a single Transformer scaled to image, text, and decision-making tasks.Source: §4 · p. 12
  • Prior work on transfer mostly stays within a modality; Liang et al. (2021c) studies transfer to new modalities but requires paired data collection and modality-specific modeling.HighMMT also studies understudied modalities and generalization to new modality subsets.Source: §4 · p. 12

5 Conclusion

HighMMT uses modality and interaction heterogeneity to determine parameter groupings for high-modality representation learning. It achieves strong results across modalities and tasks and transfers to new modalities and tasks during fine-tuning.

5 Conclusion

  • HighMMT dynamically determines parameter groupings to balance total performance and parameter efficiency.The model uses information transfer to estimate modality and interaction heterogeneity and decide which modalities to process or fuse jointly.Source: §5 · p. 12
  • HighMMT achieves strong results across text, image, video, audio, time-series, sensors, tables, and sets, with tasks from different research areas.The conclusion presents the model as a single model for high-modality representation learning.Source: §5 · p. 12
  • HighMMT transfers to new modalities and tasks during fine-tuning, and the authors release code and benchmarks.Source: §5 · p. 12

A.1 Modality and interaction heterogeneity measures

The appendix defines modality heterogeneity through transfer between modalities and examines metric properties under assumptions. Its empirical checks find approximate triangle inequalities and low-rank heterogeneity matrices.

A.1 Modality and interaction heterogeneity measures

  • Transfer difficulty compares target-task loss after source-task initialization with loss from direct target-task training; modality distance sums the nonnegative transfer values in both directions.The transfer difference measures the difficulty of transferring a source-trained model to a target task.Source: §A.1 · p. 17
  • The modality distance is nonnegative and symmetric, but positivity may fail when positive transfer makes the distance zero for distinct modalities.Under the negative-transfer assumption, positivity is satisfied.Source: §A.1 · p. 17
  • The experiments find distinct modalities with zero distance, which the paper interprets as cases where parameters can be shared safely.Source: §A.1 · p. 18
  • The relaxed triangle inequality is derived under intermediate negative-transfer and proxy-task assumptions, with a bound using a positive constant λ.The resulting constant is c = max(1, 1/λ).Source: §A.1 · p. 18
  • The exact triangle inequality holds in 87% of unimodal and 86% of interaction cases; with c = 5.0, the relaxed version holds in 96% and 95%, respectively.The negative-transfer and intermediate negative-transfer assumptions may not hold in practice; experiments show positive and intermediate positive transfer.Source: §A.1 · p. 18
  • The top 3 singular values explain at least 97% of variance, so rank 3 approximates the heterogeneity matrices and random O(M) transfer tests can capture the approach's benefits.Source: Figure 7 · p. 19

A.2 Modality and interaction heterogeneity matrix

The heterogeneity matrices are low-rank enough to support approximate measurement, while hierarchical clustering turns their measurements into parameter groupings. HighMMT standardizes diverse inputs, processes them with shared unimodal and crossmodal components, and produces task-specific predictions.

A.2 Modality and interaction heterogeneity matrix

  • The top three singular values explain ≥97% of heterogeneity-matrix variance, making a rank-3 approximation sufficient and avoiding exhaustive transfer measurements.The approximation uses outer products of basis vectors to estimate the matrix from O(M) sampled transfer values.Source: §A.2 · p. 19

A.3 Determining parameter groupings

  • Agglomerative clustering groups modalities or modality pairs by heterogeneity, with cluster count k acting as a parameter budget between separate and fully shared models.The experiments vary k to compare performance-efficiency trade-offs.Source: §A.3 · p. 19

B HighMMT Details

  • HighMMT serializes and pads inputs, adds modality and positional encodings, applies shared unimodal and pairwise crossmodal layers, then feeds concatenated outputs to task-specific heads.The shared unimodal encoders and multimodal layers are used across tasks.Source: B HighMMT Details · p. 19

B.1 Perceiver Input Processing

  • Each modality’s channels are combined with positional and one-hot modality embeddings, then zero-padded and flattened to a common embedding size and sequence format.The processing is programmatic and has no trainable parameters.Source: §B.1 · pp. 20–21

B.2 Unimodal Perceiver Encoder

  • A shared Perceiver encoder cross-attends from trainable latent vectors to each standardized input, then applies latent self-attention and feed-forward layers.It maps variable input shapes to a common latent array, with runtime linear in input sequence length.Source: §B.2 · p. 21

B.3 Crossmodal Transformer layer

  • For each modality pair, shared crossmodal layers attend in both directions, capture correspondences, and concatenate the resulting representations.For tasks with more than two modalities, the layer is applied to each pair before concatenation.Source: §B.3 · p. 21

B.4 Task-specific classifiers

Each task uses its own classification head to map a normalized multimodal representation to predictions.

B.4 Task-specific classifiers

  • Each task-specific head concatenates crossmodal outputs, batch-normalizes them, and maps the resulting multimodal representation to task logits.The concatenated size depends on the number of modalities: 2dLS for two-modality tasks and 6dLS for three-modality tasks.Source: §B.4 · p. 21

B.5 Homogeneous multitask pre-training

HighMMT pretraining combines task losses and selects checkpoints using aggregated validation performance. Fine-tuning can split pretrained parameters according to heterogeneity, including for transfer and few-shot target tasks.

B.5 Homogeneous multitask pre-training

  • Training steps progressively add tasks with fewer batches later in each epoch, because uniform sampling can cause smaller tasks to overfit.Task losses are combined as a tunable weighted sum, and the checkpoint is selected by aggregated validation performance across tasks.Source: §B.5 · pp. 21–22

B.6 Heterogeneity-aware fine-tuning

  • Heterogeneity-aware fine-tuning initializes separate parameter groups from the homogeneous model, then fine-tunes those groups alongside modality embeddings and task classifiers.Unimodal and crossmodal parameters are grouped according to modality or interaction similarity.Source: §B.6 · p. 22

B.7 Transfer learning details

  • For transfer, the model is pretrained on source tasks and then fine-tuned on a target task, whose modalities and tasks may differ from the sources.The saved source checkpoint is chosen using aggregated validation performance on the pretraining tasks.Source: §B.7 · p. 22

B.8 Few-shot multitask learning details

  • In few-shot learning, source and target tasks are trained together, with greater loss weight assigned to the target task and checkpoints selected by its validation performance.The source-task performance is not the selection objective.Source: §B.8 · p. 22

C Experimental Setup

  • The experimental setup was designed to assess HighMMT’s multitask, transfer, and generalization capabilities.The described setup includes heterogeneous task and modality configurations.Source: C Experimental Setup · p. 22

C.1 Setup

The experiments draw on multimodal datasets across multiple research areas and evaluate combinations of tasks with different input modalities and prediction objectives. The settings range from two robotics tasks to larger cross-domain combinations.

C.1 Setup

  • The benchmark spans 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas, while the reported experimental datasets total more than 370,000 examples.The experiments cover affective computing, healthcare, multimedia, robotics, and HCI.Source: §C.1 · pp. 22–23
  • Affective-computing experiments use MOSEI and UR-FUNNY to predict sentiment, emotions, and humor from language, video, and audio time-series data.Healthcare experiments use MIMIC ICU time-series and tabular data to predict disease codes and mortality.Source: §C.1 · pp. 22–23
  • Robotics experiments use visual, force, proprioception, and contact data for object-pose and action-conditional prediction, while Enrico pairs app-screen images with app sets and locations.The robotics datasets include simulated and real robotic arms.Source: §C.1 · p. 23
  • Multimedia experiments include AV-MNIST image-audio classification and CIFAR-ESC retrieval, which links partially related label spaces through 17 shared classes.CIFAR-ESC combines 110 image classes with 50 audio classes.Source: §C.1 · p. 23
  • Three multitask settings combine different modalities and domains: two robotics tasks, three tasks across multimedia, HCI, and robotics, and four tasks across affective computing, healthcare, and multimedia.The settings contain tasks with differing modality inputs and prediction objectives.Source: §C.1 · p. 23

C.2 Hyperparameters, parameter groupings, and training details

HighMMT tunes parameter-group counts to balance sharing and computational budget, then reports additional experiments supporting its multitask, transfer, and generalization capabilities.

C.2 Hyperparameters, parameter groupings, and training details

  • The number of parameter groups k controls sharing: smaller k suits lower computational budgets, while larger k allows more groups.The evaluated k values were 2, 4, 6, 7, and 9.Source: §C.2 · p. 24
  • Unimodal and crossmodal parameter groups share selected modalities or task pairs, with the group assignments changing as their counts increase.For example, at |U| = 3, AV-MNIST image and audio share parameters; at |C| = 3, AV-MNIST and MOSEI use distinct crossmodal groups.Source: §C.2 · p. 24
  • The small, medium, and large settings cover different task collections and modality sets, with their hyperparameters listed in Tables 8, 9, and 10.The settings include robotics tasks, AV-MNIST/ENRICO/PUSH, and MIMIC/AVMNIST/MOSEI/UR-FUNNY, respectively.Source: §C.2 · p. 24, Table 8 · p. 25, Table 9 · p. 26

D Additional Results

  • Transfer experiments vary pretraining across one, two, or three source datasets before evaluating generalization to a fourth target dataset.The large setting covers MIMIC, AV-MNIST, MOSEI, and UR-FUNNY.Source: Table 10 · p. 27, Table 11 · p. 28

D.1 Generalization to new modalities and tasks

HighMMT transfers across tasks and modalities through multitask pretraining, and its transfer and few-shot gains extend to low-resource targets. It also supports a single model across fusion and retrieval tasks.

D.1 Generalization to new modalities and tasks

  • Multitask pretraining followed by target-task fine-tuning improves performance over single-task training on all four target tasks.The reported experiments pretrain on zero to three of four tasks before fine-tuning on the fourth.Source: §D.1 · p. 28
  • More pretraining datasets consistently improve fine-tuned target performance, with the largest improvement as sources increase from zero to three on UR-FUNNY.The authors interpret this pattern as learning more generalizable multimodal features.Source: §D.1 · p. 28
  • The smallest target dataset, UR-FUNNY, gains +2.4% from transfer, while MIMIC gains +0.8% despite its disjoint time-series and table modalities.Transfer gains are larger where target and source modality sets overlap more.Source: §D.1 · pp. 28–29
  • Transfer experiments also report promising results on smaller, noisier real-world datasets spanning human video, medical data, and robots.The examples include MOSEI, UR-FUNNY, MIMIC, Push, and V&T.Source: §D.1 · p. 29

D.2 Few-shot learning

  • Multitask training outperforms single-task HighMMT across labeled-data proportions in few-shot settings.The comparison matches the target-task labeled-data proportion for both training approaches; Figure 9 shows the proportions used.Source: §D.2 · p. 29, Figure 9 · p. 29

D.3 Multitask fusion and retrieval

  • For fusion and retrieval, multitask training reaches 60.5% retrieval accuracy versus 58.8% for single-task training, with similar AV-MNIST fusion performance.The shared model also halves parameter counts compared with task-specific modeling.Source: §D.3 · p. 29, Table 12 · p. 29

D.4.1 Hypothesis 1: Improved generalization

HighMMT’s parameter sharing supports competitive performance with fewer parameters, while shared encoders show common attention patterns across modalities. Multitask training also reduces overfitting, though performance can depend on task weights.

D.4.1 Hypothesis 1: Improved generalization

  • Removing either unimodal or multimodal parameter sharing worsens performance; sharing both reduces parameters by 6× while maintaining competitive performance.The comparison is with separate unimodal and multimodal layers, resembling task-specific multimodal transformers.Source: §D.4.1 · p. 30
  • Removing modality-specific embeddings changes the average score only slightly, from 70.3 to 69.6.The result suggests shared unimodal encoders can extract features across heterogeneous modalities without a modality identifier.Source: §D.4.1 · p. 30
  • Shared unimodal encoders show common attention patterns across modalities and tasks, including shared temporal focus on time-aligned inputs.The authors suggest similar attention strategies can work across modalities and tasks.Source: §D.4.1 · p. 31
  • The 20 latent tokens divide attention across specific sequence regions and broader context, with tokens 9 and 17 attending more evenly as summary tokens.Tokens 10–13 typically focus about two-thirds through the sequence, while tokens 1 and 3 focus about one-third through it.Source: §D.4.1 · p. 31

D.4.2 Hypothesis 2: Improved regularization

  • Multitask and single-task training converge at about the same rate, but multitask training has a smaller train–validation accuracy gap.The authors interpret this as regularization of joint parameters and reduced overfitting on the target task.Source: §D.4.2 · p. 31, Figure 11 · p. 31
  • Multitask performance can be sensitive to task weights when prediction objectives have different scales, and tuning weights may trade auxiliary-task performance for target-task gains.The examples include MSE versus accuracy objectives.Source: §D.4.2 · p. 31, Figure 12 · p. 32

D.5 Summary of main take-away messages

The paper presents HighMMT as a unified approach for high-modality, partially observable tasks, combining shared multitask modeling with cross-modal transfer. Its reported takeaways span performance, efficiency, transfer, and information sharing.

D.5 Summary of main take-away messages

  • HighMMT trains one model across 10 modalities, 15 prediction tasks, and five research areas, with strong performance and reduced parameter counts.The authors argue standardized modeling reduces architectural decisions and offers a unified platform for analysis.Source: §D.5 · p. 32
  • Multitask pretraining transfers to new modalities and tasks, with more pretraining tasks and low-resource targets associated with larger gains.The authors position this as a supplement to pretraining-finetuning paradigms typically conducted within one modality.Source: §D.5 · p. 32
  • HighMMT improves the performance–efficiency tradeoff over task-specific state-of-the-art models, especially in low-resource and partially observable settings.The conclusion also emphasizes fewer architectural decisions and generalization to understudied modalities and tasks.Source: §D.5 · p. 32
  • The analysis identifies information sharing across modalities and tasks, modest embedding-removal performance loss, and less target-task overfitting with well-tuned regularization weights.The authors describe both generalization and regularization effects.Source: §D.5 · p. 32

Related papers

Loading 2203.01311v4…