PaperlayerView exact revision on arXiv

Source-linked AI summary

FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion

Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, Suchi Saria

arXiv:2402.03226v4cs.LG

TL;DR

Variable modality counts, missing inputs, and irregular sampling remain challenges for multimodal fusion. FuseMoE uses a mixture-of-experts framework with Laplace gating, and evaluations report superior performance across diverse scenarios.

  • Problem

    Scalable multimodal fusion remains challenging when the number of modalities varies and inputs are missing or irregularly sampled.

    Source: §1 · pp. 1–2

  • Method

    FuseMoE uses sparsely gated mixture-of-experts layers to fuse FlexiModal inputs, with a Laplace gating function.

    Source: §1 · p. 2

  • Results

    FuseMoE demonstrated superior performance across diverse scenarios.

    Source: §5 · p. 10

  • Takeaways & Limitations

    FuseMoE addresses multimodal data characterized by random missingness or irregularity, a relatively unexplored challenge.

    Source: §5 · p. 10

    The theoretical results presented in the main text cover the over-specified setting; the exact-specified setting is deferred to Appendix J.

    Source: §3 · p. 5

Abstract

from arXiv · show

As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce ``FuseMoE'', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks.

1 Introduction

Multimodal fusion still lacks scalable, unified ways to handle variable modality sets, missing inputs, and irregular sampling. FuseMoE is an MoE framework designed to fuse such FlexiModal data.

1 Introduction

  • Prior methods face limits from two-modality designs, costly pairwise comparisons, or simple concatenation, and often do not handle missing or irregular data effectively.Some require all modalities to be observed; imputation-based approaches can yield suboptimal performance.Source: §1 · p. 2
  • FlexiModal data includes varying combinations of modalities, arbitrary missingness, and irregular sampling, commonly found in clinical records.EHRs may combine tabular data, images, notes, physiological time series, and vital signs.Source: §1 · p. 2
  • FuseMoE introduces a mixture-of-experts framework for multimodal fusion of FlexiModal data.Its contribution also includes a Laplace gating function with theoretically better convergence rates than traditional Softmax functions.Source: §1 · p. 2
  • FuseMoE is designed to accommodate an unlimited array of input modalities and dynamically adjust expert influence when modalities are missing.Its designated experts specialize in particular data types.Source: §1 · p. 2
  • FuseMoE uses modality-specific feature extractors and multi-time attention before routing inputs to combinations of MLP experts whose outputs are gated into fused embeddings.The multi-time attention mechanism addresses temporal irregularities [Shukla and Marlin, 2021].Source: Figure 1 · p. 3

2 FuseMoE: Enhance Predictive Performance for FlexiModal Data

FuseMoE combines a sparse mixture-of-experts fusion layer with encoders for modality and sampling irregularity. Its components address irregular sequences, router choices, expert utilization, and missing modalities.

2.1 Sparse MoE Backbone

  • FuseMoE uses sparse MoE layers and a novel Laplace gate, which the authors report offers enhanced convergence guarantees and superior predictive performance, particularly for FlexiModal applications.The gating network routes inputs to experts, whose outputs are combined using gate weights.Source: §2.1 · p. 3

2.3 MoE Fusion Layer

  • Router options include a joint router for concatenated modalities, modality-specific routers using shared experts, and modality-specific routers with dedicated expert pools.These alternatives offer finer control over inter-modal and intra-modal relationships.Source: Figure 2 · p. 4, §2.3 · pp. 4–5

2.2 Modality and Irregularity Encoder

  • mTAND uses time attention to map irregular observations into γ regular bins, then combines discretized and continuous representations to mitigate information loss and variable-sampling-rate concerns.The procedure creates an embedding for each modality, including multivariate observations.Source: §2.2 · pp. 3–4

2.3 MoE Fusion Layer

  • Entropy regularization encourages stable expert preferences within modalities while promoting diverse expert selections across modalities.It is an auxiliary loss that maximizes mutual information between modalities and experts.Source: §2.3 · p. 5
  • For modalities missing throughout a trajectory, FuseMoE substitutes a learnable missing-indicator embedding that routers assign lower weights to than the experts processing those embeddings.The embedding is adjusted during training to minimize task-specific and entropy-regularization losses.Source: §2.3 · p. 5

3 Theoretical Contribution

This section analyzes maximum-likelihood convergence for Laplace-gating Gaussian MoE under exact and over-specified expert counts. It establishes density and parameter-estimation guarantees and compares Laplace gating with standard Softmax gating.

Theoretical Contribution

  • The paper argues Laplace gating improves convergence over standard Softmax; its L2-distance score is less magnitude-biased and gives experts more balanced selection opportunities.The authors connect this routing behavior to heterogeneous modalities and graceful degradation with missing data.Source: §3 · pp. 5, 8
  • Under exact specification (k = k*), Laplace-gating Gaussian-MoE density estimation converges at a parametric rate in n.The exact-specified setting assumes the true number of experts is known.Source: §3 · pp. 5–6
  • Figure 3 reports empirical MLE-to-truth discrepancy trends in exact- and over-specified settings as support for the theoretical convergence rates.The plots show mean discrepancy with two-standard-deviation error bars and fitted log-log regression lines.Source: §3 · p. 7, Figure 3 · p. 7
  • For over-specified k > k*, Theorem 3.2 relates expected conditional Total Variation error to D2(G, G*) for any mixing measure G in Gk.D2 is the parameter loss used in the over-specified analysis.Source: §3 · pp. 6–7
  • In over-specified fits, parameters fitted by one component have O(n^-1/2) rates up to logarithmic factors, while multi-fit gating parameters have O(n^-1/4) rates.Rates for other parameters fitted by multiple components vary with the number of fitted components.Source: §3 · p. 7

4 Experiments

FuseMoE is evaluated across sentiment, vision, clinical prediction, and activity-recognition tasks, including settings with irregular or missing modalities. The reported results favor MoE fusion and Laplace gating, while router and modality ablations examine missingness and added inputs.

4.1 Main Results

  • Across CMU-MOSI and MOSEI, an MoE backbone improves performance over representative fusion baselines regardless of router configuration or gating function.The authors attribute this to allocating components to distinct modalities and better addressing inter- and intra-modal relationships.Source: §4.1 · p. 9
  • On CIFAR-10, Laplace gating outperforms standard Softmax gating, consistent with the paper’s theoretical claims.The experiment uses Vision-MoE and replaces an even number of ViT FFN layers with MoE layers.Source: §4.1 · p. 9
  • FuseMoE-based methods surpass baselines in most MIMIC-IV scenarios, while the irregularity encoder with MoE fusion is also effective on PAM.HAIM performs strongly on 48-IHM and LOS but more moderately on 25-PHE.Source: §4.1 · p. 9

4.2 Ablation Studies

  • Adding CXR and ECG to FuseMoE significantly enhances performance over its two-modality versions, with additional information notably from CXR.The evaluated setup uses a per-modality router and entropy loss E; the Table 5 caption describes averaged results across five random experiments.Source: §4.2 · p. 9, Table 5 · p. 10
  • Most baselines other than HAIM are not designed for varying modality counts, and adapting them to extra or missing inputs requires substantial changes that may compromise performance.HAIM’s added-modality results did not benefit, possibly because it does not adequately address dynamics between modalities.Source: §4.2 · p. 10
  • With missing modalities, per-modality or disjoint routers and entropy loss E improve performance; 48-IHM and LOS outcomes exceed those from complete-modality data.The authors attribute this to separating present and missing modalities, reducing the influence of experts for absent inputs and enabling broader sample use.Source: §4.2 · p. 10

5 Discussions and Limitations

The discussion presents FuseMoE as an approach to multimodal data with missingness or irregularity, while noting possible over-parameterization for small inputs. The related-work and data sections situate this challenge among existing fusion methods and describe FlexiModal clinical data.

5 Discussions and Limitations

  • FuseMoE combines MoE fusion layers, modal embeddings, and router configurations for multimodal inputs, but its irregularity encoding may over-parameterize small inputs.The authors identify simpler, more efficient irregularity handling that preserves performance as future work.Source: §5 · p. 10

A Related Works

  • Prior multimodal fusion work includes kernel, graphical-model, neural-network, tensor-fusion, and attention-based approaches.Some methods target specific modalities or tasks, including word-level vision, language, and audio alignment.Source: A Related Works · p. 20
  • Clinical fusion approaches include concatenation pipelines and two-modality cross-attention, while the cited work identifies scalability and missing-modality handling as unresolved challenges.The discussion notes that existing approaches may rely on pairwise relationships or imputation.Source: A Related Works · p. 20
  • MoE assigns distinct parameters to inputs through sparse activation, but its use in complex FlexiModal settings remains largely unexplored.Prior applications cited include vision-language modeling and dynamic image fusion.Source: A Related Works · p. 20

B FlexiModal Data and Tasks of Interest

  • Clinical FlexiModal data can combine tabular records, images, notes, time series, and vital signs, with modalities sampled irregularly and often missing or sparse.The data section uses multimodal EHRs as an example of this setting.Source: B FlexiModal Data and Tasks of Interest · p. 21

B.1 MIMIC-IV and MIMIC-III Datasets

The section describes clinical, sentiment, activity-monitoring, and image-classification datasets and tasks, then compares three MoE routing designs that trade off cross-modal interaction, modality specialization, and efficiency.

B.1 MIMIC-IV and MIMIC-III Datasets

  • The MIMIC tasks predict in-hospital mortality and near-term discharge from the first 48 ICU hours, or classify 25 conditions using the full stay.The section also describes task-specific cohort sizes, missing-modality analyses, and F1/AUROC evaluation.Source: §B.1 · pp. 21–22, Figure 5 · p. 21
  • MIMIC-IV links ICU records to chest X-rays, radiological notes, and ECGs; the dataset section also reports missing-modality cohort sizes and modality coverage.The linked records come from 73,181 ICU stays, with 70% allocated to training and the remainder split evenly between validation and testing.Source: §B.1 · p. 22, Table 6 · p. 22

B.2 MOSI and MOSEI Datasets

  • CMU-MOSI and CMU-MOSEI use video, text, and audio to predict sentiment polarity and intensity, evaluated with MAE, correlation, accuracy, and F1.The datasets contain 1,284/229/686 and 16,326/1,871/4,659 train/validation/test samples, respectively.Source: §B.2 · p. 22, Table 7 · p. 23

B.3 PAM Dataset

  • PAM classifies eight daily-living activities from 5,333 segments measured by 17 sensors, using classification accuracy.Each segment contains 600 continuous observations sampled at 100 Hz.Source: §B.3 · p. 23

B.4 CIFAR-10 Dataset

  • CIFAR-10 is an object-recognition dataset of 60,000 32×32 color images across 10 classes.Source: §B.4 · p. 23

C.1 Joint Experts & Routers

  • Joint routing sends a concatenated representation to selected experts, capturing cross-modal interactions but potentially blending away modality-specific features.It also simplifies routing by treating the combined embedding as one input.Source: §C.1 · p. 23, Advantages · p. 23, Challenges · p. 23

C.2 Modality-Specific Router

  • Modality-specific routers send each modality to a shared expert pool, preserving modality identity but potentially weakening complex cross-modal interaction.The section reports that this router design is more effective at ameliorating missing-modality effects in the experiments.Source: §C.2 · pp. 23–24, Challenges · p. 24, Advantages · p. 24

C.3 Disjoint Experts & Routers

D Data Preprocessing

The preprocessing pipeline converts clinical modalities into embeddings and handles irregular vital-sign and lab observations through discretization and a gated combination of imputation and mTAND embeddings. The section also describes comparison baselines for clinical and multimodal tasks.

D.1 MIMIC-IV

  • MIMIC preprocessing represents 30 lab and chart events, extracts 1024-dimensional CXR and 768-dimensional radiology-note embeddings, and compresses each ECG to 256 dimensions.The CXR embeddings use fine-tuned DenseNet-121, notes use BioClinicalBERT, and ECGs use a convolutional autoencoder.Source: §D.1 · pp. 24–25, Figure 6 · p. 26

D.2 PAM Dataset

  • PAM preprocessing follows Zhang et al. [2021]’s published procedure.Source: §D.2 · p. 25

E.1 Unified Temporal Discretization Embeddings

  • Unified Temporal Discretization Embeddings combine mTAND with imputation-based discretization to represent irregular vital and lab observations in regular target bins.For each bin, the scheme uses the closest prior observation when available and otherwise the global mean, then applies a causal convolution to produce embeddings.Source: §E.1 · pp. 25–26

E.2 Unifying imputation and mTAND embeddings

  • A one-layer MLP gating function combines the imputation and mTAND embeddings to derive the final vital-sign embeddings.The two embedding sets are concatenated to compute the gate.Source: §E.2 · p. 26

F.1 MISTS

  • The MISTS baseline treats time series and notes as irregular multivariate time series and fuses them with self- and cross-attention plus Time2Vec time encodings.Source: §F.1 · p. 27

F.2 MulT; F.3 MAG; F.4 TFN

  • MulT uses pairwise bidirectional cross-modal attention before modality-specific transformers, while MAG conditions BERT or XL-Net representations on visual and acoustic data.TFN instead combines modality embeddings with a tensor-fusion layer that models interaction terms.Source: §F.2 · p. 27, §F.3 · p. 27, §F.4 · p. 27

F.5 HAIM

  • The HAIM baseline summarizes 30 vital/lab events into 330 time-series features and uses XGBoost, with modality embeddings averaged over the task’s observation window.For comparison, the pipeline includes only stays with all modalities observed, the most recent five notes, and 128 vital measurements.Source: §F.5 · p. 27

F.6 Implementation

The implementation integrates the comparison methods into the workflow and reports computational resources and hyperparameter references. In additional MIMIC-III and MIMIC-IV settings, FuseMoE-based methods show noticeable advantages over baselines.

F.6 Implementation

  • The authors integrate the comparison methods using their released implementations and adapt HAIM’s feature extraction and fitting code.For HAIM, ECG embeddings are averaged across included observations.Source: §F.6 · p. 27

G.1 Computational Resources; G.2 Hyper-Parameters

H.1 FlexiModal Experiments

  • FuseMoE-based methods demonstrate noticeable advantages over baselines on MIMIC-III with vital signs and notes and MIMIC-IV with vital signs and CXR.The reported experiments use the joint experts-and-router configuration; the supplied table captions state results are averaged across five runs.Source: §H.1 · p. 28, Table 9 · p. 28, Table 10 · p. 29

H.2 Ablation Study on FuseMoE Building Blocks

The building-block ablations compare irregularity, time-series, image, and text encoders, while MoE ablations examine expert count, resource use, and modality weights. The supplied numerical-experiment passage states that convergence rates are studied under exact- and over-specified settings.

H.2 Ablation Study on FuseMoE Building Blocks

  • The irregularity encoder performs best on average against four baselines, while mTAND improves overall performance, especially for vital signs and clinical notes.The comparisons average five random runs; the mTAND ablation uses all four MIMIC-IV modalities.Source: Figure 7 · p. 29, §H.2 · p. 29, Figure 11 · p. 31
  • The Transformer outperforms CNN and LSTM as the time-series encoder, and Clinical-Longformer performs best among the tested text encoders.Both comparisons average five random runs.Source: §H.2 · p. 29, Figure 8 · p. 30, Figure 10 · p. 30
  • ViT can further improve predictions on some tasks by providing better CXR embeddings.The CXR encoder comparison holds the rest of FuseMoE’s components constant and averages five runs.Source: §H.2 · p. 29, Figure 9 · p. 30

H.3 Ablation Study on MoE Architecture

  • FuseMoE sits approximately mid-range in computational efficiency and resource use; performance generally improves with more experts but plateaus above 16, motivating top-4-of-16 routing.The sparse MoE layer adds parameters without significantly escalating computational load.Source: §H.3 · p. 31, Figure 12 · p. 31
  • Across tasks, predictions rely heavily on vital signs and clinical notes, while CXR plays a larger role in 25-PHE and 48-IHM despite fewer samples.MOSI and MOSEI weights are more spread out, with audio carrying greater weight in MOSEI.Source: Figure 13 · p. 31, §H.3 · p. 32

I Details on Numerical Experiments

I.1 Experimental Setup

The experiments evaluate exact- and over-specified settings, with the MLE converging empirically to the true mixing measure at rates close to their theoretical parametric rates. Samples are generated from the true Laplace gating Gaussian mixture-of-experts model.

I.1 Experimental Setup

  • The data are generated by sampling X uniformly on [0, 1] and then drawing Y from the true conditional density of the Laplace gating Gaussian mixture of experts.MLE uses an EM-based numerical scheme with coordinate gradient descent in maximization steps; initialization distributes components into Voronoi cells around the true components.Source: §I.1 · p. 32

I.2 Exact-specified Setting

  • O(n^-0.49) is the empirical rate for MLE convergence to the true mixing measure under the Voronoi metric D1 in the exact-specified setting.The experiments use five sample generations per configuration across ten sample sizes from 10^3 to 10^5.Source: §I.2 · p. 32

I.3 Over-specified Setting

  • O(n^-0.47) is the empirical rate for MLE convergence to the true mixing measure under the Voronoi metric D2 in the over-specified setting.The experiments use five samples per setting across ten sample sizes from 10^3 to 10^5.Source: §I.3 · p. 32

J Exact-Specified Setting

For the exact-specified Laplace gating Gaussian MoE, the paper establishes parametric-order convergence for conditional-density estimation and for the MLE under Voronoi loss, up to logarithmic factors. A total-variation lower bound connects density error to that loss.

J Exact-Specified Setting

  • The conditional-density estimate converges to the true density at a parametric rate O(n^-1/2), up to a logarithmic factor.The appendix states this result under total variation distance.Source: J Exact-Specified Setting · p. 33

K Proof of Theoretical Results

  • The proof strategy transfers density-estimation rates to mixing-measure loss by establishing a total-variation lower bound in terms of that loss.The appendix states that such a lower bound directly gives the MLE rate under the corresponding loss function.Source: J Exact-Specified Setting · p. 33

J Exact-Specified Setting

  • O(n^-1/2) up to logarithmic factors is the MLE convergence rate to the true mixing measure under exact-specified Voronoi loss D1.The total-variation lower bound relates expected conditional-density distance to D1, supporting the rate transfer.Source: J Exact-Specified Setting · p. 33
  • The Voronoi-loss rate also yields O(n^-1/2) parameter-estimation rates up to logarithmic factors, comparable to exact-specified softmax-gating Gaussian MoE rates [2023].The supplied passage identifies the comparison as Theorem 1 in Nguyen et al. [2023].Source: J Exact-Specified Setting · p. 33

K.1 Proof of Theorem J.2

The proof of the exact-specified Voronoi-loss result establishes a total-variation lower bound using density expansions and contradiction. The density-estimation proof applies M-estimator results and bounds covering and bracketing entropy.

K.1 Proof of Theorem J.2

  • The exact-specified proof establishes an expected total-variation lower bound in terms of D1 using first-order density expansions, nonvanishing coefficients, and a Fatou-lemma contradiction.The proof also treats separate local and nonlocal inequalities for mixing measures.Source: §K.1 · pp. 33–36

K.2 Proof of Theorem 3.1

  • The density-estimation proof uses M-estimator results and bounds covering numbers and bracketing entropy for the conditional-density class.The entropy argument assumes a bounded parameter set.Source: §K.2 · pp. 37–38

K.2.1 Main Proof

  • The main proof derives the density-estimation rate by bounding an entropy integral and applying the cited density-estimation lemma.The text concludes the proof of Theorem J.1 from that lemma.Source: §K.2.1 · p. 38

K.2.2 Proof of Lemma K.1

  • For bounded parameter sets, the covering-number and bracketing-entropy bounds scale as log(1/η).The proof constructs a cover by discretizing parameter sets and uses h^2 ≤ ||·||1 to obtain the bracketing-entropy bound.Source: §K.2.2 · pp. 39–40

K.3 Proof of Theorem 3.2

The over-specified proof seeks a total-variation lower bound relative to the Voronoi loss D2. Its argument handles local and nonlocal losses and uses density expansions, coefficient bounds, and contradiction.

K.3 Proof of Theorem 3.2

  • The proof targets a total-variation lower bound relative to D2, treating mixing measures with D2 at most ε′ separately from those with D2 greater than ε′.The text identifies the two cases as inequalities A and B.Source: §K.3 · p. 40
  • The local argument decomposes density differences by Voronoi-cell cardinality and applies Taylor expansions with remainders negligible relative to D2.The expansions use orders 1 and r̄_j for one component and orders 1 and 2 for another term.Source: §K.3 · pp. 41–42
  • The proof rules out vanishing normalized coefficients, then uses a Fatou-lemma contradiction to complete the local lower-bound argument.It concludes that the normalized coefficients cannot all converge to zero.Source: §K.3 · pp. 42–44

L Identifiability of the Laplace Gating Gaussian MoE

The lemma establishes that two Laplace-gating Gaussian MoE mixing measures with identical conditional distributions almost surely must be identical.

L Identifiability of the Laplace Gating Gaussian MoE

Related papers

Loading 2402.03226v4…