PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn to Combine Modalities in Multimodal Deep Learning

Kuan Liu, Yanen Li, Ning Xu, Prem Natarajan

arXiv:1805.11730v1stat.MLcs.AIcs.LG

TL;DR

The paper addresses multimodal learning with sample-dependent weak modalities, noise, and conflicts that existing additive approaches do not fully handle. It introduces multiplicative combination for selecting reliable modality decisions, extends it to modality mixtures, and reports consistent accuracy improvements over state-of-the-art methods across three domains.

  • Problem

    Multimodal learning must combine complementary signals while handling sample-dependent weak modalities, noise, and conflicts that existing additive approaches do not fully address.

  • Method

    The paper multiplicatively combines modality decisions to suppress weak modalities and extends this procedure to additive candidates formed from modality mixtures.

  • Results

    The methods consistently outperform existing multimodal methods on three real-world classification datasets from different domains.

  • Takeaways & Limitations

    The approach combines per-sample reliability selection with cross-modal mixture modeling in an end-to-end multimodal framework.

  • Takeaways & Limitations

    The multiplicative layer has a mismatch between minimizing its objective function and maximizing the desired accuracy.

Abstract

from arXiv · show

Combining complementary information from multiple modalities is intuitively appealing for improving the performance of learning-based approaches. However, it is challenging to fully leverage different modalities due to practical challenges such as varying levels of noise and conflicts between modalities. Existing methods do not adopt a joint approach to capturing synergies between the modalities while simultaneously filtering noise and resolving conflicts on a per sample basis. In this work we propose a novel deep neural network based technique that multiplicatively combines information from different source modalities. Thus the model training process automatically focuses on information from more reliable modalities while reducing emphasis on the less reliable modalities. Furthermore, we propose an extension that multiplicatively combines not only the single-source modalities, but a set of mixtured source modalities to better capture cross-modal signal correlations. We demonstrate the effectiveness of our proposed technique by presenting empirical results on three multimodal classification tasks from different domains. The results show consistent accuracy improvements on all three tasks.

1 Introduction

Multimodal learning must handle sample-dependent weak modalities while preserving complementary cross-modal information. The paper proposes multiplicative selection of reliable modalities and modality mixtures, with consistent improvements across three real-world classification datasets.

  • Motivation: Sample-dependent noise and conflicts can make one modality informative for some examples but weak for others.The desired system should select relevant information per sample while capturing complementariness across modalities.
  • Limitations of existing methods: Existing additive approaches aggregate modalities and are difficult to train and regularize with finite data.They assume every modality may be useful, leaving a model above the aggregate representation to learn modality reliability.
  • Proposed method: The proposed multiplicative method selects stronger modalities per sample and suppresses weaker modalities during inference and training.It combines independent modality decisions differentiably, reducing pressure on weak modalities to predict correctly from noise while enabling end-to-end joint training.
  • Proposed method: A modality-mixture extension first creates additive candidate mixtures, then multiplicatively selects appropriate candidates.This increases capacity to capture cross-modal correlations and complementariness while ignoring redundant or noisy mixtures.
  • Evaluation: The methods consistently outperform existing multimodal methods on three real-world datasets spanning image recognition, physical process classification, and user profiling.The reported evaluations cover classification tasks from different domains.

2 Background

This section introduces multimodal notation and reviews early-fusion, late-fusion, and additive deep-learning approaches. It also notes their respective representation, flexibility, and interaction-modeling trade-offs.

  • Notation: The paper denotes M modalities as dense vectors v_m and uses p_k for final prediction probability of class k.In the user-profiling example, the modalities are profile image, posted text, and friend-network information.
  • Early Fusion: Early fusion concatenates or otherwise jointly represents modality features before training a single model to learn their correlations and interactions.This approach is simple but typically requires engineered, aligned features and assumes one model suits all modalities.
  • Late Fusion: Late fusion combines separate unimodal decision values using mechanisms such as averaging, voting, or a learned model.Separate models provide flexibility and make missing modalities easier to handle.
  • Late Fusion: Late fusion is not effective at modeling signal-level interactions because it operates on inferences rather than raw inputs.Its flexibility therefore comes with a limitation in representing cross-modal signal interactions.
  • Deep Multimodal Learning: Deep multimodal methods commonly use domain-specific networks to represent modalities, merge the representations, and predict from the aggregate.The paper calls these additive combinations because their critical step adds modality hidden vectors, often nonlinearly.
  • Scope: The paper distinguishes its task from shared-semantic-space methods because it predicts latent attributes rather than only observed sample identities.This distinction motivates treating those approaches as not directly applicable to the task considered here.

3 A multiplicative combination layer

The multiplicative combination layer models sample-specific modality reliability, suppressing weak signals while allowing informative modalities to guide predictions. Its boosted objective further aligns training with classification accuracy by focusing on incorrectly classified examples.

  • 3 A multiplicative combination layer: Each modality makes an independent prediction, which the combination layer integrates while accounting for sample-specific modality reliability.Informative and noisy modalities are defined relative to each particular sample.
  • 3 A multiplicative combination layer: A modality’s penalty is down-weighted when other modalities confidently predict the correct class, reducing pressure to make every modality accurate on every example.The down-weighting factor is controlled by hyperparameter β, selected by cross-validation.
  • 3 A multiplicative combination layer: The multiplicative strategy tolerates mistakes from weaker modalities and encourages each modality to specialize in examples where it is informative.The differentiable layer supports joint end-to-end training through back-propagated errors.
  • 3.2 Boosted multiplicative training: The original multiplicative objective can misalign class-loss minimization with correct classification when multiple modalities and down-weighting factors remove normalization.It may stop reducing losses for incorrect classes or reduce losses for classes already predicted correctly.
  • 3.2 Boosted multiplicative training: Boosted multiplicative training penalizes only class losses that fail to achieve the smallest value by a margin, leaving correctly classified examples with zero loss.This focuses optimization on hard examples and better aligns training with prediction accuracy.

4 Select modality mixtures

The mixture-selection extension combines additive mixture construction with multiplicative candidate selection. It enumerates modality mixtures to capture cross-modal interactions, then suppresses redundant or noisy candidates.

  • 4 Select modality mixtures: The extension first creates modality mixtures so additive representations can capture interactions and correlations across modalities.It enumerates all 2^M − 1 non-empty modality subsets as mixture candidates.
  • 4 Select modality mixtures: Each mixture candidate independently produces a prediction, but candidates may disagree and therefore require an additional selection mechanism.The candidate prediction is represented as pc = gc(uc).
  • 4.2 Mixture selections: Multiplicative selection chooses strong mixture candidates while ignoring weak ones instead of averaging all candidates equally.Averaging would lose the ability to discriminate among modality mixtures.
  • 4.2 Mixture selections: The complete pipeline additively generates single-modality and mixed-modality candidates, then combines their predictions multiplicatively.This design aims to capture cross-modal signal correlations while filtering redundant and noisy candidates.
  • 4.2 Mixture selections: The multiplicative combination layer automatically picks the most useful modalities and modality mixtures for the prediction task.The approach is intended to combine the interaction modeling of additive methods with multiplicative noise filtering.

5 Related Work

Prior multimodal learning methods include feature-, decision-, hybrid-, kernel-, graphical-model-, and neural-network-based fusion. Related multiplicative and attention methods differ in where and how they combine modality information.

  • Multimodal learning: Traditional multimodal fusion includes early feature fusion, late decision fusion, hybrid fusion, multiple kernel learning, and graphical-model approaches.These methods represent several established strategies for combining modalities.
  • Multimodal learning: Deep neural networks have fused modalities for tasks including audio-visual emotion classification, gesture recognition, affect analysis, and video description.These approaches generally combine modality information in a joint hidden layer.
  • Multiplicative combination technique: Some multiplicative methods combine similarity probabilities or ensemble convolutional-network layers using multiplicative or down-weighting operations.These examples motivate multiplicative combination as an established machine-learning technique.
  • Attention techniques: Attention methods dynamically weight modality features before mixing them, performing multiplicative operations at the feature level rather than the decision level.This distinguishes their combination location from decision-level multiplicative methods.
  • Other multimodal tasks: Other multimodal applications use joint representations for image captioning and visual question answering rather than classification.Examples combine visual features with sequential language representations.

6 Experiments

The methods are evaluated on three multimodal classification datasets spanning image recognition, physical process classification, and user profiling. The paper reports consistent superiority over existing state-of-the-art multimodal methods.

  • 6 Experiments: The evaluation covers image recognition, physical process classification, and user profiling, with more than one modality provided for each task.The stated goal is to use the modalities effectively for good generalization performance.
  • 6 Experiments: The proposed methods consistently outperform existing state-of-the-art multimodal methods across the three tasks.The paper also states that the code is publicly available.

6.1 Setup

The experiments evaluate multimodal combination methods across image recognition, physical-process classification, and user profiling, using task-specific modalities and neural architectures. Baselines include single-modality, additive, multiplicative, mixture-based, and attention combinations.

  • Datasets: Experiments cover CIFAR-100 image recognition, HIGGS classification, and Snapchat user-gender prediction.The tasks use multiple input modalities from different domains.
  • CIFAR-100 setup: CIFAR-100 uses hidden states from three ResNet residual units as modalities, with ResNet-32 and ResNet-110 evaluated.The three layers are trained with loss weights of (0.3, 0.3, 1.0).
  • Compared methods: CIFAR-100 compares vanilla ResNet, additive fusion, multiplicative fusion, and multiplicative modality-mixture methods.The additive model uses a tuned two-layer fusion network with 256 hidden nodes.
  • HIGGS setup: HIGGS provides low-level and high-level feature modalities, with full and one-third-downsampled training sets used to assess behavior across data scales.The downsampled subset is called HIGGS-Small.
  • Gender setup: Gender prediction uses userid text, inferred first names, and dense activity features, with gender-6 and gender-22 variants differing in activity-feature richness.The dataset contains 7.5 million users, including 6 million training and 1 million test samples.
  • Evaluation: The evaluation also includes attention-based combination methods alongside the methods tested on CIFAR-100 and HIGGS.Table 2 reports test error and AUC comparisons across the tasks.

6.2 Results

Across the three tasks, multiplicative and mixture-based combinations generally improve over additive or single-modality alternatives, while deeper fusion alone does not explain the gains. Additional analyses examine depth, β smoothing, boosted training, over-learning, and attention-based fusion.

  • 6.2.1 Accuracy comparisons: Additive fusion does not consistently improve CIFAR-100 test error, whereas multiplicative fusion reduces error for both ResNet-32 and ResNet-110.MulMix and MulMix* provide significant accuracy gains on both models.
  • 6.2.1 Accuracy comparisons: On HIGGS, fusion and additive combination reduce error versus single modalities, while MulMix and MulMix* significantly improve results on both small and full datasets.Multiplicative combination is better than Add on HIGGS-small but slightly worse on HIGGS-full.
  • 6.2.1 Accuracy comparisons: On gender prediction, combining modalities produces the largest improvements, and MulMix and MulMix* outperform both Add and Mul.Add achieves less than half the error rate of the best single modality, while Mul has similar, slightly better results.
  • 6.2.2 Compared to deeper fusion networks: Increasing fusion-network depth worsens CIFAR-100 and gender results and gives only transient HIGGS gains, with the best deeper network still below the proposed approach.The authors attribute the pattern to optimization difficulty or overfitting and conclude that fusion design, rather than depth, drives performance.
  • 6.2.3 Multiplicative combination or model averaging: The authors do not provide a clear automatic rule for choosing β and recommend smaller values for overfitting models and larger values for underfitting models.The proposed explanation links smaller β to stronger regularization and larger β to greater modeling flexibility.
  • 6.2.4 Boosted training: Boosted training substantially improves untuned MulMix with β=1 and produces results nearly identical to MulMix*.The reported comparison concerns HIGGS and gender error rates.
  • 6.2.4 Error analysis: Mul and MulMix* make fewer over-learning mistakes on examples where at least one individual modality predicts correctly.The MulMix improvement on this subset is 0.18, close to its improvement on the full dataset.

7 Conclusion

The paper proposes multiplicative combination methods that account for modality-strength heterogeneity at both general and per-sample levels. Across three domains, the framework consistently improves accuracy over state-of-the-art multimodal methods.

  • Conclusion: The framework targets weak modalities by tolerating their errors while combining informative modalities multiplicatively.It also combines modality mixtures to unite multiplicative and additive strengths and capture cross-modal signal correlations.
  • Conclusion: Experiments across three domains demonstrate consistent accuracy improvements over state-of-the-art multimodal methods.The conclusion presents the framework as a general advance rather than a domain-specific solution.
Loading 1805.11730v1…