Source-linked AI summary
MFAS: Multimodal Fusion Architecture Search
Juan-Manuel Pérez-Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, Frédéric Jurie
TL;DR
Multimodal classification lacks a generally optimal manually chosen fusion architecture because useful information can occur at different feature layers. MFAS searches a broad, constrained fusion-architecture space with sequential model-based exploration and finds state-of-the-art architectures across three datasets, including NTU RGB+D.
Problem
The paper addresses the difficulty of selecting effective multimodal feature combinations and fusion depths manually because features at different layers carry different semantic information.
Method
MFAS defines a broad constrained search space for multimodal fusion and progressively samples architectures using surrogate-guided sequential model-based optimization.
Results
MFAS discovers fusion architectures that provide state-of-the-art results on three datasets, including a 90.04% average accuracy on NTU RGB+D over four runs.
Takeaways & Limitations
Treating multimodal fusion as architecture search can identify effective fusion schemes across problems with different domains and dataset sizes.
Takeaways & Limitations
The approach assumes an off-the-shelf pre-trained multilayer feature extractor for each modality, although the fusion method is not limited to neural primary extractors.
Abstract
from arXiv · showhide
We tackle the problem of finding good architectures for multimodal classification problems. We propose a novel and generic search space that spans a large number of possible fusion architectures. In order to find an optimal architecture for a given dataset in the proposed search space, we leverage an efficient sequential model-based exploration approach that is tailored for the problem. We demonstrate the value of posing multimodal fusion as a neural architecture search problem by extensive experimentation on a toy dataset and two other real multimodal datasets. We discover fusion architectures that exhibit state-of-the-art performance for problems with different domain and dataset size, including the NTU RGB+D dataset, the largest multi-modal action recognition dataset available.
1. Introduction
Multimodal classification commonly merges separately extracted modality features at one point, but the best fusion depth and combination are difficult to determine manually. MFAS formulates fusion design as a combinatorial architecture-search problem and uses sequential model-based optimization to explore it.
- Deep models commonly transfer modality-specific models and merge their features at a single point, often using late fusion.
- Features at different hidden layers carry different semantic information, making the optimal multimodal fusion strategy difficult to choose by hand.
- Existing intermediate-fusion methods require empirical selection of features, while evaluating all nested combinations becomes a large combinatorial problem.
- MFAS addresses this problem by posing multimodal fusion as a combinatorial search guided by sequential model-based optimization.
- The paper defines a multimodal fusion search space, adapts an automatic search approach, and reports automatically found state-of-the-art fusion architectures across multimodal problems.
2. Related work
Prior multimodal fusion work explored architectural combinations and constraints, while neural architecture search provided related strategies for automated model design. MFAS extends this direction by treating multimodal fusion itself as an architecture-search problem with a search space designed for sequential exploration.
- Neural networks encode hierarchical features across layers, motivating searches for ways to connect and fuse multimodal features.
- MFAS is presented as the first approach to directly formulate multimodal classification fusion as an architecture-search problem.
- Multimodal fusion research has pursued architectural choices and constraints governing relationships between modalities and network weights.
- Architectural approaches differ in where unimodal features are fused, with early and late fusion offering different performance patterns across tasks.
- The method adapts progressive exploration and weight sharing to a constrained search space that is a superset of previously introduced fusion schemes.
3. Methodology
MFAS searches multimodal fusion architectures built from pre-trained modality-specific feature extractors, using a structured search space and sequential model-based optimization to identify promising fusion schemes efficiently.
- Setup: The method assumes pre-trained, multi-layer feature extractors for each modality and searches only how their features are fused.This constrains the search burden while allowing the primary extractors to be broader than neural networks.
- Fusion search space: Each fusion layer combines the previous fusion output with one feature from each modality through a trainable fusion network.The approach introduces a third network whose fusion layers process concatenated modality features.
- Fusion search space: A fusion architecture is represented as a sequence of triplets selecting the first-modality feature, second-modality feature, and non-linearity at each fusion layer.The number of fusion layers is itself a search parameter, and architectures can contain single or composed fusions.
- Fusion search space: The search space contains many fusion architectures, including CentralNet-like networks, with size (M × N × P)^L.Its exponential growth makes exhaustive exploration intractable because evaluating one architecture requires training and evaluating it.
- Search algorithm: The procedure returns the best K architectures among all sampled candidates after progressive search.MFAS starts at one fusion layer, unrolls higher complexity levels, and retains sampled architectures and accuracies.
- Search algorithm: MFAS progressively expands fusion-network complexity and samples architectures using probabilities guided by a surrogate model’s predicted accuracies.Sampled architectures are trained and evaluated, then their results update the surrogate as the temperature is adjusted.
4. Experiments
Experiments evaluate MFAS on AV-MNIST, MM-IMDB, and NTU RGB+D, comparing searched fusion architectures with existing methods and examining search behavior. MFAS finds strong fusion architectures across these datasets, including the highest reported NTU RGB+D accuracy and the best overall AV-MNIST score.
- Experimental setup: MFAS is evaluated on the AV-MNIST toy dataset and the real MM-IMDB and NTU RGB+D multimodal datasets.The experiments cover audio-visual, visual-textual, and RGB-depth action-recognition settings.
- AV-MNIST: AV-MNIST compares random exploration with MFAS after sampling 180 architectures using the proposed search space.MFAS finds the best validation architecture and produces less variability among the top five architectures than random search.
- AV-MNIST: All multimodal fusion networks largely improve over unimodal networks, while the automatically found architecture achieves the best overall AV-MNIST test score.The architecture is selected after three iterations of progressive search with L = 4.
- MM-IMDB: The MM-IMDB search space contains 110,592 fusion configurations formed from eight image features, two text features, and three nonlinearities.The best found configuration is shown in Figure 3, and it outperforms the other listed fusion strategies in the reported evaluation.
- NTU RGB+D: 90.04% average accuracy over four runs with a variance of 0.6 is reported for MFAS on NTU RGB+D, the highest score in Table 4.The search uses convolutional and fully connected features from the Inflated ResNet-50 and deep Co-occurrence baselines, with a search space of 5,308,416 configurations.
- Search behavior: During search, sampled architectures become more stable and their mean validation error becomes significantly lower at later steps.The paper attributes stabilization to refined shared fusion weights and increasingly confident surrogate-function predictions.
- Search behavior: Temperature-driven sampling helps avoid premature commitment to architectures that appear strong early because their fusion weights have been trained only briefly.The method uses an inverse exponential temperature schedule, which the authors observed to outperform a linear schedule.
5. Conclusion
The work addresses accurate fusion-architecture discovery for multimodal classification with a novel search space and exploration algorithm. Experiments on three datasets demonstrate the method's validity and state-of-the-art fusion schemes, while future work targets greater fusion-layer flexibility.
- The paper proposes a novel multimodal search space and exploration algorithm for finding accurate fusion architectures efficiently.The search space permits convoluted architectures while containing problem complexity at reasonable levels.
- Experiments on three datasets validate the method and produce several fusion schemes with state-of-the-art results.
- Future research will improve the search space so fusion layers can be composed more flexibly.