Source-linked AI summary
CentralNet: a Multilayer Approach for Multimodal Fusion
Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, Frédéric Jurie
TL;DR
Multimodal fusion seeks better decisions by combining information from multiple media, while prior approaches typically align modality features or coordinate their representations. CentralNet links modality-specific networks through a central network and multitask loss, and the paper reports improved accuracy across four computer vision tasks.
Problem
Multimodal fusion aims to combine information from multiple media for better decisions, while prior methods typically emphasize either joint representations or coordinated representations.
Method
CentralNet connects separate modality-specific convolutional networks through a central network that provides a common embedding and uses modality-specific losses for multitask regularization.
Results
CentralNet achieves state-of-the-art results on several multimodal problems and consistently improves existing fusion approaches across four computer vision tasks.
Takeaways & Limitations
The approach combines joint representation, coordinated modality learning, and learned fusion across network levels within one multimodal model.
Takeaways & Limitations
The method assumes extracted modality features have the same dimensionality, requiring projection or zero-padding when they do not.
Abstract
from arXiv · showhide
This paper proposes a novel multimodal fusion approach, aiming to produce best possible decisions by integrating information coming from multiple media. While most of the past multimodal approaches either work by projecting the features of different modalities into the same space, or by coordinating the representations of each modality through the use of constraints, our approach borrows from both visions. More specifically, assuming each modality can be processed by a separated deep convolutional network, allowing to take decisions independently from each modality, we introduce a central network linking the modality specific networks. This central network not only provides a common feature embedding but also regularizes the modality specific networks through the use of multi-task learning. The proposed approach is validated on 4 different computer vision tasks on which it consistently improves the accuracy of existing multimodal fusion approaches.
1 Introduction and Related Work
Multimodal fusion combines information from multiple media to improve decisions, but existing methods differ in where and how they fuse or coordinate modality representations. CentralNet combines joint and coordinated representations through a central network and multitask loss.
- Motivation: Multimodal methods combine information from different media for applications including video, emotion, activity, food, language, and image analysis.Their motivation is to extract relevant information across modalities and make better decisions than using one modality alone.
- Related work: Fusion approaches commonly differ by level, such as early or late fusion, but no fusion level is consistently best across tasks.Prior work reports task-dependent preferences for prediction-level, late, early, gated, and hybrid fusion.
- Related work: Prior research also coordinates modalities using complementary-representation constraints, correlation objectives, autoencoders, or modality dropping.These methods include deep Canonical Correlation Analysis, CorrNet, modDrop, and modout.
- Contribution: CentralNet borrows from joint and coordinated representations by linking modality-specific convolutional networks with a central network and global loss.The central network projects modality features into a common space, while the global loss backpropagates constraints and combines per-modality and joint losses.
- Contribution: The proposed approach automatically identifies fusion levels and combines per-modality losses with a global joint-space loss in a multitask formulation.The paper describes this design as simultaneously satisfying modality-specific and global objectives.
2 CentralNet
CentralNet combines modality-specific neural networks through a central network that fuses their hidden representations across layers while preserving independent modality decisions. Joint optimization with unimodal losses provides multitask regularization, and learned fusion weights can adapt the effective fusion stage.
- CentralNet architecture: CentralNet takes a weighted sum of corresponding unimodal layers and the central network’s previous layer at each fusion stage.The resulting central representation is processed by a convolutional or dense operating cell followed by an activation function.
- CentralNet architecture: The first central layer combines only unimodal representations, while the final output combines unimodal and central predictions before producing the classification result.No previous central representation exists at the first layer, so only modality features contribute there.
- Optimization: Unimodal and central networks, together with trainable fusion parameters, are optimized jointly with a global loss containing central and modality-specific classification losses.The modality-loss weights are cross-validated and were set to βk = 1 in all experiments.
- Optimization: Including unimodal losses helps preserve unimodal performance and acts as multitask regularization that improves generalization.The paper names this training configuration “Multi-Task.”
- Properties and interpretation: CentralNet can build on existing unimodal architectures with fewer fusion-specific parameters than prior multilayer approaches, potentially helping prevent overfitting.Although fusion uses a linear weighted sum, nonlinear operating layers enable complex joint representations.
- Properties and interpretation: Learned fusion weights indicate where modalities are combined: near-zero modality weights correspond to early fusion, whereas near-zero central weights until the end correspond to late fusion.The architecture therefore spans fusion locations between input-level and decision-level combination.
3 Experiments
Experiments evaluate CentralNet across four multimodal datasets and compare it with five fusion approaches. Results show strong performance overall, while the useful fusion depth and modality weighting vary with the dataset and information-sharing conditions.
- Experimental setup: Experiments cover four multimodal datasets and compare CentralNet with five fusion approaches ranging from simple baselines to recent state-of-the-art methods.Performance is averaged over 64 runs, with 99% confidence intervals computed from the estimated standard deviation and Student’s law.
- Multimodal MNIST: The Multimodal MNIST setup controls the energy allocated to each modality and their share ratio, enabling evaluation across different information configurations.The dataset uses paired PCA-derived views, while the CentralNet implementation uses three LeNet5 networks.
- Multimodal MNIST: The error rate increases with energy per modality, while insufficient or excessive information sharing lowers fusion accuracy.The optimal fusion layer differs by method but is early, and the benefit of fusion can depend on dataset characteristics.
- Multimodal MNIST: CentralNet achieves the best results across configurations except when the share ratio is null, where Fusion+Ensemble performs better.The authors relate the exception to the difficulty of constructing a stable joint representation when modalities are independent.
- Audiovisual MNIST and Montalbano: On Audiovisual MNIST, all fusion methods outperform unimodal models, with CentralNet performing best and using information from all modalities at every layer.On Montalbano, CentralNet also outperforms other approaches, with weights indicating a hybrid strategy that mixes early and late fusion.
4 Conclusions
CentralNet fuses multimedia information through a central network connecting modality-specific neural-network layers. Its loss learns modality combination while constraining modality-specific networks toward complementary representations.
- CentralNet uses a central network to connect the different layers of modality-specific neural networks.
- The central network’s loss learns how to combine modalities while adding constraints to the modality-specific networks.
- The constraints enforce complementary aspects in the modality-specific representations.
- CentralNet achieves state-of-the-art results on several different multimodal problems and addresses the late-versus-early fusion paradigm.