Source-linked AI summary
MMTM: Multimodal Transfer Module for CNN Fusion
Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L. Iuzzolino, Kazuhito Koishida
TL;DR
Intermediate multimodal fusion is difficult when streams have different spatial dimensions, helping explain the continued use of late fusion. The paper introduces MMTM, which uses joint squeeze-and-excitation representations to recalibrate CNN streams, and reports improved performance across several multimodal tasks, including state-of-the-art or competitive results.
Problem
Intermediate fusion is challenging because modality-specific CNN features can have different or unaligned spatial dimensions, while late fusion remains predominant for practical reasons.
Method
MMTM uses squeeze and excitation operations to learn joint multimodal representations and recalibrate channel-wise features in separate CNN streams.
Results
MMTM improves performance beyond late fusion across gesture recognition, audiovisual speech enhancement, and RGB-body-joint action recognition experiments.
Takeaways & Limitations
The module can be inserted at different feature-hierarchy levels, but optimal placement and module count vary by application.
Takeaways & Limitations
MMTM is described for settings with more than two or no spatial dimensions, but its optimal locations and number of modules differ across applications.
Abstract
from arXiv · showhide
In late fusion, each modality is processed in a separate unimodal Convolutional Neural Network (CNN) stream and the scores of each modality are fused at the end. Due to its simplicity late fusion is still the predominant approach in many state-of-the-art multimodal applications. In this paper, we present a simple neural network module for leveraging the knowledge from multiple modalities in convolutional neural networks. The propose unit, named Multimodal Transfer Module (MMTM), can be added at different levels of the feature hierarchy, enabling slow modality fusion. Using squeeze and excitation operations, MMTM utilizes the knowledge of multiple modalities to recalibrate the channel-wise features in each CNN stream. Despite other intermediate fusion methods, the proposed module could be used for feature modality fusion in convolution layers with different spatial dimensions. Another advantage of the proposed method is that it could be added among unimodal branches with minimum changes in the their network architectures, allowing each branch to be initialized with existing pretrained weights. Experimental results show that our framework improves the recognition accuracy of well-known multimodal networks. We demonstrate state-of-the-art or competitive performance on four datasets that span the task domains of dynamic hand gesture recognition, speech enhancement, and action recognition with RGB and body joints.
1. Introduction
Multimodal fusion combines complementary sensor information, but late fusion remains common because intermediate features can have incompatible spatial dimensions. MMTM addresses this challenge by transferring information between CNN streams through channel recalibration.
- Motivation: Multimodal fusion combines relevant information from different modalities to improve performance over using a single modality.The paper notes applications including video classification, action recognition, emotion recognition, and audiovisual speech enhancement.
- Motivation: Late fusion remains predominant because prediction scores are easy to combine, whereas intermediate features may have different or unaligned spatial dimensions.Late fusion also preserves independently structured unimodal streams, supporting practical reuse of existing architectures.
- Proposed approach: MMTM recalibrates channel-wise features across CNN streams and can be inserted at intermediate levels of late-fusion backbones.Its squeeze unit forms a joint representation, while excitation emphasizes or suppresses features across modalities.
- Empirical observations: Adding MMTM to intermediate and high-level features is beneficial, while low-level placement is not consistently beneficial across applications.The authors associate this observation with stronger inter-modality correlation in higher-level features.
- Empirical observations: Experiments across gesture recognition, audiovisual speech enhancement, and RGB-body-joint action recognition improve performance beyond late fusion.The paper presents MMTM as a module for three multimodal application settings.
2. Related Work
Prior multimodal methods combine streams at decision or intermediate feature levels, but many intermediate approaches require matching spatial dimensions. Related work also motivates combining visual and pose modalities because each supplies distinct strengths and weaknesses.
- Fusion methods: Late-fusion methods combine unimodal predictions using summation, weighted averages, bilinear products, rank minimization, or attention.Gated multimodal units extend gating toward intermediate feature levels.
- Intermediate fusion: Dense intermediate fusion methods can assume identical spatial dimensions, limiting their use across streams with mismatched feature layouts.The paper contrasts this limitation with fusion after spatial dimensions have already been aggregated.
- Cross-modal learning: Cross-modal learning improves an individual modality using information from other modalities, with all modalities present during training but only one tested.The paper treats these approaches as orthogonal because their improved unimodal networks can initialize CNN streams.
- Action recognition: Video-based action recognition can struggle with background clutter, whereas pose-only methods lose contextual and global cues available in video.This motivates architectures that fuse RGB and skeleton modalities.
- Audiovisual learning: Audiovisual systems may process audio and visual information separately before integrating them through channel-wise concatenation.This is presented as an example of multimodal processing in related work.
3. Multimodal Transfer Module
MMTM receives feature tensors from separate CNN streams, compresses each into channel descriptors, learns a joint multimodal representation, and generates modality-specific gates. This design supports recalibration across streams with differing spatial dimensions.
- Inputs: MMTM takes feature tensors A and B from corresponding layers of two disjoint CNN streams and learns a global multimodal embedding.The tensors may have different spatial dimensions and channel counts.
- Scope: The formulation allows spatial dimensions to include multiple axes, none, or application-specific configurations.The paper explicitly notes that time can be treated as a spatial dimension in 3D convolutions.
- Squeeze: Global average pooling squeezes spatial dimensions into channel descriptors before multimodal fusion.The squeeze operation enables fusion between modalities with arbitrary spatial dimensions.
- Excitation: Sigmoid-gated channel-wise products suppress or excite different filters in each CNN stream.Regularization can push a gating signal toward the identity vector, limiting its effect on a feature stream.
- Joint representation: The squeezed descriptors are concatenated and mapped through a fully connected layer to produce a joint representation Z.For more than two modalities, the approach generalizes by concatenating squeezed features from all modalities.
- Excitation: Independent fully connected layers transform the joint representation into excitation signals for each modality.The signals are used to recalibrate the corresponding input features through gating.
- Cross-modal transfer: Cross-modal recalibration lets information from one modality adjust processing in another, such as using clear depth cues when RGB imagery is blurry.The module is designed to transfer information between modality-specific feature streams.
4. Applications
The paper applies MMTM to gesture recognition, audio-visual speech enhancement, and action recognition, adapting fusion to the feature structures of each task.
- Hand Gesture Recognition: Gesture recognition fuses RGB, depth, and optical-flow streams using I3D backbones with MMTMs after the last six inception modules.The squeeze operation aggregates all dimensions except channels, including the time dimension of 3D convolution outputs.
- Audio-Visual Speech Enhancement: Audio-visual speech enhancement replaces late-fusion channel-wise concatenation with MMTM-based fusion between visual and audio networks.The audio tower outputs the final result, so MMTM gates the audio stream but not the visual network.
- Audio-Visual Speech Enhancement: The audio network predicts an ideal ratio mask from a log-mel mixture magnitude spectrogram to produce an enhanced magnitude spectrogram.The enhanced spectrogram is obtained by element-wise multiplying the predicted mask with the mixture magnitude spectrogram.
- Human Action Recognition: Action recognition uses MMTM for intermediate fusion between an I3D RGB network and an HCN skeleton network.HCN processes raw skeleton data and motion, fuses those subnetworks by channel-wise concatenation, and produces features through conv5, conv6, and fc7.
- Human Action Recognition: Three MMTMs connect the last three I3D inception modules with HCN conv5, conv6, and fc7 features despite mismatched spatial dimensions.MMTM aggregates every input dimension except channels, allowing fusion between I3D and HCN representations with different shapes.
5. Experimental Results
Across gesture recognition, speech enhancement, and action recognition, MMTM-based slow fusion improves multimodal performance over late-fusion and other comparison methods. Experiments also show that intermediate/high-level placement and global receptive fields are important design choices.
- The evaluation spans dynamic hand gesture recognition, audio-visual speech enhancement, and RGB-plus-skeleton action recognition.
- EgoGesture: 0.73%: MMTM outperforms I3D late fusion on the EgoGesture hand gesture dataset.
- NVGesture: MMTM provides the top performance across the reported modalities on NVGesture, while FOANet’s 91.28% result is not directly comparable because it uses a separate hand-detection network.
- Audio-Visual Speech Enhancement: MMTM outperforms both audio-only and channel-wise-concatenation audio-visual baselines on PESQ and STOI, including gains of 0.3 PESQ, 0.01 STOI, and 0.06 PESQ over the respective baselines.
- Action Recognition: On NTU-RGBD, the proposed RGB-plus-pose fusion outperforms recent action-recognition algorithms and is reported as a new state-of-the-art result.
- Network Analysis: Convolutional MMTM variants do not noticeably improve late fusion, while removing squeeze blocks increases FLOPS by about five times.
- Network Analysis: Intermediate and high-level MMTM placement is beneficial, with six modules after the last twelve inception modules giving the best gesture-recognition performance.
6. Conclusion
The paper presents MMTM as a simple module for leveraging knowledge across modalities in CNNs, enabling slow fusion at different feature-hierarchy levels. Experiments across gesture recognition, audiovisual speech enhancement, and human action recognition show its applicability.
- MMTM is a simple neural network fusion module for leveraging knowledge from multiple modalities in convolutional neural networks.