EmbraceNet: A robust deep learning architecture for multimodal classification

Jun-Ho Choi; Jong-Seok Lee

Source-linked AI summary

EmbraceNet: A robust deep learning architecture for multimodal classification

Jun-Ho Choi, Jong-Seok Lee

arXiv:1904.09078v1Apr 19, 2019cs.LGstat.ML

TL;DR

Multimodal classification needs fusion that captures cross-modal relationships while remaining usable when data or entire modalities are missing. EmbraceNet combines compatible modality networks through docking and probabilistic embracement, and experiments report improved robustness against missing data and modalities compared with other architectures.

Problem
Multimodal classification must model cross-modal relationships while handling unavailable parts of data or entire modalities.
Method
EmbraceNet docks outputs from modality networks and probabilistically embraces their representations into a fused classification input.
Results
EmbraceNet showed improved robustness against both block-wise missing data and missing modalities compared with other state-of-the-art architectures.
Takeaways & Limitations
The architecture supports diverse network types and maintains good performance when multimodal data availability is limited.
Takeaways & Limitations
Existing missing-data remedies such as repetition, default values, and interpolation are described as workarounds rather than fundamental solutions.

Abstract

from arXiv · show

Classification using multimodal data arises in many machine learning applications. It is crucial not only to model cross-modal relationship effectively but also to ensure robustness against loss of part of data or modalities. In this paper, we propose a novel deep learning-based multimodal fusion architecture for classification tasks, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data. We employ two datasets for multimodal classification tasks, build models based on our architecture and other state-of-the-art models, and analyze their performance on various situations. The results show that our architecture outperforms the other multimodal fusion architectures when some parts of data are not available.

1. Introduction

EmbraceNet addresses multimodal classification when sensors or modalities are unavailable by combining compatible network models and probabilistically integrating their representations. The architecture is designed to preserve performance under block-wise data loss and complete modality loss.

1. Introduction: Unavailable sensors expose a gap in multimodal architectures, while common remedies repeat, default, or interpolate missing values rather than fundamentally handling the loss.The problem arises because real-world acquisition devices can disconnect unexpectedly.
1. Introduction: EmbraceNet combines docking layers with a probabilistic embracement layer to integrate multimodal representations for classification.The architecture is intended for multimodal information-based classification tasks in the wild.
1. Introduction: Any number of modalities and network types can be integrated because EmbraceNet accepts outputs from existing models and produces a fused representation for the final decision model.The stated compatible models include convolutional, dense, and recurrent architectures.
1. Introduction: Probabilistic partial selection during training models cross-modal correlations while regularization helps avoid overfitting.The embracement process selects partial information from each modality for combination.
1. Introduction: EmbraceNet is designed to maintain performance under both block-wise missing data and complete loss of some modalities.Experiments use gas sensor arrays and multimodal human activity data to demonstrate robustness.

2. Related work

Prior multimodal fusion methods include early and late integration, specialized deep architectures, and preprocessing-based missing-data remedies. These approaches variously assume synchronized inputs, ignore shared representations, depend on specific architectures, or rely on imperfect reconstruction.

2. Related work: Multimodal fusion is used across videos, human activity recognition, and biological, chemical, and medical datasets.These applications combine signals such as images, audio, motion, depth, accelerometers, and microphones.
2. Related work: Early integration concatenates modality features and models cross-modal correlations, but assumes perfect synchronization of the inputs.It can adopt unimodal classification models after converting modalities to feature vectors.
2. Related work: Late integration trains modality-specific classifiers independently, but its separate representations ignore correlated characteristics across modalities.Combining classifier outputs with a weighted sum is reported to outperform multiplication.
2. Related work: Deep learning studies commonly retain early or late integration, while other missing-data methods use interpolation, zero values, noise augmentation, or reconstruction.These remedies may improve robustness partially, but reconstruction can fail when estimated values are inaccurate.
2. Related work: Existing models that handle missing modalities include shared-representation and multi-fusion architectures, but some are limited to particular network structures.The supplied discussion identifies architecture-specific limitations among prior approaches.

3. Model description

EmbraceNet docks modality-specific outputs into equal-sized vectors, then forms an embraced vector by probabilistically selecting modality contributions component by component. The resulting vector feeds a terminal classifier while retaining information from all modalities across components.

3.1. Docking layers: EmbraceNet accepts independent modality networks, converts their outputs into same-sized dockable vectors, and passes the fused vector to a terminal decision network.The modality networks may be multilayer perceptrons, CNNs, hand-crafted feature vectors, or raw-data processors.
3.1. Docking layers: Each docking layer applies a learned affine transformation followed by an activation function to produce a c-dimensional modality representation.The activation may be ReLU, sigmoid, or hyperbolic tangent.
3.2. Embracement layer: Rather than summing modality vectors, the embracement layer uses multinomial sampling to select modality information for each vector component.The sampling probabilities can be equal across modalities or tuned during and after training.
3.2. Embracement layer: The sampled one-hot vectors are applied element-wise to docked representations before producing each component of the embraced vector.Only one modality contributes to each component in a given sampling operation.
3.2. Embracement layer: Because selection occurs independently for every component, the final embraced vector integrates information from all modalities while keeping the docked-vector length c.The embraced vector then serves as input to the terminal classification network.

4. Benefits of the EmbraceNet architecture

EmbraceNet fuses modality-specific representations while modeling cross-modal correlations and handling missing modalities by reallocating selection probabilities to available inputs.

The architecture is presented as a powerful solution for fusing multimodal information across sensor inputs.
EmbraceNet’s docking and embracement layers fuse modality-specific representations while probabilistically modeling cross-modal correlations.Docking layers produce modality representations, and the embracement layer selectively combines their components.
Handling missing data: When a modality is unavailable, setting its presence indicator to zero also sets its adjusted probability to zero, preventing invalid data from propagating.Available modalities consequently contribute more to the embracement output.
Regularization effect: The embracement selection process is equivalent to Bernoulli dropout at each output value, regularizing the model against overreliance on specific modalities.Each value is dropped with probability (1−p_k).

5. Optimizing the EmbraceNet architecture

The paper optimizes EmbraceNet’s probability parameters during and after training, using bimodal MNIST and Fashion MNIST experiments to assess robustness under missing modalities.

The probability parameters regulate activations during training and control modality contributions during testing.
When only one image half is available, EmbraceNet significantly outperforms intermediate integration despite limited recognition information.
Adjusting parameters during training: Randomly selecting one modality with probability 0.5 during training reduces missing-data error rates without changing the non-missing-data error rate.The authors attribute this improvement to advanced regularization.
Adjusting parameters after training: Post-training probability adjustment lowers error from 0.95% to 0.93% on bimodal MNIST and from 8.11% to 8.06% on bimodal Fashion MNIST.Missing-data error rates remain unchanged because p has no effect when only one modality is available.
Adjusting parameters after training: The jointly optimized model performs better than the other models for all tested modality combinations.This is the final result reported for optimization during and after training.

6. Comparison with other multimodal fusion techniques

The paper evaluates EmbraceNet by comparing it with other classification models on multimodal classification tasks.

The experiments compare EmbraceNet with other classification models using specified datasets, network structures, and experimental setups.

6.1. Datasets

The evaluation uses gas sensor arrays and OPPORTUNITY datasets covering chemical-source classification and human-activity recognition with multiple sensor modalities.

The study uses the gas sensor arrays dataset for chemical-source classification and the OPPORTUNITY dataset for human-activity recognition.
Gas sensor arrays dataset: The gas sensor dataset contains 18,000 measurement sequences from 10 chemical sources and 72 sensors.
Gas sensor arrays dataset: Gas-sensor experiments treat eight sensor types as modalities, downsample to 1 Hz, window recordings, normalize values to [0.0, 1.0], and split sensor-array locations across training, validation, and testing.
OPPORTUNITY dataset: OPPORTUNITY includes inertial and accelerometer data from four subjects across daily-living and drill activity sequences.
OPPORTUNITY dataset: The OPPORTUNITY evaluation covers locomotion modes with four classes and mid-level gestures with 17 activities, plus a null class.
OPPORTUNITY dataset: The sixth sensor-array location is excluded because butanol data there are entirely missing in the original dataset.

6.2. Performance measure

Performance is measured with weighted F1, which combines class-wise precision and recall while accounting for class imbalance.

Weighted F1 combines each class’s precision and recall, weighted by its share of the total data.The weighting term n_i/n balances the score against skewed class distributions.

6.3. Network models

The study compares five multimodal baselines with EmbraceNet across gas-sensor and OPPORTUNITY classification settings. The models differ mainly in how modalities are integrated, represented, or reconstructed before classification.

Six models are compared: early, late, and intermediate integration; compact multi-linear pooling; multimodal autoencoding; and EmbraceNet.The baseline models are designed with matching structures apart from their integration components.
Early integration concatenates modality inputs before shared convolutional processing, while intermediate integration concatenates learned representations later in the network.For OPPORTUNITY, the early model processes 113 sensor channels, whereas intermediate fusion occurs after the final convolutional layers.
Late integration duplicates a network for each modality and uses validation F1 scores as integration weights.The gas-sensor and OPPORTUNITY models use eight and 19 modality-specific networks, respectively.
Compact multi-linear pooling fuses modality features through count sketches, Fourier-domain element-wise multiplication, and inverse transformation.Its integration structure otherwise follows the intermediate-integration models.
The multimodal autoencoder reconstructs inputs with randomly masked modalities before replacing its decoder with classification layers.Randomly selected 50% of modalities are replaced by -1 during preprocessing.
EmbraceNet uses docking layers to produce modality representations and an embracement layer to combine them into one vector.The docking layers produce vectors of length 1024 for each modality.

6.4. Scenarios

The experiments evaluate robustness when complete modalities or portions of modality streams are unavailable. Missing inputs are handled differently depending on whether each model naturally supports absent data.

Complete-modality loss is evaluated across modality combinations: all 255 combinations for the gas-sensor dataset and 14,319 sampled combinations for OPPORTUNITY.For OPPORTUNITY, up to 1,000 random combinations are evaluated for each number of selected modalities.
Early, intermediate, and compact multi-linear pooling models replace missing modalities with training-set averages, while late integration, autoencoding, and EmbraceNet naturally handle missing data.For missing stream segments, the first three models instead fill missing values with their previous values.
Block-wise stream loss on OPPORTUNITY removes randomly selected 300–900-step ranges from modalities until missing rates of 10%–90% are reached.The removed ranges correspond to 10–30 seconds.
The scenarios reflect real deployment conditions in which sensors may disappear or data streams may contain missing segments.Missing data are reported in several multimodal datasets, including human-action, biological, and distracted-driving datasets.

7. Results

EmbraceNet generally outperforms competing multimodal fusion architectures as modalities or data blocks are removed, while preserving strong classification performance under missing data. Its advantage is especially clear when multiple modalities remain available, where cross-modal information can still be exploited.

7.1. Missing modalities: EmbraceNet mostly outperforms early, late, intermediate, compact multi-linear pooling, and multimodal autoencoder models as modalities are lost.The comparison covers the gas sensor arrays and OPPORTUNITY datasets across different numbers of available modalities.
7.1. Missing modalities: Conventional early, intermediate, and compact multi-linear pooling models degrade sharply as modalities decrease, reaching F1 scores near 0.1 with one available modality.The result is attributed to methods that do not account for absent modalities.
7.1. Missing modalities: When more than two modalities are available, EmbraceNet outperforms late integration in all classification tasks, with larger gaps for gas types and gestures.With only one modality, late integration performs slightly better because cross-modal correlations are unavailable.
7.1. Missing modalities: EmbraceNet activations remain consistent across modality counts for the same class and distinguish different classes, even with only one or two modalities.Figure 7 visualizes averaged 1024-dimensional embracement-layer outputs reshaped to 32 × 32 matrices.
7.2. Missing block-wise data: F1 scores for early and intermediate integration fall to 0.514 and 0.496 under 90% block-wise missing locomotion data, whereas EmbraceNet maintains performance across cases.These values are reported for the OPPORTUNITY modes-of-locomotion task.
7.2. Missing block-wise data: With 50% sensor data missing, EmbraceNet F1 decreases by only 0.017 for locomotion and 0.018 for gestures.The comparison concerns block-wise missing data in the OPPORTUNITY dataset.

8. Conclusion

The paper presents EmbraceNet as a multimodal architecture that integrates heterogeneous network representations while learning cross-modal information and tolerating missing data. Across the reported comparisons, it combines compatibility with varied networks and improved robustness under limited data availability.

8. Conclusion: The architecture integrates multimodal information while handling absence of part of the data through its embracement process.The conclusion frames this as the paper’s central architectural contribution.
8. Conclusion: The paper compares EmbraceNet with state-of-the-art architectures and reports improved robustness.This comparison supports the paper’s overall conclusion about performance under data loss.
8. Conclusion: EmbraceNet can attach between feature extraction and classification parts of convolutional, dense, or recurrent networks.This supports compatibility with varied network types.
8. Conclusion: EmbraceNet learns cross-modal information and achieves similar or higher performance than early and intermediate integration when no data or modality is missing.The conclusion states that late integration performs lower in this setting.
8. Conclusion: EmbraceNet remains robust to both block-wise missing data and missing modalities.The conclusion identifies robustness to limited data availability as a principal benefit.