Gated Multimodal Units for Information Fusion

John Arevalo; Thamar Solorio; Manuel Montes-y-Gómez; Fabio A. González

Source-linked AI summary

Gated Multimodal Units for Information Fusion

John Arevalo, Thamar Solorio, Manuel Montes-y-Gómez, Fabio A. González

arXiv:1702.01992v1Feb 7, 2017stat.MLcs.LG

TL;DR

Multimodal learning needs representations that combine heterogeneous inputs while learning which modality matters for the task. The paper introduces the Gated Multimodal Unit, evaluates it for multilabel movie genre classification from plots and posters, and reports stronger performance than unimodal and other fusion approaches. It also releases the MM-IMDb dataset for multimodal movie genre prediction.

Problem
Most representation-learning models focus on one modality or use auxiliary objectives to create common representations, motivating a task-optimized approach for combining multimodal information.
Method
The Gated Multimodal Unit uses multiplicative gates inside neural networks to learn how each modality contributes to an intermediate representation.
Results
The GMU achieved the best multimodal performance, improving independent modalities in averaged measures and 16 of 23 genres while outperforming evaluated fusion strategies.
Takeaways & Limitations
Because the GMU is differentiable and reusable within neural-network architectures, it provides a learnable fusion unit for multimodal tasks.

Abstract

from arXiv · show

This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.

1 INTRODUCTION

The paper addresses multimodal representation learning by introducing a gated module that learns how different modalities should influence an intermediate neural representation. It evaluates this approach on movie genre prediction using plots and posters.

Most representation-learning efforts focus on a single information type despite real-world concepts being described through multiple modalities.The paper motivates multimodal learning as a way to exploit relationships across text, images, audio, and other data types.
The proposed module combines multiple information sources while optimizing directly for the end-goal objective.It uses gates to select input parts likely to contribute to the desired output rather than constructing a common representation through an auxiliary task.
Multiplicative gates learn feature importance from training data without manual tuning, while supporting end-to-end gradient-based optimization.The gated model is designed to be reused within different neural-network architectures and tasks.
Movie genre prediction uses both plot and poster modalities because single-modality predictions can be challenging even for human observers.The application considers assigning multiple genre labels to a movie from these complementary inputs.
The paper evaluates its multimodal approach through an experimental setup covering methods, the MM-IMDb dataset, and movie genre-classification results.The paper structure places the proposed model and baselines before dataset details, evaluation, results, and conclusions.

2 RELATED WORK

Prior multimodal work includes feature concatenation, generative models, semantic embeddings, and mixture-of-experts strategies, while movie genre studies use varied modalities and datasets. The paper identifies limited comparability and dataset scale as gaps addressed by its released dataset.

Multimodal systems combine modalities such as video, audio, images, and text to support automatic analysis, with prior work often reporting advantages over unimodal approaches.Representation learning has become a common way to characterize multimodal information.
Feature concatenation is straightforward but ignores inherent correlations between modalities.It produces a single representation intended to simplify downstream classification or prediction.
Prior fusion strategies include RBMs, autoencoders, and deep Boltzmann machines that learn or reconstruct multimodal representations.These methods have been applied to tasks such as retrieval, annotation, and modeling interactions between modalities.
Other approaches optimize task-specific objectives or map modalities into a common latent semantic space.These methods assume transformations can express the same semantic concepts across modalities.
The GMU differs from common mixture-of-experts usage by serving as a representation-learning component rather than primarily combining final task decisions.It can be used with differentiable objectives for classification, regression, or unsupervised learning.
Movie genre studies use textual, visual, audio, video, ratings, and handcrafted features, but prior datasets and tasks vary substantially.Earlier work includes single-label and multilabel settings with relatively small datasets.
The paper releases a MovieLens 20M-derived dataset with genre, poster, plot, and more than 50 IMDb characteristics to improve systematic comparison.The authors also plan to release code for automatically adding movies and genres.

3 METHODS

The methods combine multimodal representations with a Gated Multimodal Unit that learns modality contributions through gates, alongside text, visual, and classification components.

3.1 GATED MULTIMODAL UNIT FOR MULTIMODAL FUSION: The Gated Multimodal Unit learns to decide how modalities influence a hidden unit's activation using gates.It is designed as an internal neural-network unit for finding an intermediate representation from multiple modalities.
3.1 GATED MULTIMODAL UNIT FOR MULTIMODAL FUSION: Each modality is encoded with a tanh feature neuron, while a gate receives all modality vectors and controls that modality's contribution.The general model supports more than two modalities; the bimodal simplification ties gates and trades off modalities with fewer parameters.
3.1 GATED MULTIMODAL UNIT FOR MULTIMODAL FUSION: The bimodal GMU combines visual and textual features through a learned gate and remains differentiable for stochastic-gradient training.Its equations define a gated interpolation of visual and textual hidden representations, with concatenation used as the gate input.
3.2 TEXT REPRESENTATION: Text representations include n-gram, word2vec, and recurrent neural-network approaches.The evaluated recurrent variants use either transferred word2vec vectors or word vectors learned from scratch.
3.3 VISUAL REPRESENTATION: Visual representations use transfer learning with VGG or end-to-end convolutional-neural-network training.The VGG approach extracts the last hidden activations from an ImageNet-trained network.
3.4 CLASSIFICATION MODEL: Feature vectors are mapped to genre predictions using logistic regression or a two-layer maxout multilayer perceptron.The maxout function selects among linear transformations for each hidden unit.

4 EXPERIMENTAL EVALUATION

The evaluation uses the MM-IMDb multimodal dataset and compares textual, visual, and fusion-based approaches for multilabel movie-genre classification. It also examines whether normalization and regularization improve neural-network training and test performance.

4.1 MULTIMODAL IMDB DATASET: Movie plots average 92.5 words, the longest contains 1,431 words, and each movie has 2.48 genres on average.The task is defined as movie-genre prediction from the plot and image poster, with multiple genres allowed per movie.
4.2 EXPERIMENTAL SETUP: The train, development, and test subsets contain 15,552, 2,608, and 7,799 samples, respectively, with stratified 60%, 10%, and 30% genre proportions.
4.2 EXPERIMENTAL SETUP: Evaluation reports samples, micro, macro, and weighted f-score averages because multilabel measures can differ substantially.Samples averages per-sample f-scores; micro aggregates predictions, while macro averages per-genre f-scores.
4.2 EXPERIMENTAL SETUP: Text representations use pretrained Google Word2vec embeddings, producing a 41,612-word vocabulary without preprocessing beyond lowercasing.The textual architecture was also evaluated with a single fully connected layer to examine the effect of network depth.
4.2 EXPERIMENTAL SETUP: Visual representations use either VGG Transfer features or a CNN processing images scaled and cropped to 160 × 256 pixels.The CNN contains five convolutional layers with squared filters of sizes 5, 3, 3, 3, and 3, followed by the MaxoutMLP architecture.

5 RESULTS

The GMU learned which modality carried useful information in synthetic data and improved multimodal movie-genre classification over independent modalities and alternative fusion strategies.

5.1 EVALUATION OVER SYNTHETIC DATA: The GMU learned the latent modality selector in synthetic experiments, with gate activations correlating 1 with the generating variable M.Across 1000 experiments, it outperformed logistic regression in 370 and matched it in the remainder.
5.1 EVALUATION OVER SYNTHETIC DATA: Figure 7 maps each modality's feature space to gate activations and predictions, showing where the model favors visual or textual input.White regions indicate z = 1 and greater importance for xv; gray regions indicate z = 0 and greater importance for xt.
5.2 GENRE CLASSIFICATION RESULTS: MaxoutMLP w2v increased 7Genre performance from 0.841 to 0.854 while using only text, unlike the baseline's additional HTML information.It also achieved state-of-the-art results on the ki-04 dataset.
5.2 GENRE CLASSIFICATION RESULTS: Pretrained visual models outperformed visual models trained from scratch, while the visual modality exceeded the textual modality for Animation.The authors relate the visual-model result to the dataset's limited size for learning poster complexity.
5.2 GENRE CLASSIFICATION RESULTS: The GMU achieved the best multimodal performance, improving averaged independent-modality measures and results in 16 of 23 genres while outperforming all evaluated fusion strategies.Concatenation and linear combination did not model intermodal correlation sufficiently, and MoE models were no better than simpler approaches on this relatively small dataset.
5.2 GENRE CLASSIFICATION RESULTS: The model generally relied more on text, but visual input influenced predictions more for Animation and Family, consistent with their stronger visual performance.Gate subsets were selected using mutual information between predictions and z activations.
5.2 GENRE CLASSIFICATION RESULTS: Qualitative examples show the model exploiting the more accurate modality and sometimes removing false positives, including cases confusing to humans without one modality or additional context.

6 CONCLUSIONS

The paper presents the GMU as a gated strategy for learning multimodal fusion and reports successful synthetic and movie-genre applications. It also releases a large public multimodal movie dataset and identifies deeper GMU architectures, attention, and interpretability as future directions.

6 CONCLUSIONS: The GMU receives two or more sources and learns how much each modality affects unit activation through multiplicative gates.
6 CONCLUSIONS: Synthetic experiments showed that the GMU learned hidden latent variables, while the real-world application outperformed single-modality approaches.
6 CONCLUSIONS: The differentiable GMU can be coupled to other neural-network architectures and trained with standard gradient-based optimization.
6 CONCLUSIONS: The released MM-IMDb dataset contains around 27,000 movie plots, images, and other metadata and is described as the biggest multimodal dataset used for movie-genre classification and the first publicly available one.