Source-linked AI summary
Supervised Multimodal Bitransformers for Classifying Images and Text
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, Davide Testuggine
TL;DR
Multimodal classification requires models that can use image information alongside increasingly prevalent web text, while existing tasks often make multimodal contributions difficult to assess. The paper introduces a supervised multimodal bitransformer that fuses unimodally pretrained encoders and evaluates it on standard and hard classification sets. It outperforms strong fusion baselines and approaches or matches multimodally pretrained ViLBERT without multimodal pretraining.
Problem
Web classification is often multimodal, but text-heavy tasks and existing evaluations can make it difficult to determine whether multimodal information improves classification.
Method
The paper introduces a supervised multimodal bitransformer that projects image features into BERT’s token-embedding space and jointly processes them with text.
Results
The model outperforms competitive fusion techniques and approaches or matches multimodally pretrained ViLBERT across multimodal classification tasks without multimodal pretraining.
Takeaways & Limitations
The architecture provides a straightforward, extensible baseline for multimodal classification and for evaluating future multimodal models.
Takeaways & Limitations
The usefulness of multimodal self-supervised models beyond this setting remains unclear because the architecture relies on individually pretrained unimodal components.
Abstract
from arXiv · showhide
Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised multimodal bitransformer model that fuses information from text and image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks, outperforming strong baselines, including on hard test sets specifically designed to measure multimodal performance.
1 Introduction
The paper presents a supervised multimodal bitransformer for classification, motivated by the prevalence of multimodal web data and limitations of existing multimodal evaluation. It combines unimodally pretrained components, evaluates text-heavy tasks and hard multimodal test sets, and achieves competitive or superior performance against alternative fusion methods and multimodally pretrained models.
- Multimodal classification is increasingly important because web text commonly appears alongside images, sounds, videos, or other modalities.
- The proposed supervised bidirectional transformer fuses unimodally pretrained text and image components for multimodal classification.The model is designed as a simple baseline that performs multimodal fusion through a BERT-like architecture.
- The model is evaluated on three text-heavy tasks and novel hard test sets containing examples that unimodal methods fail to classify correctly.The hard sets are intended to measure multimodal-specific performance more directly.
- The bitransformer outperforms competitive fusion techniques, including alternatives with strictly more parameters.
- Its performance approaches or matches multimodally pretrained ViLBERT models without multimodal pretraining.
2 Multimodal Bitransformers
The multimodal bitransformer combines pretrained language and vision representations by projecting image features into BERT’s token-embedding space before joint transformer processing. Its design supports multiple dense image inputs, multiple modalities, and staged fine-tuning, while retaining standard task-specific classification losses.
- The architecture combines text-only self-supervised representations with convolutional neural network features from computer vision.
- Image Encoder: ResNet-152 produces N separate 2048-dimensional image embeddings from pooled K×M image grids rather than one pooled vector.This allows the transformer to process arbitrary numbers of dense image inputs.
- Multimodal Transformer Input Layer: Learned mappings W_n project each image embedding into BERT’s D-dimensional token input-embedding space.The n-th image-encoder pooling output is transformed into the input representation consumed by the bidirectional transformer.
- Multimodal Transformer Input Layer: The model assigns segment and positional embeddings to text and image inputs and generalizes to an arbitrary number of modalities.Additional segment embeddings are initialized for cases such as the three-input V-SNLI task.
- Classification: The first final-layer output feeds a classifier using sigmoid with binary cross-entropy for multilabel tasks and softmax with cross-entropy for multiclass tasks.
- Fine-tuning: Fine-tuning explores staged freezing and unfreezing because image embeddings must first be mapped into an appropriate subspace while text may dominate.The image encoder is treated as a component whose unfreezing schedule is a hyperparameter.
3 Approach
The evaluation compares multimodal classification systems across MM-IMDB, FOOD101, and V-SNLI, using unimodal and multimodal baselines plus hard test sets designed to probe multimodal performance.
- Datasets: The study evaluates MM-IMDB, FOOD101, and V-SNLI, covering movie genres, food labels, and three-way visual textual entailment classification.V-SNLI extends beyond two input types with premise, hypothesis, and image triplets.
- Datasets: MM-IMDB is a multilabel task pairing movie plot outlines with posters to predict one or more genres.The dataset was introduced to address the scarcity of high-quality multimodal classification datasets.
- Datasets: FOOD101 pairs textual recipe descriptions with images and requires predicting one of 101 food labels for each recipe-image combination.The images were obtained through Google Image Search and may correspond to noisy categories.
- Baselines: The baselines include bag-of-words, text-only BERT, image-only ResNet-152, feature concatenation, late fusion, and FiLMBert.ConcatBert directly combines BERT and image encoder outputs, while late fusion averages separate classifier scores.
- Hard test sets: Hard test sets select the top 10% of examples where unimodal Bert and Img predictions differ most from the ground-truth classes.The selection criterion maximizes p(a̸ = t|I)p(a̸ = t|T), targeting examples requiring more sophisticated multimodal reasoning.
- Evaluation: Results are reported as MM-IMDB Macro F1 / Micro F1 and Accuracy for the other tasks, with comparisons against several multimodal baselines.The main-results table compares GMU, CentralNet, Word2vec+VGGNet, Bilinear-gated, and V-BiMPM; models are evaluated over five random-seed runs with standard deviations.
4 Results
Across standard, hard, missing-image, and ViLBERT comparisons, MMBT consistently performs strongly against alternative fusion methods and competitive multimodal models. Additional analyses support staged unfreezing, robustness to missing images, and an advantage over larger ConcatBert classifiers.
- Main results: MMBT outperforms alternative fusion techniques, with multimodal models surpassing their direct unimodal counterparts across tasks.The authors attribute MMBT’s improvement over ConcatBert to self-attention enabling interaction between modalities at different levels.
- Main results: ∼3% Macro-F1 on MM-IMDB and ∼6% on Food101 represent MMBT’s improvements over BERT.The Food101 improvement corresponds to approximately 1300 additional correctly classified examples.
- Hard test sets: 80.4 on V-SNLIhard compares with 73.75 for Vu et al. (2018), while MMBT preserves the main-results pattern of outperforming alternatives.The hard sets select examples where unimodal classifiers differ most from the ground truth, providing a stronger test of multimodal performance.
- Freezing strategy: Staged training first combines the components, then unfreezes the image encoder, and finally unfreezes the bitransformer; the optimal epoch count is task-dependent.The analysis reports that early image-encoder unfreezing works best.
- Parameter analysis: MMBT remains superior to ConcatBert after increasing the baseline to 174M or 175M parameters with deeper classifiers.For MM-IMDB, ConcatBert-2 and ConcatBert-3 obtain Macro-F1 values of 60.21 ± .5 and 59.71 ± .4, respectively.
- Missing modalities: MMBT remains more robust than ConcatBert when images are missing from part of the training data, although performance drops as fewer images are available.Figure 3 evaluates MicroF1 on MM-IMDB as the percentage of training examples without images increases.
- Comparison to ViLBERT: MMBT matches task-specific ViLBERT models on MM-IMDB Macro-F1 and matches ViLBERT on its hard subset.On FOOD-101, performance is similarly close and occasionally exceeds task-specific models, particularly on the hard subset.
- Comparison to ViLBERT: Using BERT-Large makes MMBT outperform ViLBERT without multimodal retraining, whereas ViLBERT would require retraining from scratch.The authors present this as an illustration of how easily newer unimodal encoders can be incorporated into MMBT.
5 Related Work
Related work spans multimodal fusion methods, multimodal applications, and self-supervised multimodal architectures. MMBT differs by fusing unimodally pretrained components through a supervised bitransformer.
- Fusion methods: Prior multimodal fusion methods include concatenation, bilinear gated combinations, multimodal transformers, CentralNet, MFAS, and Tensor Fusion Networks.Kiela et al. found bilinear fusion with gating effective for fixed pretrained representations.
- Applications: Multimodal NLP applications include classification, cross-modal retrieval, image captioning, visual question answering, and multimodal machine translation.The literature also uses multimodal information for learning human-like meaning representations.
- Self-supervised architectures: Recent self-supervised multimodal architectures include ViLBERT, VisualBERT, LXMERT, VL-BERT, and VideoBERT.These systems are presented as multimodal counterparts to large pretrained language models.
- MMBT’s position: Unlike those architectures, MMBT uses components pretrained only unimodally and combines them through supervised multimodal fusion.This design is described as straightforward and intuitive, with distinct advantages and disadvantages relative to multimodal pretraining.
6 Conclusion
The supervised multimodal bitransformer outperforms existing state-of-the-art methods and strong baselines, including on hard multimodal test sets. Its simple, intuitive design is easy to implement and provides a useful baseline for evaluating self-supervised multimodal models.
- Conclusion: The proposed supervised multimodal bitransformer significantly outperforms existing state-of-the-art methods and strong baselines across varied tasks and hard multimodal test sets.The hard test sets specifically target cases where unimodal methods fail.
- Conclusion: A strong baseline with more parameters and a deeper classifier was still outperformed after multimodal optimization analyses.The analyses explored freezing and unfreezing strategies alongside parameter counts.
- Conclusion: The architecture uses individually unimodally pretrained components and performs competitively with ViLBERT while remaining much simpler.The paper notes that the general usefulness of multimodal self-supervised models remains unclear.
- Conclusion: Supervised multimodal bitransformers are straightforward, intuitive, and easy to implement with existing self-supervised encoders.This supports their use as practical baseline architectures.