Source-linked AI summary
Multimodal Transformer with Multi-View Visual Representation for Image Captioning
Jun Yu, Jing Li, Zhou Yu, Qingming Huang
TL;DR
Existing image-captioning attention models neglect intra-modal interactions and may lack sufficiently comprehensive visual representations. The paper introduces a Multimodal Transformer with multi-view visual features, and reports significant gains on MSCOCO, including first-place real-time leaderboard performance for a seven-model ensemble.
Problem
Existing image-captioning methods model inter-modal co-attention while neglecting intra-modal interactions, and region-based features may not cover all image objects.
Method
The MT model jointly captures self-attention within modalities and co-attention across modalities, while multi-view learning handles aligned and unaligned visual features.
Results
The method significantly outperforms existing approaches on MSCOCO, and an ensemble of seven models achieves the best real-time leaderboard performance.
Takeaways & Limitations
Deep multimodal attention and multi-view visual representations provide the basis for the paper’s reported image-captioning improvements.
Abstract
from arXiv · showhide
Image captioning aims to automatically generate a natural language description of a given image, and most state-of-the-art models have adopted an encoder-decoder framework. The framework consists of a convolution neural network (CNN)-based image encoder that extracts region-based visual features from the input image, and an recurrent neural network (RNN)-based caption decoder that generates the output caption words based on the visual features with the attention mechanism. Despite the success of existing studies, current methods only model the co-attention that characterizes the inter-modal interactions while neglecting the self-attention that characterizes the intra-modal interactions. Inspired by the success of the Transformer model in machine translation, here we extend it to a Multimodal Transformer (MT) model for image captioning. Compared to existing image captioning approaches, the MT model simultaneously captures intra- and inter-modal interactions in a unified attention block. Due to the in-depth modular composition of such attention blocks, the MT model can perform complex multimodal reasoning and output accurate captions. Moreover, to further improve the image captioning performance, multi-view visual features are seamlessly introduced into the MT model. We quantitatively and qualitatively evaluate our approach using the benchmark MSCOCO image captioning dataset and conduct extensive ablation studies to investigate the reasons behind its effectiveness. The experimental results show that our method significantly outperforms the previous state-of-the-art methods. With an ensemble of seven models, our solution ranks the 1st place on the real-time leaderboard of the MSCOCO image captioning challenge at the time of the writing of this paper.
I. INTRODUCTION
Image captioning requires recognizing objects and their relationships, but existing approaches neglect intra-modal interactions and may use insufficient visual representations. The paper proposes a Multimodal Transformer with multi-view features and reports strong MSCOCO performance.
- Image captioning generates natural-language descriptions while requiring recognition of objects and their relationships.
- Existing attention mechanisms model inter-modal co-attention but neglect intra-modal word-to-word and object-to-object interactions.
- The MT model extends the Transformer to jointly capture self-attention within modalities and co-attention across modalities.
- Multi-view learning supplies more diverse and discriminative visual representations, with separate strategies for aligned and unaligned features.
- Extensive MSCOCO experiments show that MT significantly outperforms previous state-of-the-art approaches, while an ensemble ranks first on the real-time leaderboard.
II. RELATED WORK
Image captioning research includes template-based, retrieval-based, and generation-based approaches. Attention mechanisms substantially improve generation-based models, while related co-attention methods can neglect dense cross-modal interactions.
- Image captioning approaches are categorized as template-based, retrieval-based, or generation-based.
- Template-based approaches: Template-based methods align sentence fragments with predicted labels and generate captions using predefined language templates.
- Template-based approaches: Template-based captions depend on template quality and have severely restricted diversity.
- Retrieval-based approaches: Retrieval-based methods search large caption databases using cross-modal similarities between image and caption segments.
- Generation-based approaches: Generation-based methods learn language models that produce novel captions with flexible syntactical structures, often using neural encoder-decoder architectures.
- Attention mechanisms: Attention is a major improvement for generation-based models, while many co-attention frameworks learn separate modality-specific distributions and neglect dense interactions.
III. MULTIMODAL TRANSFORMER
The Transformer is a deep encoder-decoder architecture built from stacked multi-head attention and feed-forward modules. Its encoder uses self-attention, while its decoder combines masked self-attention with encoder-guided attention.
- The Transformer was originally proposed for machine translation and has since been applied to many natural-language-processing tasks.
- Scaled dot-product attention: Scaled dot-product attention computes attended features from queries, keys, and values using dot products, scaling, and softmax weighting.
- Multi-head attention: Multi-head attention uses parallel attention heads to attend to diverse information from different representation subspaces.
- Feed-forward networks: The feed-forward network applies two linear layers with ReLU activation and dropout to transform attended features.
- Encoder-decoder architecture: Stacked attention blocks form a deep encoder-decoder: encoder blocks are self-attentional, whereas decoder blocks combine self-attention and guided attention.
B. Multimodal Transformer for Image Captioning
The image-captioning MT architecture encodes detector-derived visual regions with stacked self-attention and decodes captions using image-guided attention. Caption generation is trained for next-word prediction and performed sequentially at test time.
- The MT image encoder and textual decoder transform visual features into captions through stacked attention blocks.
- Image encoder: A pretrained Faster-RCNN detector extracts and confidence-ranks region features, retaining the top-m objects as an image feature matrix.
- Image encoder: The projected visual features pass recursively through L encoder attention blocks to obtain attended image representations.
- Caption decoder: Caption words use 300-dimensional GloVe embeddings, zero-padding for short captions, and a one-layer LSTM for temporal representations.
- Caption decoder: The decoder contains self-attention, image-guided attention, and a feed-forward module; decoder self-attention masks subsequent positions.
- Caption generation: Training applies softmax cross-entropy to predict each next word, while testing generates words sequentially and recursively feeds predictions back into the model.
IV. IMAGE ENCODER WITH MULTI-VIEW VISUAL REPRESENTATION
The paper extends its image encoder to multi-view visual representation, addressing the limited fine-grained coverage of global multi-view features and correspondence challenges across detector views.
- Existing multi-view learning approaches mainly integrate global image features, which may fail to preserve fine-grained visual information.
- Objects extracted by different detectors are naturally unaligned, making correspondence learning across views challenging.
- Aligned Multi-View Image Encoder: AMV obtains aligned features by using unified bounding boxes before feeding them into the multi-view image encoder.
- The paper introduces Aligned Multi-View and Unaligned Multi-View image encoders to handle multi-view visual features.
A. Aligned Multi-View Image Encoder
The AMV encoder aligns detector views through shared object boxes, then concatenates corresponding features so they can replace single-view image features.
- The AMV model selects one of M pre-trained Faster R-CNN detectors to generate unified bounding boxes for all views.
- For m unified boxes, each view produces X^(i) ∈ R^(m×d_i), and the view features are concatenated across columns.
- The concatenated aligned features replace the single-view feature and are fed into the image encoder.
- Using unified boxes may reduce multi-view feature diversity and restrict detectors to Faster R-CNN, limiting one-stage detector usage.
B. Unaligned Multi-View Image Encoder
The UMV encoder directly integrates unaligned detector features by learning adaptive alignment through a primary view and multi-head attention, with depth improving visual representations.
- UMV directly integrates unaligned multi-view features from different object detectors.
- Each view is projected into a common space, while a primary view guides attention learning for the other views.
- The attended features from other views are aligned to the primary view’s shape and integrated through element-wise summation.
- UMV can be stacked in depth to learn more accurate cross-view interactions and produce more discriminative visual features for caption generation.
V. EXPERIMENTS
Experiments evaluate the proposed models on MSCOCO using established splits, detector-derived multi-view features, caption preprocessing, and standard caption-quality metrics.
- The experiments evaluate the proposed MT models on the MSCOCO 2015 image captioning dataset.
- Datasets: Visual Genome object and attribute annotations are used to pre-train bottom-up-attention models for visual feature extraction.
- Datasets: MSCOCO uses Karpathy splits with 113k, 5k, and 5k images for training, validation, and testing, respectively.
- Datasets: Caption quality is evaluated with BLEU, ROUGE-L, METEOR, and CIDEr.
- Implementation Details: Captions are lowercased, whitespace-tokenized, and filtered into a 9,343-word vocabulary.
- Implementation Details: Multi-view representation uses up to three Faster R-CNN detectors with ResNet-101, ResNet-152, and ResNeXt-101 backbones.
C. Ablation Studies
Ablations on MSCOCO examine caption representations, attention-block depth, detector backbones, and multi-view features, showing that representation choices and deeper reasoning materially affect MT performance.
- Caption Representations: Pre-trained GloVe embeddings significantly improve performance over randomly initialized embeddings, while LSTM temporal modeling and fine-tuning provide smaller additional gains.Directly fine-tuning GloVe from the cross-entropy stage performs worse, suggesting that training from scratch can reduce embedding representation capacity.
- Number of Attention Blocks: MT model size grows linearly with the number of attention blocks, while performance improves with depth before saturating.The deeper model can capture more complex object relationships but also has greater overfitting risk.
- Number of Attention Blocks: Self-critical training selects L=6 as optimal, compared with L=4 for cross-entropy training.The self-critical loss explores the hypothesis space more diversely and can better use larger models.
- Single-view vs. Multi-view: Multi-view features significantly improve captioning over single-view features, and unaligned multi-view MTumv slightly outperforms aligned MTamv.MTsv with the ResNet-152 backbone also steadily outperforms its ResNet-101 counterpart.
- Number of Views: Increasing MTumv views from M=2 to M=3 yields only a slight performance improvement, indicating near saturation.The study therefore does not introduce more than three views to the image encoder.
D. Comparison with the State-of-the-Art
The proposed MT models outperform prior image-captioning systems on the MSCOCO Karpathy test split, while a seven-model ensemble leads the official leaderboard at submission.
- Results on the Karpathy test split: MTumv achieves new state-of-the-art single-model performance across all evaluation metrics on the MSCOCO Karpathy test split.The comparison uses methods with the same ResNet-101 backbone, and MTsv also outperforms most state-of-the-art methods.
- Results on the official test server: A seven-model MT ensemble significantly outperforms all other leaderboard solutions across every reported evaluation metric.The result is reported for the official MSCOCO test server at the time of submission on April 21, 2019.
E. Qualitative Analysis
The qualitative analyses show that deeper attention blocks learn clearer object, word, and word-object relationships, while multi-view encoding aligns objects across views. MTumv often improves captions, but both variants retain complementary strengths and can fail on small objects.
- Attentions of the MTsv Encoder: Deeper encoder self-attention shifts from near-diagonal similarity to focused relationships around key visual objects.The first block shows limited pairwise interaction, whereas the sixth highlights objects such as the girl and skateboard.
- Attentions of the MTsv Decoder: Deeper decoder self-attention jointly represents word importance and pairwise similarities, highlighting relationships such as “woman” and “skateboard.”Focused columns for “woman” and “riding” emerge in the sixth decoder block.
- Attentions of the MTsv Decoder: Decoder guided-attention becomes semantically clearer with depth, concentrating co-attention on key objects and their word-object relationships.The first block contains diffuse and sometimes incorrect associations, while the sixth has clearer meanings.
- Attentions of the MTumv Encoder: Multi-view guided attention adaptively aligns objects across views and captures contextual relationships involving object parts and background scenes.The multi-view encoder learns cross-view interactions that provide a fine-grained understanding of image content.
- Predicted Captions: In one example, MTumv identifies two people and a dog in a park, whereas MTsv describes one person walking with a dog.The ground-truth caption mentions a man taking a walk with two dogs.
- Predicted Captions: MTumv outperforms MTsv in four illustrated cases, while MTsv wins two; both models also produce incorrect captions, often involving small objects.Their differing strengths create diversity when the models are combined in an ensemble.
VI. CONCLUSIONS
The paper presents a Multimodal Transformer framework for image captioning, combining deep attention-based visual encoding, caption decoding, and multi-view learning. Evaluations on MSCOCO show significant improvements over existing approaches, with a seven-model ensemble achieving the best real-time leaderboard performance.
- The Multimodal Transformer uses deep self-attention in its image encoder and transforms visual features into textual captions with a decoder.
- Two MT variants model aligned and unaligned multi-view features through multi-view learning in the image encoder.The variants are named MTamv and MTumv.
- The proposed models significantly outperform existing approaches on MSCOCO, and an ensemble of seven models achieves the best real-time leaderboard performance.