Source-linked AI summary
Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network
Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun, Fuhai Chen, Gen Luo, Yongjian Wu, Yue Gao, Rongrong Ji
TL;DR
Transformer-based captioning models primarily represent image regions and therefore lack image-level information needed for broader visual reasoning. GET extracts intra- and inter-layer global representations and adaptively fuses them into decoding; experiments report state-of-the-art MS COCO performance, including a 0.4% CIDEr advance over the current state of the art.
Problem
Transformer-based captioning representations focus on image regions without incorporating global image information, limiting comprehensive multimodal reasoning and contributing to object missing and relationship bias.
Method
GET combines a Global Enhanced Encoder for intra- and inter-layer global representations with a Global Adaptive Controller that adaptively fuses global information into the decoder.
Results
GET surpasses other approaches on BLEU-4, METEOR, and CIDEr, achieves competitive SPICE and ROUGE-L performance, and advances current state of the art on CIDEr by 0.4%.
Takeaways & Limitations
GET provides more comprehensive visual information by connecting local parts and adaptively guiding caption generation on MS COCO.
Abstract
from arXiv · showhide
Transformer-based architectures have shown great success in image captioning, where object regions are encoded and then attended into the vectorial representations to guide the caption decoding. However, such vectorial representations only contain region-level information without considering the global information reflecting the entire image, which fails to expand the capability of complex multi-modal reasoning in image captioning. In this paper, we introduce a Global Enhanced Transformer (termed GET) to enable the extraction of a more comprehensive global representation, and then adaptively guide the decoder to generate high-quality captions. In GET, a Global Enhanced Encoder is designed for the embedding of the global feature, and a Global Adaptive Decoder are designed for the guidance of the caption generation. The former models intra- and inter-layer global representation by taking advantage of the proposed Global Enhanced Attention and a layer-wise fusion module. The latter contains a Global Adaptive Controller that can adaptively fuse the global information into the decoder to guide the caption generation. Extensive experiments on MS COCO dataset demonstrate the superiority of our GET over many state-of-the-arts.
INTRODUCTION
Traditional Transformer-based captioning models encode and attend to image regions but ignore image-level characteristics, limiting global guidance for object and relationship reasoning. GET addresses this gap by extracting intra- and inter-layer global representations and adaptively fusing them into caption decoding.
- Motivation: Transformer captioning models attend to encoded image regions but their vectorial representations remain region-biased and ignore image-level characteristics.This limitation can contribute to object missing and relationship bias when local objects are treated in isolation.
- Motivation: Pooling-based global extraction can introduce contextual noise, while multi-step self-attention may accumulate representation damage.The resulting semantic ambiguity can reduce global-representation accuracy.
- Method: GET uses Global Enhanced Attention and recurrent layer-wise aggregation to model intra- and inter-layer global representations.The encoder aggregates local information within layers and discards useless information from previous layers during sequential inter-layer fusion.
- Method: A Global Adaptive Controller adaptively fuses distilled global information into the decoder through alternative gating modules.The controller regulates global guidance because its need varies during caption generation.
- Contribution: GET leverages global representation to provide more comprehensive visual information and connect local parts, addressing object missing and relationship bias.The paper presents this global guidance as fundamental to the image captioning task.
- Results: GET achieves new state-of-the-art performance on the MS COCO dataset.The paper reports this outcome from extensive experiments.
RELATED WORK
Image captioning has evolved from CNN-RNN encoder-decoder systems toward advanced models that encode images and decode sentences using increasingly sophisticated architectures.
- Image Captioning: Earlier image captioning approaches commonly used CNNs to encode images and RNNs to decode their outputs into sentences.These systems were inspired by encoder-decoder architectures developed for machine translation.
PRELIMINARIES
Transformer image captioning uses stacked attention-based encoders and decoders to model interactions among image regions and words. GET’s overview extends this encoder-decoder pipeline with global-information extraction and adaptive decoder fusion.
- Transformer Architecture: The Transformer decoder applies self-attention to textual words and cross-attention to encoder representations while generating captions word by word.The encoder consists of stacked self-attention and feed-forward layers.
- Attention: Scaled Dot-Product Attention takes queries, keys, and values and produces weighted-average vectors through attention.Q contains nq queries, K and V contain nk keys and values, and d is the scaling factor.
- GET Overview: GET first extracts intra- and inter-layer global information alongside region-level representations, then adaptively fuses them into the decoder.The architecture overview omits residual connections, layer normalizations, and embedding layers.
- Attention: Multi-head attention expands subspace exploration by concatenating independently projected attention heads and applying an output transformation.Transformer uses this module to extend the capacity of attention-based representation learning.
OUR METHOD
GET uses a global-enhanced encoder to extract intra- and inter-layer global representations, then adaptively fuses them with local visual features during decoding. Its training combines supervised pre-training with reinforcement-learning fine-tuning using CIDEr-D rewards.
- Global-enhanced Encoder: GET maps object-detector visual features into local representations while extracting global information through its encoder stack.The encoder receives visual regions and produces highly abstract local representations alongside intra- and inter-layer global representations.
- Global-enhanced Encoder: Global-enhanced attention jointly processes region and global features to capture comprehensive image-level information across encoder layers.Residual connections and feed-forward processing are applied after attention; the design addresses the limitation of region-only representations.
- Global-enhanced Encoder: An LSTM fuses global representations from different layers, preserving low- and high-level information in the final global representation.The forgetting gate controls information retained from previous layers while aggregating representations from the first through the L-th layer.
- Global Adaptive Decoder: The decoder adaptively incorporates global contextual information with local visual features through a global adaptive controller before predicting caption words.The controller determines how much global information should be considered, and the decoder generates captions word by word.
- Global Adaptive Decoder: GET offers gated and multi-head adaptive controllers for refining local representations with global information during generation.The gated controller uses contextual signals, while the multi-head controller computes weighted fusion over region and global vectors.
- Training: Training uses supervised cross-entropy pre-training followed by self-critical sequence fine-tuning with CIDEr-D as the reward.Fine-tuning directly optimizes the evaluation metric using sequences sampled with beam search.
EXPERIMENTS
Experiments on MS COCO evaluate GET against state-of-the-art systems through offline, online, qualitative, attention, and ablation analyses. GET improves captioning metrics, produces more detailed descriptions, grounds words to image regions, and benefits from its proposed global-representation modules.
- Offline Evaluation: GET surpasses other approaches on BLEU-4, METEOR, and CIDEr while remaining competitive on SPICE and ROUGE-L; it advances CIDEr by 0.4%.
- Offline Evaluation: Four-model GET ensembles achieve the best performance on all metrics, increasing CIDEr by 0.6 points over the current state of the art.
- Online Evaluation: Online COCO evaluation shows GET surpassing the current state of the art on most metrics, with a 1.0-point CIDEr improvement over the best performer.
- Qualitative Analysis: GET generates more accurate and descriptive captions than a plain Transformer, adding details such as “a green uniform” and “a man” and producing more precise phrases.
- Ablation Study: Increasing the number of encoding and decoding layers slightly decreases performance, so subsequent experiments use three layers; ablations also support the proposed global modules and layer-wise fusion.
- Attention Visualization: Attention visualization indicates that GET helps ground generated words to correct image regions by exploiting its global representation.
CONCLUSION
The paper presents GET for image captioning, addressing the neglect of global contextual information through enhanced encoding and adaptive decoding. It reports superior quantitative and qualitative performance on MS COCO.
- GET addresses traditional Transformer captioning's neglect of global contextual information, which limits image-captioning reasoning.
- The Global Enhanced Encoder captures intra- and inter-layer global representations to provide more comprehensive visual information and connect local image parts.
- The Global Adaptive Decoder adaptively fuses global information into the decoder to guide caption generation.
- GET demonstrates superior performance quantitatively and qualitatively on the MS COCO datasets.