PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Transformer Based Model for Image Captioning

Yiyu Wang, Jungang Xu, Yingfei Sun

arXiv:2203.15350v1cs.CV

TL;DR

Image captioning commonly relies on pre-trained Faster R-CNN encoders and has faced limitations from LSTM-based decoding and two-stage training. PureT replaces this pipeline with an end-to-end pure Transformer model using SwinTransformer grid features, refinement, and multimodal fusion, achieving state-of-the-art MSCOCO performance.

  • Problem

    Pre-trained Faster R-CNN encoders divide image captioning into two stages, while LSTM decoders have limited training efficiency and expression ability.

  • Method

    PureT uses SwinTransformer grid features, a refining Transformer encoder, global-feature refinement, and a decoder with pre-fusion for end-to-end caption generation.

  • Results

    138.2% CIDEr is achieved by the single model on the offline “Karpathy” test split, and 135.3% (c5) and 138.0% (c40) by the 4-model ensemble online.

  • Takeaways & Limitations

    PureT provides a pure Transformer-based, one-stage image-captioning baseline that avoids object-detector pretraining and supports end-to-end training.

Abstract

from arXiv · show

CNN-LSTM based architectures have played an important role in image captioning, but limited by the training efficiency and expression ability, researchers began to explore the CNN-Transformer based models and achieved great success. Meanwhile, almost all recent works adopt Faster R-CNN as the backbone encoder to extract region-level features from given images. However, Faster R-CNN needs a pre-training on an additional dataset, which divides the image captioning task into two stages and limits its potential applications. In this paper, we build a pure Transformer-based model, which integrates image captioning into one stage and realizes end-to-end training. Firstly, we adopt SwinTransformer to replace Faster R-CNN as the backbone encoder to extract grid-level features from given images; Then, referring to Transformer, we build a refining encoder and a decoder. The refining encoder refines the grid features by capturing the intra-relationship between them, and the decoder decodes the refined features into captions word by word. Furthermore, in order to increase the interaction between multi-modal (vision and language) features to enhance the modeling capability, we calculate the mean pooling of grid features as the global feature, then introduce it into refining encoder to refine with grid features together, and add a pre-fusion process of refined global feature and generated words in decoder. To validate the effectiveness of our proposed model, we conduct experiments on MSCOCO dataset. The experimental results compared to existing published works demonstrate that our model achieves new state-of-the-art performances of 138.2% (single model) and 141.0% (ensemble of 4 models) CIDEr scores on `Karpathy' offline test split and 136.0% (c5) and 138.3% (c40) CIDEr scores on the official online test server. Trained models and source code will be released.

Introduction

PureT addresses limitations of pre-trained object-detector encoders and LSTM decoders by using a single-stage Transformer architecture for end-to-end image captioning. It combines SwinTransformer grid features with refining and decoding modules, achieving state-of-the-art MSCOCO results.

  • The model uses a Transformer-based decoder to generate captions word by word instead of relying on LSTM limitations in training efficiency and expression ability.
  • PureT integrates image captioning into one stage without object-detector pretraining, enabling end-to-end training.
  • SwinTransformer replaces Faster R-CNN to extract grid-level image features, while a refining encoder models relationships among them.
  • PureT adds global-feature refinement and pre-fusion between visual features and generated words to increase multimodal interaction.
  • 138.2% CIDEr is achieved by the single model on the offline “Karpathy” test split, while the 4-model ensemble reaches 140.8%.

Related Work

Image-captioning research evolved from CNN-LSTM systems with grid features to CNN-Transformer systems commonly using Faster R-CNN region features. Despite strong performance, region-level pipelines incur pretraining and computational costs that hinder end-to-end training.

  • Existing image-captioning models are broadly divided into CNN-LSTM and CNN-Transformer architectures.
  • Earlier systems used pre-trained CNNs for grid-level features and LSTM attention mechanisms for word-by-word caption generation.
  • Faster R-CNN introduced region-level features and became a typical encoder pattern in subsequent image-captioning models.
  • Transformer-based decoders were explored to address LSTM shortcomings in training efficiency and expression ability.
  • Region-level approaches require costly Faster R-CNN pretraining and feature extraction, leading many systems to train on cached features rather than images.

Model

PureT uses a SwinTransformer backbone, windowed self-attention refining encoder, and Transformer decoder to convert image grid features into captions. Global mean-pooled features are refined alongside grids and incorporated into multimodal decoding.

  • Model: The encoder extracts image grid features with SwinTransformer and refines them through stacked blocks, while the decoder generates captions word by word.
  • Attention Mechanism: Attention computes similarity between queries and keys, then forms outputs as weighted sums of the values.
  • Attention Mechanism: MSA models word-sequence and word-grid relationships in the decoder, whereas W-MSA and SW-MSA model image-grid relationships in the encoder.
  • W-MSA and SW-MSA: W-MSA and SW-MSA apply self-attention within local windows, reducing the quadratic cost of global MSA.
  • W-MSA and SW-MSA: Shifted windows follow regular windows to restore connections across window boundaries and improve modeling ability.
  • Encoder: The mean-pooled global feature is added as an extra key and value in windows and refined as an extra query over all grid features.

G + (S)W-MSA

The refining encoder processes grid and global image features together, producing refined visual content for the decoder. The decoder then generates captions word by word from these refined features.

  • G + (S)W-MSA: The refining encoder outputs refined grid features and a refined global feature for the decoder’s visual input.The global and grid features from block N are fed into the decoder.
  • G + (S)W-MSA: The decoder conditions word-by-word caption generation on the refined global and grid features from the encoder.It comprises N sequential blocks, with N set to 3 in this paper.
  • G + (S)W-MSA: Each decoder block contains multiple modules that support language processing and interaction between visual content and generated words.The supplied passage introduces the block structure before detailing its modules.

1) Pre-Fusion Module,

The decoder first fuses the refined global image feature with previously generated words, then models language dependencies and cross-modal visual attention. Training uses cross-entropy followed by self-critical sequence training targeting CIDEr.

  • 1) Pre-Fusion Module,: The pre-fusion module combines the refined global feature with previously generated words before each decoder block.This is the first interaction between natural language and visual content in the decoder.
  • 1) Pre-Fusion Module,: The pre-fusion design addresses the limited use of global contextual information in recent Transformer-based captioning models.The paper connects this limitation to reasoning capability and introduces global-feature fusion to capture global visual context.
  • 1) Pre-Fusion Module,: Masked self-attention models word-to-word relationships while restricting each word’s attention to earlier generated words.This preserves autoregressive decoding at timestep t.
  • 1) Pre-Fusion Module,: Cross-attention models word-to-vision relationships by using language features as queries and refined grid features as keys and values.The module captures local visual context information through a second multimodal interaction.
  • Objective Functions: The model is first optimized with cross-entropy loss and then with self-critical sequence training to optimize CIDEr.The latter uses sampled and greedily decoded captions to estimate the training gradient.

Experiments

Experiments use MSCOCO 2014 with the Karpathy split for training, validation, and offline evaluation, alongside an official online test set. Caption quality is assessed with five standard metrics.

  • Experiments: MSCOCO 2014 contains 123287 images, each annotated with 5 reference captions.The dataset includes 82783 training and 40504 validation images in its original organization.
  • Experiments: The Karpathy split assigns 113287 images to training, 5000 to validation, and 5000 to offline evaluation.MSCOCO also provides 40775 images for online testing.
  • Experiments: Evaluation uses BLEU, METEOR, ROUGE-L, CIDEr, and SPICE to measure generated-caption quality.These five metrics are adopted for fair evaluation.

Experimental Settings

The model uses fixed Transformer dimensions and a two-stage optimization schedule, first training with cross-entropy and then with a CIDEr-oriented objective.

  • Experimental Settings: The embedding size is D = 512, with 8 Transformer heads and 3 blocks in both the refining encoder and decoder.The final configuration uses three blocks for each component.
  • Experimental Settings: Training runs for 20 epochs under XE loss followed by 30 epochs under LR with a fixed learning rate of 5 × 10^-6.Adam is used in both stages, with 10000 warmup steps in the first stage.
  • Experimental Settings: Validation and evaluation use beam size 5.The supplied passage states the beam-size setting as part of the training configuration.

Comparisons with State-of-The-Art Models

PureT outperforms the compared state-of-the-art models on MSCOCO offline and online evaluations, with strong gains in CIDEr and broadly across metrics.

  • Offline Evaluation: 138.2% CIDEr for the single model and 141.0% for the four-model ensemble establish PureT's strongest offline results.The ensemble improves over DLCT and GET by 3.5% and 5.9%, respectively.
  • Offline Evaluation: PureT achieves the best single-model performance across all reported offline metrics against the compared models.Its CIDEr score exceeds RSTNet and DLCT by 2.6% and 4.4%, respectively.
  • Computational Implications: Compared with region-feature models, PureT offers a more balanced computational cost and supports end-to-end training from images to descriptions.It avoids predicting object-region coordinates and does not require cached region-level features.
  • Online Evaluation: 136.0% CIDEr (c5) and 138.3% CIDEr (c40) are achieved on the official online MSCOCO test server.These scores improve over the best performer DLCT by 2.7% and 2.9%, respectively.
  • Online Evaluation: PureT obtains the best online scores in every reported metric except BLEU-1 (c40), where it is 0.1% below DLCT.The online evaluation reports results using both five and forty reference captions.

Ablation Study

The ablations evaluate attention-window settings, pre-fusion, model depth, backbones, and refining encoders. Results favor SwinTransformer grid features, pre-fusion, three blocks, and the proposed refining encoder.

  • Attention windows: Global MSA outperforms window-only W-MSA because W-MSA lacks connections across windows.The comparison uses window size ws and shift size ss configurations with three refining-encoder and decoder blocks.
  • Pre-Fusion: Pre-Fusion improves performance in all metrics for both PureT and the standard Transformer.Adding Pre-Fusion to the standard Transformer also improves every reported metric.
  • Model depth: More than 2.0% CIDEr improvement is obtained with models using more than 2 blocks rather than 1 block.The authors select 3 blocks because 4 blocks increase parameters while 3 blocks already perform sufficiently well.
  • Backbone: SwinTransformer grid features significantly outperform ResNet-101 region features and ResNeXt-101 or ViT grid features.For M2 Transformer, SwinTransformer increases CIDEr by more than 3.7%, while also improving all metrics.
  • Backbone: CNN backbones suit LSTM or non-standard-MSA Transformer decoders, whereas SwinTransformer suits standard-MSA Transformer decoders including PureT.This pattern motivates exploring lighter and simpler Transformer-based models.
  • Refining Encoder: The proposed Refining Encoder provides the maximum gain and best performance compared with deleting it or replacing it with M2 or X-Transformer encoders.Deleting the module still performs better than most existing SOTAs, but adding a refining encoder yields significant gains.

Visualization Analysis

Qualitative examples compare captions from standard Transformer, M2 Transformer, and PureT, while attention heatmaps show where PureT attends during generation. PureT captures fine-grained details and focuses on semantically relevant image regions.

  • Caption examples: PureT generates more accurate and descriptive captions by capturing additional fine-grained information.The examples compare PureT with standard Transformer, M2 Transformer, and ground-truth captions.
  • Attention heatmaps: PureT attends to corresponding object regions for nouns and nearby zebra-head areas when generating “grazing.”The heatmaps show attention shifting to the bodies of zebras, rainbow, field, and sky regions for corresponding words.

Conclusion

The paper proposes an end-to-end pure Transformer image-captioning model using SwinTransformer, mutual refinement of local and global visual features, and decoder pre-fusion. Experiments on MSCOCO report new state-of-the-art performance.

  • Conclusion: PureT uses SwinTransformer for end-to-end image-to-description training without object-detector pretraining.A refining encoder processes grid and global features, while the decoder fuses refined global features with generated words.
  • Conclusion: The refining encoder combines local and global attention, while decoder fusion strengthens interaction between visual and language features.The authors describe these components as improving modeling capability.
  • Conclusion: Experiments on MSCOCO demonstrate new state-of-the-art performance for the proposed model.
Loading 2203.15350v1…