Source-linked AI summary
CPTR: Full Transformer Network for Image Captioning
Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, Jing Liu
TL;DR
Image captioning commonly combines a CNN-based image encoder with a Transformer decoder, while CPTR instead feeds sequentialized raw-image patches into a full, convolution-free Transformer. The model can model global context from the beginning of every encoder layer and surpasses conventional CNN+Transformer methods on MSCOCO.
Problem
Most captioning methods retain CNN or object-detector encoders even when using Transformer decoders, limiting the architecture's ability to model global context from the beginning.
Method
CPTR replaces the CNN encoder with a convolution-free Transformer that embeds sequentialized raw image patches and uses Transformer decoding for caption generation.
Results
CPTR surpasses conventional CNN+Transformer captioning models on the MSCOCO dataset.
Takeaways & Limitations
The full Transformer architecture models global context at every encoder layer from the beginning and supports visualizations of patch self-attention and words-to-patches attention.
Abstract
from arXiv · showhide
In this paper, we consider the image captioning task from a new sequence-to-sequence prediction perspective and propose CaPtion TransformeR (CPTR) which takes the sequentialized raw images as the input to Transformer. Compared to the "CNN+Transformer" design paradigm, our model can model global context at every encoder layer from the beginning and is totally convolution-free. Extensive experiments demonstrate the effectiveness of the proposed model and we surpass the conventional "CNN+Transformer" methods on the MSCOCO dataset. Besides, we provide detailed visualizations of the self-attention between patches in the encoder and the "words-to-patches" attention in the decoder thanks to the full Transformer architecture.
1. INTRODUCTION
CPTR reframes image captioning as sequence-to-sequence prediction with a totally convolution-free Transformer encoder that directly processes sequentialized raw image patches. It is designed to model global context from the beginning and outperforms conventional CNN-based captioning models on MSCOCO.
- Motivation: Recent captioning systems commonly retain CNN or object-detector encoders while replacing recurrent decoders with Transformers.These encoders extract spatial or bottom-up features before the decoder predicts words through attention.
- CPTR: CPTR replaces the conventional CNN encoder with a totally convolution-free Transformer encoder for image captioning.It divides raw images into fixed-size patches, flattens them, and feeds the resulting sequence through patch and positional embeddings.
- CPTR: CPTR directly sequentializes raw images into patch sequences instead of consuming features extracted by CNNs or object detectors.This design treats image captioning as a sequence-to-sequence prediction task.
- Design rationale: Self-attention lets CPTR model long-range dependencies among image patches from the first encoder layers, unlike convolutional encoders that enlarge receptive fields gradually.The decoder also models words-to-patches attention during caption generation.
- Results: On MSCOCO, CPTR outperforms both CNN+RNN and CNN+Transformer captioning models.The paper presents this as evidence for the effectiveness of the full Transformer architecture.
2. FRAMEWORK
The CPTR framework converts resized images into embedded patch sequences for a Transformer encoder, then uses a Transformer decoder to generate captions. Its encoder uses stacked self-attention and feed-forward layers, while the decoder combines masked self-attention, cross-attention, and vocabulary prediction.
- Encoder: CPTR resizes each image, divides it into N patches, flattens them, and maps the patch sequence into a latent space with positional embeddings.The experimental patch size is P = 16.
- Encoder: The encoder stacks identical layers containing multi-head self-attention followed by a positional feed-forward sublayer.Multiple attention heads attend to different subspaces before their outputs are linearly aggregated.
- Decoder: The decoder stacks masked self-attention, multi-head cross-attention, and feed-forward sublayers to process word embeddings and encoder features.Sinusoidal positional embeddings are added to word embeddings before decoding.
- Decoder: A linear output layer maps the final decoder representation to the vocabulary to predict the next word.Training minimizes cross-entropy loss over the ground-truth caption sequence.
- Training: The model is additionally fine-tuned with self-critical training, consistent with other captioning methods.This training stage follows the cross-entropy objective.
3. EXPERIMENTS
Experiments evaluate CPTR on MS COCO using standard splits and captioning metrics, compare it with CNN-based baselines, examine training and architecture choices, and visualize attention behavior.
- 3.1. Dataset and Implementation Details: CPTR is evaluated on MS COCO using the Karpathy splits, with 113,287 training, 5,000 validation, and 5,000 test images.Results are reported on both the offline Karpathy test split and the online MS COCO test server.
- 3.2. Performance Comparison: The study compares CPTR with CNN+RNN and CNN+Transformer models using BLEU, METEOR, ROUGE, and CIDEr metrics.The compared methods use CNN- or object-detector-derived image features, whereas CPTR directly uses raw images.
- 3.2. Performance Comparison: 129.4 CIDEr is achieved by CPTR on the offline Karpathy test split, surpassing both CNN+RNN and CNN+Transformer models.The online COCO test-server results also demonstrate CPTR's effectiveness.
- 3.3. Ablation Study: Using ImageNet-21K-pretrained ViT parameters improves performance over training from scratch, while ImageNet 2012 fine-tuning adds one CIDEr point.The best decoder setting is 4 layers with 768 dimensions, producing 111.6 CIDEr in the ablation study.
- 3.3. Ablation Study: Increasing image resolution from 224 × 224 to 384 × 384 raises CIDEr from 111.6 to 116.5 while increasing the patch sequence from 196 to 576.The authors relate this gain to finer image division and more patch interactions through encoder self-attention.
- 3.4. Attention Visualization: Attention visualizations show local and global contexts in shallow encoder layers, primary-object focus in middle layers, and broader scene coverage in the last layer.Decoder visualizations show appropriate image patches attended for each generated word.
4. CONCLUSIONS
The paper concludes that CPTR reframes image captioning as sequence-to-sequence prediction with a convolution-free full Transformer. It reports stronger performance than conventional CNN+Transformer networks and attention that aligns words with visual patches.
- 4. CONCLUSIONS: CPTR replaces the conventional CNN+Transformer procedure with a full Transformer model that directly processes sequentialized raw images.The model is totally convolution-free.
- 4. CONCLUSIONS: CPTR models global context at every encoder layer from the beginning, unlike the progressively expanding context of conventional convolutional encoders.The conclusion attributes this capability to the full Transformer architecture.
- 4. CONCLUSIONS: On MS COCO, CPTR surpasses conventional CNN+Transformer networks.The paper reports this as evidence of the method's effectiveness.
- 4. CONCLUSIONS: Visualizations show long-range patch dependencies from the beginning and decoder words-to-patches attention attending to corresponding visual patches.These visualizations provide interpretive evidence for the model's encoder and decoder behavior.