VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

Jun Chen; Han Guo; Kai Yi; Boyang Li; Mohamed Elhoseiny

Source-linked AI summary

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

Jun Chen, Han Guo, Kai Yi, Boyang Li, Mohamed Elhoseiny

arXiv:2102.10407v5Feb 20, 2021cs.CVcs.AIcs.CLcs.MM

TL;DR

Image captioning often requires large paired datasets, motivating methods that learn effectively from small in-domain samples. VisualGPT adapts a pretrained language-model decoder with self-resurrecting encoder-decoder attention, outperforming strong baselines on low-data captioning and achieving state-of-the-art on IU X-ray.

Problem
Image captioning needs costly or scarce paired data, while adapting unimodal pretrained language models to multimodal captioning remains under-investigated.
Method
VisualGPT initializes the caption decoder from a pretrained language model and uses self-resurrecting activation units to balance visual and linguistic inputs.
Results
VisualGPT outperforms strong baselines on 0.1%, 0.5%, and 1% of MS COCO and Conceptual Captions data and achieves state-of-the-art on IU X-ray.
Takeaways & Limitations
The work demonstrates efficient adaptation of linguistic knowledge from textual pretraining to image captioning and medical report generation with limited multimodal data.
Takeaways & Limitations
The gap between VisualGPT and baseline models gradually vanishes as in-domain training data increase, especially on COCO.

Abstract

from arXiv · show

The ability to quickly learn from a small quantity oftraining data widens the range of machine learning applications. In this paper, we propose a data-efficient image captioning model, VisualGPT, which leverages the linguistic knowledge from a large pretrained language model(LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder ona small amount of in-domain training data. The proposed self-resurrecting activation unit produces sparse activations but has reduced susceptibility to zero gradients. We train the proposed model, VisualGPT, on 0.1%, 0.5% and 1% of MSCOCO and Conceptual Captions training data. Under these conditions, we outperform the best baseline model by up to 10.8% CIDEr on MS COCO and upto 5.4% CIDEr on Conceptual Captions. Further, Visual-GPT achieves the state-of-the-art result on IU X-ray, a medical report generation dataset. To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data. Our code is available at: https://github.com/Vision-CAIR/VisualGPT.

1. Introduction

VisualGPT targets image captioning with very limited in-domain data by adapting pretrained language models while balancing linguistic priors against visual information. Its self-resurrecting attention connects modalities, and experiments report stronger performance on small-data benchmarks.

Motivation: Image captioning data are costly to annotate, while web-collected pairs can remain incorrect and specialized-domain data are scarce.These constraints motivate improving data efficiency for captioning systems.
Problem: The paper studies captioning performance when models train on a small subset of in-domain data, distinct from novel object captioning.The target is data efficiency within the domain rather than zero-shot coverage of unseen objects.
Problem: MS COCO uses 75% more nouns and 14% fewer verbs than WikiText-2, revealing a domain gap between captioning and pretrained text.This motivates balancing pretrained linguistic knowledge with visual input information.
Method: VisualGPT initializes its decoder from pretrained language models and uses self-resurrecting activation units to balance visual and linguistic inputs.The gates produce sparse activations while remaining less vulnerable to zero gradients than regular gates.
Results: On 0.1%, 0.5%, and 1% of MS COCO and Conceptual Captions data, VisualGPT outperforms several strong baselines and reaches state-of-the-art on IU X-ray.Ablations assess the contributions of pretrained language models and self-resurrecting attention.
Contribution: The work explores quickly adapting pretrained language models as caption decoders using only a small amount of in-domain training data.It presents this as an efficient adaptation approach for a cross-modal task.

2. Related Work

Prior image-captioning research developed encoder, decoder, and attention improvements, while related formulations addressed novel objects, unpaired supervision, and auxiliary data. VisualGPT instead adapts language models pretrained on textual data using limited multimodal data.

Image Captioning: Image captioning progressed from template filling to end-to-end systems that encode images and decode captions word by word.Subsequent work improved image encoders, decoders, and attention mechanisms.
Alternative Formulations: Novel object captioning targets objects absent from training data, while other work studies unsupervised captioning and auxiliary unpaired image-caption data.These formulations differ from VisualGPT’s small in-domain supervised-data setting.
Self-supervised NLP Models: Self-supervised NLP models use autoregressive or masked language-modeling objectives, including GPT and BERT-related methods.These models provide the pretrained linguistic knowledge used by VisualGPT.
VisualGPT’s Position: Unlike multimodal-pretrained networks, the proposed adaptation requires only a small amount of multimodal training data and transfers knowledge learned from text.The method focuses on adapting textual linguistic knowledge to a multimodal task.

3. Preliminaries: Transformer for Captioning

Transformer captioning uses multi-head dot-product attention to combine decoder queries with encoder keys and values. In visual captioning, encoder-decoder attention queries visual features using the current decoder state, followed by residual normalization.

Transformer Attention: Transformer attention takes query Q, key K, and value V matrices and computes weighted representations using trainable projections and scaled dot products.Multi-head attention repeats this process across multiple projection sets and concatenates the results.
Visual Captioning: The visual encoder represents an image as I ∈ R^O×S, where O is the object-sequence length and S is the hidden dimension.The decoder then generates caption words sequentially.
Encoder-Decoder Attention: At decoding step t+1, encoder-decoder attention uses the current decoder state H as query and visual encoding I as both key and value.This is expressed as EncDecAttn(H, I) = Attn(H, I, I).
Encoder-Decoder Attention: The AddNorm operator adds the attention output to H and applies layer normalization.This combines a residual connection with normalization before subsequent decoder processing.
Attention Variants: AoA combines visual encoding I with decoder language information H, whereas M2 Transformer connects all decoder layers to all encoder layers.Figure 3 contrasts these architectures with VisualGPT.

4. VisualGPT

VisualGPT uses a pretrained language model as a caption decoder and injects visual information through encoder-decoder attention. Its self-resurrecting activation units balance visual and linguistic signals while producing sparse activations that can escape zero outputs.

Architecture: VisualGPT uses a pretrained language model as the caption decoder and feeds visual information through encoder-decoder attention.The decoder is initialized from a pretrained language model, while the encoder-decoder module is randomly initialized.
Architecture: Complementary gates combine visual encoder-decoder attention with linguistic decoder representations.The module outputs Bvis ⊗ EncDecAttn(H, I) + Blan ⊗ H, using element-wise multiplication.
Self-Resurrecting Activation Unit: SRAU applies thresholded sigmoid gates that introduce sparsity while preserving a path for adapting pretrained linguistic knowledge.Values below threshold τ are set directly to zero, preventing gradients through those gates from disrupting pretrained weights.
Self-Resurrecting Activation Unit: The asymmetric SRAU gates can escape zero outputs because the nonzero gate remains available to change the zero-outputting gate during optimization.This self-resurrecting behavior distinguishes SRAU from symmetric normalized gates.
Ablation: SRAU outperforms both ordinary complementary gates and normalized SRAU in ablation experiments.Figure 4 visualizes the three gate designs, with inputs on the x-axis and outputs on the y-axis.

5. Experiments

Experiments evaluate VisualGPT on standard and specialized datasets under limited-data settings, comparing pretrained-decoder and attention designs against established baselines. Results show gains in low-data captioning, medical report generation, human preference, object coverage, and visual-language integration, while the advantage decreases with more in-domain data.

Datasets and Evaluation Metrics: VisualGPT is evaluated on MS COCO, Conceptual Captions, and IU X-ray, with IU X-ray providing a specialized medical report-generation setting.MS COCO has 123,287 images with five captions each; IU X-ray has 2,770 training reports.
Quantitative Results: VisualGPT outperforms the best baseline by 4.1, 6.4, and 2.5 CIDEr on MS COCO using 0.1%, 0.5%, and 1% training data.On Conceptual Captions, the corresponding gains are 4.2, 3.5, and 0.3 CIDEr.
Quantitative Results: VisualGPT achieves a new state-of-the-art on IU X-ray and outperforms most medical report-generation baselines across evaluation metrics.The dataset contains only around 2,770 medical reports in training, making it a highly data-constrained domain.
Quantitative Results: 20.6 CIDEr is VisualGPT’s improvement over Kim et al. without additional unpaired images and captions under Kim et al.’s 1%-image split.The comparison differs from Table 1 because Kim et al. use 1% of images rather than 1% of image-caption pairs.
Ablation Studies: SRAU outperforms alternative cross-attention modules, while replacing it with Normalized SRAU lowers CIDEr across all tested MS COCO and Conceptual Captions setups.The decreases from full VisualGPT are 2.7, 1.0, and 0.3 on MS COCO, and 2.2, 1.3, and 0.6 on Conceptual Captions.
Analysis: VisualGPT receives the largest share of human votes across the 0.1%, 0.5%, and 1% training splits, and has less hallucination with higher object coverage than baselines.The vote shares are 39.2%, 39.1%, and 37.4%, with statistically significant differences at p < 0.05.

6. Conclusions

VisualGPT is a data-efficient image captioning model that leverages pretrained language-model knowledge to address low-resource and specialized-domain settings. It achieves state-of-the-art performance on IU X-ray and is intended for situations where large annotated datasets are difficult to obtain.

VisualGPT leverages linguistic knowledge from a pretrained language model for data-efficient image captioning.
VisualGPT achieves the state-of-the-art result on IU X-ray, a small medical imaging report dataset.
The model may help address low-resource languages and highly specialized domains where collecting large annotated datasets is difficult.

A.1. Additional implementation details

VisualGPT is trained with autoregressive cross-entropy followed by reinforcement learning, using visual features, subword tokenization, and a beam-search-based training procedure. The cross-entropy objective predicts target caption tokens autoregressively.

Image and Word Features: Image features come from Faster R-CNN with a ResNet-101 backbone, while captions use Byte Pair Encoding and learnable positional encoding.
Training Details: Models are first trained with cross-entropy loss and then fine-tuned using reinforcement learning.
Training Details: The cross-entropy loss is the traditional autoregressive classification loss for predicting caption words.
Training Details: Reinforcement learning samples L sentences with beam search and uses their mean reward as the baseline.
Training Details: The reinforcement-learning gradient weights sampled-caption log probabilities by the difference between each CIDEr-D reward and the baseline.

A.2. Train VisualGPT with more COCO and Conceptual Caption Datasets

With more training data, VisualGPT remains strongest in low-data regimes but no longer dominates MS COCO at the largest scales. Its advantage persists across the tested Conceptual Captions sampling range, while dataset vocabulary and content diversity may explain differing behavior.

MS COCO: VisualGPT outperforms other baselines on MS COCO when sampling ≤20% of the training data.The additional evaluations use 5%, 10%, 20%, 50%, and 100% of MS COCO data.
Conceptual Captions: VisualGPT consistently outperforms all baselines on Conceptual Captions when sampling ≤2.5% of training images.The Conceptual Captions scaling experiment extends to 2.5% of the dataset.
The experiments highlight VisualGPT’s effectiveness in low-data regimes.
MS COCO: M2 Transformer surpasses VisualGPT when using 50% and 100% of COCO training data.
The authors hypothesize that GPT knowledge helps more when captions have limited word coverage, whereas abundant in-domain data may favor existing captioning models.COCO has less vocabulary diversity than Conceptual Captions, which the authors associate with their different performance patterns.

A.3. Attention over Different types of words

VisualGPT assigns higher visual attention to content-oriented parts of speech and lower attention to more linguistic parts of speech. Nouns, verbs, and adjectives receive the highest reported attention scores among the listed categories.

Nouns, verbs, and adjectives receive high mean visual attention scores of 0.71, 0.71, and 0.72, respectively.
Pronouns, punctuation, and determiners receive lower mean visual attention scores of 0.53, 0.58, and 0.61, respectively.
Figure 10 presents attention scores across different parts of speech.

A.4. More Qualitative Examples

VisualGPT’s qualitative examples show attention concentrated on image-relevant content words while remaining low for function words. Across MS COCO examples using 0.1%, 0.5%, and 1% data splits, its captions are described as more accurate than those from several baseline models.

VisualGPT assigns higher visual attention to image-relevant words such as “steam engine,” “elephants,” “horse,” “lush,” and “cabinets.”Figure 11 visualizes high scores in blue and low scores in red.
VisualGPT assigns lower visual attention to determiners and prepositions such as “to” and “at.”
VisualGPT captions are described as more accurately depicting image content than captions from Transformer, M2 Transformer, and AoA Transformer baselines.
Qualitative caption comparisons cover MS COCO training-data splits of 0.1%, 0.5%, and 1%.The comparisons are presented in Tables 6, 7, and 8, respectively.