PaperlayerView exact revision on arXiv

Source-linked AI summary

SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation

Rita Ramos, Bruno Martins, Desmond Elliott, Yova Kementchedjhieva

arXiv:2209.15323v2cs.CVcs.CL

TL;DR

Image captioning increasingly relies on costly large-scale training and finetuning. SmallCap instead combines frozen CLIP and GPT-2 components with retrieved captions, achieving competitive in-domain performance and training-free transfer across domains, with further gains from diverse retrieved data.

  • Problem

    Increasingly large image-captioning models make pre-training, finetuning, and adapting separate versions for different domains costly.

  • Method

    SmallCap freezes CLIP and GPT-2, trains connecting cross-attention layers, and conditions generation on captions retrieved from a replaceable datastore.

  • Results

    SmallCap is competitive on COCO, transfers across domains without retraining, and benefits from diverse web and human-labeled retrieval data.

  • Takeaways & Limitations

    Retrieval augmentation can provide non-parametric information as an alternative to expensive training and finetuning for multimodal captioning.

  • Takeaways & Limitations

    The study reports that only the experiments from Table 1 were repeated because of the academic computing budget.

Abstract

from arXiv · show

Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pre-training and finetuning. As an alternative to large models, we present SmallCap, which generates a caption conditioned on an input image and related captions retrieved from a datastore. Our model is lightweight and fast to train, as the only learned parameters are in newly introduced cross-attention layers between a pre-trained CLIP encoder and GPT-2 decoder. SmallCap can transfer to new domains without additional finetuning and can exploit large-scale data in a training-free fashion since the contents of the datastore can be readily replaced. Our experiments show that SmallCap, trained only on COCO, has competitive performance on this benchmark, and also transfers to other domains without retraining, solely through retrieval from target-domain data. Further improvement is achieved through the training-free exploitation of diverse human-labeled and web data, which proves to be effective for a range of domains, including the nocaps benchmark, designed to test generalization to unseen visual concepts.

1. Introduction

SMALLCAP addresses the cost of increasingly large image-captioning models with retrieval-augmented captioning. It uses frozen pretrained components and retrieved captions to support lightweight training, domain transfer without retraining, and training-free use of additional data.

  • Large-scale image-captioning models increase pre-training and finetuning costs, especially when separate versions are needed for different visual domains.
  • Earlier lightweight models freeze pretrained vision and language components but still require separate training for every use case.These approaches reduce trainable parameters to approximately 43M and accelerate training.
  • SMALLCAP conditions caption generation on an input image and captions retrieved from an external text datastore.
  • SMALLCAP freezes CLIP and GPT-2, trains only cross-attention layers, and uses 7 million trainable parameters while remaining competitive on COCO.This is an 83% reduction in trainable parameters relative to the referenced lightweight models.
  • Replacing the datastore lets SMALLCAP access new-domain styles and concepts or broader knowledge without additional training.The paper reports benefits on VizWiz and nocaps from retrieved target-domain, human-labeled, and web data.
  • SMALLCAP matches lightweight-training models in-domain and outperforms them by a large margin out-of-domain.

2. Related Work

Image captioning commonly relies on encoder-decoder systems and large pretrained vision-language models, while lightweight and retrieval-augmented alternatives reduce training demands. SMALLCAP combines frozen pretrained components with retrieved task demonstrations to support captioning and transfer.

  • Current captioning systems pass an image through a visual encoder and generate text with an autoregressive language decoder.
  • Large vision-language models pretrain on extensive image-text data and require separate downstream models for different captioning datasets.
  • Lightweight methods freeze pretrained components to reduce gradient updates, GPU memory use, and training cost.
  • Retrieval-augmented generation conditions output on information retrieved from an external datastore, a strategy still relatively unexplored in image captioning.
  • SMALLCAP uses image-specific prompts containing retrieved captions as task demonstrations for a pretrained language model.

3. Proposed Approach

SMALLCAP connects frozen CLIP and GPT-2 components through trainable cross-attention and prompts decoding with captions retrieved for the input image. The datastore can be changed at inference time to add knowledge or transfer domains without retraining.

  • 3.1. Model: The model generates captions from encoded image features together with k retrieved captions supplied as a decoder prompt.
  • 3.1. Model: SMALLCAP freezes CLIP and GPT-2 and trains only cross-attention layers connecting their different vector spaces.The projection dimensionality controls the number of trainable parameters.
  • 3.2. Prompting with Retrieved Captions: Image-to-text retrieval selects datastore texts whose CLIP representations are nearest to the input image under cosine similarity.
  • 3.2. Prompting with Retrieved Captions: The fixed prompt places retrieved captions before a cue for the target image, and the decoder generates a caption conditioned on image features and the demonstration.
  • 3.2. Prompting with Retrieved Captions: Training minimizes autoregressive cross-entropy over the target caption using the image features and retrieved-caption prompt.
  • 3.2. Prompting with Retrieved Captions: The datastore may be expanded or replaced at inference time, enabling added knowledge and domain transfer without retraining.

4. Main Experiments

Experiments evaluate lightweight training and retrieval-based generalization on COCO and nocaps. SMALLCAP remains competitive in-domain and shows especially strong out-of-domain performance, further improving when diverse web and human-labeled data are retrieved.

  • Experimental Setup: SMALLCAP uses frozen CLIP-ViT-B/32 and GPT-2Base components, with cross-attention variants containing 7M, 3.6M, or 1.8M trainable parameters.
  • Experimental Setup: Training uses COCO captions with retrieved prompts, while inference uses beam search after image-to-text retrieval.The reported setup retrieves k = 4 captions during training.
  • Experimental Setup: Evaluation reports BLEU-4, METEOR, CIDEr, and SPICE on COCO and nocaps.
  • COCO: On COCO, 7M-parameter SMALLCAP performs better or on par with ClipCap and I-Tuning, while 1.8M-parameter SMALLCAP remains better than those larger models.
  • nocaps: On nocaps, SMALLCAP clearly outperforms other lightweight methods out-of-domain and remains competitive in-domain and near-domain.
  • nocaps: Adding diverse web and human-labeled data to the datastore improves nocaps generalization, with SMALLCAP+W+H outperforming OSCARLarge by over 10 points out-of-domain.

5. Training-Free Use of Data

SMALLCAP transfers across domains without finetuning by replacing or augmenting its datastore at inference time. In-domain, web, and human-labeled data improve performance, with benefits varying by dataset and task.

  • 5.1. In-domain Data: Replacing COCO captions with in-domain data substantially improves validation performance on Flickr30k, VizWiz, and MSR-VTT without finetuning.The model adapts to retrieved information, with the largest improvement reported for VizWiz.
  • 5.2. Augmenting the Datastore: Web data improves all three datasets, with larger gains on Flickr30k and MSR-VTT than using in-domain data alone.VizWiz shows comparatively small improvement from web augmentation.
  • 5.2. Augmenting the Datastore: Human-labeled data improves Flickr30k and MSR-VTT over in-domain data alone but not VizWiz.The datastore includes text from image, video, audio, and localized-narrative captioning sources.
  • 5.2. Augmenting the Datastore: Combining web and human-labeled data with in-domain data is not beneficial for any of the three datasets.The paper evaluates the combined In-domain + Web + Human-labeled configuration against the separate augmentations.
  • 5.3. Domain-agnostic Data: Without in-domain data, web retrieval remains close to or better than in-domain performance, while human-labeled data benefits MSR-VTT most.The optimal MSR-VTT configuration is Web + Human-labeled.
  • 5.4. Results with the Best Configuration: SMALLCAP outperforms ClipCap and CaMEL without additional training, improving over CaMEL by 5.4 CIDEr points on Flickr30k, 17.4 on VizWiz, and 7.7 on MSR-VTT.The best datastore uses In-domain + Web for Flickr30k and VizWiz, and In-domain + Human-labeled for MSR-VTT.

6. Discussion

SMALLCAP uses retrieved captions to incorporate domain-specific and unseen concepts while retaining visual grounding. Retrieval improves performance across model sizes and supports benefits from diverse caption sources, with a moderate dependence on retrieval quality and decoder choice.

  • 6.1. Qualitative Examples: Retrieved captions help SMALLCAP generate domain-specific concepts such as tutu, Swanson, and Pokemon that were absent or rare in COCO training data.Tutu and Swanson were absent from COCO, while Pokemon appeared only six times.
  • 6.2. Analysis of the Retrieved Captions: Most retrieved text comes from web data for unseen concepts, while retrieval also favors corresponding or similar domains.VizWiz has the highest rate of in-domain retrieval, whereas MSR-VTT also retrieves from other video datasets.
  • 6.2. Analysis of the Retrieved Captions: SMALLCAP benefits from human-labeled text beyond image captions, including video captions that help Flickr30k, VizWiz, and MSR-VTT.Table 5 evaluates image captions, video captions, localized narratives, and audio captions as datastore sources.
  • 6.3. The Impact of Retrieval: Reducing trainable parameters from 28M to 7M causes only a 0.6 CIDEr-point drop with retrieval augmentation.Retrieval-augmented performance remains stable across the tested model sizes.
  • 6.3. The Impact of Retrieval: Without retrieval, the performance gap between the largest and smallest models is 4.3 CIDEr points, compared with 1.7 points with retrieval.Ablation results show both lower performance and faster degradation without retrieved captions.
  • 6.3. The Impact of Retrieval: A blank-image ablation yields a CIDEr score of 90.1, indicating that SMALLCAP uses visual input rather than only paraphrasing retrieved captions.The ablation sets visual encoder features to zero.
  • 6.4. Decoder Analysis: SMALLCAP also performs well with OPT-125M and OPT-350M decoders, supporting decoder agnosticism.These decoders are presented as equivalents in size to GPT2-Base and GPT2-Medium.

7. Conclusion

SMALLCAP is a retrieval-augmented captioning model designed to reduce trainable parameters while enabling domain transfer without retraining. Its cross-attention architecture connects frozen CLIP and GPT-2 components, while retrieval supplies non-parametric information and supports broader data use.

  • SMALLCAP transfers across domains without retraining by conditioning caption generation on retrieved target-domain data.
  • SMALLCAP leverages non-parametric datastore information instead of storing all captioning knowledge in trainable parameters.
  • Retrieval augmentation enables training-free use of target-domain, web, and human-labeled data across multimodal captioning settings.
  • The model uses frozen CLIP and GPT-2 components connected through newly added cross-attention layers.The attention mechanism processes queries, keys, and values through multi-head cross-attention.

B. Design Choices and Hyperparameters

The design study tunes retrieval and model configuration for competitive captioning with small trainable components. It identifies four retrieved captions as optimal and shows that larger decoders can be used by controlling cross-attention dimensionality.

  • B.3. Number of Retrieved Captions: k = 4 is the optimal number of retrieved captions in COCO validation tuning.Retrieving more captions can improve robustness against incorrect information in individual retrieved captions.
  • Prompting: The prompt template has little impact on model performance across tested separators and formatting choices.The final template places four retrieved captions between a prompt prefix and the caption-generation continuation.
  • D. nocaps: SMALLCAP+W+H outperforms other lightweight-training models and even OSCAR on nocaps validation, especially out-of-domain.
  • Decoder scaling: Larger GPT-Medium and GPT-Large decoders are beneficial across cross-attention dimensionalities d = 4, 8, and 16.Controlling cross-attention dimensionality allows larger decoders without a massive increase in trainable parameters.

E.2. OPT decoders

OPT-decoder experiments compare models with and without retrieval to assess retrieval's role in SMALLCAP. Removing retrieval causes a large performance drop, indicating that retrieval remains central with this alternative decoder.

  • Removing retrieval causes a large performance drop for SMALLCAP variants using the OPT decoder.The same retrieval dependence was observed for SMALLCAP using GPT-2.
  • Table 9 reports COCO validation CIDEr results for OPT-based models with and without retrieval.

F. Data

The paper evaluates inference efficiency and retrieval-only captioning performance alongside datastore data choices. SMALLCAP is faster than CaMEL in the reported GPU measurement, while retrieval substantially improves caption quality over retrieval alone.

  • Datastore data: The datastore uses cleaner web data with synthetic texts and human-labeled resources beyond image-captioning datasets.
  • Inference time: 0.22 seconds per image versus 0.58 seconds makes SMALLCAP faster than CaMEL on 1,000 sampled COCO images.The measurements were taken on an NVIDIA A100 GPU; the paper attributes the difference likely to CaMEL’s dual decoder architecture.
  • Retrieval-only comparison: 117.3 CIDEr for SMALLCAP versus 65.5 for retrieval alone on the COCO validation set.

H. More Qualitative Examples

Qualitative examples show that retrieval helps SMALLCAP describe COCO images more accurately and adapt captions to new domains. In-domain retrieval also reduces bias toward frequent concepts.

  • With retrieval, SMALLCAP better describes COCO images; without it, a brush is mistaken for a cell phone.
  • Figure 8 compares captions generated with and without retrieval augmentation on COCO.Retrieved captions are marked with •, generated captions with ▶, and captions from the no-retrieval model with ø.
  • Replacing the datastore with in-domain data lets SMALLCAP adapt captions to Flickr30k, VizWiz, and MSR-VTT.
  • In-domain retrieval makes SMALLCAP less biased toward frequent concepts, favoring camel over horse, diet coke over soda, and nail polish over brush.
Loading 2209.15323v2…