Source-linked AI summary
Retrieval-Augmented Transformer for Image Captioning
Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
TL;DR
Accurately describing image concepts remains difficult, while scaling models to improve memorization raises parameter and training costs. The paper uses visually retrieved captions in a retrieval-augmented Transformer, and reports improved COCO caption quality, including a significant margin over a RETRO adaptation.
Problem
Accurately describing image concepts remains challenging, while scaling models to improve memorization increases learnable parameters and training costs.
Source: §1 · p. 1Method
A visual-similarity kNN retriever supplies external captions, which a Transformer attends alongside the image and past token context.
Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 2Results
On COCO, external memory significantly improves generation quality, and the proposed architecture significantly outperforms the RETRO adaptation.
Source: §1 · p. 1Takeaways & Limitations
The COCO results support retrieval abilities as a promising direction for image captioning and further research at larger scale.
Source: §5 · p. 6Retrieved captions can be noisy and unreliable even for small k, so the language model must selectively use content coherent with the image.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
Image captioning models aim at connecting Vision and Language by providing natural language descriptions of input images. In the past few years, the task has been tackled by learning parametric models and proposing visual feature extraction advancements or by modeling better multi-modal connections. In this paper, we investigate the development of an image captioning approach with a kNN memory, with which knowledge can be retrieved from an external corpus to aid the generation process. Our architecture combines a knowledge retriever based on visual similarities, a differentiable encoder, and a kNN-augmented attention layer to predict tokens based on the past context and on text retrieved from the external memory. Experimental results, conducted on the COCO dataset, demonstrate that employing an explicit external memory can aid the generation process and increase caption quality. Our work opens up new avenues for improving image captioning models at larger scale.
1 INTRODUCTION
Accurately describing image concepts remains challenging, while scaling models to improve memorization increases learnable parameters and training cost. The paper proposes external-memory retrieval for captioning and reports improved generation quality on COCO.
1 INTRODUCTION
- Scaling a model to improve memorization increases its learnable parameters and, ultimately, its training cost.The paper frames accurate description of image concepts as an ongoing challenge.Source: §1 · p. 1
- External memory can relieve language models of memorization requirements and provide access to larger-scale data through approximate nearest-neighbor search.The introduction presents this as an alternative to relying solely on model activations.Source: §1 · p. 1
- The proposed retrieval-augmented captioner combines approximate kNN visual retrieval with attention to past context and retrieved text, which the authors describe as the first such integration in image captioning.The model uses an external memory to provide language-model hints.Source: §1 · p. 1
- On COCO, the authors report that external memory significantly improves generation quality and that their architecture significantly outperforms the RETRO adaptation.The comparison also includes a fully attentive baseline without external memory.Source: §1 · p. 1
2 RELATED WORK
Image captioning research has progressed from CNN–RNN systems toward attentive and Transformer-based architectures, with gains also associated with more data and large-scale training. This paper situates retrieval as a way to connect captioning models to external text and memory.
Image Captioning
- Earlier captioners used CNN encoders and RNN language models, while later systems adopted attention and Transformers for visual encoding and language modeling.The survey also notes large-scale multimodal models as a source of image features.Source: §2 · pp. 1–2
- Recent captioning performance gains have also come from increasing training data, model size, and large-scale training.Source: §2 · p. 2
- The paper follows Transformer-based captioning work while adding retrieval abilities to a fully attentive architecture.Prior work includes self-attention variants and vision-language early-fusion approaches.Source: §2 · p. 2
Retrieval-Augmented Approaches
- The approach retrieves captions for visually similar images, giving the model access to the training set and, in principle, data beyond training.Related retrieval-augmented language models connect retrieved text through specialized attention; this paper uses external memory rather than internal activations and a single scalar gate rather than a per-head learned parameter.Source: §2 · p. 2
3 PROPOSED METHOD
The method separates caption generation into visual retrieval and token prediction conditioned on the image and retrieved descriptions. A Transformer decoder combines attention to prior tokens with attention to retrieved captions, and the model is trained with cross-entropy before reinforcement-learning fine-tuning.
3 PROPOSED METHOD
- Caption generation is decomposed into retrieving descriptions related to an image and predicting captions conditioned on both the image and retrieved descriptions.The formulation models the caption distribution conditioned on the image and marginalizes over retrieved captions.Source: §3 · p. 2
3.1 Knowledge Retriever
- The retriever uses approximate kNN search over image embeddings, ranks corpus images by inner-product similarity, and returns their associated captions.The visual encoder uses CLIP; its visual-semantic space allows image-based retrieval without forwarding a textual encoder during search.Source: §3.1 · p. 2
3.2 Retrieval-Augmented Language Model
- A vanilla encoder-decoder Transformer processes image features in its encoder and token context in its decoder, whose layers combine kNN-augmented attention, image cross-attention, and feed-forward processing.The decoder uses a causal mask to predict the next token.Source: §3.2 · pp. 2–3
- The augmented attention attends both to the past token subsequence and to independently encoded retrieved captions, using the same queries for both attention paths.A learned gate combines the local-context and external-memory outputs; training uses cross-entropy pre-training followed by SCST fine-tuning.Source: §3.2 · p. 3
4 EXPERIMENTAL EVALUATION
On COCO, the experiments assess retrieved-caption quality, model design, benchmark performance, and qualitative behavior. Results show stronger captioning performance alongside imperfect retrieval.
4.1 Setup
- Experiments use Karpathy COCO splits, a training-caption retrieval index that excludes the current training image, approximate HNSW search, and standard caption metrics.The reported metrics are BLEU, METEOR, ROUGE, CIDEr, and SPICE.Source: §4.1 · pp. 3–4
4.2 Quality of Nearest Neighbor Captions
- At k=40, the oracle caption reaches 130.4 CIDEr, but increasing k degrades mean scores and retrieved captions remain noisy.Even small retrieval sets are not fully reliable, so the language model must selectively copy content coherent with the image.Source: Table 1 · p. 4, §4.2 · p. 4
4.3 Model Ablation and Analysis
- Mean pooling, k=10 retrieved captions, and a one-layer memory encoder give the best pre-training CIDEr score, 124.5; the RA-Transformer also consistently beats chunked RETRO.The reported best configuration uses a single external-memory encoder layer; increasing RETRO chunk size decreases results.Source: Table 2 · p. 4, §4.3 · p. 5, Table 3 · p. 5
- The memory encoder alone adds 0.9 CIDEr (121.6 vs 122.5), while the learned gate adds 2 CIDEr over the ungated model; exact search adds 0.3 CIDEr but decoding takes 26% longer.The authors characterize approximate search as an efficacy-efficiency balance; full-model decoding rises from 0.170 to 0.215 seconds per caption.Source: §4.3 · p. 5
4.4 Comparison to the State of the Art
- After CIDEr fine-tuning on Karpathy-test, the retrieval-augmented model surpasses recent captioning approaches on all evaluation metrics.Table 4 reports the Karpathy-test comparison; the prose states the model surpasses the other approaches according to all metrics.Source: Table 4 · p. 5, §4.4 · pp. 5–6
4.5 Qualitative Results
- Most retrieved captions match image content and can help generation, but a locomotive-caption mismatch for a vase painting illustrates retrieval limits; visual features remain usable.The qualitative examples compare outputs with and without memory and include retrieved captions.Source: Figure 2 · p. 6, §4.5 · p. 6
5 CONCLUSION
The conclusion presents a retrieval-augmented Transformer for image captioning, using kNN-augmented attention to generate tokens from text retrieved from an external memory. COCO experiments demonstrate retrieval's effectiveness and motivate further research.
5 CONCLUSION
- The paper presents a retrieval-augmented Transformer for image captioning that integrates kNN-augmented attention.The attention layers generate word tokens based on textual sentences retrieved from an external memory.Source: §5 · p. 6
- The model accesses an external corpus during generation, and the authors report improved caption quality.Source: §5 · p. 6
- Experiments on COCO demonstrate the effectiveness of equipping a captioning architecture with retrieval abilities, opening further research in this direction.Source: §5 · p. 6