PaperlayerView exact revision on arXiv

Source-linked AI summary

From Show to Tell: A Survey on Deep Learning-based Image Captioning

Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, Rita Cucchiara

arXiv:2107.06912v3cs.CVcs.CL

TL;DR

Image captioning seeks to generate meaningful natural-language descriptions from images, but despite substantial architectural and training advances, the task remains unresolved. This survey synthesizes methods, variants, datasets, and metrics, quantitatively comparing major approaches and identifying continuing challenges and research directions.

  • Problem

    Image captioning must produce grammatical, image-grounded descriptions, yet research has not reached a conclusive answer and the task remains far from solved.

  • Method

    The survey reviews visual encoding, language modeling, training strategies, task variants, datasets, and evaluation metrics, and quantitatively compares relevant state-of-the-art methods.

  • Results

    The comparison shows performance improvement over recent years, while accuracy, robustness, generalization, fidelity, naturalness, and diversity remain unsatisfactory.

  • Takeaways & Limitations

    The review identifies open challenges and future directions for improving the synergy between Computer Vision and Natural Language Processing in image captioning.

  • Takeaways & Limitations

    Automatic caption evaluation is difficult because human evaluation is costly and not reproducible, while reference-based metrics do not fully capture caption quality.

Abstract

from arXiv · show

Connecting Vision and Language plays an essential role in Generative Intelligence. For this reason, large research efforts have been devoted to image captioning, i.e. describing images with syntactically and semantically meaningful sentences. Starting from 2015 the task has generally been addressed with pipelines composed of a visual encoder and a language model for text generation. During these years, both components have evolved considerably through the exploitation of object regions, attributes, the introduction of multi-modal connections, fully-attentive approaches, and BERT-like early-fusion strategies. However, regardless of the impressive results, research in image captioning has not reached a conclusive answer yet. This work aims at providing a comprehensive overview of image captioning approaches, from visual encoding and text generation to training strategies, datasets, and evaluation metrics. In this respect, we quantitatively compare many relevant state-of-the-art approaches to identify the most impactful technical innovations in architectures and training strategies. Moreover, many variants of the problem and its open challenges are discussed. The final goal of this work is to serve as a tool for understanding the existing literature and highlighting the future directions for a research area where Computer Vision and Natural Language Processing can find an optimal synergy.

1 INTRODUCTION

Image captioning describes visual content in natural language through visual encoding and language generation. The field has advanced from retrieval and handcrafted methods to deep generative models, yet remains unsolved; this survey reviews techniques, variants, datasets, metrics, and future directions.

  • Image captioning maps image pixels to meaningful, syntactically correct sentences using visual understanding and language generation.
  • Early systems relied on description retrieval, template filling, and handcrafted natural-language generation, whereas current systems use deep learning-based generative models.
  • Recent progress spans global-descriptor RNNs, attention, reinforcement learning, Transformers, self-attention, and BERT-like image-text strategies.
  • Captioning remains unsolved because accuracy, robustness, generalization, fidelity, naturalness, and diversity are not yet satisfactory.
  • The survey provides a holistic review of techniques, task variants, datasets, and evaluation metrics, alongside quantitative comparisons and open challenges.

LANGUAGE MODELS

The survey organizes captioning research through taxonomies of modeling and training approaches. It also compares datasets, metrics, and major methods to clarify their performance and characteristics.

  • The survey develops taxonomies for visual encoding and language modeling approaches, describing their key aspects and limitations.
  • Training strategies: It reviews cross-entropy, masked-language-model, reinforcement-learning, and vision-and-language pre-training strategies.
  • Datasets: It covers domain-generic benchmarks and domain-specific datasets used to investigate different image-captioning aspects.
  • Evaluation metrics: It analyzes standard and non-standard metrics and the caption characteristics each metric highlights.
  • Quantitative comparison: The quantitative comparison examines major methods with standard and non-standard metrics, relating scores to model performance and caption characteristics.
  • Open challenges: The survey discusses many task variants, open challenges, and future directions.
  • Compared with earlier surveys, it offers broader coverage of generative models, papers, metrics, task variants, and datasets.

2 VISUAL ENCODING

Visual encoding has progressed from compact global CNN descriptors to increasingly granular, relational, and self-attentive representations. The survey organizes these approaches into four categories and discusses their trade-offs and recent convergence with multimodal modeling.

  • Visual encoders fall into four categories: global CNN features, additive attention over grids or regions, graph-based encodings, and self-attentive region, patch, or early-fusion methods.This taxonomy captures the main design progression in image captioning visual encoding.
  • 2.1 Global CNN Features: Global CNN features are simple and compact but compress visual information excessively, limiting specific and fine-grained descriptions.They encode the whole image into high-level representations used to condition the language model.
  • 2.2 Attention Over Grid of CNN Features: Additive attention increases visual granularity by selecting spatial convolutional features for each generated word.Attention weights connect language-model hidden states with visual representations and produce a context vector for word prediction.
  • 2.3 Attention Over Visual Regions: Object-detector regions support bottom-up and top-down attention, while saliency-based methods incorporate human fixation information as an additional attention source.The bottom-up path proposes regions and the top-down mechanism weights them for each word prediction.
  • 2.4 Graph-based Encoding: Graph encodings model semantic and spatial relationships between detected objects, whereas self-attention connects all visual elements in a complete graph representation.Graph construction enables local information exchange and external semantic integration, while self-attention avoids manually specified interaction limits.
  • 2.6 Discussion: Recent visual-encoding research revisits region features alongside better-trained grids, self-attentive encoders, large-scale multimodal models, and end-to-end visual systems.BERT-like image-text early fusion further motivates representations that integrate textual information.

3 LANGUAGE MODELS

Image-captioning language models generate captions autoregressively by conditioning next-word probabilities on prior words and visual encoding. The survey traces a progression from LSTMs and attention to increasingly specialized and multi-stage recurrent variants.

  • A language model assigns a probability to a word sequence conditioned on visual encoding and generates each next word autoregressively from previously generated words.Generation usually ends when the model emits a special end-of-sequence token.
  • The survey categorizes captioning language models as LSTM-based, CNN-based, Transformer-based, and image-text early-fusion BERT-like strategies.The supplied section mainly develops the LSTM-based category.
  • LSTM-based approaches: Single-layer LSTM captioners use visual encoding as the initial hidden state, with teacher-forced ground-truth words during training and generated words during inference.Each hidden state is projected over the vocabulary to predict the next word.
  • LSTM-based approaches: Additive attention lets the previous hidden state select visual features and feed a context vector into the word-prediction network.The mechanism provides time-dependent visual conditioning during caption generation.
  • LSTM-based approaches: LSTM variants add visual sentinels, hidden-state reconstruction, semantic inputs, or multi-stage generation to handle non-visual words, context modeling, and caption refinement.Multi-stage systems can generate coarse sentence structure before enriching attributes.
  • LSTM-based approaches: Two-layer LSTMs specialize one layer for visual attention and another for language modeling, becoming a prevalent approach before Transformer architectures.The first layer computes attended image features that are passed to the second layer for vocabulary prediction.

3.2 Convolutional Language Models

Convolutional language models were explored as an alternative to fully recurrent caption generation. They enable parallel training but did not become widely adopted because of poor performance and the emergence of Transformers.

  • A convolutional captioning model combines a global image feature vector with word embeddings and processes words in parallel during training.Right-masking prevents future-token information from being used.
  • During inference, the convolutional model generates words sequentially rather than in parallel.This preserves autoregressive caption generation while retaining parallelism during training.
  • Convolutional language models did not gain popularity because of poor performance and the advent of Transformer architectures.

3.3 Transformer-based Architectures

Transformer architectures apply fully attentive sequence modeling to image captioning, with mechanisms that regulate visual-semantic information and attention across encoded representations.

  • 3.3 Transformer-based Architectures: Transformers provide fully attentive sequence-to-sequence architectures for image captioning.Their adoption followed the broader impact of self-attention and Transformer models in language generation.
  • 3.3 Transformer-based Architectures: Gating mechanisms modulate how visual regions, global image representations, semantic attributes, and encoder layers influence word generation.These gates combine or weight visual and semantic contributions according to the generated-word context.
  • 3.3 Transformer-based Architectures: A BERT-like language model processes image regions and word tokens through a single stream of attentive layers to generate the caption.This early-fusion design jointly handles visual and textual modalities.

3.4 BERT-like Architectures

BERT-like captioning architectures fuse visual and textual modalities early and exploit pre-training, while related work explores parallel generation and generative alternatives.

  • 3.4 BERT-like Architectures: BERT-like architectures fuse visual and textual modalities in early layers and can initialize textual processing from large-scale language pre-training.This structure has been widely adopted in image-captioning systems that exploit pre-training.
  • 3.4 BERT-like Architectures: Non-autoregressive captioning generates words in parallel to reduce inference time, using staged prediction schemes in some approaches.Some methods formulate word positions as cooperative agents that optimize a sentence-level reward.
  • 3.4 BERT-like Architectures: Recurrent models are slow to train and struggle with long-term dependencies, drawbacks alleviated by autoregressive and Transformer-based solutions.The passage contrasts these models with recent architectures that also support large-scale pre-training.
  • 3.4 BERT-like Architectures: BERT-like models suit massive pre-training but are not generative by design, motivating further exploration of generative-oriented pre-training.Generative-oriented architectures are reported as achieving performances at least on par with early-fusion counterparts.

4 TRAINING STRATEGIES

Image-captioning training combines word-level objectives, masked-language modeling, reinforcement learning, and vision-language pre-training, each targeting different modeling or optimization needs.

  • 4 TRAINING STRATEGIES: Image-captioning training commonly uses cross-entropy, masked language modeling, reinforcement learning, and vision-and-language pre-training objectives.These strategies cover token prediction, contextual reconstruction, sequence-level optimization, and multimodal representation learning.
  • 4 TRAINING STRATEGIES: Cross-entropy minimizes each word’s negative log-likelihood given previous ground-truth words, but does not model longer-range generated-word dependencies.Traditional cross-entropy training also suffers exposure bias because training inputs differ from the model’s predicted-word distribution.
  • 4 TRAINING STRATEGIES: Masked language modeling randomly masks input tokens and trains the model to recover them from both preceding and subsequent context.This objective helps build robust sentence representations from contextual information.
  • 4 TRAINING STRATEGIES: Reinforcement learning treats caption generation as policy-controlled word selection and directly optimizes sequence-level rewards such as CIDEr, BLEU, or ROUGE.Training commonly begins with cross-entropy or masked-language pre-training before reinforcement-learning fine-tuning.
  • 4 TRAINING STRATEGIES: Vision-language pre-training masks tokens across visual and textual modalities or applies contrastive objectives to connect their joint representations.Cross-entropy and prefix-based objectives are also used when pre-training on noisy captions.
  • 4 TRAINING STRATEGIES: Image captioning can serve as a pre-training task for visual representations that benefit image classification, object detection, and instance segmentation.

5 EVALUATION PROTOCOL

Image-captioning evaluation depends on datasets with varied visual and linguistic properties and metrics that balance human relevance, lexical similarity, and caption diversity.

  • 5.1 Datasets: Datasets pair images with one or more captions, whose number, length, vocabulary, relevance, and style influence algorithm design and performance.Multiple captions capture variability in human descriptions.
  • 5.1 Datasets: Benchmark datasets provide a common test bed and should represent task challenges, achievable human performance, generic images, and multiple captions.
  • 5.1 Datasets: Microsoft COCO is a widely used benchmark containing more than 120,000 images of complex everyday scenes with people, animals, and objects in context.
  • 5.1 Datasets: Large vision-language pre-training datasets broaden scale and variety but may contain noisy captions or unavailable images, while collection requires substantial resources.Examples include Wikipedia-based Image Text with approximately 5.3 million images and very large, sometimes private, image-text collections.
  • 5.1 Datasets: Domain-specific datasets target visual or semantic challenges, such as low-quality assistive-technology images or bird and flower descriptions.Their specialized distributions extend captioning beyond generic-domain benchmarks.
  • 5.2 Evaluation Metrics: Human evaluation best measures caption quality but is costly and non-reproducible, motivating automatic metrics that compare generated captions with references or images.Caption quality requires both linguistic correctness and accurate reference to the input image.
  • 5.2 Evaluation Metrics: Standard metrics can favor word similarity over meaning and discourage novel, diverse captions, so diversity metrics complement them.Diversity can be measured globally with Vocab and %Novel or locally with Div-1 and Div-2.
  • 5.2 Evaluation Metrics: Learning-based metrics use pre-trained models to compare candidate and reference captions through contextual embeddings or image-grounding representations.

6 EXPERIMENTAL EVALUATION

The evaluation compares major image-captioning approaches across multiple quality metrics and model sizes. CIDEr tracks several other scores, while complexity alone does not predict performance.

  • Experimental setup: The evaluation combines standard and non-standard caption-quality scores with parameter counts to assess model performance and computational complexity.Table 2 data come from authors’ released weights and captions or the survey’s best implementation.
  • Results: Diversity scores are especially correlated with one another, while early approaches show relatively strong diversity despite lower standard-metric values.Div-1, Div-2, and vocabulary size form a particularly correlated group.
  • Results: Text-only BERT-S does not help discriminate among image-captioning approaches according to the learning-based score trends.BERT-S is reported using its F1-score variant.
  • Results: More complex models do not necessarily achieve better CIDEr performance.Fig. 9 relates CIDEr to model complexity and other characteristics.
  • Results: CIDEr has an almost-linear relationship with other evaluation scores, with some flattening at high CIDEr values.The survey uses CIDEr because it is commonly regarded as a relevant indicator of image-captioning performance.

7 IMAGE CAPTIONING VARIANTS

The survey organizes image-captioning variants by their data, visual-input, textual-output, and user-oriented goals. These variants address limited supervision, richer visual grounding, diversity, personalization, controllability, and caption correction.

  • Taxonomy: Captioning sub-tasks are grouped into four scopes: limited training data, visual input, textual output, and user requirements.This taxonomy covers variants beyond general-purpose image captioning.
  • Lack of training data: Novel object captioning describes objects absent from training data, enabling a zero-shot setting through external unpaired visual and textual data.Early methods transferred knowledge from out-of-domain images.
  • Lack of training data: Unpaired image captioning reduces dependence on paired image-text supervision through unsupervised or semi-supervised approaches.Methods include pivot-language generation and adversarial learning.
  • Lack of training data: Continual captioning incrementally learns sequences of tasks with different vocabularies while transferring visual concepts and expanding the vocabulary.Its goal is to learn new tasks without forgetting previous knowledge.
  • Focusing on the visual input: Dense captioning localizes salient regions and describes them with short sentences, while text-based captioning reads and incorporates text appearing in images.Dense captioning generalizes object detection by replacing object tags with captions.
  • Focusing on the textual output: Diverse captioning seeks human-like variability using grouped beam search, contrastive learning, conditional GANs, or paraphrasing, though these methods can underperform.The passage identifies multiple strategies but notes a performance drawback.
  • Addressing user requirements: Personalized and controllable captioning adapt descriptions to user styles, priorities, selected regions, visual words, mouse traces, or structural guidance.Personalized captioning addresses captions that avoid obvious statements and attract user interest.
  • Addressing user requirements: Image captioning editing separates generation from polishing to correct syntactic errors, repetitions, and inconsistencies.The editing variant decouples decoding into caption generation and caption polishing.

8 CONCLUSIONS AND FUTURE DIRECTIONS

The survey concludes that image captioning has improved on standard datasets but remains unsatisfactory in accuracy, robustness, generalization, fidelity, naturalness, and diversity. Future work must address procedural, architectural, computational, and long-tail challenges.

  • Conclusions: Despite performance improvements on standard datasets, accuracy, robustness, generalization, fidelity, naturalness, and diversity remain unsatisfactory.The conclusion frames image captioning as a challenge spanning Computer Vision and NLP.
  • Procedural and architectural challenges: Large-scale pre-training is increasingly used because image-captioning models are data greedy, but public datasets are needed for reproducibility and fair comparisons.The survey notes that such datasets may be imperfectly curated.
  • Procedural and architectural challenges: Growing pre-training models raise computational concerns, motivating investigation of less intensive alternatives.The authors connect this concern to equality within the research community.
  • Procedural and architectural challenges: The field still lacks resolution between early-fusion and encoder-decoder strategies, while several visual encoding strategies appear comparable in performance.The listed alternatives include scratch pre-training, detection features, and multimodal-model features.
  • Generalization, diversity, long-tail concepts: Web-scale pre-training may improve generalization and long-tail concepts, but domain specialization and captions with different styles and aims remain open challenges.Further research is also needed for models suitable for real-world applications.

APPENDIX A FURTHER ANALYSIS OF THE EVALUATION METRICS

The appendix extends the metric analysis with a taxonomy and detailed discussion of diversity, embedding-based, and learning-based measures. It explains how embedding matching and reference variability are handled in BERT-based metrics.

  • Diversity metrics: Local diversity can be quantified with mBLEU, where lower average inter-caption BLEU indicates a more diverse generated caption set.mBLEU averages BLEU scores between each caption and the others.
  • Metric taxonomy: Table 3 organizes image-captioning metrics by taxonomy and summarizes their main characteristics.The appendix uses the table as a framework for discussing additional metric families.
  • Embedding-based metrics: BERT-S compares candidate and reference tokens through cosine similarity of embeddings and uses best matches to compute precision, recall, and F1-score.This makes the metric embedding-based rather than dependent only on exact lexical overlap.
  • Embedding-based metrics: BERT-SIRV accounts for variability among references for one image by combining their concepts into a single embedding vector for candidate comparison.The combined vector contains the concepts mentioned across the reference captions.

APPENDIX B FURTHER PERFORMANCE ANALYSIS

The appendix compares representative captioning methods across metrics and examines qualitative behavior, cross-dataset generalization, and attention visualizations. Results show gains from structured visual representations and persistent weaknesses in counting and unusual concepts.

  • Quantitative performance: Average CIDEr rises from 105.8 with grid-feature attention to 121.8 with region attention and 133.2 with graph-based or self-attention approaches.The analysis associates higher scores with more complete, structured information about visual concepts and their relations.
  • Quantitative performance: CIDEr has an almost-linear relation with the other reported scores, except for BERT-S.Figure 10 relates CIDEr to the scores and model characteristics reported in the analysis.
  • Generalization analysis: Cross-dataset evaluation shows significant drops in standard and embedding-based metrics, while diversity metrics increase and learning-based metrics are more stable on Flickr30K.Training on COCO and testing on Flickr30K performs worse under all metrics than training and testing on Flickr30K.
  • Qualitative analysis: Qualitative examples show similar caption length and structure across eight models, but current approaches struggle with counting and unusual visual concepts.Models often describe multiple instances as a group and substitute unusual entities with more represented concepts; these errors are less evident for VinVL.
Loading 2107.06912v3…