PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Approaches on Image Captioning: A Review

Taraneh Ghandi, Hamidreza Pourreza, Hamidreza Mahyar

arXiv:2201.12944v5cs.CV

TL;DR

Image captioning research requires updated synthesis as methods advance and existing surveys cover earlier work. This paper reviews deep-learning approaches through a taxonomy, dataset and metric analysis, challenge discussion, and performance comparison, finding strong but uneven results alongside persistent quality gaps.

  • Problem

    An updated synthesis is needed to track recent image-captioning progress beyond surveys focused on earlier research.

  • Method

    The paper taxonomizes recent deep-learning image-captioning methods and reviews their datasets, evaluation metrics, challenges, future directions, and reported performance.

  • Results

    The surveyed methods achieve strong results across metrics, with different models leading BLEU-1 through BLEU-4, METEOR, ROUGE, CIDEr, and SPICE.

  • Takeaways & Limitations

    The review identifies information misalignment, dataset limitations, object hallucination, and evaluation shortcomings as continuing priorities for image-captioning research.

  • Takeaways & Limitations

    Unpaired image-captioning methods still lag in performance, while image and sentence encoders cannot be shared because their modalities differ.

Abstract

from arXiv · show

Image captioning is a research area of immense importance, aiming to generate natural language descriptions for visual content in the form of still images. The advent of deep learning and more recently vision-language pre-training techniques has revolutionized the field, leading to more sophisticated methods and improved performance. In this survey paper, we provide a structured review of deep learning methods in image captioning by presenting a comprehensive taxonomy and discussing each method category in detail. Additionally, we examine the datasets commonly employed in image captioning research, as well as the evaluation metrics used to assess the performance of different captioning models. We address the challenges faced in this field by emphasizing issues such as object hallucination, missing context, illumination conditions, contextual understanding, and referring expressions. We rank different deep learning methods' performance according to widely used evaluation metrics, giving insight into the current state of the art. Furthermore, we identify several potential future directions for research in this area, which include tackling the information misalignment problem between image and text modalities, mitigating dataset bias, incorporating vision-language pre-training methods to enhance caption generation, and developing improved evaluation tools to accurately measure the quality of image captions.

1 Introduction

This review updates image-captioning scholarship by organizing recent whole-image methods into method-specific categories and examining their datasets, metrics, challenges, and performance. It focuses on work published from 2018 to 2022, including the relatively underexplored vision-language pre-training category.

  • It focuses on whole-image captioning rather than dense captioning, which produces captions for individual entities.
  • Methods are organized into separate sections by category instead of being presented through pairwise comparisons.
  • Vision-language pre-training methods receive attention because this category has been seldom explored in previous surveys.
  • The review covers image-captioning methods published from 2018 to 2022, along with common problems and challenges.
  • The survey analyzes widely used datasets and evaluation metrics, compares covered methods, and reviews field challenges and future directions.

2 Common Solutions and Techniques

Image captioning is treated as a computationally intensive sequence-to-sequence problem supported by neural techniques for visual representation, object detection, memory, and feature extraction. Common solutions combine image encoders and language-generating decoders, with recurrent, residual, and region-based components serving complementary roles.

  • Encoder-decoder methods: Encoder-decoder methods map an input image to an intermediate representation and then convert it into a sequence of caption words.
  • Encoder-decoder methods: CNNs are frequently used in encoding to detect image objects and provide rich visual representations from their final convolutional layers.
  • Object detection: Region-based CNNs address variable object shapes, sizes, and locations by generating candidate regions rather than relying on one fixed grid.
  • Object detection: Fast R-CNN improves on R-CNN by convolving the image once to produce a feature map before generating region proposals.
  • Object detection: Faster R-CNN replaces time-consuming selective search with a separate network that learns to predict region proposals.
  • Recurrent and residual networks: LSTMs use gates to control information flow and preserve important sequence information, making them widely used in encoder-decoder captioning methods.
  • Recurrent and residual networks: ResNets use skip connections and residual blocks, enabling easier and faster optimization than simple network structures.

3 Deep Learning-Based Image Captioning

The survey classifies recent deep learning image-captioning approaches by core structure, emphasizing attention-based encoder-decoder methods that guide generation toward relevant image features. It also describes soft and hard attention, multi-head attention, and spatial-semantic extensions, while noting persistent practical limitations.

  • The survey organizes image-captioning frameworks, methods, and approaches according to their core structure.
  • Attention-Based Methods: Attention-based methods use attention mechanisms to emphasize relevant image regions during caption generation.
  • Attention-Based Methods: Attention-based captioning commonly combines CNN or region-based CNN encoders with RNN decoders that use weighted feature sums at each time step.
  • Attention-Based Methods: Multi-head attention runs attention mechanisms in parallel, concatenating and linearly transforming outputs to capture different sequence dependencies.
  • Attention-Based Methods: Soft attention uses weighted image features with regular backpropagation, whereas hard attention uses stochastic sampling and Monte Carlo averaging.Hard-attention accuracy depends on sample number and quality, while soft attention assumes weighted averages adequately represent attended areas.
  • Attention-Based Methods: Adding spatial and semantic relations improves caption quality but leaves ambiguity, weak grounding, detector computation, and bounding-box annotation requirements.

3.2 Graph-Based Methods for Spatial and Semantic Relations between Image Elements

Graph-based captioning methods model spatial and semantic relations among image elements using scene graphs. These graphs are encoded into representations that language decoders use to generate captions, with approaches spanning supervised, intermediate-representation, and unsupervised frameworks.

  • Graph-based methods use scene graphs to model spatial and semantic relations between image elements.
  • Scene Graph Auto-Encoder connects image and text modalities through an I →G →D →S training pipeline.The scene graph represents objects, attributes, and relationships using directed edges.
  • Cycle-Consistent Adversarial Training maps image and caption scene graphs into a shared feature space without image-caption pairs.
  • A two-phase scene-graph method performs concept cognition before sentence reconstruction, using semantic representations as input to an RNN caption generator.
  • The typical workflow extracts visual features, builds a graph over detected regions, encodes it with GCNs, and passes the result to LSTM decoders.

3.3 Combining Attention-Based Methods and Graph-Based Methods

These methods combine attention with graph-based representations to model objects, attributes, spatial relations, and semantic interactions during caption generation. The reviewed approaches include graph neural networks, scene-graph decomposition, controllable abstract scene graphs, and direct relation-feature attention, while scene-graph extraction and relation detection remain difficult.

  • Graph and relation representations: Yao et al. combine graph convolutional networks, LSTMs, and attention-based encoder-decoder modeling to incorporate spatial and semantic relations.The method treats relationship learning as a classification problem and builds separate spatial and semantic graphs from detected image regions.
  • Graph and relation representations: Graph-based captioning methods represent relations among image elements to enrich visual understanding beyond object detection alone.Spatial and semantic relations can be integrated into image representations, while visual relations describe relative positions or interactions.
  • Scene-graph decomposition: Zhong et al. decompose scene graphs into semantic sub-graphs, select meaningful sub-graphs with an sGPN, and decode them into sentences with an attention-based LSTM.Their formulation models scene-graph extraction, sub-graph selection, and sentence decoding as components of a joint probability.
  • Controllable and scene-graph captioning: Chen et al. introduce controllable captioning using an Abstract Scene Graph containing image-grounded object, attribute, and relationship nodes.The model uses a role-aware graph encoder and a language decoder that incorporates the control signal.
  • Limitations: Scene-graph methods face extraction difficulty, relations that exceed simple pairwise structure, and graph parsers that still require improvement.Detecting interactions is also complicated by the diversity of object sizes and locations.

3.4 Convolutional Network-Based Methods

Convolutional captioning models replace recurrent language components or use CNNs throughout the captioning architecture. The survey describes their feed-forward context modeling, potential for caption diversity, and remaining performance limitations under evaluation metrics.

  • Overview: Convolutional network-based methods use CNNs to extract image features and generate captions from language-model outputs.The reviewed category applies convolutional architectures to vision-language sequence generation.
  • Convolutional architectures: Aneja et al. replace recurrent units with masked convolutions between input and output word embeddings.The middle component is feed-forward and contains no recurrent functions.
  • Convolutional architectures: Wang et al. use CNNs for vision, language, attention, and prediction in an architecture without recurrent language modeling.CNN kernels and stacked layers model context instead of the recurrent path used by RNNs.
  • Strengths and limitations: Convolutional models can produce more caption diversity and avoid vanishing gradients, but still need evaluation-metric performance improvements.The survey also notes that comparatively less attention has been devoted to this category.

3.5 Transformer-Based Methods

Transformer-based methods address limitations associated with recurrent language models by modeling visual, spatial, relational, and semantic information through attention. The reviewed architectures range from object-relation and cross-modal models to fully Transformer-based image encoders and decoders.

  • Motivation: Transformers are used to build more robust captioning solutions in response to recurrent models’ inflexibility, limited expression ability, and difficulty retaining distant inputs.The survey links recurrent processing with high-frequency phrase fragments that may disregard visual cues.
  • Spatial and relational attention: Object Relation Transformer incorporates spatial relations between detected objects through geometric attention.Bounding-box properties such as center, width, and height contribute to the attention computation.
  • Cross-modal information: GLIED globally models spatial and relational groupings of image regions and attribute words, then locally extracts information for word selection.The approach is designed for aspect-based image representation and precise word selection.
  • Multi-level Transformer architectures: M^2 uses multi-level region and relationship encoding, persistent memory vectors, and learned gating to connect encoder layers with the decoder.The decoder reads multiple encoding layers and generates captions word by word using scaled dot-product attention without recurrence.
  • Other Transformer models: Other reviewed models include Transformer-LSTM hybrids, ETA-Transformer, CPTR, PureT, ViTCAP, GRIT, and ExpansionNet v2.These systems vary in their image encoders, concept-token modules, detector features, and sequence-expansion mechanisms.

3.6 Combining Transformers and Scene Graphs

A group of captioning methods combines Transformer architectures with scene-graph or graph-based encoders to represent visual and semantic relations. These designs encode relations between regions or integrate scene-graph generation and captioning within Transformer models.

  • Category overview: The surveyed literature reports that multiple works have experimented with combining Transformers and scene graphs.This category extends Transformer captioning with structured relational information.
  • Transformer and graph integration: Transformer–scene-graph methods use multiple sub-transformers or paired visual and semantic sub-encoders to encode relations between image regions.One design combines visual semantic and spatial graphs, while another uses Faster R-CNN to propose image regions.
  • Transformer and graph integration: ReFormer embeds relation information into features and combines scene-graph generation with image captioning in one modified Transformer.It explicitly represents pairwise relationships between objects in the image.

3.7 Vision Language Pre-Training Methods for Image Captioning

Vision-language pre-training reduces reliance on fully supervised image-caption pairs by learning reusable representations from large-scale data. The reviewed approaches align visual and textual information through shared representations, object tags, or cross-modal foundation-model architectures.

  • Pre-training uses large-scale self-supervised data before generalizing the learned model to downstream tasks.
  • CLIP learns shared image-text representations from numerous image-caption pairs using contrastive loss.
  • ClipCap maps CLIP embeddings into caption prefixes that condition a fine-tuned GPT-2 language model.
  • Object-tag pre-training aligns image and language modalities in a shared semantic space using word sequences, object tags, and image-region features.The resulting pre-training can support image captioning, image-text retrieval, and visual question answering.
  • mPLUG targets cross-modal understanding and generation while addressing computational efficiency and information asymmetry through cross-modal skip-connections.

3.8 Unsupervised Methods and Reinforcement Learning

Unsupervised image captioning methods relax dependence on paired image-caption datasets, often using reinforcement learning or adversarial training. The reviewed workflows combine visual encoding, caption generation, and discriminators or retrieval-based rewards, while reducing dataset-preparation burdens.

  • Unsupervised methods address reliance on the quality and volume of paired image-caption datasets.
  • Some approaches use a pivot-language paired dataset while avoiding paired captions in the target language.
  • Feng et al. project unrelated images and a sentence corpus into a common latent space so they can reconstruct each other.
  • Without image-caption pairs, discriminators evaluate sentence realism, image relevance, and visual-content consistency to reward caption generation.
  • Conditional GAN frameworks add discriminator networks that distinguish human-described from machine-generated captions because optimizing one metric cannot guarantee improvement across metrics.
  • Self-retrieval evaluates similarity among generated captions, their input image, and distractor images to assess caption-image relevance.
  • The general unsupervised workflow uses a VGGNet image encoder, an LSTM caption generator, and an LSTM discriminator that rewards realistic captions.
  • Unsupervised settings are expected to become more favored because supervised dataset preparation and training are difficult.

3.9 Generating Multi-Style Captions

Multi-style captioning extends factual, neutral descriptions with tones such as humorous, hostile, poetic, anxious, happy, vague, dramatic, and charming. The reviewed systems condition generation or retrieval on style and may project images, captions, and personality traits into a shared space.

  • Neutral factual captions can be extended with human communication styles and tones for more engaging interactions and applications such as photo-sharing.
  • TransResNet projects images, captions, and personality traits into a shared space using an encoder-decoder framework.It considers both retrieval and generative model classes.
  • Guo et al. use five modules: image encoding, style-conditioned caption generation, caption discrimination, and additional adversarial style-related components.
  • Figure 6 compares ground-truth and generated captions from multiple method categories for each displayed image.GT denotes Ground Truth Caption, while G denotes Generated Caption.
  • An example pairs the ground-truth zebra caption with a generated caption that adds its position next to a building.
  • Style examples express anxious, happy, vague, dramatic, and charming tones for a cat image.

4 Problems in Image Captioning

Image captioning faces general sequence-modeling problems and task-specific challenges involving hallucinated objects, illumination, contextual reasoning, and referring expressions. These issues affect training stability, visual grounding, and the accuracy and usefulness of generated descriptions.

  • Image captioning inherits exposure bias, loss-evaluation mismatch, vanishing gradients, and exploding gradients alongside task-specific visual challenges.
  • Exposure bias arises because training uses ground-truth prefixes while testing feeds the model’s previous predictions back into generation.This mismatch can gradually produce more output errors.
  • Cross-entropy training and sentence-level evaluation use mismatched objectives because caption metrics are non-differentiable.Reinforcement learning has been used to optimize these metrics.
  • Vanishing gradients can prevent weight updates and halt neural-network training.
  • Exploding gradients create enormous updates that destabilize networks, produce NaN weights, or prevent learning over long sequences.
  • Object hallucination occurs when a model detects objects absent from the input image, harming caption reliability for visually impaired users.CHAIR measures the proportion of generated words corresponding to image objects, with per-instance and per-sentence variants.
  • Models optimized for CIDEr frequently hallucinated more, whereas GAN-based losses decreased hallucination by encouraging human-like sentences.Attention models performed better on CHAIR, primarily because of access to convolutional features rather than attention itself.
  • Low-light, indoor, shadowed, or unevenly illuminated images can reduce contrast and hinder recognition of visual details.Suggested remedies include contrast enhancement, color correction, and low-light image enhancement.

5 Discussion

The discussion compares major image-captioning method families, emphasizing their trade-offs in representation, computational structure, grounding, and supervision. It highlights attention, relational, convolutional, Transformer, and unsupervised or reinforcement-learning approaches alongside persistent limitations.

  • Using Attention: Attention-based methods use encoder-decoder architectures to select image regions, but single-stage designs may produce less rich captions and attention-region selection can be imprecise.Multi-stage coarse-to-fine attention passes weights and hidden vectors between decoders to reduce caption ambiguity.
  • Using Attention and Graphs: Attention-only methods often omit spatial and semantic relations between image elements, limiting distinctions that depend on object arrangement or relative size.Object Relation Transformer and graph-based approaches address relations among image elements and higher-order interactions.
  • Vision-Language Pre-training: Grounding remains difficult because object tags may not link object regions with word embeddings, while models may attend to regions differently from humans.Vision-language pre-training methods use shared image-caption anchors or unified representations to improve grounding and downstream generation or understanding.
  • Using Convolutional Network-Based Methods: Convolutional captioning models replace recurrent units with masked convolutions, enabling parallel computation and faster training, although performance still needs improvement.Attention can be added to convolutional layers through attended image vectors incorporated with word embeddings.
  • Using Transformers: Transformer-based methods replace recurrent or convolutional components to exploit strong language-modeling performance and parallel training, including fully Transformer and detector-free designs.Detector-free ViTCAP avoids object detectors, which can impose computational load and require box annotations.
  • Using Unsupervised Methods and Reinforcement Learning: Supervised captioning relies on paired images and captions but faces challenges including English-language concentration and limited multilingual data; GAN discriminators and self-retrieval support alternative training strategies.A self-retrieval module using REINFORCE improves a captioning structure trained on partially labeled data.

6 Datasets and Performance Comparison

The survey reviews commonly used image-captioning datasets, evaluation practices, and comparative results. Across reported results, leading methods frequently combine Transformers, scene graphs, or vision-language pre-training.

  • Datasets: Image-captioning datasets vary from general-purpose collections to region-, relationship-, style-, accessibility-, and novel-object-focused resources.Examples include MS COCO, Flickr30K Entities, Visual Genome, style-diverse captions, assistive datasets, and nocaps.
  • MS COCO server results: Li et al.’s model achieved the best c5 and c40 results in most metrics on the Microsoft COCO server comparisons.Aneja et al.’s convolutional-network model achieved the lowest results in both settings for almost all metrics.
  • Independent results: [157] led BLEU-1 (90.7), BLEU-4 (59.3), METEOR (40.1), and ROUGE (71.5), while [84] led BLEU-2 (69.1) and BLEU-3 (54.9).Show, Control and Tell achieved the best CIDEr (209.7) and SPICE (48.5) results.
  • Patterns in high-performing methods: The strongest reported methods commonly apply Transformers, scene graphs, and vision-language pre-training.The survey associates these techniques with modeling object relationships, visual content, and joint visual-textual representations.

7 Challenges and the Future Directions

The survey identifies unresolved challenges involving dataset coverage, modality misalignment, scene-graph construction, and the limited generative capabilities of current vision-language pre-training methods. It proposes broader use of unsupervised and unpaired approaches, scene graphs, vision-language pre-training, and applications supporting visually impaired people.

  • Challenges: Supervised captioning is constrained by dataset quality and coverage, and by the object categories recognizable to the trained detector.Datasets cannot represent the entire real world, limiting applicability beyond their covered concepts.
  • Challenges: Unpaired captioning faces information misalignment between image and sentence modalities, and current unsupervised methods remain lower in performance rankings.The survey presents these problems as motivations for developing unsupervised techniques.
  • Challenges: Scene-graph methods remain difficult because graph construction and integration are complicated, while scene-graph parsers are not yet sufficiently powerful.Object interactions can extend beyond simple pairwise relations, increasing integration difficulty.
  • Challenges: Vision-language pre-training has addressed some flaws in supervised and detector-based designs, but generation tasks require capabilities beyond those commonly developed for understanding tasks.The survey identifies this gap as an ongoing issue for image captioning.
  • Future directions: Future directions include scene graphs, vision-language pre-training, unsupervised and unpaired learning, and captioning applications for visually impaired people.The survey connects these directions with representing complex relationships, addressing modality gaps, and supporting vision assistants.

8 Conclusion

The review finds that image captioning methods have advanced substantially, yet generated captions, datasets, and evaluation metrics remain imperfect. It also identifies visual assistance for visually impaired people as an important area requiring further research.

  • The survey organizes recent image captioning methods into a taxonomy and compares their features, properties, datasets, metrics, and experimental performance.
  • Generated captions remain below human quality, while available datasets cannot represent the full diversity of the real world.
  • Current evaluation metrics are not yet ideal for measuring models’ precise captioning performance.
  • Vision-language pre-training methods and Transformers are identified as likely future components of image captioning models.
  • Visual assistants for visually impaired people require denser captions that prioritize important content and include surroundings, textures, and object relations.
  • Future visual assistants may provide shorter initial captions and progressively add detail when users ask further questions.
Loading 2201.12944v5…