PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey of Deep Learning for Image Captioning

Md. Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, Hamid Laga

arXiv:1810.04020v2cs.CVcs.LGstat.ML

TL;DR

Image captioning must recognize visual entities and relationships while producing syntactically and semantically correct descriptions. This survey organizes deep learning-based methods, reviews their datasets and metrics, and finds substantial progress but no robust method for nearly all images.

  • Problem

    Earlier surveys covered few deep learning-based novel caption-generation models despite the field’s substantial subsequent growth.

  • Method

    The paper comprehensively surveys deep learning image-captioning methods, organizing them by method categories and discussing datasets, evaluation metrics, strengths, limitations, and research directions.

  • Results

    The survey reports remarkable progress, while concluding that a robust method generating high-quality captions for nearly all images has not yet been achieved.

  • Takeaways & Limitations

    Deep learning-based image captioning remains an active research area requiring continued work on methods, datasets, and evaluation.

  • Takeaways & Limitations

    Existing methods are mainly evaluated on same-domain datasets, leaving open-domain performance uncertain.

Abstract

from arXiv · show

Generating a description of an image is called image captioning. Image captioning requires to recognize the important objects, their attributes and their relationships in an image. It also needs to generate syntactically and semantically correct sentences. Deep learning-based techniques are capable of handling the complexities and challenges of image captioning. In this survey paper, we aim to present a comprehensive review of existing deep learning-based image captioning techniques. We discuss the foundation of the techniques to analyze their performances, strengths and limitations. We also discuss the datasets and the evaluation metrics popularly used in deep learning based automatic image captioning.

1 INTRODUCTION

Image captioning combines image understanding with syntactically and semantically correct language generation. The survey reviews deep learning approaches because earlier surveys covered only a limited portion of the expanding literature.

  • Image captioning requires recognizing objects, scene context, properties, interactions, and generating well-formed language.
  • Traditional methods rely on handcrafted features and task-specific classifiers, limiting scalability to large and diverse image collections.Examples include LBP, SIFT, HOG, and SVM-based classification.
  • Deep learning methods learn features automatically from training data and commonly combine CNN-based feature extraction with recurrent caption generation.CNNs are used for feature learning and are generally followed by RNNs for caption generation.
  • Earlier surveys covered relatively few deep learning papers because much of the literature appeared after their publication.
  • The survey organizes prior work into three broad families: template-based, retrieval-based, and novel caption generation.
  • It focuses on deep learning-based novel caption generation and further groups methods into multiple architectural, learning, spatial, attention, semantic, stylized, and object-based categories.The taxonomy includes visual-space, multimodal-space, supervised, dense-captioning, whole-scene, encoder-decoder, compositional, language-model, attention, semantic-concept, stylized, and novel-object categories.

2 IMAGE CAPTIONING METHODS

Image captioning methods include template-based, retrieval-based, and novel caption generation approaches. Deep learning is the main focus because most novel caption generation methods use it.

  • Template-based approaches detect objects, attributes, and actions, then insert them into fixed sentence templates.Triplets of scene elements can be used to fill template slots.
  • Retrieval-based methods select candidate captions from visually similar training images.They produce general and syntactically correct captions but cannot generate image-specific, semantically correct descriptions.
  • Novel caption generation methods analyze visual content and use a language model to create new captions for each image.The survey states that these captions can be semantically more accurate than those from previous approaches.
  • Most novel caption generation methods use deep machine learning techniques, making them the survey’s main focus.
  • The taxonomy compares novel-generation methods across visual versus multimodal spaces and supervised, reinforcement, and unsupervised learning.The survey groups reinforcement and unsupervised learning under other deep learning.

3 DEEP LEARNING BASED IMAGE CAPTIONING METHODS

The survey’s deep learning taxonomy organizes image captioning methods by representation space, learning strategy, architecture, caption scope, and specialized modeling choices.

  • The taxonomy compares visual-space and multimodal-space methods, dense and whole-scene captioning, and supervised versus other deep learning.
  • It also distinguishes encoder-decoder from compositional architectures and separates LSTM language models from other language models.
  • Attention-based, semantic-concept-based, stylized-caption, and novel-object-based methods are grouped into an additional category.
  • Table 1 records each method’s image encoder, language model, and taxonomy category.

3.1 Visual Space vs. Multimodal Space

Visual-space methods separately process image features and captions, whereas multimodal-space methods learn a shared representation before decoding captions. Multimodal approaches include joint image-text modeling and finer-grained alignment strategies.

  • Visual Space vs. Multimodal Space: Visual-space methods independently pass image features and captions to the language decoder, while multimodal methods learn a shared image-text representation first.
  • Multimodal Space: A typical multimodal architecture contains language encoding, visual feature extraction, multimodal mapping, and language decoding stages.The vision component uses a deep CNN, while the language encoder learns word embeddings and temporal context.
  • Multimodal Space: Multimodal methods jointly learn image and text in a shared space, then generate captions from that learned information.
  • Multimodal Space: Kiros et al. used CNN image features with multimodal neural language models, later extending the approach with joint image-sentence embeddings and SC-NLM.The extended approach used LSTM sentence encoding and reported significant improvements in realistic caption generation over the earlier approach.
  • Fine-Grained Alignment: Karpathy et al. aligned image fragments with sentence fragments by decomposing images into objects and sentences into dependency-tree relations.The method reasons about latent inter-modal alignment at a finer level than direct image-sentence embedding.
  • Fine-Grained Alignment: Dependency relations do not always map cleanly to visual entities, limiting the appropriateness of fine-grained image-text alignment.The paper gives complex phrases and relations such as “each other” as examples without clear image mappings.
  • Other Multimodal Methods: Other multimodal methods use interacting image and sentence subnetworks or bidirectional mappings between images and captions.The m-RNN predicts the next caption word, while Chen et al.’s method reconstructs visual features from descriptions.

3.2 Supervised Learning vs. Other Deep Learning

The survey contrasts supervised captioning with reinforcement-learning and GAN-based approaches, emphasizing their data assumptions, architectures, and limitations.

  • Other Deep Learning: Reinforcement learning uses exploration and reward signals, while GANs learn from unlabeled data through generator-discriminator competition.
  • Supervised Learning: Supervised image-captioning methods are grouped into encoder-decoder, compositional, attention-based, semantic concept-based, stylized, novel object-based, and dense captioning categories.
  • Reinforcement Learning: A reinforcement-learning captioning architecture can combine policy and value networks to guide next-word prediction and evaluate possible sequence extensions.
  • Limitations: GAN captioning faces nondifferentiability from discrete text and difficulties estimating future rewards, motivating policy gradients and Monte Carlo rollouts.
  • GAN-Based Captioning: GAN-based captioning can generate diverse captions and, in some methods, multiple captions for one image.

3.3 Dense Captioning vs. Captions for the whole scene

Dense captioning addresses the limits of a single whole-image description by generating descriptions for localized regions, although dense regions create recognition and overlap challenges.

  • Whole-Scene versus Dense Captioning: Whole-scene methods generate one caption for the image, whereas dense captioning generates captions for individual scene regions.
  • Dense Captioning: DenseCap localizes salient image regions and generates descriptions for those regions.
  • Dense Captioning: A typical dense-captioning pipeline proposes regions, extracts region features with CNNs, and uses a language model to caption every region.
  • Motivation: Region-based descriptions are presented as more objective and detailed than a subjective description of the entire visual scene.
  • Challenges: Dense captioning must handle overlapping regions and the difficulty of recognizing each target region across visual concepts.

3.4 Encoder-Decoder Architecture vs. Compositional Architecture

The survey compares simple encoder-decoder captioning with compositional architectures that separately extract concepts, generate candidates, and rerank them using multimodal similarity.

  • Encoder-Decoder Architecture: Simple encoder-decoder methods extract global image features with a CNN and feed them into an LSTM to generate a word sequence.
  • Encoder-Decoder Architecture: NIC uses CNN image representations and an LSTM decoder, whose hidden state tracks objects already described in the text.
  • Encoder-Decoder Architecture: Feeding image information only at the beginning can create vanishing-gradient problems and weaken the influence of early generated words.
  • Compositional Architecture: Compositional methods extract semantic concepts, generate candidate captions, and rerank them with a deep multimodal similarity model.
  • Compositional Architecture: Some compositional systems use visual detectors, language models, and multimodal similarity models, with image subregions mapped to likely caption vocabulary.
  • Compositional Architecture: Compositional variants can generate semantically meaningful descriptions from structural words such as object, attribute, activity, and scene.

3.5 Others

The “Others” group covers attention-based, semantic concept-based, novel object-based, and stylized captioning methods. These approaches extend image captioning by focusing on salient regions, semantic relationships, unseen objects, or personalized context.

  • Attention based Image Captioning: Attention-based methods dynamically focus on salient image regions while generating corresponding caption words.They address whole-scene encoder-decoder limitations by selecting image areas during sequence generation.
  • Attention based Image Captioning: Typical attention pipelines extract whole-image information, generate language, focus on salient regions at each time step, and update captions dynamically.
  • Other methods: Review-based, area-based, semantic-context, and personalized methods incorporate multiple reviews, word-region associations, multiscale regions, or user context.Personalized captioning uses hashtag prediction, post generation, and prior vocabulary or writing style information.
  • Attention based Image Captioning: Adaptive attention with a visual sentinel avoids attending to visual signals for words that do not require them.Unnecessary visual signals may affect caption generation and degrade performance.
  • Attention based Image Captioning: Supervised attention models use alignment annotations or semantic labels to improve mapping between caption words and image regions.Strong supervision directly maps ground-truth words to regions, whereas weak supervision uses bounding boxes or segmentation masks.
  • Attention based Image Captioning: Bottom-up saliency-based attention prioritizes important objects and was reported to perform better on unseen data.

3.6 LSTM vs. Others

This section contrasts recurrent and convolutional language models for image captioning. LSTMs retain long-term information, while CNN-based language models offer hierarchical sentence modeling and faster processing but may require recurrent components for temporal behavior.

  • LSTM: LSTM units use memory cells to maintain information over long periods and have dominated sequence-to-sequence learning.GRUs have similar structures but use fewer gates and no separate memory cells.
  • LSTM: LSTMs ignore hierarchical sentence structure and require substantial storage for long-term dependencies.
  • Others: CNNs can learn internal hierarchical sentence structure and process sequences faster than LSTMs.
  • Others: A CNN language model alone cannot model dynamic temporal language behavior, so one approach combines it with a recurrent network.
  • Others: A CNN+CNN captioning method connects vision and language CNNs through hierarchical attention and investigates layer number and kernel width.The authors report that hyperparameter choices can improve image-captioning performance.

4 DATASETS AND EVALUATION METRICS

The survey reviews commonly used image-captioning datasets and evaluation metrics. It describes dataset scale, annotation structure, and metric behavior, including limitations in correlation with human judgments and optimization difficulty.

  • Datasets: Flickr8k, Flickr30k, and MS COCO are popular datasets, while other datasets broaden scale, language, content, region annotations, or stylistic coverage.
  • Datasets: MS COCO contains more than 300,000 images, more than 2 million instances, 80 object categories, and five captions per image.
  • Datasets: Flickr30k contains 30,000 images and 158,000 human-provided captions without fixed training, validation, and test splits.
  • Datasets: Flickr8k contains 8,000 images divided into 6,000 training, 1,000 development, and 1,000 test images, with five human reference captions per image.
  • Evaluation Metrics: BLEU, ROUGE, METEOR, CIDEr, and SPICE compare generated captions with references using lexical, consensus, or semantic representations.CIDEr uses TF-IDF for consensus, while SPICE uses scene-graph semantic concepts.
  • Evaluation Metrics: BLEU, METEOR, and ROUGE are not well correlated with human quality assessments, whereas SPICE and CIDEr correlate better but are difficult to optimize.

5 COMPARISON ON BENCHMARK DATASETS AND COMMON EVALUATION METRICS

The survey compares reported image-captioning results across benchmark datasets, method groups, and evaluation metrics, while noting that formal experimental evaluation is outside its scope. Performance varies by metric, dataset, and attention or semantic mechanisms.

  • Comparison framework: The survey reports three result sets: common datasets and metrics, attention-based methods, and other deep learning methods, including reinforcement learning and GAN-based approaches.Formal experimental evaluation was outside the paper’s scope.
  • Benchmark datasets: 0.565, 0.386, 0.256, and 0.170 are Mao et al.’s Flickr8k scores for BLEU-1, BLEU-2, BLEU-3, and BLEU-4, respectively.The corresponding Flickr30k scores are 0.600, 0.410, 0.280, and 0.190.
  • Benchmark datasets: MSCOCO produced the highest reported scores, and Jia et al.’s method outperformed Mao et al. using visual-space mapping and an Encoder-Decoder architecture.The survey attributes the higher MSCOCO results to the larger dataset’s broader scene representation and context.
  • Attention-based methods: Attention-based results varied with design: Xu et al.’s stochastic hard attention exceeded deterministic soft attention, while Jin et al.’s scene-specific updates outperformed those results.Wu et al.’s review process and Pedersoli et al.’s region-to-word mapping also achieved high reported performance.
  • Other deep learning methods: Reinforcement learning- and GAN-based methods were grouped as Other Deep Learning-based Image Captioning, although the surveyed methods lacked results on commonly used evaluation metrics.The survey notes that these methods nevertheless have potential for generating image descriptions.
  • Evaluation metrics: Performance differed across metrics: BLEU-n and METEOR favored precision, recall, and word-level similarity, while ROUGE-L, CIDEr, and SPICE assess other caption properties.Zhang et al., Rennie et al., and Lu et al. were reported as strong on adequacy, fluency, saliency, and grammaticality; Gu et al. and Yao et al. performed well semantically.

6 DISCUSSIONS AND FUTURE RESEARCH DIRECTIONS

The survey identifies common deep learning approaches, datasets, and metrics for image captioning, then outlines unresolved challenges and future directions. These include improving object and relationship recognition, language generation, domain coverage, external knowledge use, and learning with less labeled data.

  • Methods: Supervised, reinforcement, and GAN-based methods are commonly used, with supervised methods operating in visual or multimodal space depending on their mapping strategy.Visual-space methods explicitly map images to descriptions, whereas multimodal-space methods use a different mapping arrangement.
  • Methods: Attention-based methods focus on salient image regions and are reported to outperform basic Encoder-Decoder methods, while semantic concept-based methods generate semantically rich captions.Dense captioning methods generate region-based captions.
  • Datasets and metrics: MSCOCO, Flickr30k, and Flickr8k are common datasets with multiple captions per image, while Visual Genome is mainly used for region-based image captioning.The survey describes MSCOCO as a very large dataset.
  • Datasets and metrics: BLEU, ROUGE, METEOR, and SPICE evaluate different caption properties, with SPICE better suited to understanding semantic details than other listed metrics.The survey also describes BLEU for small-sentence evaluation and METEOR for varied caption segments.
  • Challenges and future directions: Generation methods can produce novel captions but still miss prominent objects, attributes, and relationships, while performance depends on syntactically correct and diverse captions.The survey identifies open-domain datasets, external knowledge, unsupervised learning, and reinforcement learning as future directions.

7 CONCLUSIONS

The paper surveys deep learning image-captioning methods through a taxonomy, architectural discussion, dataset and metric review, and brief analysis of reported results. It concludes that robust high-quality captioning for nearly all images remains an open research goal.

  • Survey contributions: The survey reviews deep learning-based image captioning methods, presents a taxonomy and generic block diagrams, and highlights the strengths and weaknesses of major groups.It also discusses evaluation metrics, datasets, reported experimental results, and potential research directions.
  • Conclusion: A robust method that generates high-quality captions for nearly all images has yet to be achieved.The paper expects automatic image captioning to remain active as new deep learning architectures emerge.
Loading 1810.04020v2…