Source-linked AI summary
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, Barbara Plank
TL;DR
Automatic image description requires combining visual understanding with natural language generation, but the literature spans distinct modeling choices and evaluation challenges. This survey organizes approaches into generation and retrieval-oriented categories, reviews models, datasets, and measures, and discusses future directions. It highlights that dataset differences limit transfer and that current automatic measures can disagree with human judgments.
Problem
Automatic image description requires producing concise, grammatically correct language from comprehensive visual understanding, while existing work uses diverse models, datasets, and evaluation measures.
Method
The survey categorizes models into direct generation, visual-space retrieval, and multimodal-space retrieval, then reviews corpora, evaluation measures, and future research directions.
Results
The survey synthesizes existing advances, model differences, image-description corpora, automatic evaluation measures, and future directions for vision and language research.
Takeaways & Limitations
Dataset collection conditions and description quality affect performance transfer, while several automatic measures can rate systems above humans despite human judgments favoring human descriptions.
Takeaways & Limitations
The survey focuses on static images, and multilingual benchmark coverage is limited to IAPR-TC12 among the available datasets.
Abstract
from arXiv · showhide
Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities. In this survey, we classify the existing approaches based on how they conceptualize this problem, viz., models that cast description as either generation problem or as a retrieval problem over a visual or multimodal representational space. We provide a detailed review of existing models, highlighting their advantages and disadvantages. Moreover, we give an overview of the benchmark image datasets and the evaluation measures that have been developed to assess the quality of machine-generated image descriptions. Finally we extrapolate future directions in the area of automatic image description generation.
1. Introduction
Automatic image description brings computer vision and natural language generation together to produce concise, grammatically correct text about salient image content. The survey organizes this literature by model type and reviews datasets, evaluation measures, and future directions.
- Task motivation: Automatic image description analyzes an image and generates a textual description of its salient visual content.Descriptions may cover objects, attributes, scenes, interactions, implied objects, or background knowledge.
- Task motivation: Image description requires both full image understanding and sophisticated natural language generation.The NLG process includes content selection, text planning, and surface realization.
- Survey organization: The survey categorizes models into direct generation, visual-space retrieval, and multimodal-space retrieval approaches.Direct generation first predicts image content and then uses that information to drive language generation; retrieval approaches reuse or synthesize descriptions from similar images.
- Scope: The survey focuses on static images, although video description additionally requires analyzing objects, attributes, and actions over time.
- Survey scope: The survey reviews model families, multimodal datasets, evaluation measures, and future research directions including visual question answering.It aims to provide a comprehensive overview of the image description literature.
2. Image Description Models
The survey organizes image-description models into direct generation, visual-space retrieval, and multimodal-space retrieval, comparing their mechanisms, strengths, and limitations. Direct generation maps predicted visual content into language, while retrieval methods transfer or rank descriptions using visual or joint representations.
- Direct generation models: Direct generation first predicts scenes, objects, attributes, relationships, and actions, then converts these outputs into words or phrases and combines them into descriptions.Generation may use templates, n-grams, or grammar rules.
- Direct generation models: Explicit visual-to-language pipelines constrain descriptions to predefined semantic classes and depend on detector accuracy, which is not always achieved in practice.Detector errors can prevent valid descriptions when important objects or attributes are missed.
- The survey defines three model categories: direct generation, visual-space retrieval with description transfer, and multimodal-space retrieval.These categories structure the overview in Table 1.
- Multimodal-space retrieval: Multimodal approaches map images and sentences into shared representations for retrieval, while newer encoder-decoder models also generate novel descriptions.Kiros et al. encode sentences with an LSTM and images with a deep convolutional network; the decoder uses the shared representation for generation.
- Comparison of existing approaches: The approaches trade off novelty, grammaticality, evaluation, and complexity: generation produces novel sentences but relies on accurate vision and sophisticated NLG, whereas retrieval transfers fluent human-authored descriptions.Multimodal retrieval additionally depends on difficult cross-modal similarity metrics.
3. Datasets and Evaluation
The survey reviews benchmark datasets for automatic image description, distinguishing image descriptions from captions, and examines how generated text is evaluated against human judgments.
- Datasets: Datasets vary in size, description format, collection procedure, and whether texts describe visible content or provide contextual captions.Image descriptions refer to depicted objects, actions, attributes, and scene type, whereas captions can add personal, cultural, or historical information.
- Datasets: The reviewed benchmarks range from Pascal1K's 1,000 images to MS COCO's 123,287 images, with multiple descriptions associated with each image.Pascal1K provides five descriptions per image, and MS COCO provides five different descriptions per image.
- Datasets: Other resources include VLT2K's two-sentence action-focused descriptions, Abstract Scenes' clip-art images, and IAPR-TC12's multilingual descriptions.VLT2K descriptions mention the main action and actors, followed by important background objects; IAPR-TC12 contains 20,000 images.
- Datasets: Flickr8K and Flickr30K contain approximately 8,000 and 30,000 Flickr images, respectively, each with five descriptions collected from Mechanical Turk workers.The images were selected through user queries for specific objects and actions.
- Evaluation: Human evaluation remains common, but automatic measures differ substantially in alignment with human judgments.Meteor was reported as the most robust measure, while BLEU and ROUGE often ranked systems above human text, indicating their unsuitability in that evaluation.
- Evaluation: Recent systems have converged on large datasets such as Flickr8K/30K and MS COCO and on Meteor or CIDEr, although BLEU and human evaluation remain inconsistently used.The survey notes that BLEU remains widespread despite limitations and that human evaluation is not universal.
4. Future Directions
Future work centers on improving datasets, evaluation, description diversity, multilingual resources, and related vision-language tasks. The survey also identifies persistent gaps in cross-domain generalization and alignment with human judgments.
- 4.1 Datasets: Larger, diversified datasets with a shared comprehensive vocabulary are needed to support more complex and transferable description models.Dataset collection processes and description quality significantly affect performance and limit transfer learning across settings.
- 4.1 Datasets: Dataset differences can degrade transfer performance, with MS COCO-trained models showing lower BLEU on SBU1M Captions or Pascal1K despite more training data.The survey attributes this degradation largely to vocabulary and description-quality differences.
- 4.1 Datasets: Reducing dependence on supervised data and leveraging larger unsupervised corpora remain open directions for richer representations and description models.The survey presents lower supervision as a possible trade-off for access to more extensive unsupervised data.
- 4.2 Measures: Automatic evaluation measures should better mimic human judgments because existing measures can rank systems above the human upper bound while human evaluation often disagrees.Figure 7 compares BLEU, Meteor, ROUGE, and CIDEr score densities against human judgments on Flickr8K.
- 4.3 and 4.4 Further Directions: Future systems should generate more diverse descriptions, investigate image specificity, and develop multilingual repositories and cross-lingual multimodal models.Current systems often repeat descriptions; only IAPR-TC12 among the cited benchmark datasets provides multilingual descriptions.
- 4.4 Further Tasks: Visual question-answering and other multilingual multimodal tasks are identified as additional open challenges for future vision-language research.The survey reviews emerging VQA datasets and methods alongside proposed multilingual research directions.
5. Conclusions
The survey organizes automatic image description research across generation and retrieval paradigms while reviewing datasets and evaluation measures. It concludes that descriptions are more human-like than keyword annotations, but current systems and metrics remain below human-level reliability.
- 5. Conclusions: The survey categorizes research into direct generation, visual-space retrieval, and multimodal-space retrieval, while reviewing corpora, evaluation measures, and future directions.These categories reflect whether descriptions are generated directly or retrieved through visual or joint visual-linguistic representations.
- 5. Conclusions: Automatic descriptions provide more human-like and complete explanations of visual content than traditional keyword-based image annotation.The survey connects this richer representation of scenes with the possibility of grounded interaction and inference.
- 5. Conclusions: Experimental results indicate that image-description systems still fall short of human performance, while current automatic measures correlate insufficiently with human judgments.The conclusion identifies evaluation measures capable of handling the task's complexity as an ongoing need.