Source-linked AI summary
A Hierarchical Approach for Generating Descriptive Image Paragraphs
Jonathan Krause, Justin Johnson, Ranjay Krishna, Li Fei-Fei
TL;DR
Single-sentence captions provide limited detail, while dense captioning produces detailed but incoherent region descriptions. This paper generates unified image paragraphs by decomposing images into semantic regions and language into sentences with hierarchical recurrent modeling, demonstrating advantages over traditional captioning and region-level knowledge transfer.
Problem
Single-sentence image captions have limited detail, whereas dense captioning lacks coherent, cohesive descriptions of entire images.
Method
The model detects semantically meaningful image regions and uses hierarchical recurrent networks to decompose paragraphs into sentences and generate their words.
Results
Experiments demonstrate advantages over traditional image-captioning methods and effective transfer of region-level knowledge to paragraph captioning.
Takeaways & Limitations
Paragraph generation provides coherent natural-language descriptions with fine-grained detail and supports interpretable generation from subsets of image regions.
Takeaways & Limitations
SPICE is poorly suited to long paragraph descriptions because it does not handle coreference or distinguish instances of the same object category.
Abstract
from arXiv · showhide
Recent progress on image captioning has made it possible to generate novel sentences describing images in natural language, but compressing an image into a single sentence can describe visual content in only coarse detail. While one new captioning approach, dense captioning, can potentially describe images in finer levels of detail by captioning many regions within an image, it in turn is unable to produce a coherent story for an image. In this paper we overcome these limitations by generating entire paragraphs for describing images, which can tell detailed, unified stories. We develop a model that decomposes both images and paragraphs into their constituent parts, detecting semantic regions in images and using a hierarchical recurrent neural network to reason about language. Linguistic analysis confirms the complexity of the paragraph generation task, and thorough experiments on a new dataset of image and paragraph pairs demonstrate the effectiveness of our approach.
1. Introduction
Single-sentence captioning is limited in detail, while dense captioning offers fine-grained regional descriptions without coherence. The paper introduces paragraph generation to combine detailed visual coverage with unified language using hierarchical modeling.
- Single-sentence captions impose a fundamental upper bound on the quantity and quality of image information they can convey.
- Dense captioning detects regions of interest and describes each with a short phrase, increasing image detail.
- Dense-captioning descriptions are not coherent because they do not form a cohesive whole describing the entire image.
- Paragraph generation combines coherent natural-language description with the fine-grained detail of regional captioning.
- The proposed model detects semantically meaningful image regions and uses a hierarchical recurrent neural network to reason over paragraphs decomposed into sentences.
- The paper validates the approach with a new image–paragraph dataset, linguistic analysis, and comparisons against numerous baselines.
2. Related Work
Prior image-captioning work ranges from retrieval and template generation to region-aware and hierarchical recurrent models. The paper’s related work motivates hierarchical architectures for long-term, multi-sentence language structure.
- Image captioning research includes retrieval-based methods and direct generation using templates or recurrent neural network language models.
- Region-aware approaches use attention, ranking, or dense captioning to connect image regions with textual descriptions.
- Dense captioning detects and describes regions of interest, but its descriptions are independent rather than paragraph-level discourse.
- Video captioning has explored multiple sentences, whereas image captioning lacks strong temporal dependencies and therefore faces a different challenge.
- Hierarchical recurrent networks address long-term linguistic structure by assigning different parts of the model to different time scales.
- Language-mirroring hierarchical architectures model sentences and words with distinct recurrent units, extending hierarchical modeling beyond fixed update schedules.
3. Paragraphs are Different
The authors collected a 19,551-image paragraph dataset and compared its descriptions with sentence-level captions. Paragraphs are substantially longer and more diverse, and they contain linguistic patterns associated with describing relationships and properties.
- The dataset contains paragraph annotations for 19,551 MS COCO and Visual Genome images, collected from qualified U.S. Mechanical Turk workers with quality checks.
- Each paragraph is roughly six times as long as an average sentence caption, while its individual sentences have comparable length.
- Paragraph sentences have a diversity score of 70.49, compared with 19.01 for sentence captions.
- Paragraphs contain somewhat more verbs and pronouns, comparable adjective frequency, and somewhat fewer nouns than sentence captions.
- Paragraphs also exhibit more complex phenomena such as coreference, with references spanning multiple sentences and entities.
4. Method
The model generates image paragraphs by decomposing images into semantic regions and paragraphs into sentences and words. A hierarchical recurrent architecture coordinates sentence planning, word generation, and paragraph termination.
- 4.3. Hierarchical Recurrent Network: Hierarchical modeling shortens the dependencies each recurrent module must learn: paragraphs average 5.7 sentences and 67.5 words, with sentences averaging 11.9 words.The architecture separates paragraph-level sentence planning from sentence-level word generation.
- 4.3. Hierarchical Recurrent Network: A sentence RNN determines the paragraph’s number of sentences and produces one topic vector per sentence for the word RNN.It predicts CONTINUE or STOP for each sentence using a halting distribution.
- 4.3. Hierarchical Recurrent Network: A two-layer word RNN generates each sentence from its topic vector, predicting words until an END token, after which generated sentences are concatenated into a paragraph.The word RNN uses learned word embeddings after receiving the topic vector and START token.
- 4.4. Training and Sampling: Training minimizes a weighted sum of sentence-level cross-entropy for stopping decisions and word-level cross-entropy for generated words.During training, the sentence RNN runs for S steps and each word RNN runs for the corresponding sentence length Ni.
- Transfer Learning: The method uses transfer learning by fixing a dense-captioning region detector and initializing word-RNN parameters from a region-level captioning language model.The stated motivations are efficiency and preventing overfitting for the fixed detector.
5. Experiments
Experiments compare region- and image-based neural models, concatenation baselines, templates, and humans using automatic metrics and qualitative and linguistic analyses. Regions-Hierarchical performs best on nearly every metric and produces more coherent, varied descriptions, though automatic outputs remain less diverse than human paragraphs and SPICE is unsuitable for this task.
- Experimental setup: The evaluation uses 14,575 training, 2,487 validation, and 2,489 testing images, with six language metrics for paragraph generation.The metrics are CIDEr, METEOR, and BLEU-1 through BLEU-4.
- Baselines: Baselines span sentence concatenation, flat image models, template generation, DenseCap concatenation, and flat region models with or without language-model pre-training.The baselines isolate effects of region representations, pre-training, captioning, detection, and heuristic recombination.
- Main results: Sentence-Concat performs worst across all metrics, while Template is competitive on CIDEr, METEOR, and BLEU-1 but weaker on BLEU-3 and BLEU-4.Template outputs benefit from region-level captions but struggle to connect regions into coherent text.
- Main results: Regions-Flat-Scratch improves over Image-Flat on every metric, pre-training improves all metrics, and Regions-Hierarchical scores highest except on BLEU-4.Regions-Flat-Pretrained is mildly superior on BLEU-4, possibly because its non-hierarchical structure reproduces sentence-boundary words more exactly.
- Qualitative results: Qualitatively, Regions-Hierarchical captures coreference, object relationships, high-level-to-background description order, and more relevant language than the baselines.Sentence-Concat is repetitive and often inaccurate, whereas Template is largely accurate but uninteresting and insensitive to importance.
- Paragraph language analysis: The hierarchical model matches human descriptions in average length and variance, generates more pronouns and a wider vocabulary than Template, but all automatic methods remain less diverse than humans.Sentence-Concat is least diverse, while Template's diversity may reflect its fixed sequential region description strategy.
6. Conclusion
The paper introduces long, descriptive image paragraphs and a hierarchical generation approach that leverages the compositional structure of images and language.
- The paper introduces image paragraph generation as a task for producing long, descriptive descriptions.
- The proposed hierarchical approach leverages the compositional structure of both images and language.
- Experiments show advantages over traditional image captioning methods and effective transfer of region-level knowledge to paragraph captioning.
- The model improves interpretability by generating descriptive paragraphs from only a subset of image regions.