Source-linked AI summary
Good News, Everyone! Context driven entity-aware captioning for news images
Ali Furkan Biten, Lluis Gomez, Marçal Rusiñol, Dimosthenis Karatzas
TL;DR
Existing image captioning systems largely remain descriptive and struggle to integrate contextual knowledge and out-of-vocabulary named entities. The paper addresses this with a news-article-aware, attention-based two-stage captioning pipeline and introduces GoodNews, reporting state-of-the-art results. The authors also identify substantial task complexity and limitations in current evaluation metrics.
Problem
Current captioning systems mainly describe visible content and lack mechanisms for integrating contextual knowledge and out-of-vocabulary named entities.
Method
The paper uses visual-guided attention over news articles and a two-stage end-to-end process that inserts article entities into generated template captions.
Results
The proposed pipeline achieves state-of-the-art results on news image captioning while improving contextual information use and named-entity treatment.
Takeaways & Limitations
GoodNews and the proposed pipeline support captioning that incorporates article context and named entities beyond standard visual-only captioning.
Takeaways & Limitations
Overall results remain low relative to simpler datasets, reflecting the task's complexity and limitations of current evaluation metrics.
Abstract
from arXiv · showhide
Current image captioning systems perform at a merely descriptive level, essentially enumerating the objects in the scene and their relations. Humans, on the contrary, interpret images by integrating several sources of prior knowledge of the world. In this work, we aim to take a step closer to producing captions that offer a plausible interpretation of the scene, by integrating such contextual information into the captioning pipeline. For this we focus on the captioning of images used to illustrate news articles. We propose a novel captioning method that is able to leverage contextual information provided by the text of news articles associated with an image. Our model is able to selectively draw information from the article guided by visual cues, and to dynamically extend the output dictionary to out-of-vocabulary named entities that appear in the context source. Furthermore we introduce `GoodNews', the largest news image captioning dataset in the literature and demonstrate state-of-the-art results.
1. Introduction
The paper targets image captions that move beyond visual description toward scene interpretation by using news-article context. It introduces a context-aware pipeline, GoodNews dataset, and two-stage handling of out-of-vocabulary named entities.
- Current captioning systems generally describe or enumerate visible content rather than integrating prior world knowledge.
- News images provide article text, metadata, expert captions, and large-scale data suitable for contextualized captioning.
- GoodNews contains more than 466,000 image-caption pairs with associated metadata and is introduced as the largest news-captioning dataset in the literature.
- The proposed model selectively draws article information using visual cues to produce plausible scene interpretations.
- A two-stage end-to-end process generates template captions and substitutes named-entity placeholders with matching article entities, including out-of-vocabulary words.
- The paper reports state-of-the-art results while finding benefits from contextual information and proper named-entity treatment, alongside difficulty for current metrics.
2. Related Work
Prior work explored richer caption styles and news-specific contextual captioning, but existing context-driven methods did not address named-entity inclusion with open vocabularies.
- Recent captioning research broadened writing style through heterogeneous annotations and romance-novel text, while content generally remained visual description.
- Earlier news-captioning studies used article sentences or templates, with datasets collected from BBC News and TIME magazine.
- BreakingNews combined word2vec article representations with CNN image features and a generative LSTM.
- Existing context-driven methods used closed dictionaries and did not address named-entity inclusion, although entities often become available only at test time.
- Generic captioning work separately explored celebrity, landmark, and named-entity generation using recognition or template-filling strategies.
3. The GoodNews Dataset
GoodNews is a large news-captioning dataset built from New York Times articles, with longer, entity-rich journalist captions than generic captioning benchmarks. Its single caption per image and narrower metadata create important comparison boundaries.
- GoodNews was assembled from New York Times URLs covering 2010–2018, with released scripts supporting broader historical retrieval.
- GoodNews captions are longer than captions in generic datasets such as MSCOCO, indicating more descriptive news-caption text.
- GoodNews provides one ground-truth caption per image, whereas MSCOCO provides five, although GoodNews captions were written by expert journalists.
- Named entities make up 20% of GoodNews caption words, and 95% of caption sentences contain at least one named entity.
- GoodNews is about five times larger than BreakingNews and spans a wider range of events and stories across a longer period.
- GoodNews covers news image captioning but provides less metadata than BreakingNews, which targets more tasks.
4. Model
The model combines image and article representations in a two-stage pipeline: it first generates template captions, then inserts contextually selected named entities. Sentence-level article encoding and attention guide both caption generation and entity insertion.
- The two-stage architecture generates a template caption with named-entity placeholders, then fills those placeholders using named entities selected from the associated article.The model uses the image and article as input streams, with attention over article text guiding entity selection.
- Template Caption Generation: Template captions are generated autoregressively from previously produced words and attended image features, using cross-entropy training.The captioning formulation follows attention-guided encoder-decoder systems, with a word produced at each timestep.
- Article Encoding Methods: Sentence-level article encoding preserves domain, purpose, and context while avoiding the higher dimensionality and weaker contextual flow of word-level encoding.The article is represented as sentence features derived from GloVe word vectors using simple averages, weighted averages, or the TBB transformation.
- Article Encoding with Attention: Attention computes a timestep-specific weighting over article sentence features, allowing the model to focus on different sentences during caption generation.The attended article features are obtained by multiplying sentence-level features by the attention vector.
- Named Entity Insertion: Named entities are inserted by category after template generation, using random selection, sentence-similarity ranking, or the model’s attention vector.AttIns uses the attention vector produced during template generation and does not require an external insertion method.
- The implementation uses ResNet-152 image features, a 512-dimensional LSTM, a 35K-word vocabulary, Adam optimization, and beam size 1.Images are resized to 256 × 256 and randomly cropped to 224 × 224; articles longer than 55 sentences are compressed at the final position.
5. Experiments
Experiments evaluate template and full caption generation, named entity insertion, qualitative outputs, and human preferences on GoodNews. The visual-textual model improves final captions and named entity insertion, while results remain substantially below human performance.
- News Image Captioning: Experiments compare visual-only and visual-textual models on template caption generation and full news-image captioning.The evaluation includes standard metrics, named entity insertion, qualitative comparisons, and human judgments.
- News Image Captioning: The two-stream visual-textual architecture performs on par with baseline results on template caption generation.Averaging GloVe sentence descriptors slightly outperforms weighted averaging and the tough-to-beat baseline in this intermediate task.
- News Image Captioning: The visual-textual architecture consistently outperforms visual-only pipelines on every full-caption metric.The comparison includes random insertion, GloVe-based context insertion, and attention-based insertion; systems without the two-stage formulation perform poorly.
- Limitations: Overall results remain low compared with simpler datasets such as MSCOCO, indicating the complexity of news-image captioning and limitations of current approaches.Human-performance estimates were obtained from two subjects captioning a subset of 20 images.
- Named Entity Insertion: AttIns clearly outperforms CtxIns for named entity insertion under both exact-match and partial-match evaluation.Exact match requires character-level agreement, whereas partial match accepts token-level agreement.
6. Conclusion
The paper presents a context-driven news-image captioning pipeline that integrates article text and handles out-of-dictionary named entities. It also introduces GoodNews and reports state-of-the-art performance.
- The pipeline uses a news article as contextual input and applies attention to selectively draw information guided by the image.This targets captions that offer a plausible interpretation of the scene.
- A two-stage end-to-end procedure incorporates named entities that are unavailable in the output dictionary at test time.
- GoodNews contains 466K samples and is presented as the largest news-captioning dataset to date.
- Experimental results show state-of-the-art performance while satisfactorily incorporating named-entity information into captions.