Source-linked AI summary
Visual News: Benchmark and Challenges in News Image Captioning
Fuxiao Liu, Yinghan Wang, Tianlu Wang, Vicente Ordonez
TL;DR
News image captioning needs to generate specific, entity-rich descriptions from images and associated articles. The paper introduces a large, diverse benchmark and an entity-aware Transformer model that combines visual and textual information, achieving strong results across three datasets while revealing continued difficulty from dataset diversity.
Problem
News captions contain specific people, places, organizations, and events, but standard captioning resources and methods are less suited to this entity-rich setting.
Method
The paper builds Visual News and proposes an entity-aware Transformer that fuses localized visual features with article words and named entities.
Results
Visual News Captioner outperforms state-of-the-art methods on all 6 metrics on GoodNews and NYTimes800k, while improving Visual News CIDEr from 13.2 to 50.5 over baselines.
Takeaways & Limitations
Tag Cleaning improves Visual News CIDEr by 1.3% and helps retrieve uncommon named entities, while raw-caption end-to-end training retains richer contextual information than template-based methods.
Takeaways & Limitations
Visual News is more challenging because its multiple agencies cover more topics and use more diverse language styles.
Abstract
from arXiv · showhide
We propose Visual News Captioner, an entity-aware model for the task of news image captioning. We also introduce Visual News, a large-scale benchmark consisting of more than one million news images along with associated news articles, image captions, author information, and other metadata. Unlike the standard image captioning task, news images depict situations where people, locations, and events are of paramount importance. Our proposed method can effectively combine visual and textual features to generate captions with richer information such as events and entities. More specifically, built upon the Transformer architecture, our model is further equipped with novel multi-modal feature fusion techniques and attention mechanisms, which are designed to generate named entities more accurately. Our method utilizes much fewer parameters while achieving slightly better prediction results than competing methods. Our larger and more diverse Visual News dataset further highlights the remaining challenges in captioning news images.
1 Introduction
News image captioning requires captions that identify specific people, places, organizations, and events rather than only generic visual content. The paper introduces Visual News and an entity-aware Visual News Captioner to address this gap using both articles and images.
- Motivation: COCO captions are generally descriptive and generic, whereas news captions capture more specific situations and named entities.News captions can explain the higher-level event depicted rather than merely listing visible objects.
- Dataset and task: Visual News contains more than one million news images paired with captions, articles, and additional metadata from diverse English-language news sources.The dataset includes author information and other metadata, supporting research on news image captioning.
- Dataset and task: News Image Captioning generates captions from both input images and their corresponding news articles.The proposed model attends to article word tokens, named entities, and localized visual features.
- Method: Visual News Captioner adapts a Transformer with multi-modal fusion, attention mechanisms, relative position encoding, and a pointer-generator decoder for named entities.A tag-cleaning step is also introduced to address long-tail and out-of-vocabulary words.
- Method: Template-based approaches may lose contextual clues carried by named entities during their first stage.The paper therefore benchmarks both template-based and end-to-end methods on large-scale news datasets.
2 Related Work
Prior image-captioning work largely focuses on generic visual descriptions, while news captioning must incorporate article context and named entities. Existing news methods include article-aware, external-knowledge, template-based, and end-to-end approaches.
- Image captioning: Earlier image-captioning systems commonly combine visual encoders with recurrent decoders and increasingly use attention over image regions or semantic concepts.These developments were enabled by progress in visual representation learning and text generation.
- Benchmarking: The benchmark reports dataset statistics involving the prevalence of named entities in sentences and words, plus overlap among entities from different agencies.The table definitions identify PERSON, GPE, ORG, and DATE as frequent entity types.
- News image captioning: News image captioning is difficult because captions frequently contain named entities that convey information about the events shown.Prior systems select article sentences, concatenate article and visual features, retrieve entities through templates, or integrate external knowledge.
- News image captioning: Template-based methods replace entities with type tags before generating captions and later retrieve entities from articles or auxiliary data.This reduces the generator vocabulary but separates caption generation from entity selection.
- Benchmarking: Agency-specific differences are evaluated through entity distributions and cross-agency captioning performance.These comparisons motivate models that can handle varied news styles and source distributions.
3 Our Visual News Dataset
Visual News is a large and diverse dataset of news articles, images, captions, and metadata collected from four agencies. Its source variation produces distinct entity distributions, language styles, and cross-agency transfer difficulty.
- Dataset composition: Visual News contains over one million images and more than 600,000 articles from The Guardian, BBC, USA Today, and The Washington Post.Examples are filtered by image size and caption length, while metadata includes titles, authors, and geolocation.
- Dataset diversity: Caption length and named-entity frequency vary substantially across agencies, indicating different captioning styles.For example, BBC captions average 14.2 words, whereas Guardian captions average 22.5 words.
- Dataset diversity: Named-entity overlap is uneven across agencies: BBC shares more entities with The Guardian than with USA Today or The Washington Post.USA Today shares more same-type entities with The Washington Post.
- Cross-agency transfer: A model trained on USA Today reaches a CIDEr score of 3.7 on its own test set but 0.6 on The Guardian test set.The cross-agency gap demonstrates that Visual News contains heterogeneous and challenging distributions.
4 Method
Visual News Captioner encodes news articles, named entities, and localized image features, then decodes captions through multi-modal attention and pointer-based copying. Its design also addresses long articles, relative word positions, and out-of-vocabulary entities through truncation, position modeling, and Tag-Cleaning.
- Text Encoder: The text encoder processes the first 300 article tokens and separately extracts named entities using spaCy before encoding both with the same encoder.The first-token limit addresses the potentially long associated articles.
- Text Encoder: Position embeddings are passed through an LSTM and combined with word embeddings to model relative word-position relationships.The hidden-state size H is set to 512.
- Multi-Modal Fusion: The Visual Selective Layer updates textual embeddings with a visual information gate to capture contextual information jointly from articles and images.This addresses prior approaches that separately encoded the image and article.
- Decoder: The decoder attends to previous tokens, image features, article tokens, and named entities through masked and multi-modal Attention on Attention layers.The decoder then uses a pointer-generator with article and entity sources, while Tag-Cleaning replaces predicted entity tags with frequent same-category entities during testing.
- Decoder: The pointer-generator combines vocabulary generation with copying from article-token and named-entity attention distributions to reduce missing rare entities.Out-of-vocabulary named entities are replaced with category tags such as LOC_ and later restored during testing.
5 Experiments
Experiments evaluate Visual News Captioner across three news-image datasets using standard captioning and named-entity metrics. The model outperforms competing methods, uses fewer parameters, and benefits from entity guidance, visual selection, pointer generation, and tag cleaning.
- Experimental Setup: Experiments use GoodNews, NYTimes800k, and Visual News, with BLEU-4, METEOR, ROUGE, CIDEr, named-entity precision, and named-entity recall.Visual News uses 400,000 training, 40,000 validation, and 40,000 test samples evenly sampled from four news agencies.
- Results: Visual News Captioner outperforms state-of-the-art methods on all 6 metrics on both GoodNews and NYTimes800k.The comparison includes multiple article-only, image-only, template-based, and transformer-based baselines.
- Results: 13.2 to 50.5 in CIDEr score: Visual News Captioner outperforms baseline methods by a large margin on the Visual News dataset.The paper also reports that the final model outperforms Transform and Tell with much fewer parameters.
- Ablation and Analysis: Entity-Guide improves results across all datasets, while the Visual Selective Layer improves caption generation by providing extra visual context to text features.The pointer-generator mechanism also connects the final token distribution with the Multi-Modal AoA Layer.
- Ablation and Analysis: 1.3% CIDEr improvement: Tag-Cleaning retrieves uncommon named entities on the Visual News dataset, and the model generates more accurate named entities.Tag-Cleaning addresses the out-of-vocabulary problem through the model’s entity-handling pipeline.
- Discussion: Models trained directly on raw captions achieve the best performances, whereas template-based methods lose contextual information from uncommon named entities.The more diverse Visual News dataset is harder than GoodNews and NYTimes800k because it spans multiple agencies, topics, and language styles.
6 Conclusion and Future Work
The paper presents Visual News Captioner as an entity-aware method validated across three datasets, alongside Visual News, a large-scale benchmark for news image captioning. Experiments report stronger performance with fewer parameters, while generated examples show more accurate and complete named-entity prediction than a competing method.
- Contributions: Visual News Captioner is an entity-aware captioning method leveraging visual and textual information, including named entities and localized visual features.The model attends to individual word tokens, named entities, and localized visual features in news articles and images.
- Evaluation: The method was validated on three datasets through extensive experiments.
- Dataset: Visual News contains over one million images with accompanying articles, captions, and other metadata.The dataset is introduced as the largest news image captioning dataset.
- Results: Generated examples indicate more accurate and complete named-entity prediction than the competing method.Correct named entities are highlighted in the comparison examples.
- Results: The model outperforms state-of-the-art methods across multiple metrics while using fewer parameters.
- Future Work: The dataset may support related NLP tasks such as abstractive text summarization and fake news detection.