Transform and Tell: Entity-Aware News Image Captioning

1. Introduction

• Generic captioning systems often miss names and places, produce commonsense-inconsistent captions, and struggle with rare words because of fixed vocabularies.Human captions tend to be longer and more diverse than generated captions.Source: §1 · p. 1
• Transform and Tell addresses entity knowledge with multi-head attention over article words, faces, and objects, and addresses rare-word generation with byte-pair encoding.The model also uses dynamic convolutions and mixes linguistic representation layers.Source: §1 · pp. 1–2
• 6.8× BLEU-4 (0.89 → 6.05) and 4.1× CIDEr (13.1 → 53.8) are the reported gains over previous work [3].The authors also report consistent gains from each new component across both datasets.Source: §1 · p. 2
• NYTimes800k, the largest news image captioning dataset to date, contains 445K articles and 793K images spanning 14 years; it has 70% more articles than GoodNews and includes image locations within article text.Source: §1 · p. 2

Related work

• Generic image captioning commonly paired CNN image encoders with closed-vocabulary RNN decoders; later work added image-region, object-region, or visual-concept attention [19] [10] [49] [53] [30] [1] [55].Source: §2 · p. 2
• News-captioning methods use article text to help generate entity names, but prior end-to-end LSTM systems could not produce names absent from training [37].Other approaches extracted phrases or retrieved representative article sentences.Source: §2 · p. 2
• Template-based methods place entity placeholders and fill them using knowledge graphs or sentence similarity [3] [29], whereas Transform and Tell generates named entities directly without an intermediate template.Source: §2 · p. 2
• Byte-pair encoding represents uncommon words as subword units, enabling open-vocabulary transformer generation; this work examines its use for rare names against template-based methods [41] [57].Prior image-captioning work using BPE did not use it for rare named entities because those entities were removed during preprocessing.Source: §2 · p. 2
• Unlike work that supplies extracted web-entity labels as inputs [57], Transform and Tell automatically identifies relevant entities from the news article.Source: §2 · p. 2

3. The Transform and Tell Model

• The model uses pretrained encoders for image, face, object, and article representations, which its decoder attends over to generate subword-level captions.Source: §3 · p. 2, Figure 2 · p. 3

3.1. Encoders

• Image encoding yields 49 ResNet-152 patch vectors; separate encoders represent detected faces and objects, while RoBERTa supplies contextual article embeddings and byte-pair encoding can represent any word made from Unicode characters.The model selects up to four faces; object detections are filtered by confidence and capped at 64.Source: §3.1 · pp. 3–4

3.2. Decoder

• At each step, the decoder conditions on previously generated tokens and encoder contexts, using dynamic convolutions for token history and multi-head attention over image, article, face, and object representations.The final transformer-block output estimates the next-token probability; adaptive softmax groups tokens by frequency to reduce computation.Source: §3.2 · pp. 4–5

• The GoodNews examples pair an article, image, and caption; 463K of its 466K image URLs were downloadable, and caption-level splits can share articles across sets.The split contains 421K training, 18K validation, and 23K test captions.Source: §4 · p. 5
• GoodNews can omit important opening paragraphs and include non-English articles or sidebar images unrelated to the main article.These issues motivated NYTimes800k’s custom parser and filtering.Source: §4 · p. 5
• 70% larger, NYTimes800k averages 963-word articles versus 451 in GoodNews and contains only English articles and images from the main article.It also records image locations; 39% of articles have at least one additional image within the text.Source: §4 · p. 5
• In NYTimes800k, 97% of captions contain a named entity, 71% of training images contain a face, and 68% of training captions mention a person’s name.The paper reports that 99% of captions contain at most four names.Source: §4 · p. 5
• A time-based split supports evaluation on novel events and names; 4% of the 100K proper nouns in test captions never appear in training captions.This differs from GoodNews’s random split.Source: §4 · p. 6

5.3. Baselines and Model Variants

• The variants compare decoder and text-encoder choices and add image, face, object, and location context; GoodNews models use its original random split.The compared models have similar parameter counts except those with additional face- or object-attention modules.Source: §5.3 · p. 6, Table 3 · p. 7

5.4. Results and Discussion

• CIDEr reaches 53.8 for the full model on GoodNews, versus 13.1 for the previous state of the art [3], which it outperforms across all reported evaluation metrics.Table 3 reports results on GoodNews and NYTimes800k, including BLEU-4, ROUGE, CIDEr, and precision and recall for named entities, people’s names, and rare proper nouns.Source: Table 3 · p. 7, §5.4 · p. 7
• CIDEr rises from 13.1 to 13.9 when the basic LSTM uses BPE instead of template generation and filling.The authors describe BPE as a competitive end-to-end alternative to template filling.Source: §5.4 · p. 7
• On NYTimes800k, the worst RoBERTa model achieves 7.47% precision and 9.50% recall for rare proper nouns, with a 12.8-point CIDEr gap over the model without RoBERTa.GloVe-based article encoders skip words outside their fixed vocabulary and have zero precision and recall for rare proper nouns.Source: §5.4 · p. 7
• On NYTimes800k, replacing the LSTM with a Transformer raises CIDEr from 24.9 to 40.3.The paper reports that the Transformer improves all evaluation metrics relative to the LSTM.Source: §5.4 · p. 7
• Face attention improves precision and recall for people’s names, while object attention improves most metrics, especially on NYTimes800k.The authors suggest specialized vision models for objects such as organizational logos or landmarks as a future direction.Source: §5.4 · pp. 7–8
• On NYTimes800k, focusing article context around the image raises CIDEr from 45.1 to 52.8; this location information is available only in that dataset.The authors note a strong correspondence between an image and nearby article text that can be exploited for captioning.Source: §5.4 · p. 8
• Captions from the model average 15 words, closer to the ground-truth average of 18 than the previous state of the art’s 10; their FRE-measured complexity is also closer to ground truth.The supplementary material reports caption length, type-token ratio, and Flesch reading ease.Source: §5.4 · p. 8

• The model sets a new state of the art for news image captioning.The authors describe it as a combination of techniques from multiple machine-learning subfields.Source: §6 · p. 8
• Byte-pair encoding, contextualized article embeddings, specialized face and object encoding, and transformer-based generation are its key modeling components.Source: §6 · p. 8
• The authors frame the approach as a promising step for image description with contextual knowledge in web pages, social media feeds, or medical documents.Future directions include broader entity-specific visual models, extending image context to linked or recent articles, and applying similar techniques to other domains.Source: §6 · p. 8

7.1. Live Demo

• The live demo accepts a New York Times article URL, scrapes the page to extract article text and an image, then generates a caption.Source: §7.1 · p. 12

7.2. Entity Distribution

• The NYTimes800k training captions contain four prominent entity types—people, geopolitical entities, organizations, and dates—with people’s names comprising one-third of named entities.Source: §7.2 · p. 12, Figure 5 · p. 12

7.3. Model Complexity

• The model variants have 148–159 million trainable parameters, except face attention at 171 million and object attention at 200 million because both add multi-head attention modules.Source: §7.3 · p. 12, Table 4 · p. 12

7.4. Further Experimental Results

• Table 5 reports BLEU, ROUGE, METEOR, and CIDEr on both datasets; the text notes strong correlation among the metrics and says CIDEr emphasizes uncommon words such as entity names.Source: §7.4 · p. 12, Table 5 · p. 13
• The supplementary analyses report that captions are closer to ground truth in length and Flesch-measured complexity, though gaps remain in length, Flesch score, and type-token ratio.The material also reports entity precision and recall, including for new proper nouns absent from both training captions and article text.Source: §7.4 · p. 12, Table 6 · p. 14
• Table 7 reports precision and recall for geopolitical entities, organizations, and dates on GoodNews and NYTimes800k.Source: Table 7 · p. 15
• The caption examples show errors in spelling unseen names and interpreting article context, while object attention correctly identifies Windsor, reporters, and a pharmacy in one example.Source: Figure 6 · p. 16, Figure 7 · p. 17