PaperlayerView exact revision on arXiv

Source-linked AI summary

TextCaps: a Dataset for Image Captioning with Reading Comprehension

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, Amanpreet Singh

arXiv:2003.12462v2cs.CVcs.CL

TL;DR

Existing image captioning and OCR approaches do not jointly include and interpret written text in image descriptions. The paper introduces TextCaps and adapts a reading-capable model for this task, finding promising performance but a substantial remaining gap to human captions.

  • Problem

    Current captioning models focus on visual objects and fail to recognize and reason about text in scenes, despite text being important for understanding images.

  • Method

    The paper collects TextCaps and adapts the M4C multimodal transformer with dynamic pointer decoding to generate captions using visual content and OCR tokens.

  • Results

    M4C adapted to TextCaps generates impressive captions involving multiple copied OCR tokens, while human evaluation confirms automatic results and finds humans significantly better.

  • Takeaways & Limitations

    TextCaps establishes image captioning with reading comprehension as a task requiring semantic, visual, and spatial reasoning over text and objects.

  • Takeaways & Limitations

    Model predictions still contain scene-understanding, OCR-placement, repetition, and OCR-detection errors, leaving room for better semantic and reasoning capabilities.

Abstract

from arXiv · show

Image descriptions can help visually impaired people to quickly understand the image content. While we made significant progress in automatically describing images and optical character recognition, current approaches are unable to include written text in their descriptions, although text is omnipresent in human environments and frequently critical to understand our surroundings. To study how to comprehend text in the context of an image we collect a novel dataset, TextCaps, with 145k captions for 28k images. Our dataset challenges a model to recognize text, relate it to its visual context, and decide what part of the text to copy or paraphrase, requiring spatial, semantic, and visual reasoning between multiple text tokens and visual entities, such as objects. We study baselines and adapt existing approaches to this new task, which we refer to as image captioning with reading comprehension. Our analysis with automatic and human studies shows that our new TextCaps dataset provides many new technical challenges over previous datasets.

1 Introduction

Image captioning with reading comprehension requires models to read OCR tokens, connect them to visual context, and generate coherent descriptions. TextCaps is introduced to study these capabilities and exposes challenges that existing captioning models do not handle.

  • Motivation: Text in images can be critical for understanding scenes, especially for visually impaired users seeking descriptions or visual dialogue.In VizWiz, 21% of questions from visually impaired people concerned image text.
  • Research gap: Current captioning models focus on visual objects while OCR systems focus on text, leaving models unable to recognize and comprehend scene text jointly.OCR tokens also differ from vocabulary tokens because they often cannot be predicted from preceding context and must be selected from the scene.
  • Technical challenges: Successful systems must relate OCR tokens to one another and to visual context using semantic, spatial, and visual information.They must decide which text to mention, which tokens to join, and how tokens relate to objects and scene context.
  • Technical challenges: The task also requires switching between vocabulary words and OCR tokens, handling zero-shot text, and paraphrasing or inferring from OCR.These capabilities are presented as additional challenges beyond conventional captioning.
  • Findings: Standard captioning models fail on TextCaps, whereas adapting M4C produces encouraging results, while human evaluations still show a significant gap between the best model and people.The analysis identifies TextCaps as an avenue for further image-captioning research.

2 Related work

Prior captioning datasets generally emphasize prominent objects or remove specific text, whereas TextCaps treats scene text as an additional modality for descriptive generation. Text-oriented VQA datasets require reading and reasoning but are not directly designed for generating captions.

  • Image Captioning: Flickr30k and COCO Captions were collected through crowdsourcing, with COCO becoming a major training base for state-of-the-art captioning algorithms.COCO contains 995,684 captions for 164,062 images.
  • Image Captioning: COCO annotation instructions emphasize important scene parts and prominent objects, while Conceptual Captions filtering removes proper names and titles that can contain unique OCR tokens.Conceptual Captions contains 3.3 million image-caption pairs.
  • TextCaps: TextCaps differs by treating text as an additional modality that models must read, comprehend, and include in generated sentences.Many OCR tokens appear only at test time, creating a zero-shot requirement.
  • Optical Character Recognition: OCR generally consists of detecting text locations and extracting characters, whereas TextCaps additionally requires understanding token importance, semantics, and relationships.OCR is therefore a subtask of image captioning with reading comprehension.
  • Visual Question Answering with Text Reading Ability: TextVQA, ST-VQA, and OCR-VQA require reading and reasoning about image text in context, but their image-question-answer format is not directly suited to descriptive sentence generation.TextCaps is similar in spirit while targeting captions.

TextCaps Dataset

TextCaps is a dataset designed to study image captioning with reading comprehension by requiring captions to combine visual scenes with text read from images. Its statistics show substantially more OCR content, longer multi-word text use, rare and unseen tokens, and frequent switching between OCR and vocabulary words than conventional datasets.

  • Dataset collection: TextCaps uses verified text-containing Open Images images shared with TextVQA, supporting multi-task and transfer-learning scenarios.Images were verified through OCR and human annotation; captions were collected in two stages.
  • Dataset collection: Five independent captions were collected per image, and majority voting filtered captions for reading requirement, truth, sentence form, grammar, and objectivity.Evaluator quality was controlled with gold captions of known quality.
  • Dataset collection: 39.5% of ground-truth OCR tokens were covered by the collected human annotations on average.This estimate uses ground-truth OCR tokens present in each image.
  • Caption characteristics: TextCaps captions combine scene descriptions with copied, paraphrased, inferred, summarized, or reasoned-about image text.Examples include reporting scoreboard points, inferring a product price, and describing switch positions.
  • Dataset statistics: 12.4 words is TextCaps’ average caption length, slightly exceeding SBU’s 12.0, Conceptual Captions’ 9.7, and COCO’s 10.5.TextCaps captions typically include both scene description and image text in one sentence.
  • Dataset statistics: 39.7k unique OCR tokens occur in TextCaps, and 75.7% appear fewer than five times, creating a long-tailed vocabulary.Additionally, 2,901 of 6,329 unique test-set caption tokens are zero-shot, appearing in neither training nor validation.
  • Dataset statistics: 81.3% of TextCaps captions and 96.9% of its images contain at least one OCR token, versus 2.7% and 12.7% for COCO.Multi-word reading appears in 56.8% of TextCaps captions, compared with 0.38% of COCO captions containing two or more OCR tokens.
  • Model challenges: Most TextCaps captions require at least two switches between OCR tokens and vocabulary words, unlike most COCO and TextVQA outputs.This switching requirement is one of the dataset’s new model challenges.

4 Benchmark Evaluation

The benchmark compares conventional captioners with M4C-Captioner, which incorporates OCR features and copying into caption generation. Results show that OCR-aware modeling improves text-focused captioning, while substantial errors and a machine–human gap remain.

  • 4.2 Experimental setup: The benchmark evaluates conventional captioners, OCR-aware captioning, ground-truth OCR inputs, and human performance on TextCaps.Human scores average six runs using an additional caption for each test image, while model comparisons focus on CIDEr because it emphasizes informative n-grams such as OCR tokens.
  • 4.1 Baselines: M4C-Captioner adapts M4C’s multimodal transformer and dynamic pointer decoder to generate captions from detected objects and OCR tokens.The ablations remove OCR access, copying, or individual OCR feature types to test their contributions.
  • 4.3 Results: M4C-Captioner achieves the highest CIDEr score and benefits substantially from OCR inputs, while direct OCR copying further improves performance over restricted variants.The results indicate that spatial, visual, and semantic OCR features work best together, especially when relationships among multiple tokens matter.
  • 4.3 Results: Ground-truth OCR reduces but does not close the gap between the best machine and human performance, leaving room to improve text recognition and reasoning.The remaining errors include scene-understanding mistakes, OCR detection errors, incorrect token-context assignments, repetition, and insufficient OCR use.
  • 4.3 Results: Qualitative examples show that the full model learns to read and combine multiple OCR tokens, whereas models without OCR inputs rarely mention scene text.The full model can also express inferred or counted content, but it sometimes places OCR tokens in the wrong context, repeats them, or omits them.
  • 4.3 Results: Training with TextCaps alongside COCO enables M4C-Captioner to generate text-containing captions on COCO examples where COCO-only training describes visual objects without mentioning text.Supplemental quantitative results report lower COCO evaluation scores when training on COCO plus TextCaps, while higher TextCaps sampling improves qualitative OCR description but worsens COCO CIDEr.

5 Conclusion

TextCaps establishes image captioning with reading comprehension as a challenging task requiring joint handling of text and visual content. The dataset and analyses reveal substantial gaps in current systems while providing a basis for future progress.

  • 5 Conclusion: Image captioning with reading comprehension requires models to read text, recognize image content, and comprehend both modalities jointly.
  • 5 Conclusion: TextCaps contains 142k captions combining visual descriptions with OCR tokens copied or rephrased from images.Most captions require integrating OCR tokens, including many unseen tokens and frequent switches between OCR and vocabulary tokens.
  • 5 Conclusion: Current state-of-the-art captioning models cannot read when trained on existing captioning datasets.
  • 5 Conclusion: Adapting M4C VQA to TextCaps produces captions that copy multiple OCR tokens and integrate them correctly on TextCaps and COCO.Human evaluation strongly correlates with automatic metrics, although human captions remain significantly better.
  • 5 Conclusion: The remaining challenges include semantic understanding between image and text, missing reasoning capabilities, and reading long text or single characters.

(Supplementary Material)

The supplementary material documents additional analyses, qualitative examples, and dataset comparisons supporting the study of text-aware image captioning. It also reports that COCO captions rarely involve reading comprehension.

  • Supplementary Material: The supplement includes analyses of ground-truth OCR influence, Rosetta OCR precision and recall, COCO evaluation, and additional qualitative predictions.
  • Supplementary Material: COCO reference captions rarely involve reading comprehension, indicating that COCO is not a good dataset for training or evaluating this task.
  • Supplementary Material: Additional supplemental sections compare TextCaps test and validation sets and describe the data-collection user interface.

A Analysis of the influence of GT-OCR on M4C-Captioner model

The ground-truth OCR analysis examines how OCR inputs affect M4C-Captioner performance. Ground-truth OCR improves generated prediction quality, but training directly on it does not improve out-of-the-box performance when automatic OCR is used at test time.

  • A Analysis of the influence of GT-OCR on M4C-Captioner model: Ground-truth OCR tokens significantly improve generated prediction quality on TextCaps images with OCR annotations.The evaluation covers annotated subsets containing 96% of validation images and 92% of test images.
  • A Analysis of the influence of GT-OCR on M4C-Captioner model: Training with ground-truth OCRs does not improve performance out-of-the-box when models are tested with automatic Rosetta OCRs.Results are very close between models trained with automatic OCRs and ground-truth OCRs.
  • A Analysis of the influence of GT-OCR on M4C-Captioner model: The vocabulary size of predicted tokens and OCR tokens does not change significantly between automatic-OCR and ground-truth-OCR settings.The quality of selected OCR tokens increases, as indicated by precision.
  • A Analysis of the influence of GT-OCR on M4C-Captioner model: OCR precision is the average ratio of predicted OCR tokens matching annotator-used OCR tokens among all predicted OCR tokens in each sentence.

B Rosetta OCR performance analysis

The analysis evaluates Rosetta OCR tokens on TextCaps against human-annotated text using precision and recall under the ICDAR-13 end-to-end text-recognition protocol.

  • B Rosetta OCR performance analysis: Rosetta-en is used to extract OCR tokens from TextCaps images.The English-only version of Rosetta is referred to as Rosetta-en in prior work.
  • B Rosetta OCR performance analysis: Rosetta OCR performance is measured by comparing extracted tokens with human-annotated ground-truth OCRs on validation and test images.
  • B Rosetta OCR performance analysis: The evaluation follows the ICDAR-13 protocol for end-to-end text recognition.

C Automatic evaluation on COCO captioning

Training M4C-Captioner with TextCaps lowers COCO captioning metrics because COCO captions rarely require reading comprehension, despite producing captions that copy relevant text.

  • Higher TextCaps sampling ratios improve qualitative text description but worsen COCO validation performance.Sampling TextCaps captions more frequently encourages the model to describe more OCR tokens in generated captions.
  • Human COCO captions rarely involve reading comprehension, so captions that mention image text can receive lower CIDEr scores.M4C-Captioner trained on COCO + TextCaps learns to read and copy relevant text, yet its predicted captions have noticeably lower CIDEr scores.
  • Training on TextCaps leads to lower automatic evaluation metrics on the COCO validation set than training only on COCO.The reported explanation combines the mismatch between COCO captions and reading comprehension with domain shift between datasets.

D Illustration of most frequent words in TextCaps

TextCaps captions frequently copy product and label text, while annotators also use explicit wording to incorporate those tokens into descriptions.

  • Frequently copied OCR tokens mainly consist of brand names and words found on products and labels.Examples include ‘samsung’, ‘nokia’, ‘colgate’, and ‘ale’.
  • Annotators often use words such as ‘sign’, ‘says’, and ‘written’ when incorporating text tokens into captions.

E Comparison of TextCaps Test and Validation sets

TextCaps validation and test sets differ in image distributions and caption statistics, helping explain higher validation model performance and a domain shift between the sets.

  • Validation performance is always higher than test performance, likely because training and validation images share an OpenImages distribution while test images come from a different distribution.The training and validation sets use OpenImages training images, whereas the test set uses OpenImages test images with slightly different label distributions and categories.
  • Validation images and captions contain more OCR tokens on average, causing more switches between vocabulary words and OCR tokens.
  • Test captions more often contain no copied OCR tokens, suggesting more paraphrasing, reasoning, and reformulation of OCR tokens.
  • Caption-length distributions are almost identical between the validation and test sets.
  • BLEU-4, METEOR, ROUGE L, and CIDEr are higher on human-written test captions than on validation captions, indicating slight domain shift rather than inherently greater test difficulty.The comparison uses five human captions per image with leave-one-out evaluation.

F Data collection User Interface

TextCaps data collection used guided interfaces for annotation and evaluation, with image viewing tools that support reading small or difficult text.

  • The annotation and evaluation interfaces present instructions, a central image panel, and an answer field.Instructions remain available in a left-side panel, while prompts and response fields appear on the right.
  • Users could open images full size or use an interactive 3x magnifier for small or hardly readable text.
  • Users annotated images in mini-batches of 5 and evaluated captions in mini-batches of 10.There was no time restriction except an extreme limit of 15 minutes per mini-batch.
Loading 2003.12462v2…