Source-linked AI summary
Captioning Images Taken by People Who Are Blind
Danna Gurari, Yinan Zhao, Meng Zhang, Nilavra Bhattacharya
TL;DR
Existing captioning datasets rarely represent real users’ images, although blind people have used human-based services to understand pictures they take. The paper introduces VizWiz-Captions, analyzes its content against eight vision datasets, and benchmarks captioning algorithms, finding that lower-quality images remain difficult. It positions the dataset as a foundation for more generalized captioning systems serving diverse real-world needs.
Problem
Most publicly available captioning datasets use contrived, web-curated images and therefore do not directly represent the real-use needs of people who are blind.
Method
The paper constructs VizWiz-Captions from 39,181 images submitted by blind users, crowdsources captions, collects image metadata, compares content with eight vision datasets, and benchmarks captioning algorithms.
Results
Captioning algorithms struggle with lower-quality images, while VizWiz-Captions contains distinct visual concepts from existing datasets and often answers users’ visual questions.
Takeaways & Limitations
VizWiz-Captions provides a foundation for developing captioning algorithms that address diverse real-user needs and broader real-world visual challenges.
Abstract
from arXiv · showhide
While an important problem in the vision community is to design algorithms that can automatically caption images, few publicly-available datasets for algorithm development directly address the interests of real users. Observing that people who are blind have relied on (human-based) image captioning services to learn about images they take for nearly a decade, we introduce the first image captioning dataset to represent this real use case. This new dataset, which we call VizWiz-Captions, consists of over 39,000 images originating from people who are blind that are each paired with five captions. We analyze this dataset to (1) characterize the typical captions, (2) characterize the diversity of content found in the images, and (3) compare its content to that found in eight popular vision datasets. We also analyze modern image captioning algorithms to identify what makes this new dataset challenging for the vision community. We publicly-share the dataset with captioning challenge instructions at https://vizwiz.org
1 Introduction
VizWiz-Captions addresses the mismatch between contrived captioning datasets and images taken by blind users in real-world conditions. It introduces a large public dataset and analyzes its content and algorithmic difficulty.
- Dataset and motivation: Real users’ images differ dramatically from contrived benchmark images, contributing to poor algorithm performance after deployment.Examples include text, variable image quality, and diverse content.
- Dataset and motivation: 39,181 images submitted by people who are blind were crowdsourced with captions and metadata about text presence and image-quality severity.The dataset is called VizWiz-Captions.
- Analysis and benchmarking: VizWiz-Captions contains visual concepts distinct from eight popular vision datasets and often provides answers to users’ visual questions.The comparison spans image captioning, visual question answering, and image classification datasets.
- Analysis and benchmarking: Modern captioning algorithms struggle particularly with lower-quality images in this real-user dataset.The paper benchmarks contemporary captioning systems against the dataset.
- Broader significance: The dataset is proposed as a foundation for more generalized algorithms that handle varied image quality and textual information in real-world scenarios.The paper connects these challenges to applications including wearable lifelogging, autonomous vehicles, and robots.
2 Related Work
The related work contrasts web-curated captioning datasets with authentic image-description scenarios involving blind photographers. VizWiz-Captions extends this real-use orientation while adding metadata for systematic analysis of text and image quality.
- Captioning Images for People Who are Blind: People who are blind have sought human-powered descriptions of images they take, while services still face uncertainty about what content to describe.The related work notes a lack of guidance for images taken by blind users.
- Image Captioning Datasets: Most publicly shared captioning datasets use crowdsourced captions for images curated from online databases, especially Flickr, rather than authentic user scenarios.This curation pattern supported growth from thousands to hundreds of thousands of images.
- Image Captioning Datasets: VizWiz-Captions focuses on the distinct use case of captioning images taken by blind photographers and contains nearly 40,000 images.Prior authentic-scenario datasets contained 3,361 and 20,000 images.
- Image Captioning Datasets: To the authors’ knowledge, VizWiz-Captions is the first captioning dataset with per-image metadata for text presence and image-quality severity.These annotations enable systematic analysis around both factors.
- Content in Vision Datasets: The paper compares VizWiz-Captions with popular vision datasets to assess how well contrived datasets reflect the interests of real users of image-description services.The comparison is intended to highlight existing datasets’ value and reveal ways to improve them.
3 VizWiz-Captions
VizWiz-Captions was designed to characterize captions and content from images taken by blind photographers, including quality variation, textual content, and overlap with mainstream datasets. Analyses show diverse, often domain-shifted content, captions that answer many visual questions, and shared concepts alongside substantial dataset differences.
- Dataset construction: Crowdsourcing produced five captions for each of 39,181 submitted images, alongside metadata on text presence and image-quality severity.The task adapted a concrete vision-community captioning design for Amazon Mechanical Turk.
- Caption characteristics: Captions typically contain roughly 13 words, including four to five nouns, one to two adjectives, one verb, and two spatial relations.Caption composition is nearly identical for captioning-use-case images and VQA-use-case images.
- Caption characteristics: 63% of images contain text, and text-containing images have more nouns and fewer adjectives than images without text.Crowdworkers often combine generic and text-derived object descriptions, such as product names read from packaging.
- Content diversity: The dataset contains over 24,000 unique words, while captions average 13 words versus 11.3 in MSCOCO-Captions.The greater length reflects more nouns, adjectives, and spatial-relation words in VizWiz-Captions.
- Comparison with existing datasets: 54.4% of the 3,000 most common words overlap with MSCOCO-Captions, indicating substantial domain shift despite shared vocabulary.VizWiz-Captions also contains novel concepts absent from MSCOCO-Captions.
- VQA relevance: At least one third of visual questions have answers appearing in image captions, although exact string matching provides only a lower bound.A qualitative review increased coverage for yes/no and number questions, with yes/no coverage rising from 0% to 35%.
- Comparison with existing datasets: VizWiz-Captions contains all concepts shared across paired mainstream object, scene, and attribute datasets, but usually has considerably fewer examples per concept.Its smaller image count makes the prevalence comparison a lower bound.
4 Algorithm Benchmarking
The paper benchmarks modern captioning and text-detection algorithms on VizWiz-Captions, finding that real-user images remain challenging despite gains from dataset-specific training. Performance varies with image quality and text presence, revealing distinct failure modes.
- Dataset Splits: The dataset split contains 23,431 training, 7,750 validation, and 8,000 test images, excluding canned low-quality and rejected spam captions.
- Baselines and Evaluation: The benchmark evaluates nine captioning algorithms derived from Up-Down, SGAE, and AoANet using eight standard captioning metrics.The evaluation uses BLEU-1-4, METEOR, ROUGE-L, CIDEr-D, and SPICE.
- Overall Performance: Training on VizWiz-Captions yields roughly 10 percentage points higher BLEU-1 and 30 percentage points higher CIDEr-D across three algorithms, yet scores remain below MSCOCO-Captions.The top VizWiz-Captions algorithm is over 20 points lower in BLEU-1 and almost 20 points lower in METEOR than the top MSCOCO-Captions result.
- Overall Performance: Pretraining versus training from scratch produces no considerable overall difference, with the preferred strategy varying by algorithm.Up-Down and AoANet perform better from scratch, whereas SGAE performs better after MSCOCO-Captions pretraining.
- Fine-Grained Analysis: Algorithms perform better on images containing text, although the text detector performs very poorly and text integration remains a key challenge.The authors hypothesize that text-containing images follow simpler caption templates, making them easier to learn.
- Fine-Grained Analysis: Captioning performance parallels human difficulty, with algorithms performing best on images judged easiest by people.This highlights the practical difficulty of handling the low-quality images common in real-user captioning services.
5 Conclusions
The paper positions VizWiz-Captions as a foundation for captioning algorithms serving a socially important real-world use case. It also points toward broader vision systems that account for real-user complexity and additional sensors.
- 5 Conclusions: VizWiz-Captions provides a foundation for designing image captioning algorithms for a natural, socially important use case.
- 5 Conclusions: The analysis identifies broader challenges for generalized algorithms across settings with diverse image quality and textual information.The paper gives wearable lifelogging devices, autonomous vehicles, and robots as examples of affected scenarios.
- 5 Conclusions: Future work includes incorporating additional sensors such as GPS, sound waves, and infrared to better meet real users’ needs.
Appendix
The appendix supplements the main paper with implementation details, annotation-consistency analysis, low-quality image examples, dataset-diversity analyses, and blur-based data-augmentation results.
- Supplementary Materials: The appendix documents the crowdsourcing system implementation and analyzes consistency among captions collected from different workers.
- Supplementary Materials: It provides examples of images judged insufficient or low quality for captioning.
- Supplementary Materials: It visualizes and quantitatively analyzes VizWiz-Captions content diversity relative to MSCOCO-Captions and image-classification datasets.
- Supplementary Materials: It reports algorithm performance when blurred images are used for training-time data augmentation.
A.1 Crowdsourcing Task Design
The crowdsourcing task uses a controlled interface and layered quality checks to collect substantial, one-sentence image descriptions from qualified workers while filtering inadequate responses.
- Task Interface: The interface required each image description to satisfy specified criteria before workers could proceed or submit.
- Task Constraints: Captions had to contain at least eight words, use one sentence, and avoid generic prefixes such as “There is” and “The image”.
- Worker Quality Control: Five batches enabled the authors to review and block problematic workers between batches.Review targeted statistical timing outliers, excessive capitalization, and frequent use of canned text.
- Task Responses: The task included a canned response for images whose quality was too severe to recognize visual content, while other workers were instructed to describe image content.
- Worker Quality Control: Workers were screened using prior performance, requiring more than 500 completed HITs and at least a 95% acceptance rate.
- Post-Processing: Microsoft Azure spell-checking corrected misspellings, grammar, capitalization, and common brand-name errors in accepted captions.
B Caption Consistency (supplements Section 3.2)
VizWiz-Captions has broadly similar caption-consistency scores to MSCOCO-Captions, but its scores are more concentrated near zero, reflecting greater variation in annotator detail and content.
- Specificity scores measure agreement among five captions per image, with values nearer 1 indicating greater consistency.Scores are reported for VizWiz-Captions and the MSCOCO-Captions validation set.
- VizWiz-Captions has a distribution skewed more toward zero than MSCOCO-Captions despite similar overall distributions.
- The greater annotation differences reflect annotators providing different levels and types of detail.Figure 6 illustrates caption diversity across a range of specificity scores.
C Dataset Analysis (supplements Section 3.2)
The dataset analysis documents image-quality variation and compares caption vocabulary between VizWiz-Captions and MSCOCO-Captions through crowdworker judgments and word-frequency visualizations.
- Five crowdworkers identified examples deemed insufficient or lower quality for meaningful captioning.Figure 7 shows unanimous judgments, while Figure 8 covers medium and high difficulty based on agreement.
- Medium-difficulty images were identified by 1-2 workers and high-difficulty images by 3-4 workers judging image quality insufficient.
- Wordclouds compare the 100 most popular nouns, verbs, and adjectives in VizWiz-Captions, MSCOCO-Captions, and VizWiz-only vocabulary.
C.2 Caption Characterization
The analysis characterizes caption vocabulary and compares the prevalence of object, scene, and attribute concepts in VizWiz-Captions versus mainstream vision datasets.
- Caption Characterization: Caption vocabulary is compared using the most common words in VizWiz-Captions and MSCOCO-Captions, including words unique to VizWiz-Captions.The analysis covers nouns, verbs, and adjectives and complements a 3,000-word overlap comparison.
- Implementation: Person detections were obtained with a COCO-pretrained Faster R-CNN, retaining detections above a 0.3 confidence threshold.Only the person category was counted from the model’s 80 categories.
- Dataset Comparison: Relative category prevalence is reported across VizWiz-Captions and mainstream datasets for object recognition, scene classification, and attribute recognition.Figure 12 complements the main-paper comparison, while Figures 13-15 show overlapping visual concepts.
- Dataset Comparison: Figures 13-15 provide exemplar VizWiz-Captions images containing concepts common in existing datasets for objects, scenes, and attributes.
D Algorithm Performance with Data Augmentation (supplements Section 4)
Data augmentation with blurred training images was tested to improve captioning of low-quality images, but it reduced performance for the selected top-performing algorithm.
- AoANet was retrained from scratch with augmented copies of blurred training images to assess robustness to low-quality test images.The algorithm was selected because it performed best when trained from scratch.
- 60.5 to 56.2: CIDEr performance decreased with data augmentation on the overall evaluation set.The results are reported in Table 7 and analyzed across easy, medium, and difficult subsets in Table 6.
- The authors suspect artificial distortions were unsuitable for mimicking real-world quality issues and distracted model training.
- Human performance was estimated on easy images with five valid captions by using one caption as prediction and four for evaluation.