Source-linked AI summary
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, Jeffrey P. Bigham
TL;DR
VizWiz addresses the lack of VQA datasets grounded in the real visual questions of blind people. The paper constructs and evaluates a dataset from users' photographs and spoken questions, with crowdsourced answers and answerability judgments. The resulting dataset is difficult for modern algorithms and is released to support generalized assistive-technology research.
Problem
Existing VQA datasets are built in artificial settings rather than from goal-oriented visual questions asked by blind people in daily life.
Method
The authors filter privacy- and safety-sensitive questions, crowdsource answers and answerability judgments, characterize the data, and evaluate algorithms for answer prediction and answerability prediction.
Results
VizWiz is difficult for modern algorithms that answer visual questions or predict whether they are answerable.
Takeaways & Limitations
The dataset and code provide a shared basis for developing more generalized algorithms intended to address blind people's visual and accessibility needs.
Takeaways & Limitations
VizWiz includes poor-quality images, conversational or imperfect spoken questions, and questions that may be unanswerable from their images.
Abstract
from arXiv · showhide
The study of algorithms to automatically answer visual questions currently is motivated by visual question answering (VQA) datasets constructed in artificial VQA settings. We propose VizWiz, the first goal-oriented VQA dataset arising from a natural VQA setting. VizWiz consists of over 31,000 visual questions originating from blind people who each took a picture using a mobile phone and recorded a spoken question about it, together with 10 crowdsourced answers per visual question. VizWiz differs from the many existing VQA datasets because (1) images are captured by blind photographers and so are often poor quality, (2) questions are spoken and so are more conversational, and (3) often visual questions cannot be answered. Evaluation of modern algorithms for answering visual questions and deciding if a visual question is answerable reveals that VizWiz is a challenging dataset. We introduce this dataset to encourage a larger community to develop more generalized algorithms that can assist blind people.
1. Introduction
VizWiz addresses the gap between artificial VQA benchmarks and the natural, goal-oriented visual questions of blind people. Its real-world images and spoken questions create challenges including poor image quality, conversational audio, and unanswerable questions.
- Motivation: Existing VQA datasets use artificially created settings, whereas VizWiz targets visual questions originating from blind people.The dataset is intended to encourage more generalized algorithms that address blind people's interests.
- Motivation: VizWiz builds on a mobile application in which users photograph their surroundings and ask natural questions about what they want to know.This use case connects visual question answering to daily visual and social accessibility challenges.
- Challenges: Poor lighting, focus, framing, blur, inadequate lighting, and fingers covering the lens often make the images difficult to interpret.Because photographers cannot verify what their images capture, the visual content may not match the question.
- Challenges: Spoken questions can be conversational and may contain clipping or background audio that affects their content.Some questions also include auxiliary information or multiple sentences to clarify the desired response.
- Challenges: Visual questions may be unanswerable because the image fails to capture the visual content being asked about.Figure 1 contrasts answerable and unanswerable examples.
2. Related Works
Prior VQA datasets largely rely on contrived images and questions and generally assume that questions are answerable. VizWiz instead captures real spoken questions from blind users, explicitly labels answerability, and exposes challenges from poor-quality images.
- Data Quality and Privacy: Poor-quality images from blind photographers introduce new challenges for modern vision algorithms and require review before public release.Images collected in the wild may also contain inappropriate or private content.
- VQA Datasets: Existing VQA datasets use existing or artificially constructed images and questions generated by systems, crowd workers, or in-house participants.Their questions are described as contrived rather than authentically motivated by users' attempts to understand the visual world.
- VQA Datasets: VizWiz is the first dataset to represent visual questions authentically asked by blind people about their physical surroundings.The same person creates each image and question, and the questions are spoken rather than typed or automatically generated.
- Answerability: ∼28% of VizWiz visual questions are deemed unanswerable by crowd workers.This occurs despite automated systems intended to help blind photographers improve focus, lighting, or composition.
- Answerability: Existing VQA collection generally assumes that questions can be answered from the associated images.VizWiz instead includes questions that may be unrelated because the images are too poor in quality.
- Answerability: VizWiz explicitly asks crowd workers whether each visual question is answerable while collecting answers.Experiments find it difficult for the freely shared algorithm designed to predict answerability.
3. VizWiz: Dataset Creation
VizWiz is built from real visual questions asked by blind people, preserving the spoken, user-driven nature of the interaction. Dataset preparation compares this setting with existing VQA datasets, removes privacy and safety risks, and collects answers for algorithm training and evaluation.
- 3.1. Visual Question Collection Analysis: VizWiz collects questions from blind users who photograph their surroundings and record spoken questions about the images.The source application accrued 72,205 questions from 11,045 users; 48,169 were eligible for anonymous sharing.
- 3.1. Visual Question Collection Analysis: Unlike existing VQA datasets, VizWiz combines images from blind photographers, spoken questions, and a setting where askers cannot see the captured images.These properties produce blur, poor lighting, poor framing, conversational language, audio errors, and questions unrelated to the images.
- 3.2. Anonymizing and Filtering Visual Questions: The dataset preparation addresses privacy and safety risks through anonymization, vulnerability taxonomy, crowd-worker screening, and expert review.Anonymization removes voices and image metadata, while experts review remaining questions using vulnerability categories.
- 3.2. Anonymizing and Filtering Visual Questions: ∼31% of visual questions (14,796) were filtered from the original 48,169 candidate questions.The filtering taxonomy covers personally identifying information, location, indecent content, suspicious complex scenes, and suspicious low-quality images.
- 3.3. Answer Collection: 31,173 final visual questions received 10 crowdsourced answers each for training and evaluating visual-question-answering algorithms.Workers were shown each question and image and could answer “Unsuitable Image” when image quality prevented answering.
4. VizWiz: Dataset Analysis
VizWiz is analyzed as a natural VQA dataset through the diversity of its questions, images, answers, and answerability. The analysis finds conversational and incomplete questions, diverse and often unsuitable images, many unique answers, and substantial human agreement.
- Question diversity: 27.88% of VizWiz questions begin with a first word occurring in less than 5% of questions, versus 13.4% for VQA 2.0.The authors associate this difference partly with conversational language and recordings that begin after speech has started.
- Question diversity: The median and mean question lengths are five and 6.68 words, while the 25th and 75th percentiles are four and seven words.Three-word questions such as “What is this?” are common, but longer questions can provide disambiguating information or include captured background audio.
- Image diversity: 28% of VizWiz images received at least two “unsuitable image” labels, and the average image is gray, indicating no particular structure across all images.The average-image analysis was used to assess whether the dataset was dominated by high-quality images of single iconic objects.
- Answer diversity: VizWiz contains approximately 58,789 unique answers, with only 824 of its top 3,000 answers also appearing among VQA 2.0’s top 3,000.The authors use this low overlap to explain why existing prediction systems perform poorly on VizWiz.
- Answerability and agreement: 28.63% of visual questions are unanswerable, while independent people agree on an answer for 97.7% of visual questions.Unanswerability is identified when at least half of the crowdsourced answers say “unanswerable” or “unsuitable image”; agreement is measured by exact string matching.
5. VizWiz Benchmarking
VizWiz is challenging for existing VQA and answerability algorithms, while adapting models to VizWiz improves performance but leaves substantial gaps to human accuracy. Image-based cues and question information help predict unanswerable questions.
- 20,000 training, 3,173 validation, and 8,000 test questions define the VizWiz benchmark, approximately a 65/10/25 split.
- Nine VQA methods compare original, fine-tuned, and VizWiz-only training variants across Accuracy, CIDEr, BLEU4, and METEOR.The methods combine image, question, and attention mechanisms.
- 5.1. Visual Question Answering: ∼0.14 accuracy: existing VQA models perform poorly on VizWiz because their VQA 2.0 training data contains only 824 of VizWiz’s top 3,000 answers.
- 5.1. Visual Question Answering: Fine-tuning and VizWiz-only training significantly improve performance over unchanged models, with little difference between the two adaptation strategies.Further work is needed to approach the reported human performance of 0.75 accuracy.
- 5.1. Visual Question Answering: Attention models consistently outperform image-and-question-only models, but their gains are relatively small, partly because many images contain few objects or are difficult for attention models.
- 5.1. Visual Question Answering: Fine-tuning and training from scratch produce the greatest gains for unanswerable questions and the smallest gains for number and other questions.VizWiz questions are categorized as yes/no, number, other, or unanswerable, with unanswerable questions comprising 34.6%.
- 5.2. Visual Question Answerability: 0.218–0.318 accuracy: models trained or fine-tuned on VizWiz generalize poorly to the VQA 2.0 test set, indicating a domain shift.
- 5.2. Visual Question Answerability: 71.7 AP: the Q+I answerability model outperforms the status quo’s 30.6 AP, while image-only prediction reaches 64 AP before adding question information.Performance is evaluated with precision-recall curves and average precision, the area under the curve.
6. Conclusions
The paper concludes that VizWiz is a difficult, natural-use VQA dataset created from blind people’s images and questions. It releases the dataset and code to support research on generalized assistive technologies.
- VizWiz is a VQA dataset originating from blind people taking images and asking questions about them in a natural use case.
- VizWiz is difficult for modern algorithms, while improving performance on it can educate researchers about blind people’s technological needs.
- The authors share the dataset and code to facilitate future work on assistive technologies that address accessibility barriers.
Supplementary Materials
The supplementary materials document filtering, answer collection, dataset analysis, and algorithm benchmarking procedures supporting the main paper.
- The supplement covers filtering visual questions to support Section 3.2.
- The supplement covers collecting answers to visual questions to support Section 3.3.
- The supplement covers VizWiz dataset analysis and algorithm benchmarking, supporting Sections 4 and 5.
I - Filtering Visual Questions
The filtering process begins by identifying images containing personally identifying information and developing a taxonomy of vulnerabilities relevant to an in-the-wild VQA dataset.
- Crowdsourcing first identifies images showing personal-identifying information before iterative vulnerability-taxonomy development.The taxonomy was designed to protect people associated with the dataset.
Answer Post-Processing
VizWiz uses standardized answer cleaning and crowdsourced annotation procedures, then analyzes question, image, and answer distributions alongside model training and evaluation. The results emphasize diverse, often atypical language, broad answer tails, image variability, and domain-shifted performance.
- Answer Post-Processing: Answers are normalized by lowercasing, converting numbers to digits, removing punctuation and articles, correcting spelling, and removing filler words.Two automated spell-checkers provide candidate corrections during post-processing.
- Answer Collection: Crowd workers provide 10 brief answers per question and label poor-quality images or questions that cannot be answered as “Unsuitable Image” or “Unanswerable”.Workers were recruited through AMT with US citizenship, high prior approval rates, and adult-content qualification.
- Image Analysis: The average image excluding unanswerable questions is gray, reflecting diverse images without a particular common structure.The dataset’s image diversity is also examined through average-image visualization.
- Answer Analysis: Answers show uncommon vocabulary, lengths up to six words, and a long tail of unique responses; after “Unsuitable Image” and “Unanswerable”, yes, no, and colors are most common.Cumulative answer coverage is plotted against the most frequent answers to illustrate this long tail.
- Model Evaluation: Q+I remains the top-performing answerability method under both average precision and average F1, while cross-dataset results highlight a difficult domain shift to VQA 2.0.The compared implementations combine question, caption, and image features in different configurations and use either fine-tuning or training from scratch.