WebQA: Multihop and Multimodal QA

Yingshan Chang; Mridu Narang; Hisami Suzuki; Guihong Cao; Jianfeng Gao; Yonatan Bisk

Source-linked AI summary

WebQA: Multihop and Multimodal QA

Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, Yonatan Bisk

arXiv:2109.00590v4Sep 1, 2021cs.CLcs.AIcs.CVcs.LG

TL;DR

WEBQA addresses the gap between text-centric QA and the multimodal, multi-hop nature of web search. It introduces a benchmark requiring unified retrieval, aggregation, reasoning, and natural-language generation across image and text sources. Results show that current models remain far from human-level accuracy, especially on image-based queries, motivating unified cross-modal representations.

Problem
Existing QA systems largely treat the web as text-only, while multimodal benchmarks do not adequately require unified reasoning, retrieval, aggregation, and free-form generation across images and text.
Method
WEBQA benchmarks open-domain, multi-hop questions requiring systems to select relevant image or text sources, aggregate their information, and generate natural-language answers.
Results
Current state-of-the-art models remain far from human-level accuracy; text-based queries are easier than image-based queries, and VLP scores 22.6 end-to-end on the proposed metric.
Takeaways & Limitations
WEBQA provides a benchmark and playground for developing unified multimodal reasoning, knowledge aggregation, and open-domain visual understanding systems.
Takeaways & Limitations
Performance is bottlenecked by PICa’s lossy textual representation of images, and the evaluation leaves comparison questions outside its current NLG scope.

Abstract

from arXiv · show

Scaling Visual Question Answering (VQA) to the open-domain and multi-hop nature of web searches, requires fundamental advances in visual representation learning, knowledge aggregation, and language generation. In this work, we introduce WebQA, a challenging new benchmark that proves difficult for large-scale state-of-the-art models which lack language groundable visual representations for novel objects and the ability to reason, yet trivial for humans. WebQA mirrors the way humans use the web: 1) Ask a question, 2) Choose sources to aggregate, and 3) Produce a fluent language response. This is the behavior we should be expecting from IoT devices and digital assistants. Existing work prefers to assume that a model can either reason about knowledge in images or in text. WebQA includes a secondary text-only QA task to ensure improved visual performance does not come at the cost of language understanding. Our challenge for the community is to create unified multimodal reasoning models that answer questions regardless of the source modality, moving us closer to digital assistants that not only query language knowledge, but also the richer visual online world.

1. Introduction

WEBQA targets open-domain web QA as a multimodal, multi-hop process requiring systems to retrieve, reason over, and summarize information from images and text.

1. Introduction: Text-only QA systems overlook visual knowledge and can answer some questions less efficiently than directly using relevant images.The paper argues that the text-based web is impoverished and illustrates that an image can immediately answer whether a park has picnic tables.
1. Introduction: The benchmark motivates unified representations that treat snippets and images as knowledge carriers rather than routing each modality to separate reasoning systems.This design also targets extraction, integration, and summarization across heterogeneous information.
1. Introduction: WEBQA is a benchmark for open-domain, multi-hop multimodal question answering modeled on knowledge-seeking web use cases.Its questions require systems to incorporate both text and images, retrieve relevant knowledge, aggregate multiple sources, and generate natural-language answers.
1. Introduction: State-of-the-art multimodal reasoning and text-generation models still fail on WEBQA, indicating unresolved challenges in language-groundable visual representations and reasoning.The benchmark is described as difficult for large-scale models despite being trivial for humans.

2. Related Work

WEBQA extends prior multimodal and knowledge-seeking QA by requiring unified multimodal knowledge aggregation, source retrieval, and fluent free-form answers.

2. Related Work: Earlier VQA commonly simplified open-domain reasoning into classification over a fixed vocabulary, while newer QA work increasingly uses free-form generation and multi-hop reasoning.The paper positions WEBQA at the intersection of these transitions.
2. Related Work: MultiModalQA requires reasoning over snippets, tables, and images, but template-generated questions can reduce the task to detecting templates and filling modality-specific blanks.WEBQA instead emphasizes knowledge aggregation and unified representation.
2. Related Work: ManyModalQA focuses primarily on selecting the answer modality, whereas WEBQA focuses on representing world knowledge in a unified space.MIMOQA is complementary because it outputs text-and-image pairs, while WEBQA requires aggregation and summarization into a natural-language answer.
2. Related Work: WEBQA differs from prior benchmarks by requiring complete free-form natural-language answers rather than extractive spans or finite answer elements.It evaluates both fluency and accuracy, placing greater emphasis on digesting, aggregating, and summarizing information.
2. Related Work: WEBQA adds source retrieval to VQA and produces sentence-form answers intended to transfer to conversational agents and voice assistants.This more closely simulates the full reasoning pipeline of web search.

3. Task Formulation

WEBQA represents each example as a question, candidate snippet or image-description sources, distractors, and a natural-language answer, with restricted and full retrieval settings.

3. Task Formulation: Each example contains a question, positive sources, distractor sources, and an answer, where each source is either a snippet or an image-description pair.Image descriptions supply names or geographic information that may be absent from the image and can link to references in the question.
3. Task Formulation: WEBQA provides both a restricted retrieval setting with n≈40 candidates and a full setting with n≈900K candidates.
3. Task Formulation: The task is decomposed into source identification followed by question answering over the selected sources as context.A joint single-stage system is left for future work because existing approaches cannot consume sufficiently large multimodal contexts.

4. WEBQA

WEBQA combines image- and text-based knowledge-seeking questions with multi-hop requirements, hard distractors, broad topics, and separate modality folds.

4.1. Answers from Images: Image-based questions include multi-image and complex single-image cases, while annotators were instructed to avoid simple, text-searchable, image-specific questions.At least one of three generated QA pairs had to require two distinct images.
4.2. Answers From Text: The text fold contains 24K questions requiring 2.0 text sources, paired with 14.6 text and 11.6 visual distractors.Text questions were collected by combining knowledge from at least two snippets and were randomly sampled because clear categorization criteria were unavailable.
4. WEBQA: WEBQA contains over 34K training QA pairs, with an additional 5K development and 7.5K test examples.Overall statistics are summarized by modality fold.
4.4. Dataset Statistics: Forty-four percent of image-based queries and 99% of text-based queries require two or more knowledge sources.Image questions also require caption understanding, so even single-image queries involve multi-source reasoning.
4.4. Dataset Statistics: WEBQA spans natural-world, event, location, science, and mythology topics, with image queries tending toward physical entities and text queries toward abstract topics.

5. Metrics

WEBQA evaluates generated answers through both fluency and accuracy, combining language-based scoring with keyword-based correctness measures tailored to visual and textual queries.

5. Metrics: WEBQA measures performance using both fluency and accuracy because answers are expected to be complete natural-language sentences.Five human full-sentence references and one succinct keyword answer are collected for each test sample.
5. Metrics: Fluency is normalized BARTScore against the reference’s identity score and bounded by 1, emphasizing semantic agreement while penalizing disfluency.The formulation is intended to handle short answers and distinguish critical differences such as colors and shapes better than BERTScore.
5. Metrics: Accuracy uses keyword overlap: closed visual categories receive F1, while diverse visual and textual answers receive recall.F1 tests precision for categories such as color, shape, number, yes/no, and multichoice; recall is used for unrestricted answer domains.
5. Metrics: The final evaluation reports FL*Acc, the average combined fluency-and-accuracy score across all test samples.The metric is reported as one system-level result.
5. Metrics: The metric does not fully solve natural-language-generation evaluation, especially when extra elements can be correct for comparison questions.This limitation is outside the scope of the work and is left to future NLG evaluation research.

6. Baseline Models

The baselines combine multimodal fine-tuning and few-shot prompting, with separate retrieval and generation components for handling WEBQA’s large heterogeneous source space.

6.1. Fine-tuning Approach: Fine-tuned VLP models perform source retrieval and autoregressive answer generation, using separate models trained from released VLP weights.Retrieval scores candidate sources, while question answering generates answers conditioned on selected sources.
6.1. Fine-tuning Approach: VLP represents text with BERT tokenization and images with detected-region features, while also testing VinVL visual representations.VinVL uses a larger backbone, C4 features, and attribute-enriched pretraining relative to the ResNeXt-101 FPN representation.
6.2. Model Variants: Model variants VLPI and VLPT train on image-only or text-only queries to expose the effects of modality-specific versus combined training.The standard VLP model is trained on the full mixed-modality data.
6.1. Fine-tuning Approach: End-to-end retrieval spans approximately 390k images and 540k text sources, so BM25 or CLIP first performs coarse filtering before VLP-based processing.The full collection is too large for direct VLP retrieval, motivating sparse and dense filtering approaches.
6.3. Few-shot Question Answering Approach: PICa adapts few-shot GPT-3 prompting to WEBQA, using oracle sources for an upper-bound comparison and textual descriptions for images.Its prompt concatenates training examples, context, and the test question; the model uses an 8-shot setting when input length permits.
6.3. Few-shot Question Answering Approach: PICa cannot be fairly compared with the fine-tuned baselines across the full pipeline because GPT-3 cannot be fine-tuned and behaves unstably with more than four source choices.Most classification-based VQA models are also unsuitable because WEBQA uses a large generative answer space.

7. Results & Analysis

WEBQA exposes substantial gaps between current models and human performance, especially in retrieval at scale, visual-source use, and multi-hop reasoning despite fluent outputs.

7. Results & Analysis: >94 Acc and >55 FL are achieved by humans, while models reach high fluency but remain far from human-level accuracy.VLP scores 22.6 end-to-end, and knowing gold sources yields only modest improvement.
7.1. Source Retrieval: VLP exceeds 68% F1 with restricted candidates, but full-corpus retrieval substantially degrades performance; dense retrieval drops more than sparse retrieval.Reranking CLIP’s top 20 sources with VLP doubles F1, supporting a coarse-to-fine retrieval design.
7.2. Question Answering: Both VLP and PICa outperform question-only baselines on text-based queries, and VLP performs favorably against text-only VLPT.These results indicate source use and benefits from combined training for text-based questions.
7.2. Question Answering: Image-based queries remain substantially harder: VLP and VLPI are no better than question-only baselines, while PICa gains little from visual sources.PICa’s limited visual-source utilization is especially evident for Y/N, MC, shape, number, and other categories.
7.2. Question Answering: The authors argue that PICa is bottlenecked by lossy textual image representations and call for unified representations supporting multimodal aggregation.They also propose exploring symbolic or compositional representations for reasoning beyond simple extraction.
7.3. Qualitative Analysis: Qualitative failures show fluent but incorrect visual answers and text answers that copy facts without performing required arithmetic or reasoning.The analyzed questions require only simple implication, addition, or visual extraction patterns, yet these remain out of reach for current models.

8. Conclusion

WEBQA is designed as an open-domain, multi-hop multimodal QA challenge that requires source retrieval, cross-modal information integration, reasoning, and fluent answer generation.

8. Conclusion: WEBQA requires systems to determine relevant sources, aggregate and reason over information, and generate fluent natural-language answers across text and images.Its evaluation measures both fluency and accuracy.
8. Conclusion: WEBQA mirrors heterogeneous web information while providing a research setting for multimodal reasoning, knowledge aggregation, and open-domain visual understanding.
8. Conclusion: The benchmark targets a single model that retrieves documents and integrates multiple paragraphs and images across a large context window before answering.

A. Data Annotation Details

The dataset annotation process used qualification, image filtering, batchwise QA creation, validation, and multiple-reference generation, while test-set analysis shows that answers are mostly unique.

A. Data Annotation Details: Annotators passed a qualification task before contributing, with quality requirements including US or Canada location, over 1,000 approved HITs, and at least 95% approval.The qualification included 15 hard-coded QA pairs, some violating annotation guidelines.
A. Data Annotation Details: The image-filtering step presented ten API-returned images and asked annotators to select three distinct but related images and label their commonality.Related image triples were later paired into groups.
A. Data Annotation Details: QA-pair creation was released batchwise with spot checks, targeted feedback, and de-qualification for workers who repeatedly violated guidelines.The process addressed recurring fixed-pattern behavior in crowdsourced answer generation.
A. Data Annotation Details: The annotation workflow also included dedicated QA-pair validation and multiple-human-reference generation stages.
A. Data Annotation Details: Test-set full-sentence answers are almost entirely unique, putting classification-based approaches at a significant disadvantage on WebQA.Top-K image-based training keywords above 5k provide only approximately 70% coverage.

D. Additional Results on Full-scale Retrieval

In full-scale retrieval, CLIP retrieves image and text answers effectively when the answer modality is known, but modality-agnostic retrieval exposes a severe image-query mismatch.

D. Additional Results on Full-scale Retrieval: 91% and 64% recall are achieved for image- and text-based queries, respectively, with 2,000 candidates when answer modality is known.These figures are reported for CLIP retrieval.
D. Additional Results on Full-scale Retrieval: Without modality knowledge, image-query recall falls to zero because question-image similarities are systematically lower than question-text similarities.The passage identifies finetuning dense multimodal retrieval models as a possible way to narrow this gap.

E. Comparing WebQA and recent benchmarks

WebQA differs from recent benchmarks by treating images as retrievable knowledge, using free-form natural-language answers, and requiring aggregation or summarization across modalities.

E. Comparing WebQA and recent benchmarks: WebQA differs from prior benchmarks across knowledge awareness, modality involvement, data construction, answer schema, and reasoning level.The comparison covers both unimodal and multimodal knowledge-seeking datasets.
E. Comparing WebQA and recent benchmarks: Unlike OK-VQA, WebQA treats images as knowledge sources that must be retrieved before processing rather than as part of the query.
E. Comparing WebQA and recent benchmarks: WebQA uses annotator-written, structurally diverse queries and unifies answers as complete natural-language sentences rather than modality-specific schemas.This contrasts with template-generated questions and fixed or structured answer formats in MultiModalQA.
E. Comparing WebQA and recent benchmarks: WebQA emphasizes unified world-knowledge representation and free-form generation, whereas ManyModalQA primarily addresses answer-modality choice and restricts answers to a single word.The free-form formulation is intended to better resemble real-world use cases and downstream conversational applications.
E. Comparing WebQA and recent benchmarks: Unlike MIMOQA's selection of a text span and image, WebQA requires numerical or logical aggregation and summarization of source information.This raises the reasoning level beyond selecting multimodal output pairs.

F. Additional Qualitative Analysis

WEBQA combines multimodal and textual question answering with complete-sentence answers, keyword annotations, and web-derived knowledge sources. Its image-based split is adversarially selected to reduce shortcut learning, while examples illustrate errors involving visual properties, colors, and question understanding.

Qualitative findings: Qualitative errors include choosing a common visual-property word, predicting the wrong color, and treating a relational question as binary.A correct dog prediction is attributed to precise object detection, showing that some visual entities remain recoverable.
Dataset composition: WEBQA instances pair knowledge sources, questions, and answers, with sources consisting of captioned images or web snippets.The dataset includes 22,423 image-based queries and 24,343 text-based queries, supported by 600K images and 750K snippets.
Answer format: Each instance uses a complete natural-language answer, while test examples additionally include multiple full-sentence answers and a succinct keyword answer.Human annotators supplied five full-sentence answers and one keyword reformulation for each testing sample.
Evaluation design: The image-based test split was adversarially selected to prevent spurious shortcut learning, whereas the text-based fold uses a random split.The dataset was designed to test visual reasoning beyond majority-answer correlations.
Data collection and validation: Questions and answers were crowdsourced, while knowledge sources were mined from directly observable web content using Bing Visual Search and Wikipedia APIs.Samples were validated by two or three crowdworkers, and problematic or low-agreement testing samples were discarded.