PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiModalQA: Complex Question Answering over Text, Tables and Images

Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, Jonathan Berant

arXiv:2104.06039v1cs.CLcs.AIcs.LG

TL;DR

MMQA addresses the limited availability of QA benchmarks requiring integrated reasoning across text, tables, and images. The paper constructs such a dataset with an automated composition and paraphrasing framework and introduces ImplicitDecomp for multimodal multi-hop reasoning. ImplicitDecomp reaches 51.7 F1 versus 38.2 for a single-hop approach, while human performance reaches 90.1 F1.

  • Problem

    Prior QA work has relatively little coverage of questions requiring information integration across multiple modalities.

  • Method

    The paper constructs MMQA by linking Wikipedia tables with images and text, composing cross-modal questions, paraphrasing them, and executing predicted reasoning programs with dedicated modality models.

  • Results

    51.7 F1 versus 38.2 for a single-hop approach on multimodal questions, while humans reach 90.1 F1.

  • Takeaways & Limitations

    MMQA establishes a benchmark for multimodal multi-hop QA and shows that the best model remains substantially below human performance.

  • Takeaways & Limitations

    The automatically generated question distribution does not come from a natural source.

Abstract

from arXiv · show

When answering complex questions, people can seamlessly combine information from visual, textual and tabular sources. While interest in models that reason over multiple pieces of evidence has surged in recent years, there has been relatively little work on question answering models that reason across multiple modalities. In this paper, we present MultiModalQA(MMQA): a challenging question answering dataset that requires joint reasoning over text, tables and images. We create MMQA using a new framework for generating complex multi-modal questions at scale, harvesting tables from Wikipedia, and attaching images and text paragraphs using entities that appear in each table. We then define a formal language that allows us to take questions that can be answered from a single modality, and combine them to generate cross-modal questions. Last, crowdsourcing workers take these automatically-generated questions and rephrase them into more fluent language. We create 29,918 questions through this procedure, and empirically demonstrate the necessity of a multi-modal multi-hop approach to solve our task: our multi-hop model, ImplicitDecomp, achieves an average F1of 51.7 over cross-modal questions, substantially outperforming a strong baseline that achieves 38.2 F1, but still lags significantly behind human performance, which is at 90.1 F1

1 INTRODUCTION

MMQA addresses the limited work on QA requiring information integration across modalities by introducing a large-scale dataset and generation framework for text, tables, and images. It also presents ImplicitDecomp, a model for multi-hop multimodal reasoning, and evaluates it against baselines and humans.

  • Dataset and motivation: MMQA targets a gap in prior QA research, which has mostly addressed paragraphs, tables, or images separately rather than integrating modalities.
  • Task and model: Answering MMQA questions can require decomposing a question, routing subquestions to text, image, or table sources, and combining their answers.
  • Dataset construction: The dataset-generation framework harvests Wikipedia tables, links images and paragraphs through entities, automatically composes multimodal questions, and paraphrases them.
  • Task and model: ImplicitDecomp predicts and executes a reasoning program with dedicated text, table, and image models without explicit question decomposition.
  • Evaluation: F1 rises from 38.2 to 51.7 for ImplicitDecomp over a single-hop approach on multimodal questions, while humans reach 90.1 F1.
  • Dataset and motivation: 29,918 MMQA questions include 35.7% requiring cross-modality reasoning over free text, semi-structured tables, and images.

2 DATASET GENERATION

MMQA is generated by anchoring contexts in Wikipedia tables, enriching them with linked images and text, and combining single-modality questions into multimodal questions that workers paraphrase.

  • Context construction: Wikipedia tables serve as anchors, enriched with associated images and text paragraphs to form multimodal contexts.The selected tables are connected to images from table cells or linked WikiEntity pages and to paragraphs from reading-comprehension datasets.
  • Single-modality questions: Single-modality questions are generated over tables, images, and text paragraphs for reuse as building blocks.Table questions support matching and numeric operations; image questions cover single images or image lists; text questions draw on existing reading-comprehension datasets.
  • Question generation: A formal language combines modality-specific questions through operations such as TABLEQ, TEXTQ, IMAGEQ, IMAGELISTQ, COMPOSE, and INTERSECT.The resulting pseudo-language preserves how compositional questions were constructed and supports multi-hop reasoning operations.
  • Paraphrasing: AMT workers paraphrased automatically generated pseudo-language questions into natural language, with validation and incentives for substantial reformulation.Each question was paraphrased by one worker and validated by one to three others; 971 workers participated.
  • Dataset composition: 29,918 examples were produced and split into 23,817 training, 2,441 development, and 3,660 test examples.Development and test context components were disjoint and built from disjoint single-modality questions.
  • Limitation: The automatically generated question distribution does not come from a natural source, although the method provides controlled, cheap, and scalable dataset creation.This is identified as a shortcoming of the generation method.
  • Dataset contents: MMQA examples include questions, answers, formal pseudo-language representations, distractors, and gold context across modalities, supporting open-domain and context-dependent QA.The formal representation is ignored by the models.

3 DATASET ANALYSIS

MMQA contains many compositional and multimodal questions spanning diverse topics, while its paraphrases differ substantially from the generated pseudo-language in wording and vocabulary.

  • Key Statistics: 29,918 questions comprise the MMQA dataset, with about 60% described as compositional.Questions average 18.2 words, while answers average 2.1 words; list answers account for 7.4% of the data.
  • Domain Diversity: Entertainment is prominent in MMQA, with Films representing 36% and TV 19%, alongside a wide variety of other topics.The topic proportions are presented as evidence of domain diversity.
  • Lexical Richness: The average normalized edit distance between natural-language and pseudo-language questions is 0.7.Natural-language questions are shorter on average and use more unique words than the pseudo-language questions.
  • Lexical Richness: Natural-language questions average 20.02 words versus 22.16 for pseudo-language questions, with 39,319 versus 37,108 unique words.These statistics indicate shorter paraphrases with richer vocabulary.

4 MODELS

The models combine single-modality QA modules with routing and implicit decomposition to answer questions requiring multiple modalities.

  • 4.1 SINGLE-MODALITY QA MODULES: Single-modality baselines independently answer questions using text, tables, or images.The text module selects answer spans or yes/no/not-present outcomes; the table module selects cells and an aggregation operation; the image module uses visual features and question text.
  • 4.2 MULTIMODALITY QA MODELS: ImplicitDecomp predicts a program specifying reasoning steps across modalities and executes it with dedicated text, table, and image models.It performs multimodal multi-hop reasoning without explicitly decomposing the question.
  • 4.2 MULTIMODALITY QA MODELS: The question-type classifier predicts one of 16 types that specify relevant modalities, their order, and logical operations.For example, Compose(TextQ,TableQ) conducts the first hop on the table and the second on text paragraphs.
  • 4.2 MULTIMODALITY QA MODELS: AutoRouting predicts the answer modality, routes the question and context to the corresponding module, and returns its output.This provides a single-hop baseline without cross-modal reasoning.
  • 4.3 TRAINING AND SUPERVISION: Training uses question types and intermediate results as supervision, with one modality-specific model reused for both ImplicitDecomp hops.The question-type classifier is trained with cross-entropy loss, while QA modules are trained using modality-appropriate samples.

5 EXPERIMENTS

Experiments evaluate single-modality, multimodality, and all-question settings using average F1 and Exact Match, while analyses examine human performance and multi-hop dependence.

  • 5 EXPERIMENTS: 94.5% of human answers are identical or semantically equivalent to the gold answer, while 4.8% are human errors.An additional 0.7% of cases contain an error in the question; reported human errors are attributed to context length and fatigue.
  • Analysis: When the first-hop answer is correct, ImplicitDecomp achieves 63.9 F1; when incorrect, final F1 falls to 37.4.The analysis covers Compose, Compare, and Intersect multimodal questions, indicating reliance on the first-hop answer.
  • 5 EXPERIMENTS: A question-only baseline may exploit comparison questions because one candidate usually appears in the question, enabling 50% accuracy by guessing.The authors present this as a conjecture explaining its higher performance than the context-only baseline.
  • Analysis: The question-type classifier achieves 91.5% accuracy on the test set.This result is reported alongside the first-hop and final-answer analysis.
  • Analysis: A qualitative analysis finds that 92% of sampled compositional questions require multi-hop reasoning.The remaining examples include 6% weak distractors and 2% redundant evidence.

6 RELATED WORK

MMQA extends prior multimodal QA by requiring reasoning across free text, semi-structured tables, and images rather than merely routing questions to a modality.

  • 6 RELATED WORK: The task differs from earlier visual question answering by including complex multi-hop questions spanning text, tables, and images.Earlier visual QA work ranges from synthetic images to scientific plots.
  • 6 RELATED WORK: MANYMODALQA chooses among image, text, and table contexts but focuses on routing questions to the correct modality.MMQA instead emphasizes reasoning over the combined context.
  • 6 RELATED WORK: HYBRIDQA requires cross-modality reasoning over tables and text but does not include visual inference.MMQA adds image-based cross-modal reasoning and offers 16 major question types versus HYBRIDQA’s 6.
  • 6 RELATED WORK: MMQA uses automatically annotated intermediate multi-hop answers to perform linking implicitly over full tables and text.This contrasts with HYBRIDQA’s heuristic for linking text passages to table cells.

7 CONCLUSION

MMQA is a 29,918-example dataset with 35.7% cross-modality questions, and its results show that effective systems must exploit multimodality and implicit multi-hop reasoning.

  • 7 CONCLUSION: MMQA contains 29,918 examples, with 35.7% requiring cross-modality reasoning.The paper describes a framework for generating complex multimodal questions at scale.
  • 7 CONCLUSION: The best model exploits dataset multimodality and performs multi-hop reasoning through implicit decomposition.The conclusion frames this as the main evaluation finding.
  • 7 CONCLUSION: Human performance substantially exceeds the best model, establishing the need for further research on multimodal question answering systems.The authors hope MMQA will drive that research.

A APPENDIX - DATASET GENERATION

The appendix describes extracting Wikipedia tables, filtering them by size and image availability, parsing them, and classifying columns for question generation.

  • The 01-01-2020 English Wikipedia dump contained roughly 3M tables, of which 700K met the 10–25-row and at-least-three-images criteria.
  • A customized WikiTextParser extracted table titles and addressed row- and column-span issues.
  • Columns were classified as numeric, date, or index according to whether cells parsed as numbers, dates, or consecutive numeric values.

A.2.1 IMAGES

The dataset links tables to in-table and WikiEntity images, retrieves and caches image files, and filters cases where entity-image matching is unreliable.

  • In-table images: In-table image columns are identified with a linear classifier using position, uniqueness, WikiEntity, short-text, and header features.
  • Images from WikiEntities: WikiEntity images are linked by mapping entities to Wikipedia image files, retrieving URLs through the MediaWiki API, and caching images for reuse across contexts.
  • Entity-image matching cannot cover all entities because files may be removed, images may be absent or non-representative, and table cells may lack links.
  • Frequently reused non-representative images, such as shared film logos, were excluded from question generation.
  • The collection contains 57,713 images: 889 in-table images and 56,824 WikiEntity images.

A.3.1 GENERATING IMAGE QUESTIONS

Image questions were crowdsourced from single images and image lists, constrained toward stable features and filtered for suitable table anchors before validation and paraphrasing.

  • Single Images: Crowdworkers generated questions about an image with its WikiEntity as the focus, producing 2,268 single-image questions.
  • Single Images: Workers were discouraged from asking about temporary visual features so questions would remain suitable for open-domain use.
  • Image Lists: Image-list questions used images from one table column and asked about a distinctive feature shared by a subset of entities.
  • Image Lists: Image-list anchor columns required at least 4 WikiEntities, no more than 3 duplicate images, and no more than 2 entities without images.
  • Paraphrasing and validation: Automatically generated questions were verified by separate workers, while pseudo-language questions were paraphrased into natural language with contextual information and feedback.
  • Text questions: Text questions from NQ, BoolQ, and HotpotQA were linked to anchor tables through shared WikiEntities.

B APPENDIX - DATASET ANALYSIS

The appendix analyzes paraphrase richness and introduces additional model-module information, including the image question-answering component.

  • Paraphrasing Richness: Crowdworkers received bonuses for substantially modifying generated questions, and Figure 9 reports normalized edit-distance distributions between natural-language and pseudo-language questions.
  • The appendix provides additional information about modules and their application.
  • Image QA Module: The image QA module starts from VILBERT-MT, using image-region features extracted by Faster R-CNN with a ResNet-101 backbone pretrained on Visual Genome.

D APPENDIX - EXPERIMENTS

The appendix describes evaluation interfaces for rephrasing validation and human performance, and reports question-type classification performance. It also explains how classifier predictions are analyzed and how users inspect multimodal context.

  • 91.4% overall accuracy was obtained by the RoBERTa-large question-type classifier on the test set.
  • The classifier predicts one of 16 question types and its confusion matrix marks correct predictions on the diagonal.
  • The human-evaluation interface lets users inspect 10 text paragraphs, images, and a table before submitting a free-text answer.
  • The rephrasing validation interface presents each generated question, its human rephrasing, and relevant multimodal context for worker judgments.
Loading 2104.06039v1…