PaperlayerView exact revision on arXiv

Source-linked AI summary

ManyModalQA: Modality Disambiguation and QA over Diverse Inputs

Darryl Hannan, Akshay Jain, Mohit Bansal

arXiv:2001.08034v1cs.CLcs.AIcs.CV

TL;DR

ManyModalQA addresses QA in which the answer may be in text, an image, or a table, and the question alone does not make the answer modality clear. It constructs a crowdsourced Wikipedia challenge and a selector-plus-unimodal-QA baseline; updated results improve modality selection while the dataset remains more ambiguous than existing datasets.

  • Problem

    In multimodal QA, the answer modality may be unknown from the question, making it important to resolve where the answer is before answering.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    The paper builds a Wikipedia-based challenge and routes each question, via a modality selector, to a corresponding pretrained QA model.

    Source: §3 · p. 3, §6.3 · p. 7

  • Results

    81.35% test performance for RoBERTa modality selection, up from 77.49%, while ManyModalQA remains substantially more ambiguous than existing datasets.

    Source: §6.5 · p. 7

  • Takeaways & Limitations

    The challenge is intended as a test bed for transfer learning and end-to-end modality disambiguation with multimodal QA.

    Source: §7 · p. 8

    Requiring questions to be answerable only by combining all three modalities proved extremely challenging for crowdworkers.

    Source: §4 · p. 3

Abstract

from arXiv · show

We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities: text, images, and tables. We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs. Our questions are ambiguous, in that the modality that contains the answer is not easily determined based solely upon the question. To demonstrate this ambiguity, we construct a modality selector (or disambiguator) network, and this model gets substantially lower accuracy on our challenge set, compared to existing datasets, indicating that our questions are more ambiguous. By analyzing this model, we investigate which words in the question are indicative of the modality. Next, we construct a simple baseline ManyModalQA model, which, based on the prediction from the modality selector, fires a corresponding pre-trained state-of-the-art unimodal QA model. We focus on providing the community with a new manymodal evaluation set and only provide a fine-tuning set, with the expectation that existing datasets and approaches will be transferred for most of the training, to encourage low-resource generalization without large, monolithic training sets for each new task. There is a significant gap between our baseline models and human performance; therefore, we hope that this challenge encourages research in end-to-end modality disambiguation and multimodal QA models, as well as transfer learning. Code and data available at: https://github.com/hannandarryl/ManyModalQA

1 Introduction

ManyModalQA presents a Wikipedia-based QA challenge spanning text, images, and tables, with questions designed to make the answer modality difficult to infer from the question alone. It pairs a modality selector with pretrained unimodal QA models as a baseline, while aiming to support low-resource transfer from existing datasets.

1 Introduction

  • Questions are designed to obscure whether text, an image, or a table contains the answer, reflecting real-world settings where users consult different modalities.The challenge contains 2,036 fine-tuning, 3,055 development, and 5,099 test examples; each includes text and a table, an image, or both.Source: §1 · pp. 1–2
  • The paper prioritizes using existing unimodal and bimodal datasets and approaches, providing a fine-tuning set to encourage low-resource generalization rather than a large standalone training set.Source: §1 · p. 1
  • 39.73% test accuracy for the baseline architecture versus 91.58% for upper-limit human performance leaves a substantial gap on ManyModalQA.The baseline first selects a modality and then runs the corresponding pretrained QA model.Source: §1 · p. 2
  • The challenge covers text, images, and tables, bringing tables—an underexplored QA modality—into a three-modality task.Source: §1 · p. 2
  • The challenge spans varied knowledge domains, including politics, science, sports, people, and places, unlike more narrowly scoped multimodal QA datasets.Source: §1 · pp. 1–2

2 Related Work

Prior multimodal QA often centers on text and images or video, while ManyModalQA adds tables and emphasizes ambiguity about where an answer is found. Its Wikipedia-based, open-ended task also differs from existing datasets in domain breadth and question format.

2.1 Multimodal QA Datasets

  • ManyModalQA adds tables to multimodal QA, where prior work often considers only text and images or text and video, and emphasizes ambiguity about the answer modality.Source: §2.1 · p. 2
  • Compared with TQA, RecipeQA, and TVQA, ManyModalQA uses open-ended human questions from varied Wikipedia topics rather than multiple-choice questions in narrower domains.The related-work passage also contrasts its questions with TVQA's compositional templates and RecipeQA's automatically generated pairs.Source: §2.1 · pp. 2–3

2.2 Unimodal Text and Table QA Datasets

  • Text QA includes dialogue-based CoQA and multi-document reasoning datasets such as QAngaroo and HotpotQA.Source: §2.2 · p. 3
  • Table QA is less studied; ManyModalQA's single-table, open-ended Wikipedia questions are closest to WikiTableQuestions, unlike multiple-choice TabMCQ or template-based baseball questions in MLB.Source: §2.2 · p. 3

3 The Task: ManyModal QA Reasoning

ManyModalQA asks an agent to resolve which of up to three modalities contains an answer and then answer the question. Its questions are intentionally ambiguous, and the task is designed to encourage transfer from existing QA datasets for low-resource generalization.

3 The Task: ManyModal QA Reasoning

  • Each example is a simplified Wikipedia page containing at least two of text, images, and tables, and the agent must disambiguate the answer modality before answering.Source: §3 · p. 3
  • Questions are made as ambiguous as possible without becoming unnatural, so the answer modality is not obvious from wording alone.The paper says this ambiguity makes the task more representative of real-world QA and distinguishes it from a mere combination of unimodal tasks.Source: §3 · p. 3
  • The challenge provides a fine-tuning set rather than a large new training corpus, aiming to promote low-resource generalization from existing unimodal and bimodal QA datasets.Source: §3 · p. 3

4 Data Collection

The authors collect Wikipedia-based examples and use crowdsourcing to create ambiguous questions and validate their answers. Their collection strategy uses an adversarial framing, while fully requiring answers to combine all three modalities proved too challenging for crowdworkers.

4 Data Collection

  • The challenge data comes from English Wikipedia, and human annotators create questions through Mechanical Turk to obtain diverse, natural questions.Source: §4 · pp. 3–4
  • Requiring questions to be answerable only by combining all three modalities proved extremely challenging for crowdworkers.Source: §4 · p. 3
  • Crowdworkers were asked to create questions that would fool a QA robot, an adversarial framing intended to encourage ambiguity without listing every collection criterion.Source: §4 · p. 4
  • Image captions were added during collection to provide context and make image questions more complex and ambiguous while still requiring the image for an answer.Source: §4 · p. 4
  • 92.58% inter-annotator agreement was obtained by validating each example with three workers and accepting answers when at least two agreed.The paper treats this as evidence that the data is high quality and answerable by humans.Source: §4 · p. 4

5 Data Analysis

ManyModalQA includes questions across text, images, and tables, with prompts often ambiguous about which modality contains the answer. Its analysis describes broad topic coverage, overlapping question types across modalities, and open-ended text and table answers.

5 Data Analysis

  • 10,190 questions span image (2,873), text (3,789), and table (3,528) modalities, with a 20/30/50% fine-tuning/dev/test split.The test set is reserved for final evaluation; the fine-tuning and dev sets are publicly released.Source: §5 · p. 4

5.1 Question Properties

  • Questions are often ambiguous about answer modality, and some table questions could plausibly be answered from text instead.Image questions were filtered to ensure they were not answerable using only the caption.Source: §5.1 · p. 4
  • Pages span 13 categories, with places and sports/media together accounting for 53.7% of the data.Science and political pages account for 4.0% and 8.7%, respectively.Source: §5.1 · p. 5
  • “What” and “how many” are common across all three modalities, making broad question type a poor feature for determining modality.Differences become apparent only at a more fine-grained level.Source: §5.1 · p. 5, Figure 4 · p. 5

5.2 Answer Properties

  • Text and table answers are open-ended: answers need not appear in context, unlike span-based QA, where the answer span always appears.Answers appear in context in 6.33% of text examples and 9.38% of table examples.Source: §5.2 · p. 5

6 Baseline Models and Results

The section introduces a question-only modality selector and a routed baseline that invokes a pre-trained QA model for the predicted modality. Results show persistent modality ambiguity and a gap between baseline and human accuracy.

6 Baseline Models and Results

  • The baseline combines the modality selector with three existing QA models and also uses the selector to analyze question ambiguity.The section also reports preliminary approaches to repurposing and combining existing datasets for a manymodal setting.Source: §6 · p. 5

6.1 Modality Selection Network

  • The modality selector classifies questions into text, image, or table using ELMo embeddings, an LSTM, feedforward layers, and a final softmax.It is trained on the fine-tuning set and evaluated against a selector trained on questions from SQuAD, VQA v2, and WikiTableQuestions.Source: §6.1 · pp. 5–6

6.2 Modality Selection Network Results

  • 77.49% selector accuracy on MANYMODALQA versus 92.65% on three unimodal datasets supports greater ambiguity; the most-common baseline scores 37.14%.LIME analysis finds visually oriented words such as “color” and “wearing” informative for image questions, while proper nouns can contribute to misclassification.Source: §6.2 · p. 6

6.3 Modality Selection and Question-Answering Network

  • The routed system predicts a modality, then feeds its context and question to the corresponding unimodal QA model; it is not end-to-end.The component QA systems are trained separately on existing unimodal datasets and fine-tuned on ManyModalQA data.Source: §6.3 · pp. 6–7

6.4 Modality Selection and Question Answering Network Results

  • 39.73% test accuracy for the routed baseline trails the 46.26% test oracle and 91.58% human accuracy.The voting baseline scores 21.12%, while the most-common and nearest-neighbor baselines score 2.41% and 5.61%, respectively.Source: Table 2 · p. 7, §6.4 · p. 7

6.5 Recent Results with RoBERTa and LXMERT

  • RoBERTa raises selector test accuracy to 81.35% from 77.49%, while LXMERT raises image-QA score to 37.11% from 27.17%.On existing datasets, selector accuracy is 95.44% from 92.65%; the LXMERT oracle score is 40.79% from 29.05%.Source: §6.5 · pp. 7–8

7 Conclusion

The paper presents ManyModalQA as a challenge to promote research in manymodal QA. It aims to support work on transfer learning and end-to-end modality disambiguation with multimodal QA, and plans to expand its data to stronger cross-modal questions and additional modalities.

7 Conclusion

  • ManyModalQA is presented as a new challenge intended to promote research in manymodal QA.The authors hope it will serve as a test bed for further work.Source: §7 · p. 8
  • The challenge is structured to encourage work on transfer learning and end-to-end modality disambiguation with multimodal QA.Source: §7 · p. 8
  • Future work plans to collect questions requiring multiple modalities and add video and audio.Source: §7 · p. 8

Appendices

The text QA appendix describes a BERT-based model with a pointer-generator output, alongside TF-IDF paragraph selection to shorten context.

A Text-based BERT QA Model

  • TF-IDF selects the paragraph whose vector is closest to the question, shortening the context before text-model processing.Source: A Text-based BERT QA Model · p. 9
  • The text QA model adds a pointer-generator output to BERT so it can produce answers such as “yes” and “no” that may be absent from the context.Source: A Text-based BERT QA Model · p. 9
  • The output probability combines candidate-answer generation with selecting a word from the document, using pgen as the generation-versus-selection weight.The model forms candidate-answer probabilities from a BERT representation and computes document-word selection probabilities with a pointer.Source: A Text-based BERT QA Model · p. 9

B Question Lengths

Question lengths are similar across text, image, and table modalities, with text questions slightly longer on average.

B Question Lengths

C Modality Selector Confusion Matrix

The modality selector often confuses text and table questions, while image-question errors tend to favor text. The collection instructions ask crowdworkers to make questions that could fool a task-solving robot and emphasize ambiguity.

C Modality Selector Confusion Matrix

  • Text and table questions are frequently misclassified, while misclassified image questions tend to be labeled text.The authors suggest text's broader range of potential question types may explain the image-to-text pattern.Source: C Modality Selector Confusion Matrix · p. 9

D Sample Mechanical Turk Instructions

  • Crowdworkers are instructed to create questions that could fool a robot completing the task, with ambiguity emphasized as critical to the challenge.The instructions are kept short to increase the likelihood that workers read them completely.Source: D Sample Mechanical Turk Instructions · p. 9

E Data Samples

The samples illustrate questions whose answers are hard to assign to a modality from the question alone, even when the text, image, and table share a topic. They also include an example of the crowdsourcing instructions.

E Data Samples

  • Figure 10 asks when “alkaloid” was introduced; although its image and table concern alkaloids, the answer is in the text, and the question could also plausibly be answered from a table or paragraph.The agent must determine the answer-bearing modality before responding.Source: E Data Samples · pp. 9–10, Figure 10 · p. 10
  • Figure 9 shows a sample of the instructions used for crowdsourcing.Source: Figure 9 · p. 10
  • Figure 11 asks where the manual alphabet and numbers came from; its image, table, and text all concern American Sign Language, while the answer is in the image caption or text.The agent must locate the answer across the modalities before responding.Source: E Data Samples · p. 10, Figure 11 · p. 10
Loading 2001.08034v1…