Source-linked AI summary
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, Tong Wang
TL;DR
Existing MRC and QA datasets are often too small, synthetic, or based on crowd-written questions, limiting real-world benchmarking. MS MARCO introduces a large-scale dataset from Bing and Cortana queries with retrieved passages, human answers, and three benchmark tasks; experiments indicate that its v2.1 dataset is more difficult than v1.1.
Problem
Existing MRC and QA datasets are often too small, synthetic, or based on crowd-generated questions, motivating large-scale real-world evaluation data.
Method
MS MARCO constructs a dataset from anonymized Bing or Cortana queries, retrieved document passages, and crowd-sourced answers, supporting answerability, answer-generation, and passage-ranking tasks.
Results
The v2.1 dataset is more difficult than v1.1, with BiDAF performing substantially worse on answerability and declining on well-formed answer generation.
Takeaways & Limitations
MS MARCO’s scale and use of Bing search queries and web documents make it attractive for benchmarking machine learning models for MRC and neural IR.
Abstract
from arXiv · showhide
We introduce a large scale MAchine Reading COmprehension dataset, which we name MS MARCO. The dataset comprises of 1,010,916 anonymized questions---sampled from Bing's search query logs---each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages---extracted from 3,563,535 web documents retrieved by Bing---that provide the information necessary for curating the natural language answers. A question in the MS MARCO dataset may have multiple answers or no answers at all. Using this dataset, we propose three different tasks with varying levels of difficulty: (i) predict if a question is answerable given a set of context passages, and extract and synthesize the answer as a human would (ii) generate a well-formed answer (if possible) based on the context passages that can be understood with the question and passage context, and finally (iii) rank a set of retrieved passages given a question. The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering. We believe that the scale and the real-world nature of this dataset makes it attractive for benchmarking machine reading comprehension and question-answering models.
1 Introduction
MS MARCO is introduced as a large-scale, real-world machine reading comprehension dataset designed to address limitations of existing MRC and QA benchmarks. It supports evaluating answerability and answer synthesis, answer generation, and passage ranking using information distributed across noisy real-world passages.
- Motivation: Existing MRC and QA datasets are often too small for training deep neural models and rely on cleaner text than real-world applications.Real-world text may contain typos, abbreviations, transcription errors, and conflicting information.
- Motivation: Real-world MRC systems must combine information across multiple passages or documents rather than operate only on a single entity or text span.The information needed to answer a question may be distributed across different parts of one document or across multiple documents.
- Dataset: MS MARCO provides a large-scale real-world dataset built from anonymized search queries issued through Bing or Cortana, with extracted passages from retrieved documents.The paper states that each question is annotated with segment information and paired with passages extracted from documents retrieved by Bing.
- Tasks: The dataset evaluates whether questions are answerable, whether answers can be extracted and synthesized, and whether well-formed answers can be generated from context passages.The first task combines answerability prediction with answer extraction and synthesis; the second generates an answer when possible.
- Tasks: A third task ranks retrieved passages given a question.The paper also reports preliminary benchmarking results for the proposed tasks.
2 Related work
The paper situates MS MARCO among publicly available machine reading comprehension and question-answering benchmarks. These datasets vary in scale, data sources, answer formats, and reasoning demands, while MS MARCO emphasizes large-scale real search queries and natural-language answers.
- SQuAD versus MS MARCO: SQuAD contains 107,785 question-answer pairs from 536 articles, with answers represented as text spans.Its questions are editorially generated from selected answer spans, unlike MS MARCO’s Bing query-log questions.
- Other MRC datasets: DuReader contains 200,000 questions and 420,000 answers from 1,000,000 documents drawn from Baidu search and community question answering.It annotates answers as fact-based or opinionative, with further entity, yes/no, and descriptive categories.
- Other MRC datasets: NarrativeQA contains about 45,000 question-answer pairs over 1,567 books and movie scripts, enabling questions requiring more complex reasoning.Its stories are described as more complex and diverse than the NewsQA news corpus, with longer documents.
- Other MRC datasets: SearchQA contains more than 140K question-answer pairs and 6.9M snippets, using Jeopardy questions and short exact-span answers averaging 1–2 tokens.MS MARCO instead focuses more on longer natural-language answer generation.
- Other MRC datasets: ARC provides 7,787 grade-school science multiple-choice questions and 14M relevant sentences, while ReCoRD contains 12,000 Cloze-style pairs targeting common-sense reasoning.ReCoRD minimizes question-passage text overlap while preserving at least one shared named entity.
3 The MS Marco dataset
MS MARCO is built from anonymized Bing search questions, retrieved web passages, and human-composed answers, with annotations supporting answerability and question-type analysis. Its real-query origins, human synthesis, multiple or absent answers, and challenging formulations define the dataset’s benchmarking value.
- Dataset construction: 1,010,916 questions and 1,026,758 unique answers are created by filtering Bing queries, retrieving documents and passages, and having editors synthesize answers.Questions are sampled from Bing logs, while editors annotate useful passages and compose well-formed natural-language answers.
- Questions: Questions are anonymized Bing search queries filtered for navigational and other non-question intents, then annotated for answerability using the provided passages.A machine-learning classifier performs initial filtering, and editors determine whether each selected question is answerable.
- Passages and answers: Each question includes roughly 10 passages extracted from relevant web documents, with editors marking passages used for answers or marking all passages unselected when no answer exists.Passages are selected by Bing’s passage-retrieval system and support answer synthesis strictly from the supplied context.
- Well-formed answers: Some question-answer pairs receive rewritten answers that correct grammar, reduce passage overlap, or improve response completeness and clarity.The rewrite process is performed by a separate editor reviewing the original answer.
- Dataset characteristics: The dataset’s distinguishing features include real web-document context, human-composed answers, segment labels, and subsets with multiple answers or no answers.Questions may be complex, ambiguous, or contain typographical errors, making the dataset challenging for machine reading comprehension benchmarking.
4 The challenges
The MS MARCO challenges define three machine-learning tasks with increasing difficulty: answerability and answer generation, well-formed answer generation, and passage re-ranking for information retrieval. Together, they require systems to identify unanswerable questions, synthesize appropriate answers, and rank retrieved passages by answer relevance.
- Task overview: MS MARCO proposes three machine-learning tasks with diverse difficulty levels.The tasks are novice answerability and generation, intermediate well-formed answer generation, and passage re-ranking.
- Novice task: The novice task predicts whether passages answer a question, returns “No Answer Present” when they do not, and otherwise generates the correct answer.Answerability is judged solely from the provided passage information.
- Intermediate task: The intermediate task requires answers that remain well-formed and understandable when read aloud without the question or retrieved passages.This distinguishes it from the novice task’s answer-generation requirement.
- Passage re-ranking: The passage re-ranking task ranks 1000 BM25-retrieved passages by how likely they are to contain information relevant to answering a question.It is framed as an information-retrieval challenge targeting large-scale benchmarking of emerging neural IR methods.
5 The benchmarking results
The benchmarking section evaluates generative, discriminative, and cloze-style machine reading comprehension models on MS MARCO subsets using category-appropriate metrics. It also compares span-based models with human baselines across dataset versions, finding that v2.1 is more difficult than v1.1.
- Evaluation strategy: Evaluation uses different metrics for answer categories because binary Yes/No answers and longer descriptive answers require different benchmarking strategies.The experiments build on metrics appropriate to each question type.
- V1.1 experiments: The v1.1 experiments compare vanilla Seq2Seq, memory-network, and LSTM-based discriminative models for answer generation and passage ranking.The Seq2Seq model maps question-answer pairs as source-target sequences; the memory network initializes the RNN decoder from summed memory representations, while the discriminative model ranks passages.
- V1.1 experiments: ROUGE-L evaluates the QA models, while BLEU and pairwise BLEU evaluate the memory network on a multi-answer subset.The reported evaluations include a multi-answer MS MARCO subset in addition to the general QA benchmark.
- Cloze-style evaluation: The cloze-style evaluation tests Attention Sum Reader and ReasoNet on CNN and Daily Mail and on a numeric-answer MS MARCO subset.The section reports accuracy on both datasets and precision-recall curves for the MS MARCO subset.
- V2.1 experiments: The v2.1 comparison trains a self-attention and bidirectional-attention span model and reports that the updated dataset is more difficult than v1.1.Answers are selected from passages using the highest-confidence model-generated span, and performance is compared with a newly constructed human baseline.
6 Future Work and Conclusions
The authors frame MS MARCO’s development and public release as a learning experience shaped by substantial changes to data collection, annotation, task definitions, and scope. Its future depends largely on how the broader academic community uses the dataset.
- Future Work and Conclusions: MS MARCO evolved substantially between its first version and most recent edition in data collection, annotation, task definitions, and scope.The scope was broadened to cater to the neural IR community.
- Future Work and Conclusions: The dataset’s future will depend largely on how the broader academic community makes use of it.The authors identify community use as a central factor shaping the dataset’s future.
- Future Work and Conclusions: Developing and publicly releasing MS MARCO provided the authors with a tremendous learning experience.The reflection covers changes made from the dataset’s first version through its most recent edition.