Multi-style Generative Reading Comprehension

1 Introduction

• Generative RC is less studied than answer-span extraction, and generative models face a dearth of training data for open-domain questions.Although MS MARCO and NarrativeQA provide abstractive answers, their state-of-the-art methods rely on answer-span extraction.Source: §1 · p. 1
• Answering systems need both self-contained sentences and concise phrases, but previous studies could not use and control different styles within one model.Source: §1 · p. 1
• Masque achieves state-of-the-art performance on MS MARCO 2.1 Q&A and Q&A + NLG, and on the NarrativeQA summary task.Source: §1 · p. 1
• A pointer-generator Transformer generates abstractive answers from the question and multiple passages, covering different answer styles.It generates from a vocabulary and copies words from the question and passages.Source: §1 · p. 2
• Multi-style learning controls answer style with an artificial style token that sets mixture weights over three distributions at each decoding step.The distributions generate words from a vocabulary or copy them from the question and passages.Source: Figure 1 · p. 1, §1 · p. 2

2 Problem Formulation

• The model predicts an answer distribution from a question, a set of passages, and an answer-style label.Source: §2 · p. 2
• Training examples may include an answerability label indicating whether the passages support answering the question.Source: §2 · p. 2
• Training examples may also label whether each passage is required to formulate the answer.Source: §2 · p. 2

3 Proposed Model

• Masque models answers from a question, multiple passages, and a target style, using a reader, passage ranker, answerability classifier, and answer-sentence decoder.The answer is treated as a summary of the question and passages, and the reader is shared across answer styles and task-specific modules.Source: §3 · p. 2, §3.1 · p. 2

3.1 Question-Passages Reader

• Shared Transformer blocks encode the question and passages; ELMo supplies positional information, so these encoder blocks omit position embeddings.Input words are represented using pretrained embeddings and contextualized ELMo representations before the shared encoder.Source: §3.1.1 · pp. 2–3, §3.1.2 · p. 3

3.1.3 Dual Attention Layer

• Dual attention fuses question and passage information through word-level similarities and normalized attention, then modeling encoders produce representations for downstream modules.The modeling layer passes question and passage representations to the decoder, while passage representations also go to the ranker and answerability classifier.Source: §3.1.3 · p. 3, §3.1.4 · p. 3

3.2 Passage Ranker

• The passage ranker scores passage relevance, while the answer possibility classifier estimates whether the question is answerable.The ranker uses the first-word passage representation; the classifier maps modeling-layer output to an answer-possibility probability.Source: §3.2 · p. 3, §3.3 · p. 3

3.4.1 Word Embedding Layer

• The autoregressive decoder generates answer words sequentially, and a style token at the answer’s start lets users choose the target style without changing the architecture.Placing the token in the decoder prevents the reader module from depending on answer style.Source: §3.4 · p. 3, §3.4.1 · p. 4

3.4.2 Attentional Decoder Layer

• Masked Transformer decoder blocks attend to the question and concatenated passage representations, whose attention weights are comparable across passages.The decoder consumes embeddings and produces a sequence of d-dimensional vectors.Source: §3.4.2 · p. 4

3.4.3 Multi-source Pointer-Generator

• At each decoding step, the pointer-generator mixes vocabulary generation with copying from the question and passages; combined attention helps avoid words in irrelevant passages.Separate attention layers form the copy distributions, and the final word distribution is a mixture of the three sources.Source: Figure 3 · p. 4, §3.4.3 · pp. 4–5, §3.4.4 · p. 5

3.5 Loss Function

• Training sums the decoder’s answer-generation loss with binary-cross-entropy losses for passage ranking and answer-possibility classification.The decoder loss is averaged over answerable examples; the ranker and classifier losses are averaged over all examples.Source: §3.5 · p. 5

4.1 Setup

• MS MARCO 2.1 evaluates Q&A and Q&A + NLG using real-user questions linked to ten passages, with annotations for relevant passages and answerability.NLG answers average 16.6 words; Q&A answers average 13.1 words and often omit question context.Source: §4.1 · p. 5

4.2 Results

• A six-training-run ensemble, with each model trained on both answer styles, achieved state-of-the-art ROUGE-L on both MS MARCO V2 tasks; for NLG, its single model beat competitors on ROUGE-L and BLEU-1.Whether competing models were ensembles was unreported.Source: §4.2 · p. 6, Table 2 · p. 6
• Training on both answer styles outperformed training on the NLG style alone, with non-sentence answers contributing to improved NLG performance.Table 3 reports the ablation on the NLG development set.Source: §4.2 · p. 6, Table 3 · p. 6
• The Transformer-based pointer-generator outperformed an RNN and self-attention variant, with the deep decoder capturing multi-hop interactions better than a single-layer LSTM decoder.The comparison is reported for NLG performance.Source: §4.2 · p. 6, Table 3 · p. 6
• Joint training with the passage ranker and answer possibility classifier outperformed training without them and had a regularization effect on the question-passages reader.A gold passage ranker also significantly improved RC performance.Source: §4.2 · p. 6, Table 3 · p. 6
• On the ALL development set, answerability classification reached a maximum F1 of 0.7893 at a 0.4411 answer-possibility threshold.The precision-recall curve summarizes classification performance.Source: §4.2 · p. 7, Figure 4 · p. 7
• Generated answers were shorter than references, especially for Q&A, but their lengths remained controlled by target style across query types.The shorter Q&A answers reduced BLEU scores through BLEU’s brevity penalty.Source: §4.2 · p. 7, Figure 5 · p. 7

5 Experiments on NarrativeQA

• NarrativeQA is intended to test understanding of an underlying narrative rather than shallow pattern matching.Source: §5 · p. 7

5.1 Setup

• NarrativeQA's summary setting uses one summary averaging 659 words per question, with no unanswerable questions, so the passage ranker and answer possibility classifier were not used.Source: §5.1 · p. 7
• Because NarrativeQA has no explicit multiple-style labels, the experiment added MS MARCO's NLG subset and assigned the datasets distinct answer styles.NarrativeQA answers are short, averaging 4.73 words, and frequently contain pronouns.Source: §5.1 · p. 7
• The NarrativeQA evaluation used BLEU-1, BLEU-4, METEOR, and ROUGE-L against reported extractive and generative baselines.Source: §5.1 · p. 7

5.2 Results

• The single model trained with two styles and controlled with the NQA style pushed forward the state of the art by a significant margin.Scores controlled with the NLG style were low because the two styles differ; the NQA-only model also outperformed baselines on ROUGE-L.Source: §5.2 · pp. 7–8, Table 5 · p. 8

• Masque shares its question-passages reader among RC, passage ranking, and answer-possibility classification tasks.The discussion reports this sharing as effective.Source: §6 · p. 8
• Masque uses a style-conditional decoder rather than style-specific layers, while prior multi-task RC work includes both shared-layer models and models trained across varied NLP tasks.Source: §6 · p. 8
• Earlier generative RC methods include extraction-then-synthesis for multi-passage inputs and pointer-generators for single-passage inputs, while large amounts of training data remain necessary.The authors present multi-style learning as a technique for learning from RC datasets with different styles.Source: §6 · p. 8
• For multi-passage RC, prior systems concatenate passages, rank them before reading, or compare passage scores; Masque belongs to the non-pipelined group considering all provided passages.Source: §6 · p. 8
• Unlike sentence- or word-level content selection in abstractive summarization, Masque selects content at the passage level through combined attention.Source: §6 · p. 9
• The discussion contrasts query-based summarization models that underperformed BiDAF on DuoRC or did not copy from the question with Masque's question-and-passage generation approach.Source: §6 · p. 9

• Masque achieved state-of-the-art performance on MS MARCO 2.1 Q&A and Q&A + NLG, and on NarrativeQA summarization.Source: §7 · p. 9
• The proposed model treats answers as summaries of the question and passages and learns multiple answer styles together.Source: §7 · p. 9
• The authors identify transfer of style-independent NLG ability to the target style as key, with question-passage copying shared across styles and mixture-weight control learned for each style.Source: §7 · p. 9

• The best model jointly trained on two answer styles for eight epochs with batch size 80, using eight P100 GPUs and taking roughly six days.Its hidden size was 304, with 8 attention heads and 8 decoder blocks.Source: §A.1 · p. 12
• Inputs used GloVe and ELMo embeddings, and each passage and answer was truncated to 100 words for training.The extended vocabulary contained 5,000 common words.Source: §A.1 · p. 12
• Training used Adam with a learning rate rising to 2.5×10^-4 before cosine annealing to zero, gradient clipping, and an exponential moving average.Source: §A.1 · p. 12
• Regularization combined modified L2 weight decay, dropout at 0.3, and one-sided label smoothing that set positive labels to 0.9.Source: §A.1 · p. 12
• The ensemble combined six differently initialized runs by weighted majority, using each model's development-set ROUGE-L score as its weight.The official evaluation script was used after lowercasing answer words.Source: §A.1 · p. 12

A.2 Experimental Setup for NarrativeQA

• The best model was jointly trained on NarrativeQA and MS MARCO NLG data for seven epochs, using batches of 64 containing randomly sampled multi-style answers.The rest of the configuration matched the MS MARCO experiments.Source: §A.2 · p. 13
• NarrativeQA summaries were split into ten overlapping 130-word passages that contained each sentence in full, while MS MARCO passages were truncated to 130 words.Source: §A.2 · p. 13
• Because no official evaluation script was available, answers were evaluated with Bauer et al. (2018)'s script after lowercasing words and removing punctuation.Source: §A.2 · p. 13

A.3 Output Examples Generated by Masque

• Examples show appropriate style control across question and reasoning types, alongside style, yes/no classification, numerical copying, grammatical, and multi-hop reasoning errors.Source: §A.3 · p. 13
• MS MARCO examples cover natural-language, cloze-style, and keyword questions, and include incorrect styles, inconsistencies between styles, numerical copying problems, and grammatical errors.Source: Table 6 · p. 14
• NarrativeQA examples show style control for single- and multi-sentence reasoning, with an error in multi-sentence reasoning and some grammatical errors.Source: Table 7 · p. 15