S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension

1 INTRODUCTION

• MS-MARCO uses multiple passages per question and does not require answer words to appear in the passages, unlike SQuAD’s exact-span, single-passage setting.Answers may combine multiple passage spans and include words absent from both passages and question.Source: §1 · pp. 1–2
• S-Net first predicts important passage sub-spans as evidence, then uses that evidence with the question and passage to generate the final answer.The synthesis model takes extracted evidence as additional features.Source: §1 · p. 2, Figure 1 · p. 2
• S-Net’s extraction-then-synthesis framework outperforms pure answer extraction and other existing methods on MS-MARCO.The reported comparison uses ROUGE-L and BLEU-1.Source: §1 · p. 3
• The extraction model adds passage ranking to answer-span prediction to improve evidence extraction when questions have multiple passages.The ranking task is framed as multi-task learning.Source: §1 · pp. 2–3

2 RELATED WORK

• MS-MARCO answers are human-generated from a set of passages and need not be passage sub-spans, unlike SQuAD’s span answers.Other dataset formats described include selecting among four options and predicting a missing cloze word.Source: §2 · p. 3
• Prior MS-MARCO methods largely adapt SQuAD approaches, including match-LSTM and pointer networks, co-attention, dynamic pointers, and self-matching.The cited self-matching approach is described as state of the art on SQuAD.Source: §2 · p. 3
• Sequence-to-sequence models are used across tasks such as translation, parsing, response generation, and summarization, and here generate answers using evidence-span positions as features.The evidence-span positions are used as features for answer generation.Source: §2 · p. 3

3 OUR APPROACH

• The framework trains evidence extraction and answer synthesis in two stages; “evidence extraction” denotes pure answer extraction in prior work.The synthesis model uses the extracted snippets as additional features.Source: §3 · pp. 3–4

3.1 GATED RECURRENT UNIT

• A Gated Recurrent Unit (GRU), rather than a basic RNN, models the recurrent hidden state using reset and update gates.The paper identifies r_t and z_t as the gates and h_t as the hidden state.Source: §3.1 · p. 4

3.2 EVIDENCE EXTRACTION

• The extraction framework adds ranking of candidate passages, using passage-level representations alongside text-span prediction.MS-MARCO supplies multiple passages per question and annotates which passage is correct.Source: §3.2 · p. 4

3.2.1 EVIDENCE SNIPPET PREDICTION

• Pointer networks concatenate the passages and use attention to select the start and end positions of an evidence snippet.The pointer network receives an attention-pooling vector based on the current predicted probabilities.Source: §3.2.1 · p. 5

3.2.2 PASSAGE RANKING

• Passage ranking attends from the question representation to each passage, combines the representations for a matching score, and labels the correct passage as positive.The candidate-passage scores are normalized for the ranking objective.Source: §3.2.2 · pp. 5–6

3.2.3 JOINT LEARNING

• The extraction component is trained with a joint objective that weights the span-prediction and passage-ranking losses.The hyperparameter r sets the weights of the two loss functions.Source: §3.2.3 · p. 6

3.3 ANSWER SYNTHESIS

• The sequence-to-sequence synthesis model encodes the question and passage with evidence-boundary features, then uses an attention-based GRU decoder to generate the answer.At each decoding step, the model combines the previous word embedding, context vector, and decoder state to predict the next word.Source: §3.3 · pp. 6–7

• The framework achieves better results than pure extraction and other existing methods on the MS-MARCO leaderboard's official evaluation metrics.Source: §4 · p. 7
• The experiments use the MS-MARCO dataset and compare the framework with a pure extraction model and other baseline methods.Source: §4 · p. 7
• The evaluation also includes ablation tests and a comparison with an end-to-end generation framework.Source: §4 · p. 7

• The dataset contains Bing user queries paired with passages from real web documents.Source: §4.1 · p. 7
• The data is split into 82,326 training pairs, 10,047 development pairs, and 9,650 test pairs.Source: §4.1 · p. 7
• Because human-generated answers need not be passage spans, evaluation uses BLEU and ROUGE-L, with the paper focusing mainly on ROUGE-L.The official tool averages ROUGE-L per question but normalizes BLEU across all questions; the authors favor case-by-case evaluation.Source: §4.1 · p. 7

4.2.1 TRAINING

• Evidence extraction and answer synthesis are trained in two stages.Source: §4.2.1 · p. 8
• For extraction training, the highest-ROUGE-L span is treated as gold, and only spans scoring above 0.7 are retained, yielding 71,417 training pairs.Source: §4.2.1 · p. 8
• Synthesis training uses reference answers with evidence snippets scoring above 0.5, plus extracted spans paired with the passage containing each span.Source: §4.2.1 · p. 8

4.2.2 PARAMETER

• Extraction uses fixed 300-dimensional GloVe embeddings, 150-dimensional hidden vectors, and dropout of 0.1; synthesis uses a 30,000-word vocabulary and 150-dimensional GRU states.Source: §4.2.2 · p. 8
• AdaDelta optimization starts at learning rate 1.0, and hyperparameters are selected on the MS-MARCO development set.Source: §4.2.2 · p. 8

4.2.3 DECODING

• At decoding, synthesis receives the extracted span and its passage, uses beam search of size 12, and removes duplicate phrases or refines unknown or unsupported words.If the generated answer is only <unk>, the extracted span is used as the final answer.Source: §4.2.3 · p. 8

• S-Net combines evidence extraction and answer synthesis, while S-Net (Extraction) contains only the extraction component.Source: §4.3 · p. 8
• BiDAF and Prediction are implemented as extraction baselines, alongside an extraction-only baseline without passage ranking.Source: §4.3 · p. 8
• Table 2 presents performance on the MS-MARCO test set, with an asterisk marking synthesis that uses ensemble extraction output.Source: Table 2 · p. 9
• Table 3 presents MS-MARCO development-set ROUGE-L results; its asterisk marks ensemble extraction input and its plus sign identifies Wang & Jiang (2016b)'s reported Prediction score of 37.3.Source: Table 3 · p. 9
• The synthesis component is also applied to the baseline extraction results, and comparisons include FastQAExt, ReasoNet, and R-Net.Source: §4.3 · p. 9

• 46.65 ROUGE-L and 44.78 BLEU-1: the best synthesis result outperformed existing methods with a large margin.The authors report that it was very close to human performance.Source: §4.4 · p. 9
• 42.92 ROUGE-L and 44.97 BLEU-1: a 13-model extraction ensemble achieved state-of-the-art extraction results.Models were selected by greedy search after training 30 models with the same setting.Source: §4.4 · p. 9
• 3.72% and 3.65%: synthesis improved single and ensemble extraction models, respectively, in development-set ROUGE-L.Source: §4.4 · p. 9
• The analysis also applied answer synthesis to other answer-span predictors, including BiDAF and Prediction.Source: §4.4 · p. 10

4.5.1 ABLATION TEST ON PASSAGE RANKING

• 38.9 passage-selection accuracy, 4.3 above pure answer prediction: joint passage ranking and extraction outperformed the two-step ranking-then-extraction approach.The authors attribute the joint-learning advantage to stronger supervision from answer spans for passage ranking.Source: §4.5.1 · p. 10

4.5.2 EXTRACTION VS. SYNTHESIS

• Synthesis improved ROUGE-L by 2.0 for questions with extraction upper bounds of 0.6–0.8, but slightly worsened exact-matchable answers.For ROUGE-L<0.2 cases, the authors note that yes/no answers can be inferred from evidence, while their brevity limits BLEU impact.Source: §4.5.2 · p. 10

4.5.3 COMPARISON WITH THE END-TO-END GENERATION FRAMEWORK

• 6.28 ROUGE-L: the end-to-end model using matching information underperformed the extraction-then-synthesis framework; question-and-answer input scored 8.9, while adding all passages scored 28.75.The authors conclude that the two-step approach is effective; they identify matching between question and passage as important for answer generation.Source: §4.5.3 · p. 11, Table 6 · p. 11

• S-Net extracts question-related passage sub-spans as evidence, uses passage ranking to improve extraction, and synthesizes answers from question information and evidence.Source: §5 · p. 11
• The authors report that S-Net outperforms pure answer extraction and other existing methods on MS-MARCO.Source: §5 · p. 11
• The synthesis model uses only one evidence snippet, limiting its ability to answer questions requiring multiple evidences.The authors propose ranking separate candidates to supply multiple evidence snippets as future work.Source: §5 · p. 11