PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives

Yi Tay, Shuohang Wang, Luu Anh Tuan, Jie Fu, Minh C. Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, Aston Zhang

arXiv:1905.10847v1cs.CLcs.AIcs.IR

TL;DR

Reading comprehension over long narratives remains challenging because of long contexts and narrative reasoning demands. The paper trains a pointer-generator with curriculum-based variation in context difficulty and adds block-based alignment reasoning. It reports state-of-the-art NarrativeQA results, with relative improvements of +17% to 51% across metrics over the best ASR model.

  • Problem

    Reading full narratives remains challenging because long contexts combine with the reasoning demands of narrative text.

    Source: §1 · p. 1

  • Method

    A curriculum varies answerability and retrieved-document size while a pointer-generator learns from diverse contexts, including those without the answer.

    Source: §1 · pp. 1–2

  • Results

    +17% to 51% relative improvement across metrics over the best ASR model is reported on NarrativeQA.

    Source: §3.2 · p. 6

  • Takeaways & Limitations

    The authors conclude that curriculum-based sampling of diverse story views is potentially more effective than techniques designed for open-domain question answering.

    Source: §5 · p. 9

    NarrativeQA evaluation can give no credit to semantically correct answers when their form differs, word-level matching fails, or they are more specific than annotators’ answers.

    Source: §3.4 · p. 7

Abstract

from arXiv · show

This paper tackles the problem of reading comprehension over long narratives where documents easily span over thousands of tokens. We propose a curriculum learning (CL) based Pointer-Generator framework for reading/sampling over large documents, enabling diverse training of the neural model based on the notion of alternating contextual difficulty. This can be interpreted as a form of domain randomization and/or generative pretraining during training. To this end, the usage of the Pointer-Generator softens the requirement of having the answer within the context, enabling us to construct diverse training samples for learning. Additionally, we propose a new Introspective Alignment Layer (IAL), which reasons over decomposed alignments using block-based self-attention. We evaluate our proposed method on the NarrativeQA reading comprehension benchmark, achieving state-of-the-art performance, improving existing baselines by $51\%$ relative improvement on BLEU-4 and $17\%$ relative improvement on Rouge-L. Extensive ablations confirm the effectiveness of our proposed IAL and CL components.

1 Introduction

Reading full narratives remains difficult because of their length and reasoning demands, while summary-based settings bypass long-context challenges. The paper proposes curriculum-trained pointer-generator reading with an Introspective Alignment Layer and reports strong NarrativeQA results.

1 Introduction

  • Reading full stories and novels remains challenging because narratives combine long contexts with reasoning demands, while summary settings bypass long-context reading.The paper focuses on the full-story setting rather than the summary setting.Source: §1 · p. 1
  • The curriculum varies difficulty by answerability and retrieved-document size, targeting the gap between training-time and inference-time retrieval.The paper relates this training scheme to domain randomization.Source: §1 · p. 1
  • A pointer-generator can generate an answer even when it is absent from the context, enabling training on diverse, noisy views of the same story.The paper frames this as a form of generative pretraining during training.Source: §1 · pp. 1–2
  • The Introspective Alignment Layer applies block-style local self-attention to decomposed alignments for finer-grained local-global comparison of soft-aligned representations.Ablations report improvement over vanilla self-attention.Source: §1 · p. 2
  • 17% relative improvement over prior state-of-the-art and 10 times the BLEU-4 score of BiDAF are reported for IAL-CPG on NarrativeQA.The paper describes IAL-CPG as achieving state-of-the-art performance on the challenge.Source: §1 · p. 2
  • The authors report negative results for reinforcement learning to improve generated answers and argue that NarrativeQA evaluation can give correct answers zero credit.They cite Paulus et al. (2017) and Bahdanau et al. (2016) in describing the reinforcement-learning work.Source: §1 · p. 2

2 Our Proposed Framework

For extremely long sequences, IAL-CPG retrieves relevant chunks before reading and combines alignment-based reasoning, pointer-generator decoding, and curriculum-controlled training. Its components provide local reasoning over aligned representations and vary training contexts by answerability and chunk size.

2 Our Proposed Framework

  • For extremely long sequences, the framework first retrieves relevant chunks using the answer or question as a cue, with curriculum learning controlling retrieval.The overall framework is illustrated in Figure 1.Source: §2 · p. 2

2.1 Introspective Alignment Reader

  • IAL enriches soft context-question alignments with comparison features, then applies block-based self-attention within a local window to reason over matches.Because context sequences can exceed 2,000 tokens, the layer restricts scoring to positions where |i−j|≤b; the initial alignment already provides a global view.Source: §2.1 · p. 3

2.2 Pointer-Generator Decoder

  • The pointer-generator combines an abstractive vocabulary distribution with an extractive context distribution, allowing answers absent from the context and diverse training views.The decoder’s generation-or-point decision is conditioned on the question as well as the context.Source: §2.2 · pp. 3–4, §2.3 · p. 4

2.3 Curriculum Reading

  • Curriculum reading starts with answer-retrieved examples, swaps in question-retrieved examples after validation stalls, then changes chunk size when the easy set is depleted.The two difficulty dimensions are answerability and chunk-size-based understandability; the swap within a chunk size uses a small percentage δ.Source: §2.3 · pp. 4–5

3 Experiments

On NarrativeQA, IAL-CPG reaches state-of-the-art performance against prior work, while ablations favor its IAL and curriculum components. Qualitative analysis also reveals evaluation mismatches and answer-generation errors.

3.1 Experimental Setup

  • Experiments use NarrativeQA reading comprehension, comparing ASR, open-domain R3, and RNET with pointer-generator and curriculum variants.The reported benchmark results use the full-story setting.Source: §3 · p. 5, §3.1 · p. 6, Table 1 · p. 6

3.2 Experimental Results

  • IAL-CPG achieves state-of-the-art NarrativeQA results, improving over the best ASR baseline by +17% to 51% relative across metrics and about +1% to +3% absolute.Table 1 reports results in the full-story setting.Source: §3.2 · p. 6, Table 1 · p. 6

3.3 Ablation Study

  • Removing IAL or its enhancement reduces performance; replacing block-based attention with regular self-attention hurts most, though the tiny batch size may contribute.The regular-attention comparison required a batch size of 2 to manage memory.Source: §3.3 · p. 7
  • Removing the pointer-generator sharply reduces performance; answerability matters more than understandability, and curriculum variation beats training on only the easy or hard set.Training on the hard set alone still gives decent performance comparable to the easy set.Source: §3.3 · p. 7
  • Reinforcement learning with BLEU or Rouge rewards provides no significant benefit for answer generation.The authors report that it does not significantly harm performance either.Source: §3.3 · p. 7
  • For paragraph-size curricula, starting small and increasing generally performs better; reversing the order may hurt, and curriculum often beats static training.The passage reports these as general patterns across the tested understandability settings.Source: §3.3 · p. 7

3.4 Qualitative Error Analysis

  • Qualitative examples show semantically correct answers receiving zero credit, alongside frequent nonsensical or partially correct generations, including a gender mismatch.One scoring mismatch arises when an answer is more specific than the annotators' answer.Source: §3.4 · p. 7

4 Related Work

Prior work addresses reading comprehension with open-domain retrieval, curricula, attention, and generated answers. This paper positions diverse training views as an alternative focus for long-context narrative reading.

Open-domain question answering

  • Open-domain QA techniques often construct cleaner reader contexts, while this work presents diverse sampling as a training approach for neural reading comprehension.The passage describes prior techniques for passage selection, ranking, denoising, and evidence aggregation.Source: §4 · p. 8

Curriculum learning

  • Curriculum learning faces the challenge of deciding which samples are easy or hard; prior QA work uses self-paced learning, while automatic curricula infer progress from training signals.The cited QA example applies self-paced learning; automatic curriculum learning extracts progress signals.Source: §4 · p. 8

Attention and long contexts

  • Cross-sentence and self-attention are common in reading comprehension, but self-attention memory complexity challenges long contexts; NarrativeQA's summary setting bypasses that challenge.The summary setting reverts to a more familiar reading-comprehension setup.Source: §4 · pp. 8–9

Answer generation

  • Most prior work emphasizes span prediction or multiple-choice QA, while generation-based QA includes S-NET's two-stage retrieve-then-generate framework.The passage characterizes generation-based QA as a more recent line of work.Source: §4 · p. 9

Pointer-generator models

  • Pointer-generator networks provide a copy-or-generate mechanism used in summarization and other sequence-to-sequence tasks.The passage also mentions CopyNet's copy mechanism and pointer-generators in a multi-task NLP architecture.Source: §4 · p. 9

5 Conclusion

The paper concludes that curriculum-based pointer-generator networks achieve state-of-the-art performance on NarrativeQA. Its diverse story sampling and curriculum scheme compare favorably with techniques designed for open-domain question answering.

Conclusion

  • The proposed curriculum-based pointer-generator model addresses reading comprehension over long narratives.The conclusion names the model as IAL-CPG.Source: §5 · p. 9
  • IAL-CPG achieves state-of-the-art performance on the NarrativeQA benchmark.Source: §5 · p. 9
  • The authors conclude that diverse story views trained with a curriculum are potentially more effective than techniques designed for open-domain question answering.The conclusion also notes extensive ablation studies and qualitative analysis.Source: §5 · p. 9

Related papers

Loading 1905.10847v1…