Source-linked AI summary
Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia
Xinya Du, Claire Cardie
TL;DR
The paper targets question-answer generation from Wikipedia that covers content beyond a single sentence, addressing the limitations of sentence-level systems and costly dataset creation. It proposes CorefNQG, a gated neural model using coreference-cluster representations, and finds that it outperforms state-of-the-art baselines. Applied to 10,000 top-ranking Wikipedia articles, the framework produces a 1.26 million-pair corpus.
Problem
The paper addresses the need for high-quality QA pairs that can use information beyond a single sentence, while existing crowdsourced datasets are costly and sentence-level systems miss cross-sentence information.
Method
CorefNQG is a neural sequence model with a novel gating mechanism that uses continuous representations of coreference clusters for paragraph-level question generation.
Results
CorefNQG outperforms state-of-the-art baselines on SQuAD, with a larger performance gap on examples requiring coreference resolution.
Takeaways & Limitations
Applying the framework to the 10,000 top-ranking Wikipedia articles produces a corpus of 1.26 million question-answer pairs for QA research.
Takeaways & Limitations
The task permits questions to rely on preceding sentences, but the supplied formulation does not state a broader scope boundary.
Abstract
from arXiv · showhide
We study the task of generating from Wikipedia articles question-answer pairs that cover content beyond a single sentence. We propose a neural network approach that incorporates coreference knowledge via a novel gating mechanism. Compared to models that only take into account sentence-level information (Heilman and Smith, 2010; Du et al., 2017; Zhou et al., 2017), we find that the linguistic knowledge introduced by the coreference representation aids question generation significantly, producing models that outperform the current state-of-the-art. We apply our system (composed of an answer span extraction system and the passage-level QG system) to the 10,000 top-ranking Wikipedia articles and create a corpus of over one million question-answer pairs. We also provide a qualitative analysis for this large-scale generated corpus from Wikipedia.
1 Introduction
The paper addresses the need for scalable, high-quality QA generation from Wikipedia paragraphs that can use information beyond a single sentence. It introduces CorefNQG, which improves paragraph-level question generation by encoding coreference information, and applies the system to create over one million QA pairs.
- Motivation: Large-scale QA datasets support reading-comprehension research, but high-quality crowdsourced examples are expensive and smaller than synthetic alternatives.Synthetic datasets are large and inexpensive but differ from explicit QA questions; crowdsourced datasets require expertise and careful design.
- Motivation: Existing automatic QA generation from knowledge bases is limited by errors, incompleteness, acquisition difficulty, and questions restricted to single facts.
- Problem: Nearly 30% of human-generated SQuAD questions rely on information beyond one sentence, including coreference across sentences.For example, pronouns such as “His” and “He” must be linked to “Tesla” in an earlier sentence.
- Approach: CorefNQG uses a novel gating mechanism over continuous coreference-cluster representations for paragraph-level question generation.Coreference clusters contain the mentions used to refer to each entity.
- Evaluation: CorefNQG significantly outperforms single-sentence baselines and a model encoding preceding context plus the input sentence on SQuAD.Its performance gap is even larger on examples requiring coreference resolution.
- Corpus: Applying the system to the 10,000 top-ranking Wikipedia articles produces a dataset with over one million QA pairs and a qualitative analysis.
2 Related Work
Prior question-generation and QA-dataset work includes rule-based, neural, knowledge-base, crowdsourced, and synthetic approaches. This paper differs from related raw-text methods by incorporating contextual information beyond the sentence level through coreference knowledge.
- Question Generation: Early question-generation systems used hand-designed rules, templates, or overgenerate-and-rank pipelines to transform text into questions.
- Question Generation: Neural approaches use encoder-decoder or attention-based sequence models to generate questions from knowledge-base triples, graphs, sentences, or paragraphs.
- Question Generation: This method differs from related raw-text approaches by using coreference knowledge to incorporate contextual information beyond the sentence level.
- QA Datasets: QA datasets vary in scale and construction, including crowdsourced datasets such as SimpleQuestions, SQuAD, and TriviaQA.SQuAD contains questions written from Wikipedia paragraphs, while the examples cited include 5,810 WebQuestions and 108,442 SimpleQuestions questions.
- QA Datasets: Semi-synthetic datasets are easier to build at large scale but can be noisy and often use cloze-style questions.The cited CNN and Daily Mail dataset contains over a million examples, and later work found it noisy.
- QA Datasets: The closest related work generates human-like QA pairs from Freebase using a neural triple-to-sequence model.
3 Task Definition
The task is to harvest question-answer pairs from article paragraphs through candidate answer extraction and answer-specific question generation. Generated questions may use preceding sentences when forming questions about candidate answer spans.
- Task Formulation: The pipeline first extracts question-worthy candidate answer spans from an input paragraph.The candidates are represented as ans = (ans1, ans2, ..., ansl).
- Task Formulation: For each candidate answer, the system generates a token sequence Q based on the sentence S containing that candidate.The question is modeled as Q = y1, ..., yN.
- Question Criteria: A generated question should ask about a potentially interesting aspect of the candidate answer for a human reader.
- Question Criteria: A generated question may rely on information from sentences preceding the candidate’s sentence in the paragraph.
- Question Model: The question-generation probability conditions on the output prefix, the containing sentence S, and preceding-sentence context C.C is defined as the set of sentences preceding S in the paragraph.
4 Methodology
The framework first extracts candidate answer spans, then generates answer-specific questions with CorefNQG, which enriches paragraph inputs with coreference and answer features before decoding.
- 4 Methodology: The pipeline extracts candidate answer spans from an input paragraph and then generates corresponding questions.At test time, answer extraction precedes question generation.
- 4 Methodology: CorefNQG transforms input sentences by appending representative antecedent mentions to pronouns using coreference clusters.Proper nouns are preferred when selecting the representative antecedent.
- 4 Methodology: A gating network refines coreference position features using mention-pair scores to learn the importance of coreferent phrases.The refined representation is combined with answer features and word embeddings.
- 4 Methodology: The encoder input concatenates refined coreference, answer-position, and word embeddings before bidirectional LSTM encoding.Coreference and answer spans use BIO-style position features.
- 4 Methodology: An attention-equipped decoder generates questions token by token and can copy words directly from the source sentence.The output distribution combines vocabulary-generation and source-copy probabilities over a dynamic vocabulary.
- 4 Methodology: Candidate answer extraction is formulated as sequence labeling with feature-rich embeddings and a BiLSTM-CRF model.Each token representation combines word, named-entity, and character-level features; the CRF scores complete tag sequences.
5 Experiments
Experiments evaluate question generation and answer extraction on SQuAD, compare CorefNQG with baselines and ablations, and assess noisy predicted-answer training; the full system is applied to 10,000 Wikipedia articles.
- 5 Experiments: SQuAD provides over 100k crowdworker questions from Wikipedia articles, with each answer represented as a text segment.The dataset was built from articles selected from Wikipedia’s top 10,000 articles.
- 5 Experiments: The authors augment training with examples using predicted answer spans to quantify their effect on question generation.These noisy examples constitute 42.21% of the augmented training set.
- 5 Experiments: The complete system is applied to Wikipedia’s 10,000 top-ranking articles to generate a corpus of one million question-answer pairs.The application uses the answer extraction and question generation systems together.
- 5 Experiments: Question generation is evaluated with BLEU and METEOR, while answer extraction uses precision, recall, F-measure, and overlap metrics.Binary and Proportional Overlap supplement Exact Match because answer boundaries can be ambiguous.
- 5 Experiments: CorefNQG is compared with sentence-level sequence-to-sequence baselines, copy-augmented models, full-context ContextNQG, and coreference ablations.The ablations remove gating and refined coreference positions or set mention-pair scores to zero.
- 5 Experiments: BiLSTM answer extraction variants outperform the off-the-shelf NER system, while BiLSTM-CRF with character and NER features achieves the best F-measure.NER has higher recall despite lower overall performance than the BiLSTM variants.
6 Results and Analysis
CorefNQG consistently outperforms baseline question-generation models, including on examples requiring pronoun resolution and when trained with noisy predicted answer spans. Human evaluation shows that CorefNQG improves over ContextNQG, while human questions remain stronger in sense, answerability, and naturalness.
- Automatic Evaluation: CorefNQG outperforms seq2seq baselines and both Seq2seq+Copy variants, demonstrating benefits from copy, answer, and coreference features.The authors attribute additional gains over full-context Seq2seq+Copy models to explicitly gated coreference knowledge.
- Automatic Evaluation: 36.42% of test examples require pronoun resolution, and performance gaps between CorefNQG and baselines remain significant on this harder subset.All three systems perform worse on the pronoun-resolution subset than on the full test set.
- Automatic Evaluation: CorefNQG still outperforms baseline models across all metrics after training with noisy predicted answer spans, despite a consistent but acceptable performance drop.The drop is attributed to inaccuracies in the predicted answer spans.
- Automatic Evaluation: BiLSTM answer extractors outperform the off-the-shelf NER system, while BiLSTM-CRF with character-level and NER features achieves the best F-measure.The NER system has higher recall despite lower overall performance than the BiLSTM variants.
- Human Study: Human questions are preferred over both neural systems for sense and answer correspondence, especially when preceding context is required, but neural models achieve near-human grammaticality.CorefNQG performs statistically significantly better than ContextNQG across all human-evaluation metrics.
- The Generated Corpus: 1,259,691 question-answer pairs are generated, with nearly 126 questions per article; question-type distributions resemble SQuAD for several categories but differ for others.The corpus contains more “What is”, “What was” and “What percentage” questions, whereas SQuAD contains more “What did”, “Why” and “Which” questions.
7 Conclusion
The paper presents a coreference-aware neural model for paragraph-level question generation and applies the framework to create a large Wikipedia QA corpus.
- CorefNQG improves paragraph-level question generation by encoding coreference knowledge.The model uses a novel approach to better represent coreference information.
- 1.26 million question-answer pairs were produced by applying the question generation framework.
- 2F-1 measures average overlap between predicted and ground-truth answer spans.
A.1 Example Question-Answer Pairs from the Corpus
The corpus appendix provides examples of generated question-answer pairs drawn from Wikipedia passages covering national history, identifiers, and country descriptions.
- The corpus includes a France passage describing its cultural sites, tourism, economy, and household wealth.
- A United States passage is paired with questions about the war ending legal slavery and the war confirming the country’s status.
- An ISNI passage supports questions about identifier names, digit count, and display format.The passage states that an ISNI has 16 digits and may be divided into four blocks.
- An India passage introduces the country’s location, population, area ranking, and neighboring regions.
A.2 Human Rater Guidelines
The appendix presents human-rater instructions and specifies a five-point rating scale for judging each category.
- The rater guidelines are introduced as instructions for evaluating generated question-answer pairs.
- Ratings range from 1 to 5, with 5 fully satisfying the scheme and 1 completely failing to satisfy it.A rating of 3 represents borderline cases.
A.3 Training and Implementation Details
The training setup uses fixed vocabulary and model dimensions, dropout, random initialization, and stochastic gradient descent with minibatches.
- The input and output vocabularies retain the 50k most frequent training-data words.
- Word embeddings have size 128, while LSTM hidden states have size 256.
- Dropout uses probability p = 0.3, and parameters are initialized uniformly between −0.1 and 0.1.
- Stochastic Gradient Descent is used with mini-batches of size 64.