PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Question Generation from Text: A Preliminary Study

Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, Ming Zhou

arXiv:1704.01792v3cs.CL

TL;DR

Question-generation systems traditionally depend on rigid rules that are difficult to transfer across domains. This paper proposes the neural NQG encoder-decoder framework, which uses answer-position and lexical features to generate answer-focused questions. Experiments with SQuAD show fluent and diverse outputs, with reported gains over heuristic and sequence-to-sequence baselines.

  • Problem

    Existing question-generation methods mainly rely on rigid, human-designed rules that are difficult to adopt across domains.

  • Method

    NQG uses a neural encoder-decoder with bidirectional recurrent encoding of sentence, answer-position, and lexical features, followed by attention-based question decoding.

  • Results

    NQG outperforms PCFG-Trans and s2s+att in BLEU-4, while NQG++ exceeds PCFG-Trans by 0.76 in human-judge score.

  • Takeaways & Limitations

    The experiments show that neural models can generate fluent and diverse questions from text, supporting answer-focused question generation without predefined rules.

Abstract

from arXiv · show

Automatic question generation aims to generate questions from a text passage where the generated questions can be answered by certain sub-spans of the given passage. Traditional methods mainly use rigid heuristic rules to transform a sentence into related questions. In this work, we propose to apply the neural encoder-decoder model to generate meaningful and diverse questions from natural language sentences. The encoder reads the input text and the answer position, to produce an answer-aware input representation, which is fed to the decoder to generate an answer focused question. We conduct a preliminary study on neural question generation from text with the SQuAD dataset, and the experiment results show that our method can produce fluent and diverse questions.

1 Introduction

The paper frames neural question generation as a way to produce answer-focused questions from text without the rigid, domain-sensitive rules used previously. It studies this approach with SQuAD, reporting fluent and diverse generated questions.

  • 1 Introduction: Automatic question generation produces questions from natural-language text whose answers are contained in passage sub-spans.The task has potential educational value and can support large-scale question-answer corpora.
  • 1 Introduction: Earlier systems rely mainly on human-designed transformation and generation rules that are difficult to adapt across domains.
  • 1 Introduction: The NQG framework applies neural encoder-decoder models to generate natural-language questions without predefined rules.Its encoder incorporates answer position and lexical features, while an attention-based decoder generates answer-specific questions.
  • 1 Introduction: SQuAD provides more than 100K crowdsourced questions with answer spans, making it suitable for training and developing neural question-generation models.
  • 1 Introduction: Experiments on SQuAD show that the neural models can produce fluent and diverse questions from text.

2 Approach

The NQG approach combines a feature-rich bidirectional recurrent encoder with an attention-based decoder and a copy mechanism. It encodes sentence, answer-position, and lexical information to generate questions focused on a specified answer.

  • 2 Approach: The NQG framework consists of a feature-rich encoder and an attention-based decoder.
  • 2.1 Feature-Rich Encoder: A bidirectional GRU encodes concatenated word, lexical-feature, and answer-position vectors into contextual hidden representations.The encoder reads the inputs in both forward and backward orders and concatenates the resulting hidden states.
  • 2.1 Feature-Rich Encoder: BIO answer-position tags identify the target answer span so the encoder can support answer-focused question generation.B marks the span's start, I continues it, and O marks words outside the answer.
  • 2.1 Feature-Rich Encoder: Word case, part-of-speech, and named-entity tags provide additional lexical information to the encoder.The features are intended to encode linguistic information and help detect named entities in Wikipedia-based SQuAD text.
  • 2.2 Attention-Based Decoder: The attention-based GRU decoder combines previous-word embeddings, context vectors, and decoder states to predict the next question word.A maxout layer and softmax over the decoder vocabulary produce the next-word prediction.
  • 2.3 Copy Mechanism: A copy mechanism uses the decoder state and context vector to copy rare words from the source sentence.The system reuses attention probabilities to choose which source word to copy.

3 Experiments and Results

Experiments on SQuAD compare rule-based, sequence-to-sequence, feature-rich, and ablated NQG configurations using BLEU-4, human judgments, ablations, and question-type precision and recall. NQG++ improves automatic and human-evaluated results, while answer position and linguistic features contribute to answer-focused generation.

  • Experimental Setup: The SQuAD-derived training, development, and test sets contain 86,635, 8,965, and 8,964 sentence-answer-question triples.SQuAD contains more than 100K questions from 536 Wikipedia articles; the public test set was unavailable, so the development set was split.
  • Experimental Setup: The experiments compare PCFG-Trans, s2s+att, NQG, NQG+, NQG+Pretrain, NQG+STshare, NQG++, and feature ablations.NQG extends an attention-based sequence-to-sequence baseline with a feature-rich encoder; NQG+ adds copying, while later variants add pretrained or shared embeddings.
  • Automatic Evaluation: NQG++ outperforms PCFG-Trans and s2s+att by a large BLEU-4 margin, while NQG+ gains 2.05 BLEU and NQG++ gains 1.11 BLEU over NQG+.The reported gains are attributed to addressing rare words and improving word representations through pretrained vectors and shared encoder-decoder embeddings.
  • Human Evaluation: NQG++ outperforms PCFG-Trans by 0.76 score in human evaluation, and the raters show moderate agreement.Three human raters judged 200 test-set questions for meaningfulness and matching to the sentence and answer span.
  • Ablation Test: Removing the answer position indicator causes a severe performance drop, while removing case, POS, or NER features also affects question generation.The answer position indicator supplies the decoder with information about the target answer subsequence.
  • Question-Type Analysis: NQG++ performs well in both precision and recall for WHAT, HOW, WHO, and WHEN questions, but neither metric is acceptable for WHICH questions.WHICH questions represent about 7.2% of the training data, and alternative formulations may contribute to the weaker results; the same data sparsity may affect WHY questions.

4 Conclusion and Future Work

This preliminary study applies neural encoder-decoder models to natural-language question generation. The NQG framework uses answer position, POS, and NER information to generate answer-focused questions, while future work considers effects on question answering.

  • The study applies a neural encoder-decoder model to generate answer-focused questions from natural-language sentences.
  • The NQG framework enriches encoding with answer position, POS, and NER tag information.
  • Experiments show the effectiveness of the proposed NQG method.
  • Future work will investigate whether automatically generated questions can improve question answering systems.

A.1 Model Parameters

The model uses fixed vocabulary, embedding, hidden-state, feature, dropout, and beam-search settings for question generation.

  • The shared encoder-decoder vocabulary contains the top 20,000 frequent training-data words.
  • Word embeddings have size 300, GRU hidden states have size 512, and lexical and answer-position features use 32-dimensional embeddings.
  • Dropout is applied with probability p = 0.5.
  • Testing uses beam search with beam size 12.

A.2 Lexical Feature Annotation

Lexical feature annotation uses Stanford CoreNLP to assign part-of-speech and named-entity tags with default pretrained resources.

  • Stanford CoreNLP v3.7.0 annotates sentences with POS and NER tags.
  • Annotation uses CoreNLP’s default configuration and pre-trained models.

A.3 Model Training

Model parameters are randomly initialized and training uses Adam followed by simple SGD, with specified Adam hyperparameters.

  • Model parameters are randomly initialized using a Gaussian distribution with the Xavier scheme.
  • Training has two phases: Adam optimizes the loss first, followed by simple SGD.
  • Adam uses learning rate α = 0.001, momentum parameters β1 = 0.9 and β2 = 0.999, and ϵ = 10^-8.

B Human Evaluation Examples

Human evaluation compares PCFG-Trans and NQG++ using judges who rate whether generated questions match the input sentence and target answer. Table 5 illustrates the rating scheme with examples scored 3, 2, and 1.

  • Human judges evaluate PCFG-Trans and NQG++ using a rating scheme based on sentence-answer matching.
  • A score-3 question makes sense and supports the target answer, while score-2 and score-1 examples become progressively less adequate or incorrect.
  • Table 5 presents example inputs and outputs, with underlined words identifying the target answers.
Loading 1704.01792v3…