PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning Based Graph-to-Sequence Model for Natural Question Generation

Yu Chen, Lingfei Wu, Mohammed J. Zaki

arXiv:1908.04942v4cs.CL

TL;DR

The paper addresses limitations in natural question generation caused by insufficient use of text structure, answer information, and cross-entropy-only training. It proposes an end-to-end RL-based Graph2Seq model with bidirectional graph encoding, hybrid evaluation, and deep answer alignment. On SQuAD, the method achieves new state-of-the-art results and significantly outperforms existing methods.

  • Problem

    Natural question generation needs methods that use text structure and answer information while avoiding exposure bias and train/test evaluation inconsistency from cross-entropy training.

  • Method

    The model combines a Graph2Seq generator with a bidirectional gated graph neural network, a hybrid cross-entropy/RL evaluator, and a Deep Alignment Network for word- and contextual-level answer integration.

  • Results

    The method achieves new state-of-the-art results on SQuAD and significantly outperforms existing methods; removing DAN lowers G2Ssta BLEU-4 from 16.96% to 12.62%.

  • Takeaways & Limitations

    Graph2Seq with reinforcement learning and deep answer alignment provides an end-to-end approach for generating more effective questions from passages and answers.

  • Takeaways & Limitations

    The formulation assumes supervised passage-answer-question pairs and treats passages as collections of word tokens.

Abstract

from arXiv · show

Natural question generation (QG) aims to generate questions from a passage and an answer. Previous works on QG either (i) ignore the rich structure information hidden in text, (ii) solely rely on cross-entropy loss that leads to issues like exposure bias and inconsistency between train/test measurement, or (iii) fail to fully exploit the answer information. To address these limitations, in this paper, we propose a reinforcement learning (RL) based graph-to-sequence (Graph2Seq) model for QG. Our model consists of a Graph2Seq generator with a novel Bidirectional Gated Graph Neural Network based encoder to embed the passage, and a hybrid evaluator with a mixed objective combining both cross-entropy and RL losses to ensure the generation of syntactically and semantically valid text. We also introduce an effective Deep Alignment Network for incorporating the answer information into the passage at both the word and contextual levels. Our model is end-to-end trainable and achieves new state-of-the-art scores, outperforming existing methods by a significant margin on the standard SQuAD benchmark.

1 INTRODUCTION

Natural question generation is limited by inadequate use of text structure and answer information, as well as cross-entropy training issues. The paper addresses these gaps with an RL-based Graph2Seq model combining graph encoding, hybrid evaluation, and deep answer alignment.

  • Existing QG methods may ignore rich text structure beyond simple word sequences, limiting their effectiveness.
  • Cross-entropy sequence training can suffer from exposure bias and train/test evaluation inconsistency, reducing performance on discrete generation metrics.
  • Early neural QG methods often omitted answer information, although answer semantics can make generated questions more relevant and readable.
  • The proposed model uses an RL-based generator-evaluator architecture to exploit hidden structure, improve text validity, and model passage-answer interactions.
  • The approach constructs static syntax-based or dynamic semantics-aware graphs, then encodes them with Graph2Seq and a bidirectional gated graph neural network.
  • A hybrid evaluator combines cross-entropy and RL losses, while a Deep Alignment Network incorporates answer information at word and contextual levels.

2 AN RL-BASED GENERATOR-EVALUATOR ARCHITECTURE

The architecture generates questions from a passage and target answer by combining answer-aware alignment, graph-based passage encoding, and sequence decoding. It uses a bidirectional graph neural encoder and reinforcement-learning-based evaluation to address structural, semantic, and training-objective limitations.

  • Problem formulation: The model generates answerable natural-language questions from a text passage and a target answer.
  • Deep Alignment Network: Deep Alignment Network uses attention-based soft alignment at word and contextual levels to incorporate answer information into passage representations.The aligned answer representations are concatenated with passage representations before further encoding.
  • Passage graph construction: The encoder constructs passage graphs from text and applies Graph2Seq modeling to exploit syntactic or semantic relations among passage words.The paper explores both static and dynamic graph construction approaches.
  • Bidirectional gated graph neural network: BiGGNN updates node representations using incoming and outgoing edges in an interleaved fashion, fusing information from both directions at each computation hop.This differs from independently learning both directions and concatenating them only at the final step.
  • Hybrid evaluator: The hybrid evaluator combines cross-entropy and reinforcement-learning objectives to improve training-test measurement consistency and guide syntactically and semantically valid generation.Its reward functions include BLEU-4 and negative word mover’s distance, with self-critical sequence training used for the reinforcement-learning component.

3 EXPERIMENTS

Experiments evaluate the proposed models on SQuAD using automatic metrics, human judgments, ablations, and generated-question examples. The models achieve strong benchmark results, while analyses identify the contributions and trade-offs of answer alignment, graph encoding, reinforcement learning, and graph construction.

  • Data and metrics: The experiments evaluate the models on SQuAD using two data splits, automatic metrics, human ratings, ablations, and generated-question examples.Automatic evaluation uses BLEU-4, METEOR, ROUGE-L, and Q-BLEU1; human evaluation rates syntax, semantics, and relevance.
  • Experimental results: Both G2Ssta+BERT+RL and G2Sdyn+BERT+RL achieve state-of-the-art scores on both SQuAD splits and significantly outperform previous methods.G2Ssta+BERT+RL outperforms G2Sdyn+BERT+RL on all reported automatic metrics and remains fully end-to-end trainable.
  • Human evaluation: Human evaluation finds that the best model outperforms MPQG+R and performs well relative to ground-truth questions on syntax, semantics, and relevance.The main syntactic errors involve repeated or unknown words, while lower semantic quality also affects relevance.
  • Ablation study: Static graph construction performs better than dynamic construction in almost every ablation setting, but each has a distinct structural trade-off.Static graphs can encode domain knowledge but depend on prior knowledge; dynamic graphs avoid that requirement and use attention to capture structure.
  • Ablation study: 16.96% BLEU-4 falls to 12.62% without DAN, while both word-level and contextual-level answer alignments improve performance.A DAN-enhanced Seq2Seq model also reaches 16.14 BLEU-4, compared with 13.29 for NQG++, 14.71 for MPQG+R, and 15.82 for AFPQA.
  • Ablation study: Graph2Seq improves over Seq2Seq by exploiting hidden passage structure, bidirectional message passing helps, and REINFORCE further improves performance.The study also reports considerable effects from pretrained BERT embeddings and further gains from fine-tuning them.

4 RELATED WORK

The paper positions its RL-based Graph2Seq model as a response to structural, training-objective, and answer-utilization limitations in prior question-generation systems.

  • Neural and graph-based QG: Existing QG systems rely on Seq2Seq architectures with attention, copy, or coverage mechanisms, while Graph2Seq architectures have been applied in other tasks.The related-work discussion situates Graph2Seq within broader graph-based sequence modeling research.
  • Limitations of existing approaches: Prior QG approaches ignore rich text structure, rely solely on cross-entropy, and fail to fully exploit answer information.These limitations motivate the paper’s model design.
  • Proposed direction: The proposed RL-based Graph2Seq model augments question generation with a Deep Alignment Network and introduces Graph2Seq to QG.The model is presented as the first application of Graph2Seq architecture to question generation.

5 CONCLUSION

The paper concludes that its RL-based Graph2Seq model combines answer alignment with bidirectional graph encoding for question generation. On SQuAD, it reports state-of-the-art performance and identifies broader graph-learning directions.

  • Model contributions: The model uses a Deep Alignment Network for answer information and a bidirectional GNN for the directed passage graph.These components address answer incorporation and passage-graph processing, respectively.
  • Results: On SQuAD, the method outperforms existing methods by a significant margin and achieves new state-of-the-art results.The conclusion reports the benchmark outcome without specifying individual metric values.
  • Future directions: Future work includes automatically learning graph structures from text and applying Graph2Seq to structured data such as knowledge graphs or tables.These directions extend graph construction and question-generation inputs beyond the current setting.

A DETAILS ON THE RNN DECODER

The decoder attends to relevant input content while combining generation and copying through a dynamically maintained extended vocabulary. Coverage further encourages use of diverse input components.

  • Attention: At each decoding step, attention computes a context vector from the decoder state, coverage vector, and attention memory.The context representation is conditioned on both current decoding information and prior coverage.
  • Generation and copying: A generation probability acts as a soft switch between generating vocabulary words and copying words from the input sequence.The input sequence includes passages and answers through the dynamically extended vocabulary.
  • Input utilization: The extended vocabulary unions the normal vocabulary with words in source examples, while coverage encourages the decoder to use diverse input components.Source examples comprise passages and answers.

B MODEL SETTINGS

The model settings combine fixed GloVe embeddings with trainable weighted BERT representations and specify compact tag, recurrent, and hidden-layer dimensions.

  • Word representations: The model fixes 300-dimensional GloVe vectors for the 70,000 most frequent training words and computes 1024-dimensional BERT embeddings on the fly.The BERT representation is a trainable weighted sum of all BERT layer outputs.
  • Model dimensions: Case, POS, and NER tag embeddings have sizes 3, 12, and 8, respectively, while the BiLSTM hidden state is 150 per direction.The concatenated bidirectional state therefore has size 300.
  • Model dimensions: All other hidden layers use size 300.

C SENSITIVITY ANALYSIS OF HYPERPARAMETERS

The sensitivity analysis examines how changing the number of GNN hops affects the G2Ssta model on SQuAD split-2. Results indicate limited sensitivity across hop counts.

  • GNN hop sensitivity: The experiment varies the number of GNN hops in the G2Ssta model using SQuAD split-2 data.Figure 3 reports the effect of this hyperparameter.
  • GNN hop sensitivity: The model achieves reasonably good results across various numbers of GNN hops.The reported results indicate that performance does not depend strongly on a single hop count.

D DETAILS ON BASELINE METHODS

The baselines extend sequence-to-sequence question generation with attention, copying, answer modeling, position awareness, long-sequence processing, or multi-task learning.

  • Sequence-to-sequence baselines: The Transformer baseline combines a Seq2Seq model with attention and copy mechanisms and is trained from scratch using OpenNMT.The reported baseline performed very poorly on the benchmarks despite moderate hyperparameter search and extensive training.
  • Copying mechanisms: SeqCopyNet extends copying by learning to copy sequences as well as individual words from the input sentence.
  • Answer and feature modeling: NQG++ uses attention, copying, and a feature-rich encoder for answer position, POS, and NER information.
  • Answer and feature modeling: MPQG+R applies reinforcement learning with a multi-perspective matching encoder for answer information, alongside copy and coverage mechanisms.
  • Answer and feature modeling: AFPQA combines answer-focused interrogative-word generation with position-aware modeling of context-to-answer distances.
  • Long-input processing: s2sa-at-mp-gsa uses a gated attention encoder and maxout pointer decoder for long inputs; sentence-level results are reported for matched settings.
  • Answer modeling: ASs2s treats the passage and answer separately in an answer-separated Seq2Seq model.
  • Copying and generation: CGC-QG uses multi-task learning to guide accurate boundaries between copying and generation.

E DETAILS ON HUMAN EVALUATION

The human evaluation uses anonymized system outputs rated by five evaluators on a five-point question-quality scale, with responses averaged across evaluators.

  • Rating procedure: Five human evaluators rate each anonymized output using a scale from 1 to 5.The scale ranges from 1, Poor, to 5, Excellent, with higher scores indicating better quality.
  • Rating procedure: Responses from all evaluators are collected and averaged for each evaluation question.

F MORE RESULTS ON ABLATION STUDY

The comprehensive ablation study assesses the contributions of the proposed model components across static and dynamic variants on SQuAD split-2.

  • Ablation design: The ablation study evaluates BERT, RL, DAN, BiGGNN, FEAT, DAN-word, and DAN-contextual across static and dynamic model variants.The evaluation uses the SQuAD split-2 test set.
  • Ablation findings: Experimental results confirm that every evaluated component contributes to overall performance.
Loading 1908.04942v4…