PaperlayerView exact revision on arXiv

Source-linked AI summary

Question Answering and Question Generation as Dual Tasks

Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, Ming Zhou

arXiv:1706.02027v2cs.CL

TL;DR

The paper addresses whether QA and QG can improve each other by exploiting their intrinsic duality and probabilistic correlation. It jointly trains differentiable QA and QG models with a duality-based regularizer, improving both tasks on three datasets while achieving comparable QA performance to strong baselines.

  • Problem

    QA and QG are intrinsically connected, but existing studies typically train them separately and ignore their probabilistic correlation.

  • Method

    A joint training framework uses QA and QG task losses together with a regularization term enforcing their probabilistic correlation.

  • Results

    The framework improves both QA and QG on three datasets, and the improved QA model performs comparably with strong baselines on all three.

  • Takeaways & Limitations

    QA and QG can be trained as equally important dual tasks whose probabilistic relationship supports mutual improvement.

Abstract

from arXiv · show

We study the problem of joint question answering (QA) and question generation (QG) in this paper. Our intuition is that QA and QG have intrinsic connections and these two tasks could improve each other. On one side, the QA model judges whether the generated question of a QG model is relevant to the answer. On the other side, the QG model provides the probability of generating a question given the answer, which is a useful evidence that in turn facilitates QA. In this paper we regard QA and QG as dual tasks. We propose a training framework that trains the models of QA and QG simultaneously, and explicitly leverages their probabilistic correlation to guide the training process of both models. We implement a QG model based on sequence-to-sequence learning, and a QA model based on recurrent neural network. As all the components of the QA and QG models are differentiable, all the parameters involved in these two models could be conventionally learned with back propagation. We conduct experiments on three datasets. Empirical results show that our training framework improves both QA and QG tasks. The improved QA model performs comparably with strong baseline approaches on all three datasets.

Introduction

QA and QG are dual tasks whose probabilistic relationship can be used to jointly train both models. The proposed differentiable framework leverages this relationship and improves both tasks across three datasets.

  • QA selects the most relevant answer sentence for a question, whereas QG generates a question answerable by an input sentence.
  • Their duality lets QA assess whether generated questions are relevant and lets QG provide question-given-answer probabilities useful for QA.
  • Existing approaches typically train QA and QG separately, ignoring their probabilistic correlation through the shared joint probability of questions and answers.
  • The framework jointly trains QA and QG by using their probabilistic correlation as a regularization constraint on both loss functions.
  • The implementation uses a sequence-to-sequence QG model and a recurrent-neural-network QA model, with differentiable components trained by back propagation.
  • Empirical results on three datasets show improvements for both QA and QG, while the improved QA model performs comparably with strong baselines.

The Proposed Framework

The proposed framework jointly optimizes answer selection and question generation with a regularizer enforcing their probabilistic duality. It combines task-specific objectives with language-model and QA-derived probabilities, and learns both models together.

  • Task Definition and Notations: The QA task ranks candidate answer sentences for a question, while the QG task generates a question from an answer sentence.
  • The Proposed Framework: The framework contains a QA model, a QG model, and a regularization term reflecting their duality.
  • Algorithm Description: The QA-specific objective classifies question-answer pairs using labels indicating whether each candidate answer is correct, requiring negative pairs.
  • Algorithm Description: The QG-specific objective trains generation on correct question-answer pairs, without requiring negative QA pairs.
  • Algorithm Description: The third objective regularizes training by enforcing the probabilistic duality constraint for each correct question-answer pair.
  • Algorithm Description: The models are updated jointly by minimizing a weighted combination of the original QA and QG losses and the regularization term.
  • Relationships with Existing Studies: Unlike approaches that treat QG as auxiliary to a main task or rely on task-specific pretraining, this framework gives QA and QG equal roles and learns them from random initialization.

The Question Answering Model

The QA model maps question-answer pairs to relevance scores for ranking candidate answers. It uses bidirectional recurrent representations and combines question, answer, interaction, and word-co-occurrence features.

  • During inference, the QA model scores each candidate answer for a question and returns the highest-ranked answer sentence.
  • Bidirectional recurrent networks encode variable-length questions and answers into fixed-length vectors using gated recurrent units.
  • The question-answer representation concatenates question and answer vectors, their element-wise product, and an embedding of word co-occurrence.
  • The model feeds the combined representation through a linear layer and tanh, then uses softmax with negative log-likelihood for QA training.
  • Word-co-occurrence embeddings empirically improve QA performance, while the ranking loss performs worse than negative log-likelihood in experiments.

The Question Generation Model

The QG model encodes an answer sentence and decodes it sequentially into a question, using attention to select relevant answer information. It is trained end-to-end and handles rare answer-related words through post-processing.

  • Architecture: The encoder maps a variable-length answer sentence to a fixed-length continuous vector, which the decoder uses to generate a question.The encoder uses a bidirectional GRU-based RNN, while the decoder generates question words sequentially.
  • Decoder: The decoder conditions each generated word on the answer representation and previously predicted question words.This sequential conditioning defines the question-generation process.
  • Attention: Attention selectively retrieves relevant information from the answer sentence when generating each question word.The attention state is computed by weighted averaging encoder hidden states.
  • Rare-word handling: Unknown words are replaced with the most relevant answer-sentence word using attention probability as the relevance score.This post-processing targets informative low-frequency words such as named entities or numbers.
  • Training and inference: All QG components are differentiable, so the model is trained end-to-end by minimizing negative log-likelihood.During inference, beam search returns the top-K confident questions and stops at the end-of-sentence symbol.

Experiment

Experiments evaluate the framework on MARCO, SQUAD, and WikiQA using established QA metrics and BLEU-4 for QG. Joint training improves QA and QG, while analyses identify both informative generation effects and dataset- and model-level limitations.

  • Experimental setting: Experiments use MARCO, SQUAD, and WikiQA, applying the same model to all three datasets and using official WikiQA splits.MARCO and SQUAD are repurposed from reading comprehension for answer sentence selection; WikiQA is a benchmark for that task.
  • Evaluation: QA is evaluated with MAP, MRR, and P@1, while QG is evaluated with BLEU-4 overlap against reference questions.The study notes that BLEU-4 measures literal overlap and is not a perfect automatic QG evaluation method.
  • QA results: On SQUAD, Dual QA achieves the best performance among the compared methods, while on MARCO it performs comparably with ABCNN.CDSSM outperforms word co-occurrence methods on MARCO, and Dual QA exceeds Basic QA according to the reported analysis.
  • QA results: On WikiQA, the Dual QA model performs comparably to strong baseline methods.The basic QA model is described as simple yet effective under the dataset’s standard experimental protocol.
  • QG results: Joint training improves QG performance on MARCO and SQUAD despite relatively low overall BLEU-4 scores.The comparison reports QG results with and without joint training.
  • Analysis and limitations: Dual QG generates more informative words and often greater overlap with reference questions, but it may still miss the reference question’s point.The analysis links these generated-question patterns to the QA and QG scoring interactions, while noting that some generated questions are not point-correct.

Related Work

Prior work spans text-based, knowledge-based, community-based, and reading-comprehension QA, alongside diverse approaches to generating questions from text or structured knowledge.

  • QA research includes text-level, knowledge-based, community-based, and reading-comprehension tasks.
  • This paper focuses on text-based QA, where the answer is represented as a sentence.
  • Existing QG methods include semantic representation and realization, overgenerate-and-rank pipelines, ontology alignment, and template selection.
  • Other QG studies generate questions from knowledge-base facts using neural recurrent networks.

Conclusion

The paper jointly trains QA and QG models by exploiting their probabilistic duality as a regularization term. Experiments show improvements on both tasks across three datasets.

  • The framework jointly trains neural QA and QG models using their probabilistic correlation as a regularization term.
  • Experimental results show that the proposed training framework improves both QA and QG on three datasets.
Loading 1706.02027v2…