PaperlayerView exact revision on arXiv

Source-linked AI summary

Answer-based Adversarial Training for Generating Clarification Questions

Sudha Rao, Hal Daumé III

arXiv:1904.02281v1cs.CL

TL;DR

The paper targets generic clarification questions that may fail to elicit useful, specific information missing from a textual context. It uses hypothetical answers to estimate question utility and adversarially trains a generator against a utility-based discriminator. The authors report more useful and specific questions than maximum-likelihood and utility-reward reinforcement-learning models.

  • Problem

    Directly training sequence-to-sequence models on context-question pairs produces generic questions, despite the goal of eliciting useful, specific information missing from context.

    Source: §1 · p. 1

  • Method

    The approach generates a question and hypothetical answer, then uses a utility calculator as a discriminator to guide adversarial training of the question generator.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    The adversarially trained model generates more useful and specific questions than maximum-likelihood and utility-reward reinforcement-learning models.

    Source: §1 · p. 1, §5 · p. 9

  • Takeaways & Limitations

    The approach targets clarification questions whose answers would add useful information and make the textual context more complete.

    Source: §1 · p. 1

    The authors report that automatic metrics correlate with human judgments, but not enough to trust those metrics completely.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present an approach for generating clarification questions with the goal of eliciting new information that would make the given textual context more complete. We propose that modeling hypothetical answers (to clarification questions) as latent variables can guide our approach into generating more useful clarification questions. We develop a Generative Adversarial Network (GAN) where the generator is a sequence-to-sequence model and the discriminator is a utility function that models the value of updating the context with the answer to the clarification question. We evaluate on two datasets, using both automatic metrics and human judgments of usefulness, specificity and relevance, showing that our approach outperforms both a retrieval-based model and ablations that exclude the utility model and the adversarial training.

1 Introduction

The paper addresses generic clarification questions by aiming to elicit information missing from a textual context. It models hypothetical answers to estimate question usefulness and reports more useful, specific questions than baseline models.

1 Introduction

  • Directly trained sequence-to-sequence models produce generic clarification questions, while the goal is to generate questions that are useful and specific.The introduction gives “What are the dimensions?” as an example of a generic question about home appliances.Source: §1 · p. 1
  • The model generates a question and hypothetical answer, then estimates the question’s utility from whether the answer would make the context more complete.The paper casts the utility calculator as a GAN discriminator and the question generator as its generator.Source: §1 · p. 1
  • Human judgments and automatic metrics show the adversarially trained model generates questions more useful and specific to the context than baseline models.The evaluation uses Amazon product descriptions and Stack Exchange posts.Source: §1 · p. 1

2 Training a Clarification Question Generator

The training framework generates questions from context and uses a hypothetical answer and utility estimate to reward useful questions. It then trains the question generator adversarially against a utility-based discriminator.

2 Training a Clarification Question Generator

  • A context produces a question and hypothetical answer; a utility calculator scores the triple, and its reward updates the generator before adversarial training.The answer-generator-plus-utility component is then used as a discriminator between true-question and generated-question triples.Source: §2 · p. 2

2.1 Sequence-to-sequence Model for Question Generation

  • The question generator is an attention-based sequence-to-sequence model that encodes context and generates the question one token at a time.Its standard training objective maximizes the likelihood of context-question pairs.Source: §2.1 · p. 2

2.2 Training the Generator to Optimize UTILITY

  • MIXER trains the generator with a utility-based reward, beginning with maximum-likelihood steps and gradually shifting to reward optimization.A greedy-decoding reward baseline is used to reduce high variance and stabilize training.Source: §2.2 · p. 3

2.3 Estimating UTILITY from Data

  • The utility calculator estimates whether an answer would be a meaningful addition to its context by classifying context-question-answer triples.It represents the context, question, and answer with separate LSTMs and feeds their representations to a feed-forward network.Source: §2.3 · pp. 3–4

2.4 UTILITY GAN for Clarification Question Generation

  • The UTILITY calculator and answer generator serve as a discriminator, trained to distinguish true-question from model-generated triples.Positive instances pair a true question with its generated answer; the paper uses generated rather than true answers to avoid making the discriminator too strong.Source: §2.4 · p. 4

2.5 Pretraining

  • Pretraining maximizes likelihood for the question and answer generators, while the discriminator is trained on true versus randomly sampled questions.The question generator parameters are updated during adversarial training; the answer generator parameters remain fixed.Source: §2.5 · pp. 4–5

3 Experimental Results

The experiments compare generation models with retrieval and training ablations on Amazon and StackExchange data, using automatic metrics and human judgments. The human evaluation also measures annotator agreement, which is moderate for subjective usefulness and specificity judgments.

3 Experimental Results

  • The study asks whether generation beats retrieval, utility optimization beats maximum likelihood, adversarial training adds benefit, and models differ on usefulness and specificity.These questions define the experimental comparisons and the human-evaluation focus.Source: §3 · p. 5

3.1 Datasets

  • Amazon experiments use Home & Kitchen product descriptions and their paired questions and answers, with 19,119 training, 2,435 tune, and 2,305 test examples.The dataset has 3–10 questions per description, averaging seven.Source: §3.1 · p. 5
  • StackExchange data pairs posts with clarification questions and replies or post updates, with 61,681 training, 7,710 tune, and 7,709 test examples.For 500 tune and 500 test instances, expert annotators identified 1–6 additional valid questions.Source: §3.1 · p. 5

3.2 Baselines and Ablated Models

  • GAN-Utility combines a utility-discriminator GAN with the MIXER question generator; Max-Utility optimizes the utility reward without adversarial training.MLE is the pretrained maximum-likelihood question generator, and Lucene retrieves a question from one of the ten nearest training contexts.Source: §3.2 · p. 5

3.3.1 Automatic Metrics

  • Automatic evaluation measures output diversity through unique trigrams and reference similarity through BLEU and METEOR.BLEU measures n-gram precision, while METEOR also accounts for stemmed and synonym matches.Source: §3.3.1 · p. 6

3.3.2 Human Judgements

  • Human judgments assess relevance, grammaticality, whether a question seeks new information, specificity, and usefulness to product buyers or users.Specificity and usefulness use five-point scales, with usefulness ranging from useful only to the asker to suitable for a product description.Source: §3.3.2 · p. 6

3.3.3 Inter-annotator Agreement

  • Annotator agreement is high for relevance, grammaticality, and seeking new information, but moderate for usefulness and specificity.Usefulness ratings are collapsed from five categories into a binary label to reduce subjectivity.Source: §3.3.3 · p. 6

3.4 Automatic Metric Results

On Amazon, GAN-Utility performs significantly better on human-rated usefulness and product specificity, while automatic metrics show mixed comparisons. On StackExchange, it outperforms the ablations on BLEU and METEOR, but human evaluation was not conducted for that dataset.

3.4 Automatic Metric Results

  • On Amazon, GAN-Utility beats all ablations on diversity, while Lucene has the highest diversity but lower BLEU and METEOR.Lucene's human-written questions are longer; GAN-Utility beats all baselines on METEOR, but MLE has a higher BLEU score, consistent with its more generic outputs matching references.Source: §3.4 · pp. 6–7, Table 2 · p. 7
  • On StackExchange, GAN-Utility beats all ablations on BLEU and METEOR and beats MLE on diversity, while Max-Utility has higher diversity than GAN-Utility.Max-Utility produces longer outputs, but manual analysis finds them less grammatical.Source: §3.4 · p. 7

3.5 Human Judgements Analysis

  • Crowdsourced human evaluation covers Amazon, not StackExchange, because the latter data were too technical for crowdworkers to evaluate.The human-judgment results therefore do not establish performance on StackExchange.Source: §3.5 · p. 7
  • GAN-Utility performs significantly better on human-rated usefulness and product specificity; all models are more useful than Lucene and the reference questions.Lucene and reference questions are often useful only to the person asking, whereas GAN-Utility also scores significantly better on specificity.Source: §3.5 · p. 7

4 Related Work

Related work distinguishes clarification questions for missing information from reading-comprehension questions, and contrasts question generation with prior question-ranking and adversarial-generation approaches.

4 Related Work

  • Unlike reading-comprehension questions, clarification questions target answers missing from the given text and aim to fill information gaps.Source: §4 · pp. 7–8
  • Prior systems use templated questions, crowdsourced templates, images (Mostafazadeh et al., 2016, 2017), or address speech-recognition ambiguity.Source: §4 · p. 8
  • Rao and Daumé III (2018) rank candidate clarification questions, whereas this work generates them from context.Source: §4 · p. 8
  • GAN-Utility replaces SeqGAN’s policy-gradient generator with MIXER and its CNN discriminator with a utility calculator; Li et al. (2017) instead use a human-versus-machine classifier.Source: §4 · pp. 8–9

5 Conclusion

The paper reports that adversarial training produces more useful and specific clarification questions than maximum-likelihood and utility-reward training. It also notes limits in trusting automatic evaluation and calls for real-platform study.

5 Conclusion

  • The adversarial approach produces more useful and specific questions than maximum-likelihood and utility-reward models.Source: §5 · p. 9
  • The method generates a question from context, generates a hypothetical answer, scores the resulting triple for utility, and uses that score to retrain the question generator adversarially.Source: §5 · p. 9
  • Automatic metrics correlate with human judgments, but not enough for complete trust; the authors propose real-platform integration to assess utility and identify further research questions.Source: §5 · p. 9

A Sequence-to-sequence model details

The sequence-to-sequence model computes a distinct input context vector for each target step using global attention over input hidden states.

A Sequence-to-sequence model details

B Experimental Details

The experimental appendix specifies preprocessing, sequence-length limits, and sequence-to-sequence and MIXER settings.

B Experimental Details

  • Inputs are tokenized and lowercased; context, question, and answer lengths are capped at 100, 20, and 20 tokens, respectively.Longer tested lengths gave similar automatic-metric results but took longer.Source: B Experimental Details · p. 12
  • The sequence-to-sequence model uses in-domain GloVe embeddings of size 200, a vocabulary cutoff of 10, teacher forcing in training, and beam search of size 5 at test time.Source: B Experimental Details · p. 12
  • The recurrent models use hidden units of size 100, dropout 0.5, and learning rate 0.0001; MIXER starts with Δ=T and reduces it by 2 per epoch, stopping at 2.Source: B Experimental Details · p. 12

C.1 Amazon Dataset

Amazon product examples show that generated questions can be useful or specific yet still miss relevance or grammaticality. In one example, all three models ask about information already stated in the description.

C.1 Amazon Dataset

  • For the shower curtain, GAN-Utility produces a long, awkwardly phrased but specific and useful question, while Max-Utility’s longer output loses relevance.The authors report that unrelated-sentence outputs occur repeatedly in both utility-trained models.Source: §C.1 · p. 12
  • For the duvet-cover set, Reference and Lucene questions mainly serve the asker, while all three generated models ask irrelevant questions about a size stated in the description.The paper identifies such Reference and Lucene outputs as a main reason for their comparatively lower usefulness scores.Source: §C.1 · p. 12
  • For the mopping-clothes example, MLE and Max-Utility questions are specific and useful, while GAN-Utility repeats a word ungrammatically.The authors connect this repetition to sequence-to-sequence models’ difficulty maintaining long-range dependencies.Source: §C.1 · p. 12

C.2 Stack Exchange Dataset

Stack Exchange examples show GAN-Utility producing specific questions in some cases, alongside unknown-token errors on technical terms. Other systems’ examples vary in specificity and grammaticality.

C.2 Stack Exchange Dataset

  • In the boot-recovery example, GAN-Utility produces a specific, useful question; MLE’s question is generic, and Max-Utility’s is slightly ungrammatical.Reference and Lucene questions are also described as useful in this example.Source: §C.2 · p. 13, Table 6 · p. 14
  • For another post, Max-Utility and GAN-Utility produce fairly specific questions with unknown tokens, which the authors associate with technical terms and a long vocabulary tail.The Stack Exchange dataset’s technical terms lead to many unknown-token instances in both models.Source: §C.2 · p. 13, Table 6 · p. 14
  • In a third example, both Max-Utility and GAN-Utility generate specific, relevant questions, while Reference and MLE are generic.Source: §C.2 · p. 13, Table 6 · p. 14
Loading 1904.02281v1…