Source-linked AI summary
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Sudha Rao, Hal Daumé
TL;DR
The paper asks how machines can rank clarification questions whose answers will best help complete an underspecified post. It builds a neural model around expected value of perfect information, trains and evaluates it using StackExchange-derived question-answer data, and reports better performance than controlled baselines against expert annotations.
Problem
Machines need to identify useful clarification questions that fill missing information so they can better support assistance on underspecified posts.
Method
The paper combines neural networks with expected value of perfect information, ranking candidate questions by expected utility over possible answers and their effects on updated posts.
Results
The EVPI model outperforms baseline models against expert human annotations, with evaluation covering the original question and expert-marked good candidate questions.
Takeaways & Limitations
The results support EVPI as a promising formalism for clarification-question ranking and suggest extending the approach toward multi-turn conversations.
Takeaways & Limitations
Training uses noisy labels that treat the original question-answer pair as positive and nine alternatives as negative, although some alternatives may also be good questions.
Abstract
from arXiv · showhide
Inquiry is fundamental to communication, and machines cannot effectively collaborate with humans unless they can ask questions. In this work, we build a neural network model for the task of ranking clarification questions. Our model is inspired by the idea of expected value of perfect information: a good question is one whose expected answer will be useful. We study this problem using data from StackExchange, a plentiful online resource in which people routinely ask clarifying questions to posts so that they can better offer assistance to the original poster. We create a dataset of clarification questions consisting of ~77K posts paired with a clarification question (and answer) from three domains of StackExchange: askubuntu, unix and superuser. We evaluate our model on 500 samples of this dataset against expert human judgments and demonstrate significant improvements over controlled baselines.
1 Introduction
The paper frames clarification-question selection as ranking questions by the usefulness of their likely answers. It introduces an EVPI-based neural model and a StackExchange dataset for evaluating this task against expert judgments.
- Motivation: Clarification questions fill information gaps, and the best question is expected to produce a useful answer.The motivating example contrasts questions that are unlikely to help or overly specific with one that is useful and answerable.
- Task and approach: The model ranks candidate clarification questions by their usefulness to a given post.The proposed use case is a system suggesting questions while someone writes a post so missing information can be added immediately.
- Task and approach: The model uses expected utility over possible answers, estimating how useful each question would be if its possible answers were obtained.At test time, candidate questions and answers are retrieved from similar posts, answer representations are compared, and questions are ranked by expected utility.
- Contributions: The work contributes a neural EVPI model and a StackExchange-derived dataset for learning to rank clarification questions.The dataset contains approximately 77K post-question-answer triples, with expert annotations on 500 triples.
- Evaluation: The model is evaluated both on retrieving the original clarification question and on selecting any candidate question experts mark as good.The authors report that the EVPI model outperforms baseline models against expert human annotations.
2 Model description
The model ranks clarification questions by expected utility, estimating both likely answers and how much those answers would improve a post. It combines candidate generation, answer modeling, and utility prediction in a jointly trained neural architecture.
- EVPI ranks a question by the expected utility of its possible answers, combining answer probabilities P[aj|p, qi] with utility U(p + aj).The post, question, candidate answer, probability, and utility roles are defined in the model formulation.
- Candidate generation: Candidate questions and answers are retrieved from similar StackExchange posts, using Lucene to identify relevant posts and their associated questions.The retrieval process uses observed questions and answers from (post, question, answer) triples.
- Answer modeling: The answer model combines neural representations of a post and candidate question, then compares the resulting answer representation with candidate answers and question similarity.Answer likelihood uses both representation distance and cosine similarity between the candidate question and its paired question.
- Answer modeling: Training encourages an answer representation to match the original answer and answers paired with questions similar to the original question, but not dissimilar-question answers.This supports generalization across different phrasings of similar clarification questions and multiple valid answers.
- Utility calculation: The utility calculator predicts U(pi + aj) from representations of the post, candidate question, and answer, using positive original triples and negative candidate pairs.The negative-sampling scheme uses one positive and nine negative samples per post, with noisy labels because alternative questions may also be useful.
- Joint training: The model jointly trains recurrent representations and feedforward networks by minimizing the sum of answer-model and utility-calculator losses.The implementation uses GloVe embeddings, post/question/answer LSTMs, and feedforward utility modeling; the EVPI model differs from neural baselines in its joint loss.
3 Dataset creation
The dataset is constructed from StackExchange post histories and comments by pairing initial posts with clarification questions and answers extracted from subsequent edits or responses. The resulting resource contains 77,097 triples across three domains.
- Source records: The dataset represents each example as an initial unedited post, a comment containing a clarification question, and an extracted answer.StackExchange post histories, comments, and revisions provide the source records.
- Question extraction: Questions are taken from the first question comment and truncated at its question mark, with manually designed rules removing rhetorical non-clarification questions.About 7% of extracted questions are rhetorical questions that indirectly suggest a solution.
- Answer extraction: Answers are extracted from author edits that add missing information or from responses, with edits shorter than five words excluded as likely stylistic or grammatical changes.When both methods provide an answer, the semantically most similar answer to the question is selected using cosine distance between average word embeddings.
- Dataset statistics: 77,097 post-question-answer triples are collected across three StackExchange domains, and the dataset includes nine generated question-and-answer candidates per triple.The data is divided into train, tune, and test splits for the three domains.
4 Evaluation design
The evaluation ranks candidate clarification questions by usefulness and uses expert annotations to address noise in the original-question evaluation target.
- The task ranks a candidate set of clarification questions according to their usefulness to a given post.
- 35%-40% of posts receive multiple clarification questions, but the dataset retains only each post’s first clarification question.Later questions are difficult to distinguish from dialogue.
- Ten technical experts selected one “best” question and marked other acceptable questions as “valid” among ten Lucene-retrieved candidates.
- Annotation analysis: 0.15 agreement was obtained for strict “best” annotations, compared with 0.87 when one annotator’s “best” could be another’s “valid”; “valid” agreement was 0.58.
- Annotation analysis: 72% of samples had the original question marked “best” by at least one annotator, but only 20% had both annotators mark it “best”; 88% marked it “valid” at least once.These results motivate evaluation beyond the original question alone.
- Annotation analysis: 85% of posts had more than two jointly valid questions, while 60% still did after removing the original question.The candidate retrieval process therefore often contains useful alternatives.
5 Experimental results
Experiments compare random, feature-based, Community QA, neural, and EVPI models on expert annotations and alternative candidate sets, finding strongest advantages for EVPI in original-question settings.
- The study asks whether neural models improve on non-neural baselines, whether EVPI adds leverage, and how answers and original-question exclusion affect evaluation.
- Models are evaluated on 500 samples against the union of “best” annotations, the intersection of “valid” annotations, and the original question.Because annotators showed low agreement on a single best question, both union and intersection targets are used.
- Evaluating against expert annotations: Answer-aware neural baselines outperform Neural(p, q), showing that answers help select the right question.
- Evaluating against expert annotations: EVPI outperforms Neural(p, q, a) across most metrics despite matched information and parameter counts.EVPI additionally uses alternate question-answer candidates when computing its loss.
- Evaluating against the original question: Against the original question, Community QA again beats Neural(p, q) and approaches Neural(p, a), while EVPI significantly outperforms Neural(p, q, a).
- Excluding the original question: When the original question is excluded, neural models beat non-neural baselines, but differences among neural models are statistically insignificant.This setting avoids the overly optimistic assumption that the original question is in the candidate set.
6 Related work
Prior work emphasizes answerable reading-comprehension questions, question generation, search, and duplicate retrieval, whereas this dataset targets ranking clarification questions in StackExchange comments.
- Most question-generation research produces reading-comprehension questions answerable from supplied text, unlike clarification questions.
- Other approaches generate high-level templates, improve related-work writing, or create visual questions about image-inferable content.
- Community-forum research includes candidate-question generation, paraphrase ranking, and neural retrieval of similar questions.
- CQADupStack uses StackExchange forums for duplicate-question retrieval, while this dataset ranks clarification questions asked as post comments.
7 Conclusion
The paper contributes a dataset and an EVPI-integrated neural model for ranking clarification questions. Results support EVPI as a promising formalism, while future work targets generalization, generation, and evaluation.
- The authors construct a dataset for learning to rank clarification questions and propose a neural model integrating deep architectures with expected value of perfect information.The approach models how a questioner anticipates the usefulness of another party’s answer.
- The EVPI approach could be extended to reinforcement learning for multi-turn conversations.
- The results identify EVPI as a promising formalism for question generation.
- Future work includes generalizing question templates, moving from ranking to sequence-to-sequence question generation, and developing automatic evaluation methods.The evaluation challenge is motivated by expert annotation requirements and the existence of multiple valid questions.