Resolving Intent Ambiguities by Retrieving Discriminative Clarifying Questions

• Task-oriented dialogue systems extract semantic information from natural-language queries for downstream tasks including domain classification, intent detection, and slot filling.The paper situates its approach in commercial applications such as personal assistants and booking systems.Source: §1 · p. 1
• Ambiguous or underspecified queries can fail to map directly to predefined intents, such as a request to open an account that could mean savings or checking.Creating a separate general “opening an account” intent would require task workflows, extra training data, and retraining classifiers.Source: §1 · p. 1
• The proposed rule-based system uses an existing question generator to generate discriminative questions between two intents, illustrated by a conversation in which two questions clarify the user’s intent.The paper assumes a pipeline that classifies the domain before the intent.Source: §1 · p. 1, Figure 1 · p. 1
• Clarification can shift a non-deterministic classifier’s task from top-1 to top-k intent classification, while making the exchange more interactive.The user’s answer supplies clarification that gives the classifier some slack in performance.Source: §1 · p. 2

• Dialogue research established clarification taxonomies and studied entity disambiguation, while later work built clarification datasets for knowledge-based and open-domain conversations.The cited work includes Purver et al. (2003a,b), Purver (2004), Healey et al. (2003), Coden et al. (2015), Xu et al. (2019), and Aliannejadi et al. (2019).Source: §2 · p. 2
• Neural clarifying-question models have used context-question training pairs, but most still require large amounts of Wizard-of-Oz dialogue annotations.Rao and III (2019), Cao et al. (2019), and Zamani et al. (2020) are cited examples.Source: §2 · p. 2
• Other work captures discriminative attributes of confusing law articles, generates disambiguating rewrites of open-domain questions, or asks discriminating questions about image pairs.The cited examples are Xu et al. (2020), Min et al. (2020), and Li et al. (2017), respectively.Source: §2 · p. 2
• Unlike Yu et al. (2020), which manually generates questions and answer choices, this approach uses TF-IDF for potential answers and an existing question generator with a discriminative similarity measure.The authors also describe the approach as easier to integrate with pre-deployed intent classifiers and suitable for small amounts of intent training data.Source: §2 · p. 2

3 Model

• The model classifies incoming queries, selects a discriminative question, falls back to a template when needed, and classifies the reply as one of two intents or neither.The workflow uses an intent classifier, a pre-trained question generator, and a response classifier.Source: §3 · p. 2

3.1 Intent Classification and Ambiguity Detection

• A query is ambiguous if its top intent score is below a confidence threshold or the top two intent scores differ by less than a second threshold.The classifier assigns intent probabilities using a softmax layer.Source: §3.1 · pp. 2–3

3.2 Discriminative Question Selection

• Question selection draws generated question-answer pairs from training utterances for the two leading intents and scores candidate pairs for discrimination.The method favors questions similar to each other and to the user query, with answers that are dissimilar across intents.Source: §3.2 · p. 3
• The selection score combines similarity between candidate questions, similarity of each candidate question to the user query, and dissimilarity between their answers.The answer dissimilarity reflects the hypothesis that each answer should be distinctive to its intent.Source: §3.2 · p. 3

3.3 Template Based Clarification Question

• When generated questions are insufficient, TF-IDF n-grams from training data provide discriminative phrases for template questions with candidate answers.The approach addresses cases such as limited training examples, unseen ambiguity, or intents with very different verbs; candidate answers can steer users toward a workflow.Source: §3.3 · p. 4

3.4 Ambiguity Resolution

• The response classifier compares the user’s reply with the two candidate answers and a set of common “none” keywords using sentence encodings and cosine similarity.It assigns the reply to the closest of the two intents or the neither option.Source: §3.4 · p. 4

4 Experiments and Results

• Around 4% for a BiLSTM and around 3% for a linear SVM is the potential bandwidth to increase F1-score through clarification.Table 1 is identified as reporting intent-classification performance.Source: §4 · p. 4, Table 1 · p. 5

4.1 Performance of Ambiguity Detection

• With t2 = 0.3, the top-two intents were identified in 839 and 709 of 1,068 cases for the linear SVM and BiLSTM, respectively.The evaluation did not check the order of the two intents.Source: §4.1 · pp. 4–5
• 38 of 100 manually reviewed examples explicitly represented both expected user intentions rather than a shared abstract representation.Source: §4.1 · p. 4
• Close top-two scores produced false ambiguities among single-class examples, while softmax calibration pushed almost all examples into the unambiguous zone.Source: Figure 2 · p. 5, Figure 3 · p. 5

4.2 Performance of Discriminative Questions

• Examples illustrate ambiguity through a shared “add” verb, the phrase “office” paired with “change,” and a generic “do” for different verbs.The “do” example uses two answer options.Source: Figure 4 · p. 5, Figure 5 · p. 5, Figure 6 · p. 6
• 3.84/5 grammaticality and 4.17/5 relevance were the average question ratings; questions generated using only QG had 34% coverage.MTurk raters assessed 100 ambiguous examples detected by both classifiers, considering the two source utterances when rating relevance.Source: §4.2 · pp. 5–6

5 Discussion

• The approach detects ambiguity and generates discriminative questions from user utterances using an existing question generator.It is intended to improve intent classification and enhance interaction.Source: §5 · p. 6
• Separating the rule-based approach from intent-identification logic makes it easy to deploy in systems with pre-existing intent classifiers.Source: §5 · p. 6
• Coverage of the discriminative-similarity approach remains low, leaving substantial scope for improvement.Source: §5 · p. 6
• Scaling to implicit query ambiguities or clarification references in external knowledge bases remains critical.The current approach identifies discriminative attributes in surface forms of user utterances; correctly identifying such attributes is key to strong questions, as validated in the visual counterpart (Li et al., 2017).Source: §5 · p. 6