CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models

1. Introduction

• Language models rarely clarify ambiguous inputs and therefore perform poorly when answering ambiguous questions.The paper notes that ambiguity has received little attention in the LLM question-answering community.Source: §1 · p. 1
• CLAM detects ambiguous questions, asks for clarification, and improves question-answering performance while avoiding unnecessary clarification on precise inputs.The framework uses prompting to identify ambiguity, resolve it, and answer the disambiguated question.Source: §1 · pp. 1–2
• CLAM is presented as a proof of concept for metacognition: prompting a model to reason about a problem property before solving it.The authors frame this as a strategy for more reliable and safer foundation-model deployment.Source: §1 · p. 2
• The paper proposes an automatic, prompt-driven protocol in which a language model with privileged information stands in for a person during multi-turn evaluation.The authors advocate shifting evaluation research toward data-generating processes rather than fixed evaluation data sets.Source: §1 · p. 2

2. Selective Clarification QA Data Set

• The data set models mixed inputs: precise questions should be answered directly, while ambiguous questions should prompt clarification.The desired behavior is to avoid unnecessary clarification for well-defined questions.Source: §2 · p. 2
• Existing data sets ClariQ and CLAQUA do not support evaluation of every step in selective clarification QA.The paper refers readers to its related-work discussion for details of their limitations.Source: §2 · p. 2
• Ambiguous TriviaQA contains 200 pairs of ambiguous and precisely disambiguated questions derived from TriviaQA.Each ambiguous question is constructed to need just one piece of clarifying information to become precise.Source: §2 · p. 2
• Ambiguous questions are created by replacing a name or noun with a generic pronoun or replacing a noun phrase with its broader class.Examples include changing “Dame Judi Dench” to “she” and “silk producer” to “producer.”Source: §2 · pp. 2–3
• The questions are closed-book, and their typically short reference answers support more reliable automatic accuracy measurement.The questions stand alone without accompanying context.Source: §2 · p. 3

3. CLAM: Selective clarification Framework

• CLAM classifies a question first, answers unambiguous questions directly, and asks a follow-up before answering ambiguous questions.The described implementation completes one clarification round; repeating the process is proposed as future work.Source: §3 · p. 3
• The implementation uses prompting for ambiguity classification and clarifying-question generation, with zero-shot prompts on Ambiguous TriviaQA and few-shot prompts on CLAQUA.For classification, it uses the model’s next-token log probability for “True” as a continuous ambiguity predictor.Source: §3 · pp. 3–4
• State-of-the-art models can detect ambiguity but normally do not ask users for clarification.The authors conjecture that clarifying-question dialogues are scarce in pre-training and fine-tuning data, leaving a thorough investigation for future work.Source: §3 · p. 4

3.1. Meta-cognition

• For foundation models, meta-cognition can mean using information about the default sequence-completion task to choose a different task.CLAM applies this idea by prompting a model to detect ambiguity and redirect it toward asking a clarifying question rather than directly answering.Source: §3.1 · p. 4
• The paper presents CLAM as a proof of concept for meta-cognition in language models and suggests secondary classifiers could redirect model behavior in broader settings.The discussion gives toxicity detection as an example of a classifier that could select a more constructive prompt.Source: §3.1 · p. 4

4. Automatic Evaluation Protocol

• The protocol automates multi-turn dialogue evaluation without human input, addressing the expense, reproducibility, and idiosyncrasy concerns raised about human evaluation.The passage also notes potential ethics and approval costs associated with experiments involving people.Source: §4 · p. 4
• The authors say their work is the first to propose prompting language models with a parallel corpus of ambiguous and unambiguous questions to provide clarification.They suggest that model evaluation should move toward evaluation data-generating processes rather than evaluation data sets.Source: §4 · p. 4
• A language model with privileged information about the unambiguous question stands in for the user and provides clarification when asked.The parallel corpus pairs ambiguous questions with corresponding unambiguous questions, which are supplied to the oracle model to produce clarifying information.Source: §4 · pp. 4–5, Figure 4 · p. 5

5. Experiments

• Evaluation covers ambiguity types in Ambiguous TriviaQA, ClariQ, and CLAQUA, but only Ambiguous TriviaQA supports end-to-end testing of every pipeline step.Other datasets cover subsets of the pipeline; experiments use random samples of 400 examples for detection and 100 for generation and answer accuracy.Source: §5 · p. 5

5.1. Results

• The study reports text-davinci-002 results in the main text; davinci performs less well across tasks, with qualitatively similar method differences.Accuracy and clarifying-question appropriateness are manually evaluated, and AUROC measures ambiguity detection.Source: §5.1 · p. 6, §5 · p. 6

5.1.1. OVERALL PERFORMANCE

• Roughly 20 percentage points: CLAM boosts adjusted accuracy on TriviaQA containing ambiguous and unambiguous questions.Adjusted accuracy penalizes unnecessary clarification on unambiguous questions by λ = 0.8.Source: Figure 5 · p. 6, §5.1.1 · p. 7
• Without the unnecessary-clarification penalty, CLAM and always prompting perform comparably on the full set and outperform Default GPT and the Prompting baseline.On unambiguous questions, always prompting creates unnecessary turns and can sometimes lead the conversation off-topic, hurting accuracy.Source: Figure 5 · p. 6, §5.1.1 · p. 7

Step 1b: Detect Ambiguity

• Few-shot log-probability detection achieves high AUROC across datasets; Default GPT and Force clarification score roughly 0.5, while the Prompting baseline is only slightly better than random.Default GPT rarely clarifies, while Force clarification treats every question as ambiguous.Source: Step 1b: Detect Ambiguity · p. 7, Figure 6 · p. 7

Step 2: Ask clarifying questions

• Human judgments find correct clarifying questions for 84% of Ambiguous TriviaQA, 99% of CLAQUA I, and 95% of CLAQUA II examples.On TriviaQA, errors sometimes repeat the ambiguous question or ask an unrelated additional question.Source: Step 2: Ask clarifying questions · p. 7, Table 2 · p. 7

Steps 2 through 4: Final answer after clarification

• When questions are known to be ambiguous, clarification substantially improves answer accuracy across all evaluated datasets.ClariQ is omitted from this figure because it has no answers for its ambiguous questions.Source: Steps 2 through 4: Final answer after… · p. 7, Figure 7 · p. 8

Step 3: Oracle automatic evaluation

• The oracle model reliably supplies correct clarifying information on TriviaQA and CLAQUA II, but can fail to distinguish two proper-noun meanings in CLAQUA I.It sometimes provides information applicable to both possible entities rather than disambiguating them.Source: Step 3: Oracle automatic evaluation · p. 8

Related approaches

• Information-retrieval and conversational-search work has explored ambiguity detection and question generation, including logistic regression, BiLSTMs, templates, intent selection, and GANs.The overview is attributed to Keyvan & Huang (2022); examples include Trienes & Balog (2019), Dhole (2020), Wang & Li (2021), and Rao & Daumé III (2019).Source: §6 · p. 8

Datasets

• AmbigNQ’s low performance at finding precise questions and the subtlety of some AmbigQA ambiguities motivate the authors’ preference for Ambiguous TriviaQA as an evaluation tool.Ambiguous TriviaQA is described as requiring less factual knowledge.Source: §6 · p. 8

Transformer-based dialogue systems

• The authors identify Krasheninnikov et al. (2022) as concurrent work fine-tuning GPT-3, which modestly improves ambiguous-question accuracy but has a 44.5% false omission rate.Unlike that approach, CLAM requires no fine-tuning for question-answering improvement or the oracle model.Source: §6 · p. 8

• CLAM detects ambiguous questions, asks clarifying questions, and answers after clarification, increasing accuracy on ambiguous questions without affecting unambiguous ones.Source: §7 · p. 8
• Few-shot prompting reliably distinguishes ambiguous from unambiguous questions, helping avoid unnecessary clarification on precise inputs.Source: §7 · p. 8
• An automatic evaluation protocol uses an oracle language model with extra information to simulate users in multi-turn clarification dialogues.Source: §7 · p. 8

• Results with davinci are qualitatively similar to those with text-davinci-002, although at a lower level.Source: §A.1 · p. 11
• davinci distinguishes ambiguous from unambiguous questions less reliably than text-davinci-002.Source: §A.1 · p. 11
• davinci reliably generates clarifying questions across all data sets and relatively reliably provides clarification.Source: §A.1 · p. 11
• Clarification improves QA accuracy on ambiguous questions across all data sets with davinci.Source: §A.1 · p. 11

• CLAM outperforms default GPT and the baselines across different penalty terms λ; the main-paper results use λ = 0.8.Source: §A.2 · p. 11

A.3. Additional analysis of CLAM performance

• CLAM outperforms default GPT and the baselines across penalty terms λ on the full Ambiguous TriviaQA dataset.The adjusted-accuracy penalty applies when clarification is requested for an unambiguous question; default GPT and the prompting baseline incur no such penalty because they never ask.Source: Table 4 · p. 12
• On Ambiguous TriviaQA, CLAM and forced clarification improve accuracy on ambiguous questions, while performance on unambiguous questions is largely unaffected by CLAM.Forced clarification can create unnecessary turns on unambiguous questions and sometimes decrease accuracy by taking the conversation off-topic.Source: §A.3 · p. 11, Figure 10 · p. 12

B. Additional information on data sets and prompts

• Appendix B provides additional details about the datasets and prompts used in the experiments.Source: B. Additional information on data sets… · p. 13

B.1. Ambiguous TriviaQA

• Ambiguous TriviaQA pairs an ambiguous question with its precise counterpart and derives ambiguity by replacing a name or noun with a pronoun or a noun phrase with its class.Each ambiguous question is constructed to need only one piece of clarifying information.Source: §B.1 · p. 13

B.2. ClariQ

• ClariQ queries are labeled unambiguous for clarification-need ratings 1–2 and ambiguous for ratings 3–4; without reference answers, the dataset supports ambiguity detection only.Source: §B.2 · p. 14

B.3. CLAQUA I

• CLAQUA I is the single-turn dataset part with two same-name entities, ambiguity labels, and answers for ambiguous questions, but not unambiguous ones.The entity descriptions are provided to the question-answering model with the question.Source: §B.3 · p. 14

B.4. CLAQUA II

• CLAQUA II corresponds to the multi-turn portion of CLAQUA and includes a three-turn dialogue whose final turn is a question.Source: §B.4 · p. 15

C. CLAM Algorithm

• CLAM tunes its ambiguity threshold τ on a holdout set to maximize QA accuracy, then uses τ = −0.3 in the remaining experiments.The algorithm first obtains a model answer and classifies ambiguity; if the question is ambiguous, it generates a clarifying question, gets a user or oracle clarification, and returns an answer using the full dialogue.Source: C. CLAM Algorithm · p. 17