Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions

1 INTRODUCTION

• The proposed preference method simulates annotators answering clarifying questions, then evaluates the model’s subsequent responses against their expected interpretations.Preferences over clarifying questions and direct answers are aggregated across annotators.Source: §1 · pp. 1–2
• The evaluation framework measures whether systems recover each user’s expected answer and how many model turns they use.It evaluates open-domain QA with simulated user interactions using effectiveness and efficiency measures.Source: §1 · p. 2
• 4-5% improvement in F1 over standard preference annotation methods was consistent across three base LLMs when training used double-turn preferences.The paper also reports that this method can train systems to decide whether to clarify or answer directly.Source: §1 · p. 2
• The paper presents its evaluation paradigm and preference labeling scheme as support for future research on interactive LLM assistants.Its listed contributions also include training models to judge when clarification is needed.Source: §1 · p. 2

2 TASK: QA WITH SIMULATED USER INTERACTION

• Open-domain QA is used because clarification needs are not immediately identifiable, and the task measures both answer effectiveness and interaction efficiency.Effectiveness uses F1 against the gold answer set, with clarification answers required to match the corresponding user’s answer; efficiency counts system turns.Source: §2 · p. 3, §2.1 · p. 3

2.1 INTERACTION SCENARIO

• The interaction scenario assumes a query has multiple users with distinct intents and expected answers, and aims to satisfy as many as possible with few interactions.A clarifying initial response triggers simulated user replies and tailored responses; otherwise, the initial response is treated as the final prediction.Source: §2.1 · p. 3

2.2 EVALUATION FRAMEWORK IMPLEMENTATION

• The evaluation framework identifies clarifying questions by a text prefix and uses GPT-4 to simulate user replies from the query, question, and target answer.If a question does not address the ambiguity, or the simulated reply leaks the target answer, the example is treated as incorrect or abstained, respectively.Source: §2.2 · pp. 3–4

3 FINE-TUNING LLMS TO ASK CLARIFYING QUESTIONS

• The training approach constructs an instruction-tuning dataset for clarifying responses, applies supervised fine-tuning, and then performs preference learning.The preference stage compares clarifying questions using their outcomes after an additional interaction.Source: §3 · p. 4

3.1 PREFERENCE DATA GENERATION FROM USER-LLM INTERACTIONS

• Double-turn preferences use each annotator’s interpretation and later model outcome, preserving individual judgments instead of collapsing them into one majority label.Annotators answer candidate clarifying questions and rank them or direct answers by whether the final prediction matches their expected interpretation; ties favor direct answers.Source: §3.1 · p. 4
• For training, a user simulator predicts each user’s clarifying reply from the query, candidate question, and annotated answer; candidate responses are ranked by average exact-match accuracy.Direct answers are likewise evaluated against each annotator’s expected answer, with ties favoring direct answers.Source: §3.1 · pp. 4–5

4 EXPERIMENTAL SETTINGS

• Experiments use Llama2-7b, Gemma-7b, and Llama3-8b base models, with 8-bit quantization and LoRA for training and inference.Instruction-tuned models are excluded because of data-leakage concerns.Source: §4 · p. 5
• About 10% of NQ-Open training questions and 42% of development questions have multiple distinct answers tied to annotators’ interpretations.NQ-Open questions come from real Google search queries and are annotated with Wikipedia answers.Source: §4 · p. 5
• About 56% of NQ-Open queries are identified as ambiguous in AmbigQA, which recovers 2.6x more answers; its test set includes 1,172 ambiguous and 788 unambiguous questions.The AmbigQA test set contains 1,960 questions; ambiguous questions average 3.7 answers each.Source: §4 · pp. 5–6
• GPT-4 proposes clarifying questions and simulated responses using feasible answers drawn either from human annotations or from Llama-2-7b predictions.Model-predicted answers are sampled repeatedly and filtered to retain examples with at least one answer matching an NQ-Open annotation.Source: §4 · p. 6
• Each feasible-answer method yields 4,400 query–clarifying-question pairs, split into 4,000 training and 400 development examples.Together, the two datasets contain 19,807 query, question, simulated-response, and answer examples.Source: §4 · p. 6

4.1 COMPARED SYSTEMS

• Turn 2 clarification-capable systems share a Turn 4 answer model trained on all generated SFT examples, isolating differences associated with the clarifying question.A separate model handles the answer-after-clarification step; joint models are explored later in the paper.Source: §4.1 · pp. 6–7
• The comparisons include direct-answer SFT and clarification SFT, plus DPO variants ranked by a reward model, answer likelihood, or answer matching.Clarify-or-direct SFT and DPO models are also trained; the DPO model uses double-turn preferences over clarification questions and direct answers.Source: §4.1 · p. 7

5 RESULTS

• Double-turn preference labeling consistently achieves the best Answer F1, with Clarify DPO w/ Match yielding the best overall results; selective clarification trades off effectiveness and turn count.Table 1 distinguishes systems that never clarify, always clarify, or can do either; the caption says Clarify DPO w/ Match attains optimal Answer F1 at the cost of always asking.Source: §5 · p. 7, Table 1 · p. 8
• RLHF annotation struggles to identify useful clarifying questions, while double-turn preferences assessed by later outcomes consistently improve useful-question generation on both ambiguous and unambiguous queries.Source: §5 · p. 8
• For Llama2, model-ambiguity SFT data improves over human-ambiguity data alone, but improvements are minor for Gemma and Llama3.The model-ambiguity answer sets were generated from Llama2 predictions; the authors suggest base-model-specific data may improve clarifying-question quality.Source: §5 · p. 8, Table 2 · p. 8

A Joint Model for Asking Clarifying Questions and Predicting Answers Here,

• With a joint model, overall performance degrades versus separate turn-specific models, but most gains over Direct-Ans and single-turn-preference QA-with-Clarification remain.The authors identify mixture-of-experts methods and improved multi-turn learning algorithms as possible future directions for mitigating this degradation.Source: Table 3 · p. 8, A Joint Model for Asking Clarifying… · p. 9

5.1 DETERMINING WHEN CLARIFYING QUESTIONS ARE NECESSARY

• The Clarify-or-Answer system outperforms baselines on all reported metrics except Ambig Accuracy; PPDPP has nearly 10% higher ambiguity accuracy, while the proposed system has higher answer accuracy.The authors interpret the results as showing that their system more often clarifies when useful and avoids clarification when it cannot successfully use the answer.Source: §5.1 · p. 9, Table 4 · p. 9

Ambiguity in NLP

• Prior NLP ambiguity research spans machine translation, natural-language inference, entity linking, coreference, and temporal or geographical context.Cross-annotator disagreement in NLI has been linked to input ambiguity.Source: §6 · pp. 9–10

Uncertainty in LLMs

• LLM uncertainty work studies calibration, confidence expression, and how uncertainty relates to missing knowledge versus missing clarification.Prior findings cited here report that calibration can be lost after RLHF fine-tuning and that single-turn RLHF encourages overconfidence.Source: §6 · p. 10

Clarifying Question

• Earlier clarification-training work often targets tasks with fixed clarification needs, such as predetermined relevant features or multiple valid answers in a reading context.Applications cited include social and moral judgments and image classification.Source: §6 · p. 10
• Andukuri et al. (2024) also use user simulation for task ambiguity, while this work applies preference optimization to open-domain QA rather than filtering questions for supervised fine-tuning.Related work also studies simulated dialogues, methods to elicit more clarifying questions, and individualized preferences.Source: §6 · p. 10

7 LIMITATIONS AND FUTURE WORK

• The method excludes interactions needing further clarification after the user's first clarifying answer, so its scope stops at one or two turns.Future work could extend the evaluation framework and double-turn preferences to general multi-turn interactions; the paper also identifies other dialogue acts and tasks as directions for extension.Source: §7 · p. 10

8 CONCLUSION

• On QA, double-turn preferences improve performance over single-turn preference labels and can train models to decide judiciously when to clarify.The proposed evaluation framework uses simulated user interactions.Source: §8 · p. 10

• Table 5 examples show that multiple annotators’ answers can differ from one annotator’s set of all possible answers.The examples come from AmbigQA and NQ-Open.Source: A DATASET DETAILS · p. 16

A.1 HUMAN AND MODEL AMBIGUITY SFT AND RLHF TRAINING DATASET DETAILS

• SFT uses (x, q) pairs for clarifying-question systems and (x, q, a, y) pairs for clarification-based QA and user simulation; direct QA uses (x, y).The (x, q, a, y) pairs also train direct QA systems when the clarifying question and answer are ignored.Source: §A.1 · p. 16
• RLHF training uses 70,904 remaining NQ-Open questions, split into 64,584 training and 6,320 development examples.Source: §A.1 · p. 16

B.1 CLARIFY-OR-ANSWER BASELINES

• ProCoT [Deng et al. (2023a)] prompts instruction-tuned models to judge ambiguity before answering or clarifying, so its results may benefit from stronger base models.Source: §B.1 · p. 16
• The PPDPP baseline [Deng et al. (2023b)] predicts a dialogue act separately, but hyperparameter sensitivity led training to select a single option; its predictions route a DPO model.The authors omit PPDPP’s RL method because its reported performance was comparable with and without additional RL.Source: §B.1 · p. 16

• ProCoT with instruction-tuned Gemma and Llama3 never directly answers; the other methods show trends similar to Llama2 results.Source: §B.2 · p. 16

C EXPERIMENTAL DETAILS

• SFT uses a 5e-5 learning rate, batch size 32, and up to five epochs; DPO uses a 5e-6 learning rate, KL factor 0.1, and up to two epochs.Models select the best development checkpoint; DPO batch sizes and evaluation intervals differ for Llama2-7b and Gemma-7b, and joint models reuse standard SFT hyperparameters.Source: C EXPERIMENTAL DETAILS · pp. 16–17
• Table 5 presents QA examples with differing labeled answer sets, and Table 6 reports Llama3 clarify-or-answer metrics.Table 6 includes direct-answer percentage, F1, Direct-Answer Accuracy, and Ambig Accuracy.Source: Table 6 · p. 17, Table 5 · p. 17

D EXAMPLE OUTPUTS

• Example predictions illustrate clarification-assisted answer recovery for unambiguous queries and recovery of multiple annotated answers.The cited captions describe examples in Tables 13 and 14.Source: D EXAMPLE OUTPUTS · p. 17, Table 13 · p. 21, Table 14 · p. 21

E LICENSING

• The paper reports Apache License 2.0 for Transformers and NaturalQuestions, the Llama 2 Community License Agreement, Gemma Terms of Use, and CC BY-NC-SA 4.0 for created assets.AmbigQA lists no license; its underlying NaturalQuestions dataset is under Apache License 2.0.Source: E LICENSING · p. 17
• The example-output tables include model responses generated on May 21, 2024, and SFT dataset statistics and sampled examples.The listed tables include examples from both Human-Ambiguity and Model-Ambiguity SFT datasets.Source: Table 7 · p. 18, Table 8 · p. 18, Table 10 · p. 19
• The appendix includes GPT-4 prompts for SFT dataset generation and simulated user clarification answers during evaluation.The evaluation prompt table identifies GPT-4 version gpt-4-0613.Source: Table 11 · p. 20, Table 12 · p. 20