Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models

• Ambiguous questions can span multiple dimensions, such as medal type and Olympic season, while answering them also requires substantial factual knowledge.Source: §1 · p. 1
• Tree of Clarifications recursively generates and prunes disambiguated questions using retrieved passages and few-shot prompting, then generates a long-form answer covering them.Its tree structure targets distinct clarification dimensions, while external sources provide additional knowledge.Source: §1 · pp. 1–2
• TOC reaches state-of-the-art performance on ASQA and, in a 5-shot setup, surpasses fully supervised baselines by 7.3 Disambig-F1 and 2.9 Disambig-ROUGE.Source: §1 · p. 2
• TOC combines retrieval with LLMs to explore diverse clarification paths and generate comprehensive answers to ambiguous questions.Source: §1 · p. 2

• Earlier retrieve-and-read approaches generate disambiguated questions and answers but require fine-tuning on large-scale training data.Source: §2 · p. 2
• Recent LLM methods prompt for long-form answers or selective clarification, yet lacking external information may increase hallucination risk and constrain performance to parametric knowledge.Source: §2 · p. 2
• TOC differs by recursively clarifying questions and using self-verification to prune unhelpful disambiguations within a tree architecture.Source: §2 · p. 2

3 Tree of Clarifications

• TOC combines retrieval-augmented clarification, recursive tree exploration, pruning, and answer generation to cover valid interpretations of an ambiguous question.Source: §3 · p. 2

3.1 Retrieval-Augmented Clarification (RAC)

• RAC retrieves Wikipedia passages with ColBERT and Bing, combines over 200 passages, reranks them, and prompts an LLM to generate disambiguated questions and answers.Source: §3.1 · pp. 2–3

3.2 Tree Structure (TS)

• The tree begins at the ambiguous question and recursively adds disambiguated question-answer nodes, reranking passages for each query and using breadth-first search to cover broader interpretations.Source: §3.2 · p. 3
• Self-verification prunes nodes whose answers are not factually coherent with the original ambiguous question, including factually correct answers that change its scope.Source: §3.2 · p. 3
• Table 1 evaluates long-form QA on ASQA's development set, comparing fully supervised and 5-shot baselines with TOC's retrieval-augmented clarification and tree structure.Source: Table 1 · p. 3
• After tree construction, TOC aggregates valid nodes, prioritizes answer-containing passages, and generates a comprehensive long-form answer.Source: §3.2 · p. 3

4.1 Experimental Setup

• The experiments evaluate models on ASQA using Disambig-F1, ROUGE-L, and DR, with Answer-F1 additionally measuring intermediate disambiguation accuracy.ASQA contains 6K ambiguous questions, and the baselines include fully supervised and few-shot systems.Source: §4.1 · pp. 3–4

4.2 Experimental Results

• Employing the tree structure helps the model explore diverse interpretations, improving D-F1 and DR scores.Source: §4.2 · p. 4
• TOC achieves state-of-the-art ASQA performance, surpassing the best fully supervised baseline by 7.3 Disambig-F1 and 2.9 Disambig-ROUGE.The comparison is against T5-large with JPR trained on the whole training set.Source: §4.2 · p. 4
• Retrieval-augmented clarification improves performance, especially Disambig-F1, indicating that external knowledge supports factual clarification.The ablation removes disambiguations and retrieval components to measure their contributions.Source: §4.2 · p. 4, Table 2 · p. 4
• Self-verification pruning improves the accuracy of retained disambiguations by 18.4 Answer-F1 points compared with the baseline.Deduplication alone does not improve accuracy, whereas self-verification removes unhelpful disambiguations.Source: §4.2 · p. 4, Table 3 · p. 4

Ambiguity Detection

• TOC does not explicitly detect ambiguity; failed disambiguation or pruning of all generated disambiguations can instead be treated as evidence that a question is unambiguous.The framework may continue attempting clarification and generate duplicate or irrelevant disambiguations.Source: §5 · p. 5

Computational Complexity

• TOC requires multiple LLM calls, with a maximum of fewer than 20 calls per question in the described setup.Exploration terminates after obtaining 10 valid nodes, or after three consecutive failed expansions.Source: §5 · p. 5

Generalizability

• The model-agnostic tree idea could generalize to other tasks and architectures, but this remains a direction for future investigation.The paper specifically mentions tasks requiring multi-step reasoning, such as multi-hop QA.Source: §5 · p. 5

6 Conclusion

• Tree of Clarifications recursively builds a disambiguation tree through few-shot prompting with external knowledge and uses it to generate a long-form answer.The framework explores diverse dimensions of ambiguity before producing the answer.Source: §6 · p. 5
• TOC is reported to guide language models through diverse clarification paths and generate comprehensive answers.The conclusion frames the framework as a potential basis for robust clarification models in real-world scenarios.Source: §6 · p. 5

• The study does not demonstrate TOC's generalizability across different kinds or sizes of LLMs and evaluates only the ASQA benchmark.Source: Limitations · p. 5
• Repeated prompting enables diverse reasoning paths but incurs a non-negligible cost.Source: Limitations · p. 5
• Pilot experiments found that chain-of-thought prompting did not enhance performance, while future pruning and reranking methods may improve results.Source: Limitations · p. 5

A.1 Ambiguous QA Datasets

• ASQA contains 6,316 ambiguous questions with long-form answers, split into 4,353 training, 948 development, and 1,015 test examples.Source: §A.1 · p. 8
• ASQA is built from ambiguous questions identified in AmbigNQ and collected from Natural Questions.Source: §A.1 · p. 8

A.2 Evaluation Metrics

• ROUGE-L measures lexical overlap, while Disambig-F1 measures factual correctness of information extracted for ground-truth disambiguated questions.Source: §A.2 · p. 8
• Disambiguation-ROUGE is the geometric mean of ROUGE-L and Disambig-F1, and Answer-F1 measures answer accuracy for a single disambiguated question.Source: §A.2 · p. 8

A.3 Implementation Details

• The system dynamically selects 5-shot examples using MiniLM representations, nearest-neighbor search, and Faiss similarity scores.Source: §A.3 · p. 8
• Retrieval-augmented clarification uses the top-5 relevant passages, while pruning reranks answer-containing passages to select the most relevant one.Source: §A.3 · p. 8
• AC@k measures the proportion of disambiguated answers covered by the top-k retrieved passages.Source: Table 4 · p. 8
• Answer generation uses ten valid disambiguations in breadth-first order and five answer-containing passages with GPT-3 served through an API.Source: §A.3 · p. 8

• The evaluation randomly samples 100 ASQA examples and treats retrieval as successful when passages contain an answer to the target disambiguated question.Source: §B.1 · p. 8
• Answer coverage measures the proportion of disambiguated answers for an ambiguous question that appear in retrieved passages.Source: §B.1 · p. 8
• 80.1 AC@100 indicates that the combined retrieval system's passage set contains sufficient information when properly explored.Source: §B.1 · p. 8
• Combining two retrieval systems performs best across the reported metrics, whereas unre-ranked Bing performs worst at AC@10 and AC@30.Source: §B.1 · p. 8

C.1 Prompt Format

• Prompt format descriptions are added to both retrieval-augmented clarification and answer-generation prompts to encourage the language model to follow the required format.Source: §C.1 · p. 9

C.2 Question Clarification

• Retrieval systems supply external knowledge, which the language model uses to generate disambiguated question-answer pairs during retrieval-augmented clarification.Long-form answers may also be generated in this stage for formatting, but they are not used later.Source: §C.2 · p. 9
• ToC can fail on unambiguous questions, asking about different or irrelevant facts instead of clarifying the original question.Source: §C.2 · p. 9, Table 7 · p. 11

C.3 Self Verification

• Self-verification asks the language model whether each answer is factually coherent with the original ambiguous question, using True or False to decide whether to discard the node.Source: §C.3 · p. 9

C.4 Answer Generation

• Answer generation uses a prompt containing the ambiguous question, selected disambiguations, and relevant passages to produce the final response.The answer-generation format is shared with the clarification format and can encode up to ten disambiguations and five relevant passages.Source: §C.4 · p. 9, Table 5 · p. 10, Table 10 · p. 14