Source-linked AI summary
A Small Model Asks Better: Fine-Tuning Query Clarification for Retrieval-Augmented Generation
Lihong He, Lucian Popa, Marina Danilevsky
TL;DR
Conversational RAG queries can be unclear from dialogue context or become ambiguous when grounded in retrieved documents, motivating clarification before answering. The paper introduces ClariEdge and fine-tunes a LoRA for both cases. The model outperforms larger prompted baselines, while hard negatives help distinguish genuine ambiguity from context-resolvable queries.
Problem
The task is to detect whether a multi-turn conversation’s last query has no clear interpretation or multiple valid interpretations, optionally using retrieved documents, and request clarification when needed.
Source: §1 · p. 2Method
Results
The lightweight LoRA achieves the highest classification accuracy and clarification-generation quality among seven prompted baselines, including substantially larger models, and removing hard negatives sharply reduces boundary-case accuracy.
Source: §1 · p. 2Takeaways & Limitations
The same LoRA can check ambiguity before retrieval and again with retrieved documents before answer generation.
Source: §1 · p. 2The model generates a single clarifying question rather than conducting multi-turn clarification dialogues.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Query clarification is essential for retrieval-augmented generation (RAG) systems that face ambiguous user queries. We consider multi-turn grounded conversational scenarios where a user query is either ill-formed or ambiguous in the context of the prior dialog or, more interestingly, it may be well-formed but when posed against retrieved documents there are multiple relevant interpretations. The task is to recognize both such cases and then generate an appropriate clarification request to the user. We present ClariEdge, a new dataset of approximately 3,900 examples that include such ambiguous queries along with negative examples that do not require clarification. We also show that a lightweight fine-tuned LoRA achieves 96.6% classification accuracy and outperforms prompted baselines of many larger models on both classification and quality of clarification generation. Our study shows that hard negatives are an essential ingredient to separate out queries that are truly ambiguous from queries that may look ambiguous but are not (i.e., can be resolved from the context). The same LoRA achieves high performance in both pre-retrieval and post-retrieval scenarios; thus, it can be used in two places in a RAG pipeline as a check before answer generation. The ClariEdge dataset is available at https://huggingface.co/datasets/ibm-research/ClariEdge.
1 Introduction
The introduction frames clarification as necessary for queries that are unclear from conversation context or become ambiguous against retrieved documents. It presents ClariEdge and a single fine-tuned LoRA as addressing both cases, including deployment before retrieval and before answer generation.
1 Introduction
- Clarification is needed when conversation context leaves an entity or attribute unresolved, or when retrieved documents support multiple interpretations of a well-formed query.The latter case calls for naming grounded options and briefly answering each; the former can list candidates already raised in the conversation.Source: §1 · p. 1
- Existing benchmarks miss the joint conversation-plus-corpus setting: CLAMBER uses standalone questions, while AmbigQA treats multiple atomic answers as ambiguous even when they form a composite answer.The introduction notes that neither captures the setting faced by conversational RAG deployments.Source: §1 · pp. 1–2
- ClariEdge contains approximately 3,900 multi-turn examples with retrieved documents and hard negatives that appear ambiguous but can be resolved from context.Examples combine direct human annotation and synthetic LLM generation followed by human validation.Source: §1 · p. 2
- The lightweight LoRA leads seven prompted baselines, including substantially larger models, in classification accuracy and clarification-generation quality on queries it correctly flags.Removing ClariEdge hard negatives causes a sharp accuracy drop on boundary cases.Source: §1 · p. 2
- A single LoRA handles both pre-retrieval and post-retrieval ambiguity checks, gating retrieval from conversation alone and answer generation from conversation plus retrieved documents.Either invocation can trigger a clarifying question.Source: §1 · p. 2
2 Related Work
Prior work studies ambiguity across several QA and search settings, while existing systems specialize in when and how to clarify or in a particular deployment setting. The paper contrasts this with one hard-negative-trained LoRA that spans pre- and post-retrieval and outperforms larger prompted baselines.
2 Related Work
- The related work spans open-domain factual QA, standalone and conversational QA, and conversational search, including benchmarks such as AmbigQA, CLAMBER, and Qulac.Table 1 compares ClariEdge with four prior clarification datasets.Source: §2 · p. 2, Table 1 · p. 3
- The paper motivates task-specific supervision by noting that prompting yields only marginal gains on ambiguity identification in prior work.It also describes approaches using uncertainty signals or retrieval-grounded disambiguation without asking the user.Source: §2 · p. 3
- Prior clarification systems specialize in when to clarify, how to ask, or which deployment setting to target, and typically rely on prompting or uncertainty rather than task-specific supervision.Such systems remain tied to prompts, models, or iterations of their setting.Source: §2 · p. 3
- A single hard-negative-trained LoRA covers pre- and post-retrieval, distinguishes genuine ambiguity from context-resolvable references, and outperforms substantially larger prompted baselines.The paper presents this as a contrast to systems tied to particular prompts, models, or iterations.Source: §2 · p. 3
3 ClariEdge Dataset
ClariEdge spans four MTRAG domains and organizes queries into genuinely ambiguous, easy-clear, and context-resolvable hard-negative categories. Its creation combines annotation and synthetic generation, with human review also shaping clarification references.
3 ClariEdge Dataset
- ClariEdge conversations span ClapNQ, Cloud, FiQA, and Govt, and the dataset separates ambiguous, CLEAR-random, and CLEAR-hard queries.Ambiguous queries require clarification; CLEAR-random examples are easy negatives, and CLEAR-hard examples are superficially ambiguous but context-resolvable.Source: §3 · p. 3, §3.1 · p. 3
3.1 Data Categories and Statistics
- CLEAR-hard examples look ambiguous but are resolvable from conversation context, while CLEAR-random queries are straightforwardly clear.The distinction is critical because hard negatives teach the boundary between genuinely ambiguous queries and references that context resolves.Source: §3.1 · p. 3
3.2 Data Creation Process
- Ambiguous examples combine annotator-authored or synthetic final turns with retrieved-document and conversation-only variants, while annotators classify LLM-generated queries as ambiguous or context-resolvable.Human review was essential for CLEAR-hard examples because the generation model struggled to delineate the boundary.Source: §3.2 · pp. 4–5
3.3 Generation of Clarification Response
- Clarification references identify the ambiguous entity, extract and verify candidate options, then use response patterns suited to the number of options.Few options receive brief answers, medium lists are enumerated, and open-domain ambiguity prompts the user to specify an entity type.Source: §3.3 · p. 5
3.4 Inter-rater Agreement
- Cohen’s kappa was 0.875 for ambiguity labels; reviewers found clarification versions matched in content but varied in phrasing.The reported word-overlap F1 was 0.774 and BLEU-4 was 0.57.Source: §3.4 · p. 5
4 Models
The study fine-tunes parameter-efficient LoRA models on 3,509 examples, centering on small Granite models and comparing against prompted baselines spanning several model families.
4.1 Fine-Tuned LoRA Models
- LoRA models are fine-tuned on 3,509 training examples, including two small Granite models and the larger open-source GPT-OSS-20B.The Granite models target cost-effective deployment, while GPT-OSS-20B provides a larger-model comparison.Source: §4.1 · pp. 5–6
- Granite-4.0-micro and Granite-4.1-3B are the primary models, chosen for deployment efficiency, adaptation with modest data, and controlled fine-tuning comparisons.Using the same base architectures in fine-tuned and prompted configurations helps isolate fine-tuning from model-capacity differences.Source: §4.1 · p. 6
4.2 Prompted Baseline Models
- Seven prompted baselines without task-specific training span proprietary frontier, open-weight large and medium, and lightweight model families.The set includes GPT-5.6 Sol, GPT-4o, GPT-OSS-120B, GPT-OSS-20B, Granite-4.1-30B, Granite-4.1-3B, and Granite-4.0-micro.Source: §4.2 · p. 6
5 Experiments and Analysis
LoRA fine-tuning outperforms prompted baselines on classification and clarification generation across full, pre-retrieval, and post-retrieval evaluations. The results also show the value of hard negatives and indicate that the Granite-4.0-micro LoRA retains its lead in two out-of-domain domains.
5.1 Evaluation Metrics
- The evaluation measures classification accuracy and clarification quality, using option-level precision, recall, and F1 for generated clarifications.The option-focused metrics accommodate open-domain clarifications where options cannot be explicitly enumerated.Source: §5.1 · p. 6
5.2.1 Full Test Set
- 96.6% overall accuracy for Granite-4.1-3B LoRA is 17.2 pp above GPT-4o and 31.8 pp above prompted Granite-4.1-3B on the full test set.Granite-4.0-micro LoRA reaches 92.4%; the GPT-OSS-20B LoRA reaches 90.1%, 10.7 pp above GPT-4o.Source: §5.2.1 · pp. 6–7, Table 5 · p. 7
5.2.2 Pre-Retrieval
- On pre-retrieval classification, all three fine-tuned LoRAs lead their architecture-matched prompted baselines, while prompted models tend to over- or under-clarify.Granite-4.1-3B LoRA is relatively balanced, though its Ambiguous accuracy is lower than its CLEAR-hard accuracy; this setting relies only on conversational cues.Source: §5.2.2 · p. 7
5.2.3 Post-Retrieval
- In post-retrieval classification, Granite-4.1-3B LoRA reaches 100.0% accuracy and Granite-4.0-micro LoRA reaches 99.0%, ahead of GPT-OSS-120B at 82.1%.CLEAR-hard examples are not represented in this subset.Source: §5.2.3 · p. 7
5.3.1 Full Test Set
- On the full test set, Granite-4.0-micro and Granite-4.1-3B LoRAs lead on clarification F1 at 89.7% and 89.0%, respectively.Prompted Granite-4.0-micro reaches 56.8% F1; prompted GPT-5.6 Sol and GPT-OSS-20B have high precision but substantially lower recall.Source: §5.3.1 · p. 8, Table 6 · p. 8
5.3.2 Pre-Retrieval
- On pre-retrieval generation, both Granite LoRAs achieve F1 around 85%, close to several prompted models that have lower classification accuracy.The LoRAs combine clarification quality with more reliable identification of which queries require clarification.Source: §5.3.2 · p. 8
5.3.3 Post-Retrieval
- On post-retrieval generation, the Granite LoRAs attain the highest F1, combining high precision with the highest recall.Prompted models can achieve strong precision but comparatively low recall, while the LoRAs gain on both.Source: §5.3.3 · p. 8
5.4 Out-of-Domain (OOD) Evaluation
- In out-of-domain evaluation on Banking and Telco, Granite-4.0-micro LoRA maintains its lead over prompted baselines in both classification and generation quality, suggesting its clarification skill is task-general rather than domain-specific.Source: §5.4 · p. 8
6 Conclusion
The conclusion presents ClariEdge as a dataset for RAG clarification and reports that fine-tuning a small model with LoRA is more beneficial than prompting larger LLMs. Hard negatives are important, and the resulting LoRA works in both pre-retrieval and post-retrieval settings, with pre-retrieval more challenging.
6 Conclusion
- ClariEdge supports RAG question clarification, and LoRA fine-tuning on a small model is more beneficial than prompting larger LLMs.Hard negatives are an essential ingredient; the resulting LoRA works in both pre-retrieval and post-retrieval settings, although pre-retrieval is more challenging.Source: §6 · p. 8
Limitations
The study's scope is constrained to single-turn clarification and English-language evaluation.
Limitations
- The model generates one clarifying question rather than conducting multi-turn clarification; iterative strategies remain an open direction.Source: Limitations · p. 9
- The dataset and evaluation are restricted to English, leaving crosslingual transferability unexplored.Source: Limitations · p. 9
A Dataset Distributions
The dataset combines pre- and post-retrieval examples, mostly multi-turn conversations, with annotation and review procedures designed to preserve genuine ambiguity and distinguish it from queries resolvable from context. It also includes several clarification response patterns.
A Dataset Distributions
- Pre- and post-retrieval examples appear in roughly equal proportions; most conversations are multi-turn, and most post-retrieval examples include at least three documents.Figure 3 summarizes retrieval setting, conversation turns, and retrieved-document counts.Source: A Dataset Distributions · pp. 10–11, Figure 3 · p. 11
B Human Annotation and Review
- The review process relied on an existing set of general multi-turn conversations and combined human-authored examples with synthetic generation and author validation.The appendix describes human annotation for one ambiguous-example approach and LLM generation followed by manual review for another approach and CLEAR-hard examples.Source: B Human Annotation and Review · p. 11
B.1 Human-Authored Ambiguous Questions
- Human-authored ambiguous questions were inspired by corpus entity types and instances, then stitched into existing conversations, usually on a similar topic but sometimes after a topic switch.About 75% were stitched onto same- or similar-topic conversations and about 25% onto different-topic conversations; validation filtered cases where prior context resolved the query.Source: §B.1 · p. 11
B.2 Manual Review of Synthetically Generated Questions
- Synthetic queries and CLEAR-hard examples were manually reviewed because generated questions could be resolvable through coreference or ordinary dialogue flow despite appearing ambiguous.Review retained genuinely ambiguous cases and classified context-resolvable ones as CLEAR-hard.Source: §B.2 · p. 12
C Clarification Response Patterns
- The appendix illustrates three clarification patterns: hedging with 2–3 options, hedging over 4–8 options, and open-domain clarification for many or unbounded options.Table 7 pairs each pattern with a query, its conversational or document context, and a clarification response.Source: C Clarification Response Patterns · p. 12
D LoRA Implementation Details
The LoRA accepts conversation history and optional retrieved documents, and produces either CLEAR or a clarification question. The appendix also specifies shared prompts for baselines and LLM-based judgments used to evaluate clarification quality.
D LoRA Implementation Details
- LoRA models use rank 32 and scaling factor α = 64 on attention and MLP layers, with early stopping based on validation performance.Training uses one H100 GPU, a learning rate of 1 × 10−5, dropout of 0.05, and up to 15 epochs with a minimum of 5.Source: D LoRA Implementation Details · p. 12
- The model receives conversation history and optional retrieved documents, then outputs CLEAR for an unambiguous query or a clarifying question otherwise.Its decision uses conversation history alone when documents are absent and both sources when they are present.Source: D LoRA Implementation Details · p. 12
E Prompt for Baseline Models
- All seven prompted baselines use the same templates, varying only user-message content; the assistant judges ambiguity using conversation history and retrieved documents, then responds with a grounded clarifying question or exactly CLEAR.Source: E Prompt for Baseline Models · pp. 12–13
F Prompts for Generation Quality Metrics
- Four LLM prompts support generation-quality judgments by extracting options, checking option relevance and reference coverage, and deciding whether a response is open-ended.The appendix describes these judgments as the steps used to compute precision and recall for free-form clarification responses.Source: F Prompts for Generation Quality Metrics · pp. 13–14
- The open-ended check treats responses as open-ended when they allow answers beyond a fixed option set, including when listed options are illustrative or accompanied by a catch-all.A general question without concrete options also counts as open-ended.Source: F Prompts for Generation Quality Metrics · p. 14
G Examples of Generation Quality Metrics
Worked examples illustrate how the entity-option-centric precision, recall, and F1 metrics handle enumerated options and open-domain clarifications.
G Examples of Generation Quality Metrics
- The examples contrast enumerated-option clarifications with open-domain questions, for which a gold [extra] token represents the open-ended response.Any correct generated option counts toward precision in open-domain cases, and the gold [extra] is covered when the generated clarification includes an open-ended component.Source: G Examples of Generation Quality Metrics · p. 15
H Ablation Study on Hard Negatives
Hard negatives improve the balance between detecting ambiguous queries and recognizing clear queries that appear ambiguous, while their removal sharply reduces CLEAR-hard accuracy.
H Ablation Study on Hard Negatives
- The ablation compares Granite-4.0-micro LoRA trained on the full training set with the same data minus hard negatives, evaluating both classification accuracy and clarification generation quality on the full test set.The presence of hard negatives is the only training-set variable.Source: H Ablation Study on Hard Negatives · p. 15
- 35.4% CLEAR-hard accuracy without hard negatives contrasts with 97.1% Ambiguous detection, while training with hard negatives yields a better balance and improves generation quality.The authors attribute the generation-quality improvement as likely reflecting more nuanced boundaries between ambiguous and clear queries.Source: H Ablation Study on Hard Negatives · p. 15
I Out-of-Domain Evaluation Details
The evaluation tests whether the Granite-4.0-micro LoRA transfers to Banking and Telco, two domains outside the four training domains. It leads prompted baselines in overall classification accuracy and maintains balanced classification across categories in both domains.
Out-of-Domain Evaluation Details
- The OOD sets use Banking and Telco conversations, with 230 and 235 examples respectively, divided among Ambiguous, CLEAR-hard, and CLEAR-random categories.The sets include both pre-retrieval and post-retrieval examples; the original MTRAG-UN conversations supply the CLEAR-random examples.Source: I Out-of-Domain Evaluation Details · p. 15
- The authors say the OOD results match in-domain findings and suggest that ClariEdge-learned clarification skill is task-general rather than domain-specific.Their overall picture combines balanced classification, high F1 on correctly flagged clarifications, and a large gap over similarly sized prompted baselines.Source: I Out-of-Domain Evaluation Details · p. 16
- 92.6% on Banking and 90.2% on Telco, the LoRA leads prompted baselines in overall accuracy and alone balances all three categories, near 92.4% in-domain.The prompted models retain their in-domain imbalance: larger models over-clarify, while smaller Granite models under-clarify.Source: I Out-of-Domain Evaluation Details · pp. 15–16, Table 10 · p. 17
- 89.28% F1 on Telco is the LoRA's highest generation result; on Banking, GPT-4o's 89.82% and Granite-4.1-30B's 86.61% exceed its 81.56%.Those higher Banking F1 scores come from smaller correctly flagged subsets: 42/79 and 60/79, versus 69/79 for the LoRA.Source: I Out-of-Domain Evaluation Details · p. 16, Table 11 · p. 17