Source-linked AI summary
ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog
Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, Massimo Poesio
TL;DR
Task-oriented clarification benchmarks are limited, and existing datasets often constrain the questions or responses they evaluate. ClarQ-LLM pairs varied dialogue tasks and evaluation metrics with provider agents that simulate human providers. LLAMA3.1-405B seekers achieve 60.5% success, below the 85% reported for human seekers.
Problem
Task-oriented dialogue datasets focusing on clarification are scarce, and existing clarification tasks often restrict responses to yes/no or list the information needing clarification in advance.
Source: A. Task-Oriented Dialogues · p. 2Method
ClarQ-LLM evaluates seekers on dialogue scenarios using task-success metrics and GPT-4o- or LLAMA3.1-405B-powered provider agents.
Source: §I · p. 1Results
60.5% success for LLAMA3.1-405B seekers, compared with 85% for human seekers.
Source: §I · p. 2Takeaways & Limitations
ClarQ-LLM provides evaluation metrics and a provider agent to measure improvements in future seeker models.
Source: §VII · p. 10
Abstract
from arXiv · showhide
We introduce ClarQ-LLM, an evaluation framework consisting of bilingual English-Chinese conversation tasks, conversational agents and evaluation metrics, designed to serve as a strong benchmark for assessing agents' ability to ask clarification questions in task-oriented dialogues. The benchmark includes 31 different task types, each with 10 unique dialogue scenarios between information seeker and provider agents. The scenarios require the seeker to ask questions to resolve uncertainty and gather necessary information to complete tasks. Unlike traditional benchmarks that evaluate agents based on fixed dialogue content, ClarQ-LLM includes a provider conversational agent to replicate the original human provider in the benchmark. This allows both current and future seeker agents to test their ability to complete information gathering tasks through dialogue by directly interacting with our provider agent. In tests, LLAMA3.1 405B seeker agent managed a maximum success rate of only 60.05\%, showing that ClarQ-LLM presents a strong challenge for future research.
I. INTRODUCTION
ClarQ-LLM targets a limited benchmark landscape for task-oriented clarification and evaluates seekers through interactive information gathering. Its provider agent supports flexible dialogue, while reported LLM seeker success rates remain below the human-seeker result.
I. INTRODUCTION
- Existing clarification benchmarks are limited, often using yes/no answers, pre-listed knowledge points, and surface-form metrics such as BLEU.These designs allow less flexible clarification and do not directly assess semantic content or intent.Source: §I · p. 1
- ClarQ-LLM combines dialogue scenarios, evaluation metrics, and GPT-4o- or LLAMA3.1-405B-powered provider agents to assess seekers functionally.Seekers interact with providers to obtain scenario-specific information needed for task completion.Source: §I · p. 1
- Success requires gathering all necessary information through questions; guessing unspecified details is not accepted, even when the guess is correct.The benchmark is designed to assess whether seekers identify dialogue uncertainties and resolve them with relevant clarification questions.Source: §I · p. 2
- 60.5% success for LLAMA3.1-405B seekers and 50.8% for GPT-4o seekers, versus 85% for human seekers.The paper characterizes these results as showing that the tasks remain challenging for future research.Source: §I · p. 2
II. RELATED WORK
Prior task-oriented dialogue datasets provide limited coverage of clarification tasks, and existing clarification datasets often constrain the response format or information sought. ClarQ-LLM provides varied scenarios and permits flexible information gathering through dialogue.
A. Task-Oriented Dialogues
- Clarification datasets such as ShARC and ClariT often use yes/no responses and list the knowledge points in advance, limiting task complexity.Aliannejadi et al. address clarification in chat but focus solely on information seeking, which the authors say does not suit their task-oriented purpose.Source: A. Task-Oriented Dialogues · p. 2
B. Clarification Question in Dialogues
- ClarQ-LLM allows information gathering through clarification and non-clarification questions rather than requiring an assumed ideal response measured by BLEU, ROUGE, or METEOR.The related-work discussion notes that systems may skip clarification yet still provide a good user experience in some information-seeking interactions.Source: B. Clarification Question in Dialogues · p. 3
A. Overview
- The benchmark contains 31 task types and 310 instantiations, with 26 types for testing, five for development, and no training set.Tasks span 392 background items, 341 skills or actions, and 736 scenes; they include two to six uncertainties and provider response trees of depth one to four.Source: A. Overview · p. 3
B. Construction Principles
ClarQ-LLM tasks are designed to elicit relevant information gathering across varied settings. Human-provider responses are organized into trees, and the bilingual task set is built through drafting, review, adaptation, and testing.
B. Construction Principles
- Tasks target relevant questions about ambiguous references, use virtual game worlds to test adaptation to unfamiliar tasks and real-world knowledge, and follow ethical rules in scenario design.The examples include fictional monsters rather than harmful tasks and vegetarian cooking options.Source: B. Construction Principles · p. 3
C. Human Provider Response Tree
- Provider response trees order information logically, including placing sauce questions after the pizza type is established, but do not constrain what seekers may ask.The tree is intended to ensure provider-side logical correctness; a provider can repeat information when a seeker overlooks it.Source: C. Human Provider Response Tree · pp. 3–4
D. Annotation Steps
- Task construction used about 700 hours of human labor, with Chinese drafts reviewed and annotated, translated into English, culturally adapted, and tested in real conversations.A bilingual student reviewed the GPT-4 translation, followed by evaluation from a native English-speaking PhD student; students unfamiliar with the data then tested the provider agent.Source: D. Annotation Steps · p. 4
IV. PROVIDER AGENT
ClarQ-LLM uses provider agents to simulate human providers and support seeker evaluation without relying on human dialogue partners. Response-generation restrictions preserve task information while allowing context-sensitive replies to other questions.
A. Overview
- Provider agents based on GPT-4o or LLAMA3.1-405B are intended to replicate human providers so future seeker models can be tested without human providers.Human provider responses are organized as trees to support replication of their needs, language, and logic.Source: A. Overview · p. 4
B. Ensuring Valid Responses
- When a seeker asks an unrelated or task-irrelevant question, the provider can generate a context-based reply that reassures the seeker and guides the dialogue back to the task.The response restrictions are intended to avoid invalid replies and support fair comparisons between seeker models.Source: A. Overview · p. 4, B. Ensuring Valid Responses · p. 4
- The provider agent preserves the original human response when it contains task-required information, while adding auxiliary text for fluency and consistency across seekers.The agent uses a context-generated Response II only when the human response tree cannot address the seeker's request; this response does not affect task-success judgment.Source: B. Ensuring Valid Responses · p. 4
C. Implementation Details
ClarQ-LLM’s provider uses multiple expert models to respond to seeker statements, while its seeker metrics measure task completion, query discrepancy, and response length. The metrics favor successful information gathering with fewer unnecessary questions and concise responses.
C. Implementation Details
- The provider classifies seeker statements into six types, then uses the corresponding expert model to generate a response, drawing on human responses for task-detail inquiries when suitable.When no human response fits, the expert model generates a context- and task-informed response without revealing content from human responses.Source: C. Implementation Details · p. 5
A. Metrics
- Success Rate measures the share of evaluated tasks in which the seeker collects all required information from the provider.The numerator is successful tasks; the denominator is the total number of evaluated tasks.Source: A. Metrics · p. 5
- Average Query Discrepancy (AQD) indicates whether seekers ask too many questions or too few, while Average Query Length (AQL) measures seeker response length.Positive AQD may indicate excessive caution; negative AQD may indicate difficulty distinguishing uncertainties. For similar success rates, lower AQD and AQL are preferable.Source: A. Metrics · pp. 5–6
B. Automatic Task Success Evaluation
Task success is determined by whether the provider’s responses contain all required information. The evaluation mainly analyzes Response I and uses GPT-4o to check rare cases involving Response II.
B. Automatic Task Success Evaluation
- A seeker succeeds when the provider’s responses contain all required information; otherwise, the task is treated as nearly a failure.Response I retains the human provider’s original response, with auxiliary text allowed to improve fluency or fit the question.Source: B. Automatic Task Success Evaluation · p. 6
- GPT-4o checks Response II against missing required information when it may supply that information or contradict it; this occurred only three times in over a thousand reviewed dialogues.Either finding is counted as success, because contradictory content means the original human response was replaced.Source: B. Automatic Task Success Evaluation · p. 6
- Success determination mainly analyzes Response I, with the Response II check addressing rare cases beyond that primary analysis.Response I’s retained human content allows evaluators to check whether required information was conveyed.Source: B. Automatic Task Success Evaluation · p. 6
VI. EXPERIMENTS
The experiments compare seeker LLMs using ClarQ-LLM’s test set and three metrics, with GPT-4o as the default provider and evaluation implementation. The setup also describes model modes, prompting, and available alternatives.
A. Experimental Setup
- The tested seeker LLMs included the GPT series, Llama2-70B-Chat, and ERNIE 4.0; Llama-family models ran on Amazon Bedrock, with a shared baseline seeker prompt crafted for fair comparison.For the Llama family, temperature was 0.00001 and top_p was 0.9999.Source: A. Experimental Setup · p. 6
- The ClarQ-LLM test set used in this section contains 260 dialogue tasks across 26 task types, with 10 tasks per type.Source: A. Experimental Setup · p. 6
- Seekers were evaluated using success rate, average query discrepancy, and average query length; seeker LLMs used Chat mode when supported by GPT and Llama, or Completion mode supported by all LLMs.ERNIE does not support Chat mode. GPT-4o and Llama3.1-405B-Instruct perform similarly in English tests.Source: A. Experimental Setup · p. 6
B. Dialogue Between GPT-4 Seekers and Human Providers (L2H)
GPT-4 seeker performance was examined in conversations with human providers, and manual assessments were used to check automated evaluation. Success declined as provider response quantities and dialogue turns increased, while corrected manual assessments matched automated results.
B. Dialogue Between GPT-4 Seekers and Human Providers (L2H)
- In 4×260 L2H dialogues, the authors observed one GPT-4 seeker hallucination and no other issues; they also used human-provider dialogues to assess response reasonableness and verify automated evaluation.Manual assessments recorded seeker success or failure for comparison with the automatic method.Source: B. Dialogue Between GPT-4 Seekers and… · pp. 6–7
- Success fell from over 50% to around 20% as predefined provider responses and completed dialogue turns increased.The reported comparison concerns GPT-4 seeker performance in Chat and Completion modes; the text also reports that Chat mode did worse than Completion mode.Source: Figure 4 · p. 7, B. Dialogue Between GPT-4 Seekers and… · p. 7
C. Automated Evaluation
- After manual-evaluation errors were corrected, manual and GPT-4o automated evaluations aligned perfectly on the assessed tasks.The discrepancies found on review were attributed to manual-evaluation errors.Source: C. Automated Evaluation · p. 7
- Across 780 English dialogue tasks, only one Llama3.1-405B-Instruct evaluation differed from GPT-4o, and GPT-4o was correct in that case.The reported difference in final results between the two evaluation methods was less than 1%.Source: C. Automated Evaluation · p. 7
D. Dialogue Between Human Seekers and GPT-4o Providers (H2L)
The H2L and L2L experiments compare human and LLM seekers with GPT-4o providers, while error analysis identifies recurring causes of seeker failures. L2L is presented as a basis for future automated assessment of seeker agents.
D. Dialogue Between Human Seekers and GPT-4o Providers (H2L)
- In H2L, two students substantially outperformed the GPT-4 seeker, and the provider failed to give the most appropriate response in approximately 1% to 2% of conversations.Students reported that the provider understood their intentions and responded appropriately; GPT-4 seeker queries were longer on average than human queries.Source: D. Dialogue Between Human Seekers and… · p. 7
E. Dialogue Between LLMs Seekers and the GPT-4o Provider (L2L)
- GPT-4 seeker performance in Completion mode declined from H2L to L2L because the GPT-4o provider withheld unsolicited task information to discourage vague questions.L2L pairs an LLM seeker with a GPT-4o provider; the provider was stricter than human providers in this respect.Source: E. Dialogue Between LLMs Seekers and… · p. 8
- Llama2-70B-Chat had the highest Chat-mode success rate but also the highest D value, reflecting many unnecessary questions and prolonged conversations.The authors prefer GPT-4o and Llama3.1-405B-Instruct despite their lower success than Llama2-70B-Chat in Chat mode.Source: E. Dialogue Between LLMs Seekers and… · p. 8
- Llama3.1-405B-Instruct achieved the highest success rate in Completion mode, while its high L value and higher-than-human D values indicate remaining dialogue-quality issues.Llama2 models could not be evaluated in Completion mode because of severe hallucinations.Source: E. Dialogue Between LLMs Seekers and… · p. 8
- GPT-3.5-to-GPT-4o success rates improved, and Chinese-optimized ERNIE 4.0 closely matched GPT-4 in S score, though it was evaluated only in Chinese Completion mode.The authors evaluated GPT-4 only in Completion mode for L2L and ERNIE 4.0 only in Completion mode to save funds.Source: E. Dialogue Between LLMs Seekers and… · p. 8
- Completion mode had higher success rates than Chat mode, while Chat mode produced shorter, more conversational queries that still exceeded the human average.The comparison covers GPT-4o and Llama3.1-405B-Instruct in Table III and GPT-4 in Table I.Source: E. Dialogue Between LLMs Seekers and… · p. 8, Table III · p. 9
F. Error Analysis of Seeker Agents in Both L2H and L2L Setups
- GPT-4 often ended dialogues before resolving uncertainties, forgot previously identified uncertainties over longer exchanges, or mistook summary-confirmation questions for resolved details.The examples include overlooking a pizza-size choice and accepting a provider’s positive response to an unresolved summary.Source: F. Error Analysis of Seeker Agents in… · pp. 8–9
- In a random sample of 300 L2L dialogues, other LLMs showed similar failures, with lower-performing models more prone to ending after two or three turns; no hallucinations were observed.Better-performing models made these mistakes less frequently.Source: F. Error Analysis of Seeker Agents in… · p. 9
G. Qualitative Analysis of the Provider Agent
The provider’s observed errors were rare but could affect seeker outcomes, while a modified multi-info provider was introduced to handle multiple questions in one turn. The reported results associate that provider with improved task completion and fewer dialogue turns.
G. Qualitative Analysis of the Provider Agent
- Across the analyzed L2L dialogues, provider errors occurred only when it misunderstood a seeker request and gave an incorrect response; its error rate was less than 0.7 per 100 turns.The qualitative analysis covered 300 dialogues totaling 1,957 turns.Source: G. Qualitative Analysis of the… · p. 9
- In 1,957 turns, the provider made eight errors when Response I was required, with around 2% impact on task success, and five errors when Response II was required, with less than 2% impact.The first error type could lower seeker performance; the second could help it by providing needed information without an appropriate question.Source: G. Qualitative Analysis of the… · p. 9
H. Multi-Info Provider Agent
- The multi-info provider combines multiple human provider responses into one reply, addressing seeker agents that ask several questions at once.The earlier provider was designed to supply only one human-provider response per turn.Source: H. Multi-Info Provider Agent · p. 9
- The multi-info provider improved seekers’ task completion and reduced dialogue turns; the authors suggest longer dialogues can contribute to forgetting and task failure.This pattern is reported as consistent with the L2H finding.Source: H. Multi-Info Provider Agent · pp. 9–10
I. Provider Agent Based on Llama3.1-405B-Instruct
ClarQ-LLM’s Llama3.1-405B-Instruct provider performs comparably to GPT-4o in the reported evaluation, while experiments identify persistent shortcomings in LLM seekers. The conclusion places state-of-the-art models below human performance and the appendices provide prompts, task examples, and ethical practices.
I. Provider Agent Based on Llama3.1-405B-Instruct
- GPT-4o seeker performance was very similar with GPT-4o and Llama3.1-405B-Instruct providers; the latter’s error rate was below 0.6 per 100 turns.A manual review found two instances of each of two error types among 716 turns, each affecting final task success by about 2%.Source: §I · p. 10
J. Findings from Experiments
- Experiments identify four LLM shortcomings: uncertainty reasoning in Chat mode, forgetting prior dialogue, longer responses, and asking more questions than necessary.The authors speculate that instruction fine-tuning may contribute to these issues and argue that improving reasoning within conversational content and Chat mode is important.Source: J. Findings from Experiments · p. 10
VII. CONCLUSIONS
- State-of-the-art LLMs have yet to reach human performance on ClarQ-LLM’s task-oriented clarification tasks.The benchmark provides evaluation metrics and a provider agent for measuring future model improvements.Source: §VII · p. 10
- The database avoids potentially harmful tasks by using fictional monsters and vegetarian cooking scenarios.PhD annotators were paid £17.22 per hour; other fully involved undergraduate and graduate students were offered co-authorship, while Chinese H2L participants received extracurricular credits.Source: ETHICAL CONSIDERATIONS · p. 10
VIII. PROMPTS UNDER DIFFERENT WORKING MODES OF THE SEEKER
- Figures 5 and 6 provide example prompts for seeker Chat mode and Completion mode, respectively.The supplied passage specifically identifies Figure 6 as the Completion-mode prompt.Source: §VIII · p. 12
IX. DATA EXAMPLES
- Figures 7–12 provide English and Chinese examples of different ClarQ-LLM tasks.Figures 7–9 are English examples, paired with Chinese versions in Figures 10–12.Source: §IX · p. 12
X. DIALOGUE EXAMPLES
The dialogue examples pair task backgrounds with interactions involving human or model seekers and GPT-4o providers. They illustrate Completion and Chat modes, including a multi-information provider setup.
X. DIALOGUE EXAMPLES
- Figures 13–21 show H2L and L2L conversations based on the task backgrounds in Figures 9 and 12.The examples include human seekers, GPT-4 or GPT-4o seekers in Completion mode, and GPT-4o seekers in Chat mode.Source: §X · p. 12, Figure 13 · p. 17, Figure 14 · p. 17
- Figures 18–19 show GPT-4o Completion-mode seekers interacting with a multi-info GPT-4o provider.The corresponding examples use the task backgrounds in Figures 12 and 9.Source: §X · p. 12, Figure 18 · p. 20, Figure 19 · p. 21
- Figures 20–21 show GPT-4o seekers in Chat mode interacting with a GPT-4o provider.The examples use task backgrounds in Figures 12 and 9.Source: §X · p. 12, Figure 20 · p. 21, Figure 21 · p. 22
- Figure 6 presents a prompt for Completion mode; Figures 7–9 show English task backgrounds and their human providers’ responses from ClarQ-LLM, while Figures 10–11 show Chinese task backgrounds and their human providers’ responses from ClarQ-LLM.Source: Figure 6 · p. 13, Figure 7 · p. 14, Figure 8 · p. 14