Source-linked AI summary
A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models
Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra
TL;DR
Ambiguous or underspecified user queries make interactive clarification difficult to evaluate with conventional static benchmarks. The paper introduces a tri-agent framework with synthetic supply-chain data and metric-based judging, reporting 87% Overall Task Success while identifying completeness-related failures. It positions the framework as a scalable approach, but notes that human validation and synthetic-interaction realism remain limitations.
Problem
Static evaluation benchmarks may not adequately capture interactive clarification dialogues that require ambiguity management, response understanding, and final intent derivation.
Method
The paper evaluates a Question Clarifying Agent through simulated Respondent Agent dialogues and an Evaluator Agent using synthetic supply-chain data and predefined metrics.
Results
87% Overall Task Success was reported, alongside AH-DA: 0.92, QQ-Rel: 4.48, and DE-Turns: 4.83; failures often correlated with AH-CC: 4.15.
Takeaways & Limitations
The framework offers a structured approach to benchmark and iteratively improve question clarifying agents in interactive conversational settings.
Takeaways & Limitations
Validation remains preliminary because the EA pilot used a single annotator and single metric, while synthetic interactions and RA or EA biases may differ from real-world judgment.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly deployed in interactive systems where understanding user intent precisely is paramount. A key capability for such systems is effective question clarification, especially when user queries are ambiguous or underspecified. This paper introduces a novel tri-agent framework for the robust evaluation of an LLM's ability to engage in clarifying dialogue. Our framework comprises three distinct LLM-based agents: (1) a Question Clarifying Agent (QCA), the system under evaluation, tasked with identifying ambiguities and posing clarifying questions; (2) a Respondent Agent (RA), designed to simulate human user responses, potentially including irrelevant or challenging replies; and (3) an Evaluator Agent (EA), an LLM-as-a-judge, which assesses the quality of the dialogue based on a comprehensive set of metrics. We detail a methodology for synthetic data generation in the supply chain domain as an example. We propose metrics evaluating ambiguity handling, question quality, dialogue efficiency, language appropriateness, and final intent alignment. We also briefly discuss the validation of the EA against human judgments. This work provides a structured approach to benchmark, validate, and improve the clarification capabilities of conversational LLM applications.
1 Introduction
The paper addresses the difficulty of evaluating interactive question clarification and proposes a tri-agent framework, synthetic supply-chain data methodology, and multidimensional metrics for benchmarking and improvement.
- Static input-output benchmarks may miss the dynamic process of resolving ambiguity, understanding responses, and deriving actionable intent.
- The framework uses a Question Clarifying Agent, Respondent Agent, and Evaluator Agent to evaluate clarification dialogues.The QCA asks clarifying questions, the RA simulates user responses, and the EA assesses performance.
- The methodology generates synthetic clarification data tailored to the supply chain domain.
- The proposed metrics assess ambiguity handling, question quality, dialogue efficiency, language appropriateness, and final intent alignment.
- The framework is presented as a scalable approach to benchmark and iteratively improve question clarifying agents.
2 Related Work
Prior work evaluates language models and agents with static benchmarks, task metrics, and LLM judges; this framework combines interactive simulation with judge validation for clarification.
- Traditional text-generation metrics and static benchmarks often do not capture nuanced dialogue quality or interaction success.
- Clarification research studies when agents should ask questions and measures outcomes such as success rate and query discrepancy.
- LLM-as-a-judge methods provide scalable evaluation, but can exhibit position bias, verbosity bias, and prompting sensitivity.
- The proposed framework integrates an interactive simulation loop with an LLM judge to evaluate clarification across unexpected responses and domain contexts.It also emphasizes validating the judge against human judgments.
3 Proposed Evaluation Framework
The proposed framework evaluates clarification through coordinated QCA, RA, and EA agents, using synthetic test cases, iterative dialogue, and metric-based assessment against ground truth.
- QCA: The QCA analyzes an ambiguous or incomplete query, asks clarifying questions, integrates responses, and forms a final clarified question.
- RA: The RA simulates a human user by answering QCA questions according to a ground-truth query and a persona that can introduce ambiguity or challenges.
- EA: The EA receives the dialogue transcript and ground-truth query, then produces quantitative scores or qualitative feedback using predefined metrics.
- Evaluation process: Each test case begins with a potentially ambiguous query, proceeds through iterative QCA-RA exchanges, and ends with the QCA formulating its final understanding.
- Evaluation process: The EA evaluates the complete transcript, including the initial query, clarification turns, final question, and ground truth.
4 Synthetic Dataset Generation
The dataset generates supply-chain clarification dialogues from ambiguous base questions and corresponding ground-truth queries. It varies omitted entities, mandatory and optional fields, and respondent behavior to test clarification robustness.
- Dataset design: The dataset defines supply-chain base questions with entities and dimensions marked mandatory or optional for well-formed queries.A fully clarified query may leave optional entities unspecified when aggregation or partial specification is intended.
- Query-pair construction: Each instance pairs an initial query Q_orig with a ground-truth query Q_gt representing the ideal unambiguous intent.Q_orig is derived by omitting some or all entity values from Q_gt, creating different ambiguity levels.
- Query-pair construction: Initial queries range from completely unspecified base questions to partially specified or confirmation-worthy queries.This variation requires the QCA to distinguish sufficiently clarified entities from those needing further questions.
- Examples: Examples include forecast queries with optional product and site fields but mandatory dates, and inventory queries with mandatory site and date fields.The examples demonstrate how alternative omissions produce distinct Q_orig variants for one Q_gt.
- Respondent simulation: The Respondent Agent follows Q_gt while varying direct answers, aggregation responses, rephrasings, and deliberately unrelated entities or intents.These behaviors test whether the QCA can interpret broad specifiers and handle challenging responses.
5 Experimental Setup
The experiment evaluates a QCA through simulated multi-turn dialogues involving a Respondent Agent and an Evaluator Agent. The evaluator scores ambiguity handling, question quality, efficiency, language use, and final alignment against ground truth.
- Agents: The QCA, RA, and EA are instantiated with Claude 3.5 Sonnet, with reasoning and responses separated into designated XML tags.The authors report performance improvements after imposing this test-time compute process.
- Evaluation protocol: The EA scores dialogues using the transcript, the final clarified question Q_final, and the ground-truth query Q_gt.Most metrics use a 1–5 scale, while some metrics are binary or probabilistic; completeness violations deduct points.
- Evaluation metrics: Ambiguity Handling measures detection of underspecification and completeness of questioning for necessary missing entities and dimensions.Detection Accuracy is binary or scaled, while Completeness of Clarification evaluates whether all required information was addressed.
- Evaluation metrics: Question Quality evaluates relevance, clarity, and conciseness, while Dialogue Efficiency measures the number of turns needed to reach Q_final.Fewer turns generally indicate greater efficiency when clarification succeeds.
- Evaluation metrics: Language Appropriateness assesses handling of unsupported inputs, and Final Question Alignment assesses semantic fidelity and precision relative to Q_gt.Precision penalizes extraneous or incorrect entities and values not negotiated during the dialogue.
- Evaluation metrics: Overall Task Success measures whether QCA transforms Q_orig into an actionable Q_final semantically equivalent to Q_gt.The EA receives metric definitions and scales to produce quantitative scores or qualitative feedback.
6 Results and Analysis
Across 200 supply-chain dialogues, the QCA detected ambiguity and asked relevant questions effectively, but failures exposed weaknesses with unsupported inputs and incomplete or premature clarification. EA validation showed promising agreement with one author while remaining preliminary.
- Evaluation setup: 200 unique supply-chain dialogues comprised the synthetic evaluation dataset, with the RA refined through a 20-dialogue pilot before full-scale testing.The pilot monitored consistency with persona instructions and intended conversational flows.
- QCA performance: AH-DA was 0.92, QQ-Rel was 4.48, and DE-Turns was 4.83, indicating strong ambiguity detection, question relevance, and reasonably efficient successful interactions.These are the reported aggregate QCA results across the evaluation dialogues.
- QCA limitations: LA-Uns was 3.75, reflecting challenges when respondents supplied entities or dimensions outside the predefined supported scope.The QCA sometimes responded ambiguously or attempted to process unsupported inputs instead of redirecting users toward supported alternatives.
- Task success: OTS was 87%, while failures correlated with lower AH-CC of 4.15 and incomplete reconciliation of ground-truth granularity.A final query could preserve a valid broad scope such as “all products” yet omit another mandatory dimension required for backend execution.
- QCA limitations: Premature clarification closure occurred when evasive or incomplete responses left essential parameters unresolved, such as a required specific date range.The reported failure pattern concerns insufficient persistence in probing responses against Q_gt requirements.
- EA-human alignment: On AH-CC across 50 dialogues, EA scores correlated with an author’s scores at r=0.87, but the validation used one annotator and one metric.The paper calls for scaled evaluation with multiple trained annotators, calibration, and inter-coder reliability measurement.
7 Conclusion
The tri-agent framework provides a structured, validated approach for evaluating and iterating on LLM question clarification, while recognizing dependencies on synthetic data, agent quality, and human-alignment challenges. Future work targets broader scenarios, stronger EA-human validation, and more adaptive respondent behavior.
- The tri-agent framework enables dynamic, scalable, and validated assessment of LLM question clarification.It combines QCA, RA, and EA agents with synthetic supply-chain data and detailed evaluation metrics.
- Its evaluation benefits include realistic interaction, granular feedback, domain-specific testing, and robustness checks across respondent personas.
- The framework depends on RA and EA quality, differs from real-world interaction, and may not fully align EA judgments with nuanced human evaluation.
- Future work will expand synthetic scenarios, including multi-intent queries and longer dialogues, while studying EA-human alignment more rigorously.
- The authors also plan to explore adaptive respondent behaviors that produce more challenging or diverse responses over time.