PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Correctness: Resolving Underspecification in Agentic Text-to-SQL

Wen-Zhi Li, Yue Gong, Konstantinos Kanellis, Balakrishnan Murali Narayanaswamy

arXiv:2610.02739v1cs.CL

TL;DR

Text-to-SQL requests can leave multiple intended interpretations unresolved, so execution correctness alone does not establish that the agent addressed the user’s underspecified intent. PlanPool maintains clarification questions in an explicit, mutable pool; across three benchmarks it improves ambiguity coverage and reduces silent failures while retaining competitive execution accuracy.

  • Problem

    Text-to-SQL requests are often underspecified, and correct execution does not establish that the agent explicitly resolved the ambiguities underlying the intended query.

    Source: §1 · p. 1

  • Method

    PlanPool externalizes clarification needs as a mutable question pool, requiring planned questions to be asked or dropped before submission while allowing new ambiguities to be added.

    Source: §1 · p. 2

  • Results

    Across three benchmarks, PlanPool improves ambiguity coverage and grounded success, reduces silent failures, and maintains competitive end-to-end execution accuracy.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Reliable agentic reasoning requires maintaining and discharging identified clarification needs, not only generating a plan.

    Source: §7 · p. 10

    Prompt-based planning represents the plan only as conversational text and does not require the agent to account for unresolved questions before submission.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Agentic Text-to-SQL systems can interact with users to clarify underspecified queries before generating SQL. However, a correct execution result does not necessarily imply that the agent has adequately resolved the underlying underspecification: the agent may silently make unverified assumptions that happen to match the intended answer. We show that this behavior is driven in part by premature clarification termination. Although forcing an agent to ask more questions improves execution accuracy, ambiguities are concentrated in earlier interactions, making brute-force questioning inefficient. More importantly, even when explicitly prompted to plan its clarification process, the agent frequently abandons questions that it has already identified as relevant. To address this failure mode, we introduce PlanPool, which externalizes the clarification plan as a mutable question pool. Every planned question must be explicitly asked or dropped before submission, while newly discovered ambiguities can be added during interaction. Across three benchmarks derived from BIRD-Interact and Spider, PlanPool consistently improves ambiguity coverage and reduces silent failures over unconstrained and prompt-based alternatives, while maintaining competitive execution accuracy. Our results highlight an important distinction in agentic reasoning: identifying missing information is not sufficient, and the agent must also reliably maintain and resolve it before committing to an answer.

1 INTRODUCTION

Underspecified requests require systems to clarify intent, yet strong execution can coexist with unresolved ambiguity. The introduction diagnoses premature clarification termination and motivates PlanPool’s explicit, adaptive question pool.

1 INTRODUCTION

  • Execution accuracy can improve while ambiguities remain unresolved: on BIRD-Lite, agents identify 2.6 of 3.6 annotated ambiguities on average.The same passage reports accuracy rising from 22.8% to 74.3% with multiple-turn interaction.Source: §1 · p. 1
  • Execution accuracy rises from 64.9% at k = 2 to 88.9% at k = 10, but later questions increasingly target non-critical or unanswerable issues.The authors characterize forcing more questions as effective but inefficient because annotated ambiguities concentrate earlier.Source: §1 · p. 2
  • On BIRD-Lite, the agent plans 4.1 clarification questions on average but asks only 3.3, often stopping after gaining confidence from answers or datasource evidence.The abandoned questions had already been identified as relevant.Source: §1 · p. 2
  • PlanPool externalizes a ranked clarification plan as a mutable pool: questions must be asked or explicitly dropped before submission, and newly discovered ambiguities can be added.This makes clarification adaptive while preventing planned questions from being silently forgotten.Source: §1 · p. 2
  • Across three benchmarks, PlanPool improves ambiguity coverage and grounded success, reduces silent failures, and retains competitive execution accuracy.The reported comparison includes unconstrained interaction, post-hoc reflection, and prompt-based planning.Source: §1 · p. 3

2 PROBLEM FORMULATION

The formulation treats underspecified Text-to-SQL as an interaction in which the agent must determine whether it has enough information to implement the intended query. It distinguishes execution correctness from whether the agent explicitly resolves the annotated ambiguities.

2 PROBLEM FORMULATION

  • An interactive trajectory allows the agent to explore the datasource, ask the user a clarification question, or submit SQL and end the interaction.The agent must decide whether available information suffices to determine the intended implementation.Source: §2 · p. 3
  • An ambiguity is unresolved information associated with multiple plausible interpretations that can lead to different intended SQL semantics; coverage is the fraction of annotated ambiguities addressed by questions.An ambiguity counts as addressed when the agent asks a corresponding clarification question.Source: §2 · p. 3
  • Grounded success requires both correct execution and resolution of all annotated ambiguities, distinguishing it from correct answers that leave ambiguities unresolved.The formulation calls a correct but unresolved answer an ungrounded success and an incorrect unresolved answer a silent failure.Source: §2 · p. 3

3 DIAGNOSING PREMATURE CLARIFICATION TERMINATION

Clarification can improve execution accuracy, but asking more questions becomes inefficient as later questions increasingly miss important ambiguities. Prompted plans also fail to ensure that agents resolve questions they have already identified, motivating explicit tracking of unresolved needs.

3 DIAGNOSING PREMATURE CLARIFICATION TERMINATION

  • The recurring failure is premature clarification termination: agents end interaction while relevant underspecification remains unresolved.The section examines whether too few questions explain this gap, then tests question budgets and prompted planning.Source: §3 · p. 4

3.1 MORE QUESTIONS HELP, BUT WITH DIMINISHING VALUE

  • Execution accuracy rises from 64.9% at k = 2 to 88.9% at k = 10, but added questions increasingly concern non-critical or unanswerable issues.At k = 10, agents ask 2.9 non-critical and 1.8 unanswerable questions per query, alongside 4.8 questions matching annotated ambiguities.Source: §3.1 · p. 4
  • Increasing the question budget improves accuracy, but brute-force clarification is inefficient because later questions have diminishing value.The desired behavior is to follow important identified ambiguities without asking unnecessary questions.Source: §3.1 · p. 4

3.2 PROMPTED PLANNING IDENTIFIES AMBIGUITIES BUT FAILS TO FOLLOW THROUGH

  • With prompted planning, agents often proceed to SQL generation while previously planned questions remain unresolved and unaccounted for.The questions may disappear from subsequent reasoning rather than being explicitly judged irrelevant or resolved.Source: §3.2 · p. 5
  • Prompt-based plans remain conversational text, so the model can stop clarifying without explicitly accounting for identified questions.The model may become confident in one interpretation even when its earlier analysis identified unresolved issues.Source: §3.2 · p. 5

4 PLANPOOL: EXTERNALIZING CLARIFICATION PLANS

PLANPOOL externalizes clarification planning as a persistent, mutable question pool. The agent may revise the pool during interaction, but must explicitly account for every pending question before submitting SQL.

4 PLANPOOL: EXTERNALIZING CLARIFICATION PLANS

  • The mechanism addresses the challenge of maintaining clarification needs until they are explicitly addressed.PLANPOOL turns clarification planning into an explicit interaction protocol rather than relying on a soft instruction.Source: §4 · p. 5

Pool initialization

  • PLANPOOL keeps a ranked initial clarification plan as explicit external state after datasource exploration.This prevents a previously identified clarification need from silently disappearing from the agent’s reasoning.Source: §4 · p. 5

Dynamic pool management

  • The agent can remove questions made redundant or irrelevant and add ambiguities discovered during interaction.This makes the plan adaptive rather than a static checklist while keeping unresolved needs represented.Source: §4 · p. 5

Explicit accounting before submission

  • SQL submission is allowed only when the pool is empty, requiring each pending question to be asked or deliberately removed.The protocol enforces explicit accounting, not exhaustive questioning; newly discovered needs face the same requirement.Source: §4 · p. 5

5 EXPERIMENTS

Across three benchmarks, PLANPOOL improves clarification groundedness over prompt-based planning while reducing silent failures, with competitive execution accuracy. The experiments examine setup, clarification behavior, and the interaction and computational costs of these gains.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses a shared agentic framework with a system agent, user simulator, datasource tools, and fixed underlying model across methods.The three benchmarks comprise two BIRD-Interact subsets and a Spider benchmark; BIRD instances are filtered to those solvable after annotated clarifications, while Spider instances receive multiple controlled underspecifications.Source: §5.1 · p. 6

5.2 MAIN RESULTS: CORRECTNESS AND GROUNDEDNESS

  • On BIRD-Lite, PLANPOOL raises ambiguity recall from 80.8% to 87.5% and grounded success from 42.1% to 49.1%, while reducing silent failure from 10.5% to 8.2%.The same pattern holds on BIRD-Full and Spider; the comparison is against Plan-Prompt, PLANPOOL’s closest planning-based baseline.Source: Table 2 · p. 7, §5.2 · p. 7

5.1 EXPERIMENTAL SETUP

  • Methods are compared on execution accuracy and clarification groundedness, including ambiguity recall, grounded and ungrounded success, silent failure, and average questions asked.The evaluation also records token usage and monetary cost.Source: §5.1 · p. 7

5.2 MAIN RESULTS: CORRECTNESS AND GROUNDEDNESS

  • On BIRD-Full, Self-Reflection has higher execution accuracy than PLANPOOL (70.7% versus 67.1%), but lower ambiguity recall and grounded success.36.9% of Self-Reflection instances are correct despite unresolved annotated ambiguity, compared with 19.1% for PLANPOOL.Source: §5.2 · pp. 7–8

5.3 HOW DOES PLANPOOL CHANGE CLARIFICATION BEHAVIOR?

  • PLANPOOL’s question pool is modified in 34.5% of BIRD-Lite, 38.2% of BIRD-Full, and 18.5% of Spider instances, with additions more common than drops on the BIRD datasets.Questions are added in 30.4% and 32.9% of BIRD-Lite and BIRD-Full instances, versus drops in 4.7% and 10.7%, respectively.Source: Table 3 · p. 8, §5.3 · p. 8
  • PLANPOOL has the closest detected-ambiguity distribution to the annotated distribution on all three datasets, with KL divergence falling from 0.34/0.17 to 0.09 on BIRD-Lite, 0.29/0.13 to 0.07 on BIRD-Full, and 0.78/0.37 to 0.28 on Spider.Each pair gives Naive and Plan-Prompt values, followed by PLANPOOL’s value.Source: Figure 3 · p. 8, §5.3 · p. 8

5.4 INTERACTION AND COMPUTATIONAL COST

  • PLANPOOL asks more clarification questions than unconstrained and prompt-based methods, but fewer than the larger brute-force budgets where later questions increasingly become non-critical or unanswerable.The added interaction is the cost of preventing unresolved clarification needs from being silently skipped.Source: §5.4 · p. 8
  • Read-All uses the fewest tokens, partly because it asks fewer questions and terminates earlier, but this lower cost coincides with weaker execution accuracy and groundedness.Repeated context is served through prompt caching, so monetary cost differences are smaller than raw token counts suggest.Source: §5.4 · pp. 8–9, Table 4 · p. 9

6 RELATED WORK

Prior Text-to-SQL research has emphasized SQL generation and interactive clarification, while agent research has examined planning and self-refinement. PLANPOOL connects to work on structured agent state by keeping identified ambiguities represented until they are resolved or deliberately removed.

Text-to-SQL

  • Text-to-SQL research has moved from supervised semantic parsing on Spider Yu et al. (2018) to LLM methods using decomposition, self-correction, and agent collaboration Pourreza & Rafiei (2023); Wang et al. (2025a).Many methods assume specified requests and focus on SQL generation; other work studies multi-turn interaction Yu et al. (2019b;a), ambiguous requests Min et al. (2020); Bhaskar et al. (2023); Saparina & Lapata (2024); Chen et al. (2025); Dong et al. (2025); Ding et al. (2026), and BIRD-Interact Huo et al. (2026).Source: §6 · p. 9

LLM agents

  • Agent research uses planning for long-horizon execution Wang et al. (2023a); Xu et al. (2023) and post-hoc feedback or self-refinement Madaan et al. (2023); Shinn et al. (2023); Gou et al. (2024).Self-refinement has also been applied to Text-to-SQL Wang et al. (2024).Source: §6 · p. 9
  • PLANPOOL is complementary to agent-memory research Wang et al. (2025b); Xu et al. (2025); Yang et al. (2025); Wan et al. (2026), because it keeps identified ambiguities explicit until addressed or deliberately removed.Source: §6 · p. 9

7 CONCLUSION

The conclusion argues that execution correctness alone misses whether an agent resolved the assumptions underlying its answer. It presents PLANPOOL as explicit, dynamically managed clarification state and concludes that reliable agentic reasoning requires maintaining and discharging unresolved commitments.

7 CONCLUSION

  • Across three benchmarks, PLANPOOL improves ambiguity coverage and grounded success while reducing silent failures, with competitive execution accuracy.The conclusion frames the central challenge as maintaining unresolved information throughout an evolving interaction, not only identifying useful questions.Source: §7 · p. 10
  • The paper’s broader conclusion is that reliable agentic reasoning requires explicitly maintaining and discharging unresolved commitments induced by plans.Source: §7 · p. 10

AI use disclosure

  • Generative AI tools were used in experiments and research and writing workflows, and the authors state that all AI-assisted outputs were reviewed.The statement also describes execution-based validation for benchmark construction and filtering outputs involving LLMs.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The reproducibility statement points to paper and appendix details covering datasets, methods, prompts, interaction tools, and evaluation procedures.Source: REPRODUCIBILITY STATEMENT · p. 10

A BENCHMARK CONSTRUCTION AND PROCESSING

The appendix describes filtering BIRD-Interact instances to focus evaluation on ambiguity handling rather than intrinsic SQL-generation failures. It retains instances solvable after all annotated critical ambiguities are clarified.

A.1 BIRD-INTERACT FILTERING

  • BIRD-Interact-Lite and BIRD-Interact-Full contain 300 and 600 instances, with intent-, implementation-, and knowledge-level ambiguities; the filtering focuses on intent and knowledge ambiguities that affect the intended query.Only the priority question is retained for each instance.Source: §A.1 · p. 13
  • The oracle filter supplies a clarification for every annotated critical ambiguity, then retains an instance only if the agent’s resulting SQL passes execution-based evaluation.This isolates clarification strategy performance from SQL-generation failures that persist even after the ambiguities are resolved.Source: §A.1 · p. 13
  • The filter retains 171 of 300 BIRD-Interact-Lite instances and 225 of 600 BIRD-Interact-Full instances, naming the resulting benchmarks BIRD-Lite and BIRD-Full.Source: §A.1 · p. 13

A.2 SPIDER MULTI-AMBIGUITY CONSTRUCTION

The multi-ambiguity Spider benchmark extends PRACTIQ by composing applicable corruption operators within a single instance. Modified instances are execution-validated and annotated with their injected ambiguities.

A.2 SPIDER MULTI-AMBIGUITY CONSTRUCTION

  • Four corruption types target SELECT columns, values within a column, WHERE columns, and filter criteria.The operators make the intended target, value, column, or criterion underspecified.Source: §A.2 · p. 13
  • The construction composes PRACTIQ’s applicable corruption operators in one Spider instance, modifying the query, schema or cell values, and reference SQL.PRACTIQ introduces a single ambiguity per instance; this extension composes multiple applicable operators.Source: §A.2 · pp. 13–14
  • The pipeline validates modified SQL by execution, discards failures, then annotates injected ambiguities and yields 297 valid instances.The known transformations supply each ambiguity’s target and type; an LLM locates its textual span and SQL snippet.Source: §A.2 · p. 14, Figure 4 · p. 14

B AGENTIC INTERACTION FRAMEWORK

The agentic interaction framework gives methods a shared ReAct-style harness while leaving interaction choices to the agent. It pairs datasource tools with a two-stage user simulator that also supports groundedness evaluation.

B AGENTIC INTERACTION FRAMEWORK

  • All methods share a ReAct-style base prompt that encourages datasource exploration and clarification but leaves the interaction trajectory to the agent.PLANPOOL masks the base ask(question) action and mediates clarification through pool next().Source: B AGENTIC INTERACTION FRAMEWORK · p. 14
  • Datasource tools support schema inspection, retrieval of column semantics and external knowledge, value probing, and SQL execution; action-level costs are unused.The original BIRD-Interact tool interfaces are adopted.Source: B AGENTIC INTERACTION FRAMEWORK · p. 14, Table 5 · p. 14
  • The two-stage simulator maps answerable questions to annotated ambiguities, generates responses from task information, and uses the mapping to evaluate groundedness.Questions unsupported by the task specification are treated as unanswerable rather than answered using inferred information.Source: B AGENTIC INTERACTION FRAMEWORK · p. 15

C IMPLEMENTATION DETAILS

The implementation realizes PLANPOOL through explicit pool operations and uses separate harness procedures for alternative interaction strategies. These procedures share the same underlying agent harness while varying how exploration or clarification is handled.

C IMPLEMENTATION DETAILS

  • PLANPOOL initializes a ranked question pool, exposes its state, asks the highest-priority pending question, and permits explicit drops or additions.The agent can continue exploring while pending questions remain; SQL generation follows once the pool is empty.Source: C IMPLEMENTATION DETAILS · p. 15
  • The ordinary interaction branch lets the agent explore or ask questions before generating an initial SQL candidate.The Regenerate procedure then independently regenerates SQL from the context truncated at the last clarification.Source: C IMPLEMENTATION DETAILS · p. 16
  • Regenerate passes through matching normalized SQL candidates; divergent candidates trigger a fresh-context ambiguity-resolution interaction before final SQL generation.The divergence audit is returned to the original agent through a harness-level observation.Source: C IMPLEMENTATION DETAILS · p. 16
  • Read-All retrieves schema, column descriptions, and available external knowledge before the first agent turn, discouraging redundant exploration calls.Source: C IMPLEMENTATION DETAILS · p. 16

D CASE STUDY

The case study follows PLANPOOL as it resolves a museum query with several ambiguities. Its pool changes during interaction, and the resulting SQL passes execution evaluation.

D CASE STUDY

  • The query has three annotated ambiguities: urgent-care criterion, ranking method, and how to compute the domain-specific Conservation Priority Index.The CPI definition cannot be determined from the database schema alone.Source: D CASE STUDY · p. 16
  • The initial ranked pool asks about the score and ERF inputs, direct use of a CPI column, the meaning of “historical era,” and sorting or numeric rank.Source: D CASE STUDY · pp. 16–17
  • The final SQL reports artifact identifiers, names, historical eras, and CPI scores, joins artifact and rating records, and passes execution evaluation.Its score formula combines three ratings with a conservation-status adjustment and orders results by descending CPI score.Source: D CASE STUDY · p. 17
  • One planned question is dropped and one new question added, leaving five actual user questions from an initial pool of five.The example illustrates persistent accounting without requiring the initial plan to remain fixed.Source: D CASE STUDY · pp. 17–18

Related papers

Loading 2610.02739v1…