Source-linked AI summary
Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift
Yujia Liu, Jiayan Lin, Zijin Hong, Zheng Yuan, Shengyuan Chen, Hao Chen, Qinggang Zhang, Xiao Huang, Feiran Huang
TL;DR
Conversational text-to-SQL benchmarks largely miss how user intent unfolds through dependent clarification and later retraction. TIDE-BENCH addresses this with controlled samples and dedicated metrics, revealing persistent chain-identification difficulty, a drift recognition-resolution gap, and overlapping failures when both patterns co-occur.
Problem
Existing evaluations emphasize execution accuracy or turn-local ambiguity, leaving cross-turn clarification dependency and intent retraction insufficiently covered.
Method
TIDE-BENCH builds paired Chain-only, Drift-only, and Joint samples from 514 BIRD anchor SQLs and evaluates them with counterfactual, non-monotonicity, and dedicated behavioral tests.
Results
Evaluating 12 advanced LLMs reveals a persistent chain-identification bottleneck, a drift recognition-resolution gap, and overlapping failures under joint activation.
Takeaways & Limitations
The benchmark makes dialogue-level dependency and retraction measurable beyond execution accuracy, exposing distinct but overlapping failure modes in conversational text-to-SQL.
Takeaways & Limitations
Generalization beyond BIRD, chains deeper than two clarification layers, drift-then-chain ordering, and fully human user behavior remain untested or unexplored.
Abstract
from arXiv · showhide
Recent advances in large language models (LLMs) have established conversational text-to-SQL as a practical interface between users and databases, often involving multiple turns of clarification and revision. However, existing benchmarks primarily evaluate execution accuracy, leaving the unfolding and shifting of user intent across turns largely uncovered. To address this, we introduce TIDE-Bench, a benchmark for conversational text-to-SQL under chain ambiguity and intent drift evaluation, targeting two recurring patterns: chain ambiguity, where an underspecified question triggers layered clarification with conditional dependencies, and intent drift, where the user retracts and replaces a previously committed request element. Built on 514 anchor SQLs from BIRD, TIDE-Bench comprises 1,542 samples and introduces dedicated metrics for chain identification and drift recognition-resolution beyond execution accuracy. Evaluating 12 advanced LLMs reveals a persistent chain identification bottleneck unaffected by clarification frequency, a wide drift recognition-resolution gap, and overlap between failure modes when jointly activated. The corresponding code of TIDE-Bench is released for further research.
1 Introduction
Conversational text-to-SQL requires handling intent that unfolds through clarification and revision, but existing benchmarks largely assess only end-to-end execution. TIDE-BENCH targets chain ambiguity and user intent drift with controlled samples and dedicated behavioral metrics.
- Multi-turn text-to-SQL interactions refine requests and seek clarification before SQL is produced, yet strong single-turn agents can fail conversationally.
- Chain ambiguity involves layered clarification whose later question depends on resolving an earlier one, while intent drift retracts and replaces a committed request element.
- Existing benchmarks measure multi-turn SQL accuracy or turn-local ambiguity, leaving cross-turn dependency and retraction insufficiently evaluated.
- TIDE-BENCH uses counterfactual testing for conditional dependency and non-monotonicity testing over SQL-derived commitment sets.
- 514 anchor SQLs generate 1,542 samples through a four-stage pipeline with automated validation and human verification.
- Evaluations of 12 advanced LLMs find persistent chain-identification bottlenecks, a drift recognition-resolution gap, and overlapping failures when both patterns occur.
2 Problem Formalization
The paper formalizes two dialogue-level phenomena: clarification dependencies that change across layers and non-monotonic revisions of accumulated user commitments. Each is defined with an operational test that distinguishes the target behavior from superficially similar interactions.
- 2 Problem Formalization: Conversational text-to-SQL intent may be underspecified initially, clarified over time, or revised after a prior choice, motivating formalization of chain ambiguity and intent drift.
- 2.1 Chain Ambiguity: Chain ambiguity occurs when an underspecified query induces two clarification layers, with L2 conditionally dependent on resolving L1 and two plausible interpretations at each layer.
- 2.1 Chain Ambiguity: The counterfactual test asks whether L2 would remain the same under an alternative L1 resolution, yielding Eliminated, Transformed, or Invariant verdicts.
- 2.1 Chain Ambiguity: In the worked example, changing L1 from a monetary quantity to tenure removes the referent needed for L2’s refund-exclusion question, producing an Eliminated verdict.
- 2.2 User Intent Drift: User intent drift is a non-monotonic revision that withdraws or replaces an established commitment and discards withdrawn requirements from the resulting query.
- 2.2 User Intent Drift: The user’s accumulated intent is represented by SQL y and commitment set S(y), covering predicates, projections, grouping, ordering, aggregation filters, limits, and offsets.
- 2.2 User Intent Drift: Drift occurs iff Spre ⊈ Spost, allowing additions alongside removals or replacements while excluding unchanged updates and pure additions.
- 2.2 User Intent Drift: The benchmark distinguishes value-level, constraint-level, and projection-level drift according to which commitments change.
3 TIDE-BENCH Construction
TIDE-BENCH isolates chain ambiguity and intent drift through matched samples built from BIRD anchors, then validates the resulting benchmark with automated checks and human annotation. Its metrics cover execution, clarification, chain-layer identification, and drift handling.
- Paired Design: Each eligible BIRD anchor spawns Chain-only, Drift-only, and Joint samples, controlling for schema complexity, domain difficulty, and surface ambiguity.
- Construction Pipeline: The construction pipeline has four stages: anchor selection, chain injection, drift injection, and triplet assembly.
- Anchor Selection: Candidate anchors are executable BIRD SQLs with sufficient structural complexity for two-layer chain injection and at least one admissible drift subtype.
- Chain Injection: Chain injection generates two clarification layers and alternatives, then judges apply the counterfactual test to discard invariant candidates.
- Drift Injection: AST validation checks parseability, executability, distinct results, non-monotonicity, and subtype-specific constraints before semantic cross-validation.
- Triplet Assembly: 514 triplets yield 1,542 matched samples, with Chain-only, Drift-only, and Joint instances sharing each anchor and schema.
- Human Validation: Human auditing covers 250 stratified triplets, with five annotators independently applying a checklist while blinded to automated verdicts.
- Human Validation: Majority acceptance reaches 92.4% overall, while counterfactual dependency, non-monotonic retraction, and joint transition coherence pass at 93.4%, 94.8%, and 91.6%.
4 Evaluation Protocol
TIDE-BENCH evaluates conversational text-to-SQL through simulator-driven dialogues under full and L1-primed protocols, using execution, clarification, identification, and drift-handling metrics.
- Evaluation setup: Each sample runs an agent–user-simulator dialogue over database D and schema S, alternating user utterances with clarification requests or candidate SQL queries.Drift-only and Joint samples inject a pre-annotated drift turn after the agent’s first SQL query.
- Evaluation protocols: The f-protocol starts from the initial question and requires resolving both clarification layers for Chain-only and Joint samples.
- Evaluation protocols: The p-protocol prepopulates a synthetic three-turn L1 resolution, marking L1 resolved before the first live turn to isolate L2 identification.
- Metrics: Seven metrics cover execution accuracy, clarification behavior, chain-layer identification, and drift handling, with all metrics computed as proportions over sample sets.DEA requires every trajectory SQL query to execute correctly; AR measures asking, while L1-IR, L2-IRp, JIR, DRR, and DRA measure identification or drift handling.
- Metrics: Figure 3 compares per-model L1-IR under the f-protocol with L2-IRp under the p-protocol and plots JIR against ask rate.The figure caption reports that L2 identification remains low after L1 pre-resolution and that identification remains low regardless of asking frequency.
5 Experimental Results
Across 12 LLMs, conversational text-to-SQL performance remains low, with distinct weaknesses in identifying clarification chains, resolving drift, and handling their combination.
- Overall Performance: No model exceeds 46% overall on any sample type, while Claude-Sonnet-4.6 reaches 45.1% DEA on Drift-only and 27.8% on Joint.DEA is highest on Drift-only, lowest on Joint, and intermediate on Chain-only.
- Overall Performance: Larger models outperform smaller ones by only 2–3 percentage points on average, and top-to-bottom model spread remains roughly within 10 points.The low performance is not concentrated in a particular model class.
- What Makes Chain Challenging: JIR stays below 5% for every model, while even L2-IRp after externally resolving L1 ranges only from 1.2% to 13.8%.This indicates failures in both recognizing the chain and independently identifying its second layer.
- What Makes Chain Challenging: JIR remains under 4% across ask rates ranging from below 10% to over 60%, so asking more often does not reliably surface chain ambiguity.Claude-Sonnet-4.6 asks on 9.7% of Chain-only samples, versus 60.1% for Qwen2.5-72B, yet both have JIR at most 1.2%.
- Where Drift Handling Fails: DRR is at least 94.9%, but DRA stays around 45–55%, leaving every model with a recognition–resolution gap exceeding 40 percentage points.Models generally register the drift turn but fail to recover an executionally correct post-drift SQL.
- Where Drift Handling Fails: Projection-level drift is easiest for 11 of 12 models, while constraint-level drift is hardest for 10 of 12 despite producing the simplest post-drift SQL.Within one model, subtype DRA differs by up to 19.4 points; GPT-5.4 scores 61.7% on projection drift versus 42.3% on constraint drift.
- How Chain and Drift Interact: From Drift-only to Joint, DRR falls by at most 5 points, DRA by 6–12, and DEA by 13–18 across all models.The post-drift ground-truth SQL is held identical across the two conditions, isolating the cost of placing drift after a chain phase.
- How Chain and Drift Interact: Joint DEA exceeds the approximately 13% independent-failure prediction by 10–13 points, indicating that chain and drift failures largely overlap rather than occur independently.The combined analysis identifies three distinct dialogue-level failure modes that models do not handle consistently.
6 Related Work
Prior text-to-SQL benchmarks progress from static cross-domain and enterprise evaluation toward conversational interaction, but mainly score end-to-end execution. Related ambiguity work largely remains turn-local.
- Static benchmarks: Spider, BIRD, and Spider 2.0 advance evaluation across cross-domain, knowledge-grounded, and enterprise settings while assuming well-formed questions.These static benchmarks leave multi-turn interaction untested.
- Conversational and multi-turn benchmarks: SParC and CoSQL introduce context-dependent question sequences and Wizard-of-Oz dialogues, while BIRD-Interact and DySQL-Bench add user simulation and evolving-intent settings.These benchmarks bring dialogue and greater interaction realism into text-to-SQL evaluation.
- Ambiguity and clarification: Existing conversational benchmarks mainly measure end-to-end multi-turn execution, while prior ambiguity studies focus on lexical, structural, linguistic, or single-turn clarification phenomena.This leaves cross-turn dependency and retraction insufficiently evaluated.
7 Conclusion
TIDE-BENCH evaluates chain ambiguity and user intent drift in conversational text-to-SQL using a paired benchmark design. Experiments across 12 LLMs reveal persistent identification, resolution, and interaction failures, positioning the benchmark as a diagnostic tool.
- Conclusion: TIDE-BENCH targets chain ambiguity and user intent drift, two underexamined patterns in conversational text-to-SQL interaction.It evaluates how agents handle layered clarification and changing user requests.
- Conclusion: The benchmark contains 1,542 samples from 514 BIRD anchor SQLs, constructed through an automated pipeline with multi-stage validation and human verification.Its paired design supports comparison across activated interaction patterns.
- Conclusion: Across 12 LLMs, chain ambiguity exposes an ask-behavior-independent identification bottleneck, drift reveals a recognition–resolution gap, and joint failures overlap.These are reported as three failure modes of current conversational text-to-SQL systems.
- Conclusion: TIDE-BENCH is intended as a diagnostic tool for research and development in conversational text-to-SQL.The benchmark’s released code supports further research.
Limitations
TIDE-Bench’s evaluation is bounded by benchmark coverage, shallow chain structure, one joint-order configuration, and reliance on an imperfect LLM-based user simulator.
- Generalization beyond BIRD and the benchmark’s cross-domain coverage remains untested on other benchmarks.
- Chain ambiguity is limited to two clarification layers, leaving deeper chains for future work.
- Joint samples only use a chain-then-drift order, so drift-then-chain remains unexplored.
- The evaluation relies on an LLM-based user simulator whose proxy behavior remains imperfect despite validation on diagnostic suites and expert-labeled traces.The simulator achieved Fleiss’ κ = 0.802 on expert-labeled traces, but human-in-the-loop evaluation would better reflect practical user behavior and simulator-specific biases.
C.5 Dominant Failure Modes
The audit identifies different dominant rejection patterns across chain-only, drift-only, and joint samples, with joint samples combining several coherence and transition problems.
- Chain-only: 17 chain-only samples were rejected, most often because they failed counterfactual dependency.Other rejections involved L1 or L2 plausibility, while initial underspecification was rarely flagged.
- Majority pass rates were determined from five-annotator checklist judgments across the three core validation criteria.A criterion passed when at least three of five annotators marked it satisfied.
- Drift-only: 11 drift-only samples were rejected, most often because post-drift SQL did not faithfully match the drift utterance.Fewer failures involved preserving non-targeted commitments, and non-monotonic retraction was rarely flagged.
- Joint: 29 joint samples were rejected most often for counterfactual dependency, non-monotonic retraction, or chain-to-drift boundary coherence.The remaining rejections reflected less frequent failures on other checklist items.
D Evaluation Setup
The evaluation tests 12 frontier LLMs in a bounded agent–simulator dialogue harness using fixed prompts, read-only BIRD execution, and a two-stage user simulator.
- Models: 12 frontier LLMs spanning reasoning and non-reasoning models are evaluated through official APIs.The model set includes Claude-Sonnet-4.6, GPT-4o, GPT-5.4, o3-mini, o4-mini, Gemini-2.5-Flash, Grok-4-Fast, Grok-4.1-Fast, Qwen2.5-72B, Qwen-Max, Llama-3.3-70B, and DeepSeek-V4-Flash.
- Dialogue harness: Each dialogue runs for at most 20 turns, allowing either clarification requests or candidate SQL responses parsed from JSON.Malformed responses receive one retry; persistent failures count as abstentions, and candidate SQL executes read-only against the corresponding BIRD SQLite database.
- Inputs to the agent: At every turn, agents receive the database DDL, rewritten evidence, and dialogue history under a fixed prompt shared across models.
- Two-stage design: The simulator’s Stage 1 router classifies agent clarifications as AMB, LOC, or UNA before Stage 2 generates constrained responses.TIDE-Bench routes unresolved-L1 L2-targeted questions as LOC, restricts Stage 2 traversal to one atomic node, and validates generated responses for out-of-scope content.
- Configuration and validation: Stage 1 routing and Stage 2 generation both use GPT-5.1 with matched temperature and decoding settings, and simulator behavior is checked on USERSIM-GUARD and labeled traces.Expert-labeled traces produced Fleiss’ κ = 0.802.
F Full Per-Model Results
The full per-model breakdowns report chain-layer identification and ask rates alongside drift recognition and resolution by subtype, while additional analyses characterize drift SQL changes across all 514 triplets.
- Per-model breakdowns: Tables 13 and 14 provide complete per-model results for chain-layer identification, ask rates, and drift recognition-resolution across the evaluated settings.Chain metrics cover Chain-only and Joint samples; drift metrics cover Drift-only and Joint samples and are broken down by subtype.
- Drift subtype analysis: The appendix characterizes value-level drift as structurally identical pre- and post-drift SQL and constraint-level drift as the simplest subtype.
- Drift subtype analysis: 514 triplets are used for the subtype comparison, and all anchors parse successfully under the analyzer.
G.1 Per-Subtype Structural Effects
Drift changes structural complexity differently by subtype: value-level drift preserves structure, while constraint- and projection-level drift selectively reduce specific SQL components. These patterns reflect the subtype definitions and isolate which commitments change.
- Cross-subtype comparison: The structural measures distinguish drift subtypes by showing whether revisions affect literals, constraints, or projection-related SQL components.Table 12 reports mean pre-to-post changes, with negative values indicating simpler post-drift SQL and zero indicating no average change.
- Value-level: Value-level drift produces zero mean change across structural measures because it replaces only a WHERE literal while preserving predicate structure and other commitments.The AST validator enforces this invariant, and pre- and post-drift tier distributions remain identical at 44/107/24 across basic, intermediate, and advanced tiers.
- Constraint-level: Constraint-level drift reduces WHERE predicates by 1.11 and token count by 7.47 on average, while leaving JOIN, aggregation, and subquery structures unchanged.Its tier distribution shifts from 44/104/13 to 76/79/6, indicating lower post-drift complexity across tiers.
- Projection-level: Projection-level drift reduces aggregation count by 0.26 on average, while WHERE predicates and JOIN count change by only −0.03 each.The aggregation reduction can reflect movement from aggregate queries to non-aggregate top-k listings while filters are largely preserved.