PaperlayerView exact revision on arXiv

Source-linked AI summary

BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions

Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, Linheng Han, Edward Alexander, Xintong Zhu, Rui Qin, Ruihan Yu, Yiyao Jin, Feige Zhou, Weihao Zhong, Yun Chen, Hongyu Liu, Chenhao Ma, Fatma Ozcan, Yannis Papakonstantinou, Reynold Cheng

arXiv:2510.05318v3cs.AI

TL;DR

BIRD-INTERACT addresses the gap between single-turn text-to-SQL benchmarks and real database use, where ambiguity, execution feedback, and evolving tasks require dynamic interaction. It introduces an executable, simulator-based benchmark with conversational and agentic settings and finds that current models achieve low end-to-end success, including GPT-5’s 8.67% in c-Interact and 17% in a-Interact. The benchmark therefore exposes strategic interaction as a central challenge for complex text-to-SQL systems.

  • Problem

    Existing text-to-SQL benchmarks inadequately represent real database interaction because practical tasks involve ambiguity, failed attempts, clarifications, and evolving goals rather than static conversation histories.

  • Method

    BIRD-INTERACT couples executable databases with metadata, hierarchical knowledge, a function-driven user simulator, two interaction settings, and executable evaluation over interactive CRUD tasks.

  • Results

    GPT-5 completes 8.67% of tasks in c-Interact and 17% in a-Interact, while broader results show no more than 16.33% and 17.00% end-to-end success respectively.

  • Takeaways & Limitations

    BIRD-INTERACT reveals a gap between SQL generation capability and the strategic interaction skills required for effective human-AI database collaboration.

  • Takeaways & Limitations

    The current a-Interact setting imposes strict budget constraints, creating a stress-mode evaluation that pressures agents to make optimal decisions under resource scarcity.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated remarkable performance on single-turn text-to-SQL tasks, but real-world database applications predominantly require multi-turn interactions to handle ambiguous queries, execution errors, and evolving user requirements. Existing multi-turn benchmarks fall short by treating conversation histories as static context or limiting evaluation to read-only operations, failing to reflect production-grade database assistant challenges. We introduce BIRD-INTERACT, a benchmark that restores this realism through: (1) a comprehensive interaction environment coupling each database with a hierarchical knowledge base, metadata files, and a function-driven user simulator, enabling models to solicit clarifications, retrieve knowledge, and recover from errors without human supervision; (2) two evaluation settings consisting of a pre-defined conversational protocol (c-Interact) and an open-ended agentic setting (a-Interact) where models autonomously decide when to query the user simulator or explore the environment; (3) a challenging task suite covering the full CRUD spectrum for business-intelligence and operational use cases, guarded by executable test cases. Each task features ambiguous and follow-up sub-tasks requiring dynamic interaction. The suite comprises BIRD-INTERACT-FULL (600 tasks, up to 11,796 interactions) for comprehensive performance assessment, and BIRD-INTERACT-LITE (300 tasks with simplified databases) for detailed behavioral analysis and rapid method development. Our empirical results highlight BIRD-INTERACT's difficulty: GPT-5 completes only 8.67% of tasks in c-Interact and 17.00% in a-Interact. Analysis via memory grafting and Interaction Test-time Scaling validates the importance of effective interaction for complex, dynamic text-to-SQL tasks.

1 INTRODUCTION

BIRD-INTERACT addresses the mismatch between strong single-turn text-to-SQL results and real database interactions involving ambiguity, failed executions, and evolving goals. It introduces a dynamic benchmark environment, dual interaction settings, and challenging CRUD tasks, while showing that current models still struggle.

  • Motivation: BIRD-INTERACT evaluates text-to-SQL systems through dynamic interactions involving clarification, error recovery, and follow-up queries rather than isolated SQL generation.Realistic tasks require systems to resolve ambiguity, revise failed code using execution feedback, and answer questions dependent on intermediate results.
  • Benchmark design: The benchmark couples executable databases with hierarchical knowledge, metadata, and a function-driven user simulator for end-to-end evaluation without human intervention.The simulator answers clarification questions, provides action feedback, and guides models through complex tasks.
  • Evaluation settings: c-Interact tests protocol following, whereas a-Interact requires autonomous decisions about querying users, databases, and documentation.The two settings distinguish structured conversational behavior from open-ended agentic planning.
  • Task suite: The task suite spans CRUD operations across analytical and operational domains, with executable tests and ambiguous initial tasks followed by dynamic sub-tasks.These design choices target environmental uncertainty and interaction-dependent task completion.
  • Results: GPT-5 achieves only 8.67% success in c-Interact and 17% in a-Interact, highlighting the difficulty of dynamic text-to-SQL evaluation.The reported experiments identify substantial remaining challenges for current models.

2 PROBLEM DEFINITION

BIRD-INTERACT formalizes interactive text-to-SQL as multi-turn collaboration between a system and user simulator over an executable database environment, metadata, and external knowledge. Related sub-tasks proceed through interaction turns until SQL solutions pass executable correctness tests.

  • Task Definition: Interactive text-to-SQL models collaboration between system Sθ and user simulator Uγ over environment E = {D, M, K}, containing a database, metadata, and external knowledge.The task includes a sequence of related sub-tasks requiring SQL solutions generated through interaction.
  • Interaction process: The user simulator presents sub-tasks, answers clarification questions, and provides SQL feedback, releasing later sub-tasks only after earlier success.This creates stateful dependencies across the interaction sequence.
  • Correctness: Predicted SQL is judged correct when it passes all associated executable test cases, ensuring functional equivalence with the ground-truth solution.The benchmark therefore evaluates execution behavior rather than relying only on textual SQL similarity.
  • Metrics: Each task uses an ambiguous priority sub-task followed by a follow-up sub-task, with Success Rate reported separately for both during interaction.The implementation sets n = 2 and also reports normalized reward.

3 BENCHMARK CONSTRUCTION

BIRD-INTERACT converts LIVESQLBENCH tasks into interactive scenarios by injecting controlled ambiguities, adding state-dependent follow-ups, and using a function-driven simulator. The resulting 900-task benchmark spans CRUD operations and supports scalable evaluation of interactive behavior.

  • Benchmark foundation: The benchmark builds on LIVESQLBENCH because it supports DML and DDL, executable database sandboxes, metadata files, and hierarchical knowledge.These components support dynamic database states and reproducible extensions.
  • Construction process: Annotators create multi-turn scenarios through ambiguity injection and follow-up sub-task generation.The construction process explicitly transforms clear single-turn tasks into interaction-dependent problems.
  • Ambiguity injection: Injected ambiguities cover vague or underspecified requests, incomplete external knowledge, masked knowledge-chain intermediates, and environmental uncertainty.Knowledge ambiguities include one-shot omissions and broken multi-hop chains such as urgent care → AVS → IF/CPI.
  • Ambiguity control: Each ambiguity is paired with a ground-truth SQL clarification source so the simulator can generate consistent responses while preserving reconstructability after clarification.Quality control ensures tasks are unsolvable without clarification but reconstructable once it is provided.
  • Follow-up annotation: Follow-up sub-tasks introduce state dependency, requiring models to reason over modified database states or objects created by preceding queries.The follow-ups are organized using a principled five-category taxonomy.
  • Final benchmark: The benchmark contains 900 interactive tasks with ambiguous initial sub-tasks, dynamic clarification, follow-ups, environmental uncertainty, and full CRUD coverage.Its user simulator uses constrained symbolic actions before generating controlled responses to reduce leakage risks in scalable evaluation.

4 EVALUATION SETTINGS

BIRD-INTERACT evaluates interactive text-to-SQL through protocol-guided conversation and autonomous agentic interaction, under explicit limits on turns, actions, patience, and computational resources.

  • c-INTERACT EVALUATION: c-Interact stages clarification, SQL generation, validation, and follow-up completion, allowing one debugging revision after execution failure.Failure on the initial priority sub-task terminates the session, and debugging incurs a reward penalty.
  • c-INTERACT EVALUATION: Clarification turns are capped by τclar = mamb + λpat, where mamb counts annotated ambiguities and λpat represents extra user patience.The patience parameter provides additional clarification opportunities beyond the minimum ambiguity-resolution budget.
  • a-INTERACT EVALUATION: a-Interact gives agents callable access to the database, metadata, knowledge base, and user simulator for autonomous planning and execution.The setting follows a ReAct-style action space with nine discrete actions.
  • 4 EVALUATION SETTINGS: The broader budget framework combines a base budget with ambiguity and patience terms, enabling evaluation under finite computational resources and limited user patience.This design supports stress-testing under varying budgets.
  • 4 EVALUATION SETTINGS: BIRD-INTERACT reports success rate and normalized reward on FULL, with c-Interact values measured after debugging and average task cost included.The user simulator has an average cost of 0.03 USD.

5 EXPERIMENT

Experiments show that BIRD-INTERACT remains difficult, with performance shaped by interaction mode, task type, communication quality, and the number of available interaction opportunities.

  • 5.1 MAIN RESULTS: 16.33% and 17.00% are the highest end-to-end task success rates observed in c-Interact and a-Interact, respectively.Gemini-2.5-Pro and GPT-5 capture only 20.92% and 25.52% of available reward in the two modes.
  • 5.1 MAIN RESULTS: Follow-up sub-tasks are more challenging, with longer concatenated context remaining a bottleneck for interactive text-to-SQL.The evaluation distinguishes initial priority objectives from subsequent follow-up objectives.
  • 5.1 MAIN RESULTS: BI queries are significantly harder than DM tasks because they require domain-specific business logic and analytical reasoning, whereas DM operations are more standardized.The comparison covers both business-intelligence and data-management use cases.
  • 5.1 MAIN RESULTS: GPT-5 reaches 14.50% SR in c-Interact but 29.17% SR in a-Interact, making interaction mode a decisive factor for its outcome.The authors associate this difference with model-specific capabilities and hypothesize links to training distributions and architectural biases.
  • 5.2 INTERACTION ANALYSIS: Memory grafting substantially improves GPT-5 when it receives ambiguity-resolution histories from Qwen-3-Coder or O3-mini, indicating communication limitations in c-Interact.The experiment supports a distinction between SQL generation ability and interactive communication effectiveness.
  • 5.2 INTERACTION ANALYSIS: Claude-3.7-Sonnet shows clear performance scaling as user patience and interaction opportunities increase, while the idealized baseline removes ambiguity through single-turn context.ITS defines success as matching or surpassing idealized single-turn performance given enough interactive turns.
  • 5.2 INTERACTION ANALYSIS: Submit and ask comprise 60.87% of actions, showing that models favor costly execution and clarification over systematic knowledge or schema exploration.The authors characterize this pattern as a preference for direct trial-and-error execution.

6 USER SIMULATOR ANALYSIS

The user-simulator analysis evaluates reliability and human alignment, finding that the function-driven design reduces unsafe responses and better reflects human interaction patterns.

  • 6 USER SIMULATOR ANALYSIS: Failure rates on unanswerable questions reach 67.4% for conventional simulators but fall to as low as 2.7% with the function-driven approach.The approach invokes constrained handling for inappropriate or unanswerable requests.
  • 6 USER SIMULATOR ANALYSIS: The evaluation uses USERSIM-GUARD, a 2,100-question dataset with human-labeled reference actions, alongside LLM-as-Judge assessment.The benchmark specifically tests simulator behavior across question categories, including unanswerable requests.
  • 6 USER SIMULATOR ANALYSIS: Function-driven simulators show stronger alignment with human behavior across randomly sampled BI and DM tasks.The comparison correlates success rates achieved by human users and simulators across the same tasks.
  • 6 USER SIMULATOR ANALYSIS: The authors conclude that the designed mechanism produces more realistic user simulators that better reflect human-AI interaction patterns.This conclusion is supported by both objective safeguards and human-alignment comparisons.

7 RELATED WORK

Prior text-to-SQL research emphasizes strong single-turn LLM performance, while multi-turn benchmarks add dialogue but commonly rely on static, noise-free histories.

  • 7 RELATED WORK: LLM-based methods have achieved impressive results on single-turn benchmarks such as Spider and BIRD.These systems exploit large language models' reasoning and cross-domain generalization.
  • 7 RELATED WORK: Multi-turn benchmarks address ambiguity and underspecified queries through dialogue turns, but provide static, noise-free histories shared by all models.This setup omits the failed attempts, clarifications, and interaction variability emphasized by BIRD-INTERACT.

8 FUTURE WORK

BIRD-INTERACT identifies future work on improving user-simulator reliability and examining interaction strategies under and without strict resource constraints.

  • Future work will develop a post-trained, human-aligned local user simulator that improves response reliability while preserving controllability and reducing API cost.
  • The authors plan to compare stress-mode and free-mode a-Interact to study natural exploration strategies and efficiency-effectiveness trade-offs.

9 CONCLUSION

BIRD-INTERACT is introduced as a benchmark for evaluating interactive text-to-SQL systems through dynamic, multi-turn interactions. It combines a function-driven user simulator, conversational and autonomous evaluation modes, and 900 challenging tasks targeting ambiguity and state maintenance.

  • BIRD-INTERACT evaluates interactive text-to-SQL systems through dynamic, multi-turn interactions designed to reflect real-world usage.
  • The benchmark combines a function-driven user simulator with conversational and autonomous planning settings.
  • The suite contains 900 challenging tasks that test handling of ambiguities and state across turns.

ETHICS STATEMENT

The research reports compliance with the ICLR Code of Ethics and addresses privacy, fairness, and responsible dataset and code release. The work involved no crowdsourcing or human-subject research, with annotation and task creation performed by the authors.

  • The research states that it complies with the ICLR Code of Ethics, including considerations of privacy, fairness, and responsible use.
  • The study involved no crowdsourcing or human-subject research; the authors conducted all annotation and task creation.

REPRODUCIBILITY STATEMENT

The paper documents experimental configurations, containerized database execution, annotation procedures, and benchmark design principles to support reproducible evaluation. Its scope is centered on text-to-SQL, although the interaction framework is proposed as potentially adaptable to other generative domains.

  • Reproducibility Statement: Experiments document model parameters, temperature, and budget configurations, and execute each task in a freshly re-initialized PostgreSQL 14 Docker instance.
  • Reproducibility Statement: The framework is developed for text-to-SQL but is described as potentially adaptable to Python code synthesis and API call generation.
  • Reproducibility Statement: The benchmark uses a multi-stage annotator process with tutorials, qualification assignments, and validation of solution SQL against annotated test scripts.
  • Reproducibility Statement: Approximately 90% of annotator candidates passed qualification, supporting the reported quality-control process for benchmark construction.
  • Reproducibility Statement: BIRD-INTERACT injects ambiguity and follow-up tasks to make interaction necessary, while reference-based disambiguation supports reliable simulator behavior.

E.1 TASK COMPARISON

BIRD-INTERACT unifies ambiguity handling, dynamic multi-turn interaction, and environment exploration while evaluating SQL with executable tests across analytical and state-changing tasks.

  • E.1 TASK COMPARISON: The benchmark extends prior work beyond static conversation histories by integrating ambiguity handling, dynamic environments, and active clarification.Prior benchmarks often provide clean transcripts rather than failed attempts, digressions, and clarifications.
  • E.1 TASK COMPARISON: BIRD-INTERACT jointly evaluates SQL generation, ambiguity resolution, and dynamic interaction with users and database environments.Its design requires systems to clarify intent and strategically alternate between user clarification and environment exploration.
  • E.1 TASK COMPARISON: Success Rate counts a sub-task as correct only when its predicted SQL passes all executable test cases.Results are reported separately for the ambiguous priority sub-task and the follow-up sub-task.
  • E.1 TASK COMPARISON: Normalized Reward captures sub-task importance and distinguishes first-attempt solutions from solutions obtained after debugging.In c-Interact, the reward assigns 0.7 for solving the first sub-task without debugging, 0.5 with debugging, 0.3 for the second without debugging, and 0.2 with debugging.
  • E.1 TASK COMPARISON: The benchmark tests both analytical and state-changing SQL using execution-result comparisons or task-specific postcondition checks.BI tasks use normalized soft exact matching, while DM tasks may inspect created artifacts, return values, row counts, or invariants.
  • E.1 TASK COMPARISON: BIRD-INTERACT introduces ambiguity into originally unambiguous queries so active clarification is required for successful reconstruction.Annotation distinguishes intent-level ambiguity, such as vague or underspecified requests, from other ambiguity types.

J.3 ACTION SELECTION PATTERNS AND THEIR IMPACT (FULL SET)

Action-selection analysis shows that successful agentic systems balance environment exploration with user-facing actions, then shift from exploration toward execution and submission.

  • J.3 ACTION SELECTION PATTERNS AND THEIR IMPACT (FULL SET): GPT-5 and Claude-Sonnet-4 achieve the strongest reported performance with relatively balanced action strategies.GPT-5 allocates 47% of its budget to environment probes and 52% to asking or submitting, while Claude-Sonnet-4 emphasizes execution at 29.9%.
  • J.3 ACTION SELECTION PATTERNS AND THEIR IMPACT (FULL SET): Submit-call proportion correlates positively with P1, whereas execute-call proportion correlates negatively across systems.The reported correlations are Pearson r≈0.41 and Spearman ρ≈0.54 for submit, versus Pearson r≈−0.52 and Spearman ρ≈−0.54 for execute.
  • J.3 ACTION SELECTION PATTERNS AND THEIR IMPACT (FULL SET): Stronger systems explore and clarify early, then increase execution and submission in later turns to test and refine SQL.Weaker systems either submit too early or overuse execution without consolidation.
  • J.3 ACTION SELECTION PATTERNS AND THEIR IMPACT (FULL SET): Overall performance depends more on balancing exploration, user interaction, and timely submission than on sheer interaction frequency.The analysis specifically identifies wasted budget from excessive execution or poorly timed submissions.
  • J.3 ACTION SELECTION PATTERNS AND THEIR IMPACT (FULL SET): More than 80% of sampled failures stem from incomplete ambiguity resolution, with systems asking about once per task despite roughly four ambiguities per task.Remaining errors include SQL syntax mistakes, incorrect column selection, and misunderstanding database constraints.
  • J.3 ACTION SELECTION PATTERNS AND THEIR IMPACT (FULL SET): The AST-based LOC() mechanism retrieves SQL fragments relevant to clarification questions without exposing the entire ground-truth query.It parses the query, matches relevant AST nodes semantically, and extracts contextual snippets.

O.3 RESULTS AND ANALYSIS

The function-driven user simulator substantially improves reliability, especially on unanswerable questions, while high-quality interactions follow a funnel from ambiguity clarification to executable analysis plans. Human evaluation further supports the benchmark’s dataset quality.

  • User Simulator Reliability: 2.7% failure rate is achieved by the function-driven simulator on unanswerable questions, versus baseline failures reaching 67.4%.The approach classifies request intent and invokes UNA() to reject inappropriate questions rather than leaking ground-truth details.
  • User Simulator Reliability: 87.3%–97.3% UNA accuracy with the function-driven approach contrasts with baseline accuracy as low as 32.6%.The results indicate that explicit structural constraints are needed to keep simulator feedback predictable and prevent inappropriate answers.
  • Interaction Analysis: The funnel effect narrows broad intent through progressively deeper inquiries until the dialogue converges on a clear, executable analysis plan.Its phases cover concept clarification and scoping, computational and implementation details, and final formatting and confirmation.
  • Interaction Analysis: Effective interactions clarify ambiguous concepts, confirm calculation details, determine presentation rules, and re-check formulas before producing results.The Figure 18 case study illustrates this sequence for ambiguous terms, calculations, sorting, and output formatting.
Loading 2510.05318v3…