PaperlayerView exact revision on arXiv

Source-linked AI summary

ARCS: Towards Precise Text-to-SQL via Structured Disambiguation

Yihao Hu, Yanlin Feng, Naoki Otani, Nikita Bhutani

arXiv:2610.09396v1cs.CL

TL;DR

Ambiguity in natural-language questions remains a challenge for text-to-SQL, and can make outputs diverge from user intent. The paper proposes structured, constrained interaction and introduces ARCS, a benchmark of naturally occurring ambiguity over real-world databases. On ARCS, gpt-6-sol achieves 51% end-to-end execution accuracy, while no open-source model exceeds 27%.

  • Problem

    Many text-to-SQL benchmarks use unambiguous, over-specified questions, while ambiguity in practical questions can be subtle and domain- or data-specific.

    Source: §1 · p. 1

  • Method

    The paper proposes structured disambiguation, in which users resolve ambiguity through constrained choices, and constructs ARCS with complete annotations for naturally occurring ambiguities over real-world databases.

    Source: §1 · p. 2

  • Results

    51% end-to-end execution accuracy for gpt-6-sol, while no open-source LLM exceeds 27%, on ambiguous ARCS questions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present ARCS as a first step toward transparent, efficient disambiguation and hope it will catalyze research on ambiguity-aware text-to-SQL.

    Source: §6 · p. 11

    Error analysis finds models sometimes invent ambiguity, overlook database context, or fragment semantically identical interpretations.

    Source: §5.5 · p. 10

Abstract

from arXiv · show

As text-to-SQL systems move beyond demonstrations toward real-world deployment, ambiguity in user questions becomes a primary source of errors. Such ambiguities are often subtle, domain- or data-specific, and can silently cause system outputs to deviate from the user's true intent. Ambiguity is traditionally addressed through conversational clarification, which is often inefficient, cognitively demanding, and poorly aligned with real-world user workflows. We propose structured disambiguation, a new paradigm in which ambiguity is resolved through explicit, constrained interactions rather than free-form dialogue. We construct ARCS (Ambiguity Resolution Corpus for SQL), the first text-to-SQL benchmark featuring naturally occurring, unconstrained ambiguities over real-world databases, with complete annotations of all valid ambiguity points, interpretations, and SQL queries. Experimental results show that text-to-SQL remains challenging in the presence of ambiguity: gpt-6-sol achieves only 51% end-to-end execution accuracy, and no open-source model exceeds 27%.

1 Introduction

Ambiguity in natural-language questions can make text-to-SQL outputs diverge from user intent, while conventional clarification asks users to articulate constraints. The paper proposes structured disambiguation and introduces ARCS to study text-to-SQL under naturally occurring ambiguity.

1 Introduction

  • Common text-to-SQL benchmarks use unambiguous, lengthy, often over-specified questions, unlike practical questions whose ambiguity can be subtle and domain- or data-specific.The paper notes that such ambiguity can affect business decisions, such as budgets based on retrieved revenue.Source: §1 · p. 1
  • Structured disambiguation lets users select interpretations or adjust numeric thresholds, then immediately inspect and compare SQL results and revise their choices.The authors describe this approach as reducing user effort and enabling more interactive disambiguation.Source: §1 · pp. 1–2
  • 51% end-to-end execution accuracy for gpt-6-sol, while no open-source LLM exceeds 27%, on ambiguous ARCS questions.The experiments also find that frontier LLMs with similar text-to-SQL capabilities differ in disambiguation performance.Source: §1 · p. 2
  • ARCS contains 311 realistic disambiguation tasks built from real-world questions with naturally occurring, interacting ambiguity phrases and complete annotations.Its annotations cover valid ambiguity points, interpretations, and corresponding SQL queries, supporting evaluation without fixing system architecture or disambiguation strategy.Source: §1 · p. 2
  • The paper contributes a taxonomy for systematically categorizing ambiguity in text-to-SQL.Source: §1 · p. 2

2 Structured Disambiguation for Text-to-SQL

The framework turns database-contextualized ambiguity into separate resolution tasks, distinguishing finite interpretation choices from unbounded values. Users resolve those points through a panel, after which the system generates SQL for their selected resolutions.

2 Structured Disambiguation for Text-to-SQL

  • A database schema can reduce open-ended interpretation possibilities to a finite set, turning clarification from recall into recognition.The paper contrasts this with prior enumeration that lists full-sentence interpretations, which grows exponentially with multiple ambiguity points and cannot handle unbounded vagueness.Source: §2 · p. 3
  • The framework decomposes ambiguity into separate points, grouping them as finite interpretation sets or unbounded interpretation spaces represented by parameter variables.For each finite point, it identifies valid interpretations; for each unbounded point, it introduces a parameter variable.Source: §2 · p. 3
  • Users resolve finite ambiguity points with multiple-choice selections and unbounded points with value sliders and operators, and can reject an identified ambiguity.Source: §2 · p. 3
  • The selected interpretations produce a parameterized SQL query; user-provided values and operators instantiate its variables, and SQL can be regenerated after a different selection.The authors note that queries for all interpretation combinations can be pre-generated in parallel to reduce latency.Source: §2 · p. 3
  • The disambiguation and SQL-generation components are implementation-agnostic, allowing any text-to-SQL system to serve as the underlying SQL-generation module.Source: §2 · p. 3

3 Categorizing Ambiguity in Text-to-SQL

ARCS organizes text-to-SQL ambiguity along two orthogonal, exhaustive dimensions: its linguistic source and how it maps to database elements. This taxonomy distinguishes ambiguity types and helps diagnose whether errors arise in language understanding or database grounding.

3 Categorizing Ambiguity in Text-to-SQL

  • The taxonomy separates linguistic source from database mapping as two orthogonal, exhaustive dimensions.This separation supports diagnosing language-understanding versus database-grounding failures.Source: §3 · pp. 4–5
  • Semantic ambiguity concerns multiple meanings of words or phrases, while syntactic ambiguity concerns multiple sentence structures.Examples include polysemy or vague predicates and modifier attachment, respectively.Source: §3 · p. 4
  • Database ambiguity includes phrases mapping to multiple columns, tables, values, or computation strategies.These categories cover similar-semantics columns, alternative tables or joins, possible column values, and alternative aggregations or expressions.Source: §3 · p. 4
  • Treating linguistic and database dimensions separately helps distinguish parsing errors from schema-alignment errors for targeted reliability improvements.Prior work focused on one dimension in isolation or conflated them.Source: §3 · pp. 4–5

4 Benchmark Construction

ARCS is a benchmark for text-to-SQL under ambiguity, with annotations of ambiguity points, valid interpretations, and corresponding SQL queries. It supports both end-to-end interaction and fine-grained evaluation, using realistic databases and human-reviewed questions.

4 Benchmark Construction

  • ARCS covers the ambiguity taxonomy and annotates ambiguity points, valid interpretations, and corresponding SQL queries.It supports complementary end-to-end and fine-grained evaluation settings.Source: §4 · p. 5

4.5.1 End-to-End Evaluation

  • End-to-end evaluation allows systems to interact with the database and user simulator in any order before producing SQL.Execution accuracy allows additional predicted columns when they cover valid interpretations; such a prediction receives EXe2e = 1 and user effort of 0.Source: §4 · p. 5, §4.5.1 · p. 7

4.5.2 Fine-Grained Evaluation

  • Fine-grained evaluation separates identifying all ambiguity points and interpretations from SQL generation given ground-truth disambiguation.Full Recall checks whether all gold points and interpretations are found, Perfect checks whether exactly all and only gold items are found, and EXdisambiguated measures SQL execution accuracy given gold disambiguation.Source: §4 · p. 5, §4.5.2 · p. 7

4.1 Database Construction

  • The benchmark uses six existing databases selected for realism, real-world interest, ambiguity suitability, and coverage of all eight ambiguity types.Five databases come from BIRD and one from Spider 2.0; all were converted to SQLite, totaling 8.3 GB.Source: §4.1 · p. 5

4.2 Ambiguous Question Generation

  • GPT-4.1 generates candidate questions conditioned on database schemas, the taxonomy, and human-curated examples, with freedom to produce novel ambiguities.Generation is repeated at temperature 0.7 until the target is reached, producing 180 questions per database.Source: §4.2 · pp. 5–6

4.3 Ambiguity and SQL Annotation

  • Annotators select realistic, genuinely ambiguous questions, identify ambiguity points, enumerate finite interpretations, and produce parameterized SQL for interpretation combinations.They may revise questions and annotations; global conventions such as tie handling and rounding are specified in prompts rather than treated as ambiguities, and annotation averages 2.5 hours per question.Source: §4.3 · p. 6

4.4 User Simulator

  • The user simulator supports free-text, multiple-choice, and value-selection interactions, and responds using ground-truth resolutions for ambiguity points.A two-stage design first identifies relevant ambiguity points to mitigate information leakage; sampling yields 331 task instances from 101 questions and 413 unique SQL queries.Source: §4.4 · p. 6

4.5 Evaluation Metrics

  • Evaluation metrics are computed per task instance and averaged across the benchmark.ARCS supports end-to-end and fine-grained evaluation settings.Source: §4.5 · p. 7

5 Experiments

Across 23 LLMs, text-to-SQL remains difficult on ARCS, with disambiguation a primary bottleneck. Structured disambiguation matches conversational performance with less user effort, while ARCS exposes challenges that taxonomy input addresses only marginally.

5.2 Main Results

  • 57.88% end-to-end performance for gpt-5 leads, followed by gpt-6-sol at 51.77% and gpt-4.1 at 29.90%; gptoss-120b leads open models at 26.05%.The evaluation covers 23 LLMs, including 8 open-source and 15 proprietary models.Source: §5.2 · p. 8, §5.1 · p. 8
  • Across models, disambiguation scores trail SQL generation; all except gpt-5 and gpt-6-sol identify all ambiguity points and interpretations on at most 30% of tasks, and none exceed 14% Perfect Disambiguation.The paper identifies disambiguation as ARCS’s primary performance bottleneck.Source: §5.2 · pp. 8–9

5.4 Comparison with Previous Benchmarks

  • With taxonomy input, o4-mini’s end-to-end execution accuracy rises from 52% to 80% on Ambrosia but improves only marginally on ARCS, which the authors describe as harder and more diverse.Table 2 compares end-to-end EX on ARCS and Ambrosia, with and without taxonomy input; Ambrosia uses a 400-sample subset.Source: Table 2 · p. 9, §5.4 · p. 10

5.2 Main Results

  • For gpt-5, claude-opus-4.5, and gemini-3-pro, ground-truth-disambiguation execution accuracy ranges from 65.27% to 68.81%, while Full Recall spans 22.51% to 61.09%.The authors suggest disambiguation may be underrepresented in current LLM training objectives.Source: §5.2 · p. 9
  • For gpt-5, end-to-end execution accuracy rises from 37.62% at minimal reasoning effort to 57.88% at medium, with no higher-setting gains; o4-mini rises from 36.01% at low to 44.05% at high.Scaling appears in both disambiguation and SQL generation, more strongly for disambiguation.Source: §5.2 · p. 9

5.3 Comparison of Disambiguation Methods

  • Structured disambiguation matches conversational performance at lower user effort; at comparable effort, conversation performs worse, while study participants report less demand, effort, frustration, and greater transparency.Figure 5 compares overall results and end-to-end EX across ambiguity counts.Source: §5.3 · pp. 9–10, Figure 5 · p. 9

5.5 Error Analysis

  • Across models, common errors are executable SQL that matches no ground-truth query or matches one other than the user-intended query; observed failures also include spurious, open-ended, and fragmented ambiguity predictions.qwen3-coder-480b produces invalid outputs on 18% of tasks, while all three examined models produce executable SQL for most inputs.Source: §5.5 · p. 10

5.6 Analysis on Ambiguity Types

  • All three models perform poorly at identifying computation ambiguities; qwen3-coder-480b and gpt-4.1 also detect syntactic ambiguities less well than semantic ones.Figure 7 reports ambiguity-point recall by ambiguity type.Source: §5.6 · p. 10, Figure 7 · p. 11

6 Conclusion

The paper introduces structured disambiguation and ARCS as steps toward addressing naturally occurring ambiguity in text-to-SQL. Its experiments show that ambiguity remains challenging for text-to-SQL systems.

6 Conclusion

  • Structured disambiguation resolves ambiguity through explicit, constrained interactions, and the paper presents it as a new interaction paradigm.The conclusion frames it as a first step toward transparent and efficient disambiguation with minimal user effort.Source: §6 · p. 11
  • ARCS is presented as the first text-to-SQL benchmark with naturally occurring, unconstrained ambiguity over real-world databases.The conclusion says ARCS includes naturally occurring ambiguity in real-world databases.Source: §6 · p. 11
  • The experiments show that text-to-SQL remains highly challenging when questions are ambiguous, and the authors hope ARCS will catalyze further research and realistic interaction paradigms.The stated aim is to better align interaction paradigms with realistic user workflows.Source: §6 · p. 11

A.1 Text-to-SQL Benchmarks with Ambiguous Questions

Prior ambiguous text-to-SQL benchmarks limit ambiguity scope, database realism, or user-in-the-loop evaluation. The paper contrasts free-form clarification with structured feedback and describes a multi-round annotation process.

A.1 Text-to-SQL Benchmarks with Ambiguous Questions

  • Prior ambiguity benchmarks limit scope through single ambiguity points, predefined templates, synthesized or modified databases, or evaluations without a user simulator [23].Mainstream benchmarks also assume one unambiguous intent per question [14] [4].Source: §A.1 · p. 16

A.2 Disambiguation Methods

  • Structured feedback presents predefined options instead of user-authored text, simplifying response processing and reducing user effort.The surveyed text-to-SQL approaches include candidate SQL generation, iterative interpretation discovery, and user-in-the-loop clarification [2] [24] [7].Source: §A.2 · p. 16

B.1 Annotation Details

  • Annotation guidelines excluded overly vague phrases, unrealistic interpretations, unanswerable questions, and time-sensitive expressions such as “last year.”A fixed date threshold was allowed for terms such as “young.”Source: §B.1 · p. 17
  • During annotation, over 60% of questions were found to contain errors; after the second round, final verification identified errors in fewer than 2%, and annotation produced 101 unique questions yielding 311 tasks.Annotation took approximately 2.5 hours per question.Source: §B.1 · p. 18

B.2 Database Statistics

The database statistics compare ARCS with earlier benchmarks, while the question statistics describe ambiguity and query counts across questions. The appendix also points readers to annotation conventions and example database schemas.

B.2 Database Statistics

  • ARCS databases are significantly larger than prior ambiguous text-to-SQL benchmarks [23] and comparable in size to Spider 2.0 [14].Table 3 presents the database statistics and benchmark comparison.Source: Table 3 · p. 18, §B.2 · p. 18

B.3 Question Statistics

  • Question statistics cover domain distribution, ambiguity points per question, and SQL queries per question; ARCS includes all eight ambiguity types.Semantic ambiguities are described as the most common in practice.Source: §B.3 · p. 18, Figure 8 · p. 18

B.4 Global Annotation Conventions

  • The complete global annotation conventions are provided in Table 4.Source: §B.4 · p. 18

B.5 Sample Database Schema

  • Figures 10 and 11 show the schemas of the retail and basketball databases.Source: §B.5 · p. 18

B.6 Sample SQL Query

The appendix identifies a median-length SQL query as a sample and points to the conventions and data-format examples used to describe ARCS tasks.

B.6 Sample SQL Query

  • Table 5 presents a sample SQL query at the median of ARCS query length.Source: §B.6 · p. 18
  • Table 4 contains ARCS’s global annotation conventions.The conventions are provided to systems during disambiguation and SQL generation.Source: Table 4 · p. 19

B.7 Data Format

  • Tables 6 and 7 illustrate the ARCS data format with a sample task entry.Source: §B.7 · p. 19

C.1 Error Analysis

The appendix presents an error-distribution figure and database schemas, then specifies the agent tools and user-simulation procedure used in the evaluation implementations.

C.1 Error Analysis

D.1 Details for Structured Disambiguation

  • Structured disambiguation uses a ReAct agent with schema access, while SQL generation uses a separate ReAct agent with schema, column-description, keyword-search, and query-execution tools.Source: §D.1 · p. 25

D.2 Details for Conversational Disambiguation

  • Conversational disambiguation uses one ReAct agent to ask natural-language clarification questions and access the SQL-generation tools; a patience limit removes the ask-user tool after its quota is exhausted.Source: §D.2 · p. 25

D.3 Details for Unstructured Interpretation Generation [24]

  • Unstructured interpretation generation outputs standalone full-question paraphrases rather than separate ambiguity points; an added agent identifies numeric-threshold ambiguities to adapt the method for vagueness.Source: §D.3 · p. 25

D.4 Details for User Simulator

  • The user simulator first identifies relevant annotated ambiguity points, then generates a response conditioned on those points to avoid information leakage.Source: §D.4 · p. 25

D.4.1 Details for Evaluation Metrics

The appendix defines two strict evaluation metrics for disambiguation and describes a small within-subjects study comparing conversational and structured systems. The study used balanced question sets and randomized system order across eight participants.

D.4.1 Details for Evaluation Metrics

  • Full Recall requires perfect recall at both ambiguity-point and interpretation levels, while Perfect requires perfect F1 at both levels.The metrics align predicted ambiguity points to ground truth, then align interpretations within matched points.Source: §D.4.1 · p. 25
  • The structured system uses separate disambiguation and SQL-generation agents, while the conversational agent has ask-user and query-execution tools.The structured disambiguation agent has schema and final-result tools; its SQL-generation agent also has column-description, keyword-search, query-run, and finish tools.Source: Table 8 · p. 26, Table 9 · p. 27, Table 10 · p. 27
  • Full Recall = 1 with EXdisambiguated = 1 typically implies EXe2e = 1, assuming the user simulator behaves as expected.Perfect is a stricter version of Full Recall that captures precise disambiguation.Source: §D.4.1 · p. 26

E User Study

  • The user study compared both systems with eight non-author participants who regularly analyze tabular data and use LLM-based tools.Source: E User Study · p. 29

E.1 Study Design

  • Participants used both systems in a randomized, balanced within-subjects design, answering three questions per system after practice.The first two questions per system came from the benchmark with comparable ambiguity complexity; participants created the third.Source: §E.1 · p. 29

E.2 Questionnaire and Results

Structured disambiguation scored better on all six questionnaire measures, with the largest reductions in Effort and Frustration. Participants also valued the interpretation panel and preview tables for helping them choose among ambiguity options.

E.2 Questionnaire and Results

  • The questionnaire rated four NASA-TLX-adapted workload measures plus Result Confidence and System Transparency on a 1–7 scale.Source: §E.2 · p. 29, Table 13 · p. 29
  • Effort and Frustration each decreased by 0.88 points, Mental Demand decreased by 0.75 points, and System Transparency increased by 0.62 points.Result Confidence rose from 5.25 to 5.38; all six measures favored structured disambiguation, with workload scores lower and confidence and transparency scores higher being better.Source: §E.2 · p. 29, Table 13 · p. 29
  • Participants identified the interpretation panel and preview tables as useful for reducing mental workload and choosing among ambiguity options.Source: §E.2 · p. 29

Related papers

Loading 2610.09396v1…