PaperlayerView exact revision on arXiv

Source-linked AI summary

STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases

Mounica Maddela, Lingjue Xie, Daniel Preotiuc-Pietro, Mausam

arXiv:2509.19508v1cs.DBcs.CL

TL;DR

Existing Text2SQL benchmarks do not explicitly target complex analytical reasoning questions, despite analysts’ need to query large specialized databases. STARQA provides a human-created benchmark, and the paper evaluates an approach combining SQL and Python. The hybrid approaches outperform Text2SQL, although the dataset remains challenging for tested models.

  • Problem

    Existing Text2SQL benchmarks do not focus explicitly on complex analytical reasoning, including aggregate calculations, time-series analysis, and scenario understanding.

    Source: §1 · p. 1

  • Method

    The paper introduces STARQA and evaluates SQL-only generation alongside TEXT2SQLCODE, which uses SQL for data fetching and Python for complex processing.

    Source: §1 · p. 1

  • Results

    45.4% and 44.5% are the respective median performances of HYBRIDsingle and HYBRIDmulti, significantly outperforming Text2SQL, whose median execution accuracy across six LLMs is 36.5%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that combining SQL and Python generally performs better, especially when the combined approach is invoked only when necessary.

    Source: §7 · p. 9

    The study evaluates three databases and English queries, so it does not test generalizability to other languages or multilingual models.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

Semantic parsing methods for converting text to SQL queries enable question answering over structured data and can greatly benefit analysts who routinely perform complex analytics on vast data stored in specialized relational databases. Although several benchmarks measure the abilities of text to SQL, the complexity of their questions is inherently limited by the level of expressiveness in query languages and none focus explicitly on questions involving complex analytical reasoning which require operations such as calculations over aggregate analytics, time series analysis or scenario understanding. In this paper, we introduce STARQA, the first public human-created dataset of complex analytical reasoning questions and answers on three specialized-domain databases. In addition to generating SQL directly using LLMs, we evaluate a novel approach (Text2SQLCode) that decomposes the task into a combination of SQL and Python: SQL is responsible for data fetching, and Python more naturally performs reasoning. Our results demonstrate that identifying and combining the abilities of SQL and Python is beneficial compared to using SQL alone, yet the dataset still remains quite challenging for the existing state-of-the-art LLMs.

1 Introduction

STARQA targets complex analytical reasoning questions that existing Text2SQL benchmarks do not explicitly address. The paper introduces a human-created dataset and evaluates a SQL–Python approach alongside SQL-only systems.

Motivation

  • Analysts need quantitative insights from large, often proprietary datasets, while most surveyed Text2SQL benchmarks have few select clauses and mathematical or aggregation functions per query.The cited survey covers 36 benchmark datasets; almost all have fewer than two select clauses and fewer than one mathematical, scalar, or aggregation function per query.Source: §1 · p. 1

Dataset contribution

  • STARQA contains 362 realistic complex analytics and reasoning questions and answers over public databases in movies, sports, and e-commerce.Questions require one or more kinds of reasoning, including math operations, time-series analysis, nested queries, aggregate calculations, temporal reasoning, and scenario understanding.Source: §1 · p. 1

Approach

  • TEXT2SQLCODE combines SQL data operations with Python for analytics, data wrangling, and computations that are burdensome in SQL.Because the multi-step approach can compound errors, hybrid variants invoke it only for questions automatically identified as difficult from Text2SQL outputs.Source: §1 · p. 2

Results

  • 45.4% and 44.5% are the respective median performances of HYBRIDsingle and HYBRIDmulti, which significantly outperform Text2SQL on STARQA.Across six LLMs, standalone Text2SQL has a median execution accuracy of 36.5%; the authors also report that most questions cannot be solved by existing LLMs with web search.Source: §1 · p. 2

2 Related Work

Prior Text2SQL benchmarks and methods address domain, schema, and generation challenges, but the paper identifies complex analytical reasoning as underexplored. STARQA and its SQL–Python approach target that gap.

Reasoning benchmarks

  • A survey of 36 Text2SQL benchmarks identifies reasoning as underexplored, motivating STARQA’s focus on complex analytical questions.The related-work discussion contrasts this gap with benchmarks addressing domains, SQL dialects, schemas, entity linking, and external knowledge.Source: §2 · p. 2

Related datasets

  • Archer’s arithmetic, commonsense, and counterfactual questions are solvable in basic SQL, while SPIDER 2.0 targets enterprise workflows rather than annotated analytical reasoning.The paper says these differences limit Archer’s complexity and make SPIDER 2.0 less suited to evaluating analytical reasoning abilities.Source: §2 · p. 2

Text2SQL methods

  • Existing Text2SQL systems commonly use schema and entity linkers, retrieved exemplars, and prompting or repair strategies to improve SQL generation.The related-work discussion also describes semantic parsing across representations such as SQL, SPARQL, and Cypher.Source: §2 · p. 2

Combining languages

  • Prior SQL–Python combinations use artificial prompts or limited post-SQL processing, while Text2Python alone was reported to perform worse because of generation length.TPTU applies simple mathematical operations to SQL statistics; table-augmented generation succeeds only with short SQL outputs and does not study combinations with other languages.Source: §2 · pp. 2–3

3 The STARQA Dataset

STARQA is a manually created dataset for complex analytical reasoning over SQL databases, with questions, answers, reasoning categories, and reference code. It covers three specialized-domain databases and is designed primarily for development and evaluation.

3 The STARQA Dataset

  • The dataset packages questions, answers, reasoning categories, and reference code, which is released to verify gold answers but is not used in experiments.Questions are intended to cover one or more categories of complex analytical reasoning.Source: §3 · p. 3

3.1 Databases

  • The authors use IMDb, European soccer statistics, and Brazilian e-commerce data, limiting database count to prioritize diverse questions with depth in reasoning and analytics.They expect STARQA to be useful primarily for development and evaluation, rather than training.Source: §3.1 · p. 3

3.2 Dataset Creation

  • Authors with programming, SQL, and Python experience manually wrote and reviewed questions for correctness, credibility, specificity, and clarity.Entity mentions were normalized to reduce entity linking and schema adaptation as factors orthogonal to the dataset’s goal; creator-written SQL and Python code supplied the answers.Source: §3.2 · pp. 3–4

3.3 Dataset Statistics

  • STARQA contains 362 questions: 100 on IMDb, 162 on ES, and 100 on Olist; 299 answers are tuples, averaging 2.35 tuples per answer.The dataset also has 55 single-number answers and 8 null answers; tuples average 2.43 constituents and reach up to 8.Source: §3.3 · p. 4

4 The TEXT2SQLCODE Approach

TEXT2SQLCODE divides analytical question answering between SQL data fetching and Python computation. Its hybrid variants use Text2SQL self-consistency to decide when to invoke the combined approach.

4 The TEXT2SQLCODE Approach

  • SQL fetches relational data, while Python handles complex processing when SQL’s expressiveness is insufficient for the required analytical reasoning.The split may leave simple processing in SQL and move complex processing to Python.Source: §4 · p. 4
  • TEXT2SQLCODEmulti uses separate LLM calls for decomposition, SQL generation, and Python generation, whereas TEXT2SQLCODEsingle uses one prompt to produce a Python function containing the SQL query.The workflow uses three steps: a decomposer, a Text2SQL model, and a Text2Python model.Source: §4 · pp. 4–5
  • In TEXT2SQLCODEmulti, generated SQL queries fetch data into dataframes, which are then used for Python processing of complex analytical reasoning.SQL steps are generated for each decomposed Text2SQL step; the Python stage receives the original question, decomposition, and dataframe shapes.Source: §4 · p. 5
  • The framework could add retrieval for databases whose schemas exceed the prompt or explicit entity linking, but the experiments omit both for simplicity.The authors state these additions were not required for STARQA.Source: §4 · p. 5

4.1 A Text2SQL-Text2SQLCode Hybrid

  • HYBRIDmulti and HYBRIDsingle invoke TEXT2SQLCODE when three Text2SQL runs disagree; if two runs match, their answer is used instead.The authors use self-consistency as a proxy for question difficulty because early experiments indicated LLMs could not reliably assess when to use decomposition and Python.Source: §4.1 · p. 5

5 Experimental Setup

The experiments compare standalone Text2SQL and self-consistency baselines across current open-weight and commercial LLMs, using execution accuracy to assess generated answers. Additional comparisons examine parametric knowledge, web search, and generalization to Archer.

5.1 System Details

  • The study tests standalone Text2SQL and self-consistency baselines with K = 3 or 5 across open- and closed-source models, including reasoning models.The evaluated models include Claude 3.7, GPT 4.1, GPT o1, GPT o3-mini, DeepSeekV3.1, and Qwen2.5.Source: §5.1 · pp. 5–6
  • Prompts provide database schemas, sample data, and categorical values, and require tuple-list outputs to support automatic evaluation.The prompts also describe the task and instruct models not to add names, descriptions, or other information to their outputs.Source: §5.1 · p. 6

5.2 Evaluation Metrics

  • Execution accuracy compares generated-code outputs with reference answers and is insensitive to tuple order and tuple construction order.The authors report that models widely respect the requested list-of-tuples format and consider execution accuracy an accurate performance measure.Source: §5.2 · p. 6

5.3 Additional Comparisons

  • The additional comparisons test whether LLMs can answer without SQL or code using parametric knowledge, and assess a web-search agent as a non-comparable approach with access to extra information.Gemini Pro 2.5 outputs were manually processed when needed to obtain performance scores.Source: §5.3 · p. 6
  • The study also evaluates Text2SQLCode on the English validation set of Archer, comprising 104 questions from two databases.Source: §5.3 · p. 6

6 Results

TEXT2SQLCODE improves on Text2SQL alone across tested models, but STARQA performance remains moderate and varies by model and domain. Analyses show that selective Python routing and the division of SQL and Python work matter, while multi-step pipelines still make logical and decomposition errors.

6 Results

  • 48.3% is the best overall STARQA result; model gaps are within 4%, and tested reasoning models do not substantially outperform non-reasoning models overall.Different models lead on IMDb, ES, and OL, with no clear pattern of a superior model.Source: §6 · pp. 6–7
  • TEXT2SQLCODE methods outperform Text2SQL alone for every tested model, with accuracy gains reaching 14.5% on STARQA for DeepSeek and 13.1% on Archer for GPT-4.1.The authors report consistent improvements on both datasets, supporting generalizability beyond STARQA.Source: §6 · p. 7
  • Three-run Text2SQL self-consistency improves performance by 2.1% on average, but five runs degrade performance for GPT-o1 and GPT-4.1 on STARQA.The improvements require three or five LLM calls, and performance is often comparable to TEXT2SQLCODE at a similar call count.Source: §6 · p. 7
  • TEXT2SQLCODE variants lead on different domains, and the authors conclude that decomposition should be deployed selectively across questions.The post-hoc Oracle, which takes the best output of Text2SQL and TEXT2SQLCODE, exceeds either system by 3%–9%.Source: §6 · p. 7
  • Without structured data, accuracy is below 10% except GPT o1 at 11.6%; web search reaches 65.4% on EuroSoccer but performs very poorly on IMDb and Olist.The authors attribute EuroSoccer's stronger web-search result to readily available records and summary statistics, while IMDb and Olist require searching a much vaster space.Source: §6 · p. 8

6.1 Analysis on Use of Python Code

  • Non-reasoning models route at least 80% of questions to Python, while TEXT2SQLCODE routing changes by under 9% between SQL-correct and SQL-incorrect subsets; HYBRID shows up to 39% relative deviation.HYBRID uses Text2SQL consistency as a proxy for difficulty; the authors say this lets it draw on the systems' complementary strengths.Source: §6.1 · pp. 8–9

6.2 Error Analysis

  • SQL-plus-Python approaches excel at string manipulation, conditional aggregation, and large joins paired with dynamic grouping or in-memory calculations.For semi-structured IMDb genres, SQL-only runs with GPT-4.1 failed at runtime, while TEXT2SQLCODE used SQL retrieval and Python splitting and aggregation.Source: §6.2 · p. 9
  • TEXT2SQLCODE errors are mostly logical mistakes or inaccurate decomposition, including missing filters, mishandled nulls, omitted final aggregation, or lost information.A SQL-only GPT-o3-mini run on a complex per-group episode question stalled for over 20 minutes; the decomposed approach returned the correct output in under three minutes.Source: §6.2 · p. 9

7 Conclusions

STARQA is a public human-created benchmark for complex analytical reasoning over structured data, and the best reported performance reaches 48.3%. SQL/Python decomposition generally performs better, especially when used selectively.

7 Conclusions

  • STARQA is a public human-created dataset designed to measure complex analytical reasoning over structured data.Its reasoning categories include statistical analysis, time-series analysis, complex conditional logic, multicriteria filtering, commonsense knowledge, and scenario understanding.Source: §7 · p. 9
  • 48.3% was the best performance reported for current state-of-the-art LLMs on STARQA.Source: §7 · p. 9
  • SQL/Python decomposition generally produces better results than SQL alone, especially when invoked only when necessary.Source: §7 · p. 9
  • Future work could explore task decomposition in agentic workflows and extend STARQA with more reasoning types and domains.The authors identify cross-domain training as a possible study enabled by broader coverage.Source: §7 · p. 9

Limitations

  • The three evaluated databases do not represent the full complexity of all databases, and the study is limited to English queries and datasets.The authors also note scope for prompt optimization and leave generalization to other languages and multilingual models for future work.Source: Limitations · pp. 9–10

A STARQA Categorization Results

The category analysis reports that the proposed approaches generally outperform Text2SQL, though results for categories with few questions are suggestive.

B Additional Error Analysis

The error analysis finds that model reliability and prediction consistency matter for self-consistency and hybrid SQL/Python approaches.

  • SQL error profiles vary: Qwen-2.5 and DeepSeek-v3.1 show diverse failures, while GPT-o3-mini and GPT-o1 produce more uniform, predictable errors.The latter are described as easier to debug and more reliable in execution.Source: B Additional Error Analysis · p. 13
  • Self-consistency helps when predictions are reliable: GPT-o3-mini, GPT-o1, and GPT-4.1 produce stable outputs, while Qwen-2.5 repeatedly produces the same incorrect outputs.Source: B Additional Error Analysis · p. 13
  • Hybrid approaches are not universally better: Qwen-2.5 combines incorrect SQL, poor prediction diversity, and runtime failures, unlike the reliable and diverse outputs of GPT-o3-mini and GPT-o1.The reported hybrid failure chain can include consistently incorrect SQL followed by Python execution failure.Source: B Additional Error Analysis · p. 13

C Example Text2SQL Prompt for Eurosoccer

The example prompt supplies a soccer database schema and guidance, requests valid SQLite SQL, and specifies answer formatting constraints.

D Example Text2SQLCodesingle Prompt for Eurosoccer

This appendix gives a Text2SQLCode prompt template that partitions complex instructions between SQL and Python and specifies how generated code should be returned.

D Example Text2SQLCodesingle Prompt for Eurosoccer

  • The prompt directs Text2SQLCode to split complex instructions between SQL and Python, label each engine’s steps, and keep multiple SQL queries independent.SQL handles data fetching; Python performs work that is burdensome in SQL.Source: D Example Text2SQLCodesingle Prompt… · p. 22
  • A common-payment-method question is handled as a single Text2SQL step because SQL can count each method and select the most frequent.The example requires no Python step.Source: D Example Text2SQLCodesingle Prompt… · p. 22
  • The time-series example fetches AAPL daily prices from January 1 through June 30, 2024 with SQL, then uses Python to find each month’s high and date.The prompt frames this as grouping by month before computing each maximum.Source: D Example Text2SQLCodesingle Prompt… · p. 22
  • The Eurosoccer example asks for the longest undefeated run, in months, for either team in the Old Firm between 2008 and 2016.The requested output is team name and number of months.Source: D Example Text2SQLCodesingle Prompt… · p. 30
  • The code prompt requires a Python function taking a database path, using sqlite3 for SQL execution, and handling missing values in transformations.It specifies a function signature returning a list of tuples.Source: D Example Text2SQLCodesingle Prompt… · p. 30
  • The output instructions require only the requested columns in tuples, no column names or extra text, and no rounding unless requested.The final code must appear in one Python code block without intermediate explanations.Source: D Example Text2SQLCodesingle Prompt… · p. 30

E Example Text2SQLCodemulti Prompt for Eurosoccer

Appendix E illustrates how Text2SQLCode prompts decompose complex requests into SQL data retrieval and Python analysis, with explicit function and output requirements, using a Eurosoccer undefeated-streak question as an example.

E.1 Decomposition

  • The decomposition assigns database retrieval to Text2SQL and complex loops, conditionals, arithmetic, or string manipulation to Python, while simple requests remain SQL-only.Examples contrast counting the most common payment method in SQL with fetching daily AAPL prices in SQL and finding monthly highs in Python.Source: §E.1 · p. 31
  • Every Text2SQL query must be independently executable; a later query cannot depend on results from an earlier one.The invalid Zara example chains a product query into a price query, violating this constraint.Source: §E.1 · p. 31
  • Decompositions must return only requested information and include requested fields in SQL retrieval so Python can produce them.For example, outputting monthly high prices is invalid when the question asks only for their dates.Source: §E.1 · pp. 31, 37
  • For the Old Firm undefeated-month question, the prompt requests short, line-by-line steps with at most two Text2SQL prompts, not direct SQL queries.Steps are labeled Text2SQL or Python, and Text2SQL prompts must contain the information needed to answer the question.Source: §E.1 · p. 37
  • The instructions define season boundaries and require names rather than IDs, exact requested columns, and specified season, date, match, and score formats.Seasons between two years exclude seasons ending in the first year or starting in the second.Source: §E.1 · p. 37

E.2 Python

  • The Python prompt directs the model to produce directly executable code that uses already-fetched data in Pandas dataframes to generate the requested answer.Any explanation or reasoning should appear as Python comments.Source: §E.2 · p. 37
  • The example asks which team, if any, had the longest undefeated-month streak between Celtic and Rangers from 2008 to 2016.The requested result is the team name and the streak length in months.Source: §E.2 · p. 37
  • SQL retrieves match dates, home and away teams, and scores for Celtic–Rangers matches between 2008 and 2016.The prompt requests match information as home-team name versus away-team name, with scores in home-away order.Source: §E.2 · p. 37
  • Python determines which team, if any, was undefeated in each match and tracks each team’s longest streak of undefeated months.It then outputs the team name and number of months for the longest streak.Source: §E.2 · p. 37
  • listOfDFs contains one dataframe corresponding to the Text2SQL query, with columns match_date, match_info, and score; the sample includes a Celtic - Rangers row with score 2 - 4.Source: §E.2 · pp. 37–38
  • The prompt requires a `compute_result` function that accepts `listOfDFs`, returns a list of tuples, uses only standard Python libraries, and handles missing values without crashing.It also specifies that the response contain only the Python function.Source: §E.2 · p. 38
  • The output must contain only the requested columns as tuples, use entity names rather than IDs, and include every matching entity even if the question asks for one.The prompt also specifies date, match-information, and score formats, and prohibits extra output or rounding unless required.Source: §E.2 · p. 38

F Example Text2SQL Prompt for IMDb

This appendix gives an example SQLite Text2SQL prompt over IMDb, including database schema and a query about consistency across TV series seasons.

F Example Text2SQL Prompt for IMDb

  • The supplied IMDb schema includes tables for titles, localized titles, episodes, crew, principals, ratings, and people, with fields such as release year, genres, and average rating.Source: F Example Text2SQL Prompt for IMDb · pp. 39–41
  • The prompt instructs the assistant to write valid SQLite SQL for the provided tables and explicitly notes that missing values are encoded as '\\N'.Source: F Example Text2SQL Prompt for IMDb · p. 41
  • The example asks for the TV series season with the smallest rating range, considering only series with at least 10 seasons and treating each season separately.It requests the title, season number, and difference between the highest- and lowest-rated episodes.Source: F Example Text2SQL Prompt for IMDb · p. 41

G Example Text2SQLCodesingle Prompt for IMDb

This example prompt directs an LLM to split complex database instructions into SQL and Python work, while allowing simple questions to use SQL alone. It illustrates the split with monthly stock-price maxima.

G Example Text2SQLCodesingle Prompt for IMDb

H Example Text2SQLCodemulti Prompt for IMDb

This section illustrates the Text2SQLCode workflow for an IMDb question, from decomposing instructions into SQL retrieval and Python reasoning to specifying the Python output format.

H.1 Decomposition

  • Text2SQLCode assigns data retrieval to SQL and complex loops, conditionals, arithmetic, or string manipulation to Python, while keeping simple instructions as a single SQL step.When multiple SQL queries are needed, they must be independently executable rather than depend on one another.Source: §H.1 · p. 46
  • The decomposition prompt uses a monthly stock-high question to illustrate fetching daily prices with SQL and using Python to find each month’s highest price and its date.The example specifies the January–June 2024 period for AAPL.Source: §H.1 · p. 46
  • A products question illustrates using two independent SQL queries to retrieve all products and products sold above a threshold, then using Python to subtract the sets and count the remainder.The example threshold is total value higher than 5.Source: §H.1 · p. 46
  • The examples reject SQL queries that depend on earlier query outputs and decompositions that return unrequested information, such as monthly highs when only dates were requested.Each SQL query must be independently executable, and the final output should contain only requested information.Source: §H.1 · pp. 46–47
  • The decomposition prompt requires labeled Text2SQL or Python steps, limits the example to at most two Text2SQL steps, and says simple questions should be passed directly to Text2SQL without Python.Multiple Text2SQL steps must be independent and must not use another step’s output.Source: §H.1 · p. 49

H.2 Python

  • The Python stage receives fetched results as Pandas dataframes and is instructed to produce directly executable code that returns the requested answer.The IMDb example specifies a function taking a list of dataframes and returning a list of tuples, using standard Python libraries and handling missing values.Source: §H.2 · pp. 49–50
  • For the IMDb example, SQL retrieves episode ratings, season numbers, and parent-series titles; Python groups by series and season, computes rating ranges, filters for at least 10 seasons, and selects the smallest range.The requested output is the series title, season number, and range.Source: §H.2 · p. 49
  • The Python prompt requires only the requested columns, no extra information or unnecessary rounding, unnamed tuples, and code enclosed in Python fences.Source: §H.2 · p. 50

I Example Text2SQL Prompt for Olist

The example prompt supplies Olist database schemas and asks for SQL to assess the relationship between order product count and review score. It specifies constraints on query syntax and output.

I Example Text2SQL Prompt for Olist

J Example Text2SQLCodesingle Prompt for Olist

The prompt divides complex instructions between SQL data retrieval and Python reasoning, while allowing SQL alone for simpler tasks. It illustrates this split with a time-series example and code-generation requirements.

J Example Text2SQLCodesingle Prompt for Olist

K Example Text2SQLCodemulti Prompt for Olist

This section presents an Olist example of prompts for decomposing user instructions into Text2SQL and Python steps, then generating an answer from fetched data. It also specifies that Python code should return every entity matching the criteria and be enclosed in Python fences.

K.1 Decomposition

  • The decomposition prompt assigns SQL to data retrieval and Python to complex loops, conditionals, arithmetic, or string operations, while keeping simple questions in SQL alone.Each step must be labeled by its executor, and any multiple Text2SQL queries must be independently executable.Source: §K.1 · p. 59
  • For monthly AAPL highs, the example fetches daily prices with SQL and uses Python to find each month’s highest price and its date.The prompt also rejects a version that outputs the highest price when only the date was requested.Source: §K.1 · pp. 59–60
  • The decomposition guidance rejects dependent SQL queries, requiring every Text2SQL query to execute independently.The Zara example is invalid because its second query depends on products identified by the first query.Source: §K.1 · pp. 59–60
  • The Olist example asks whether order item count relates to review score, requesting the Pearson correlation coefficient and p-value rounded to two decimal places.The provided schema describes Olist tables for customers, orders, items, reviews, payments, products, sellers, and geolocation.Source: §K.1 · pp. 60–62

K.2 Python

  • The Python-generation prompt takes the instruction, decomposition, and already-fetched Pandas dataframes as input and requests directly executable code for the final answer.Its example fetches each reviewed order’s item count and review score with Text2SQL, then assigns correlation and p-value calculation to Python.Source: §K.2 · p. 63
  • The Python function must use only standard libraries, handle missing values, and return only the exact requested columns as a list of tuples.It should not add column names or other information, and should round numbers only when the question requires it.Source: §K.2 · p. 63
  • If multiple entities match the question’s criteria, output all of them, even when the question asks for one.Source: §K.2 · p. 63
  • Enclose the Python code in ```python ``` fences.Source: §K.2 · p. 64

Related papers

Loading 2509.19508v1…