Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

• GPT-4-based methods reach 91.2% execution accuracy on Spider 1.0 and 73.0% on BIRD, where text-to-SQL supports analysts’ routine queries, workflows, and business intelligence.The paper frames text-to-SQL as a way to reduce repetitive human labor and programmers’ workload.Source: §1 · p. 1
• Earlier benchmarks use small schemas, simple SQL, and limited questions while overlooking dialects, unlike enterprise workflows spanning database systems, nested schemas, project code, external knowledge, and multistep data engineering.These workflows include data wrangling, transformation, and analytics, motivating a more realistic enterprise-level benchmark.Source: §1 · pp. 1–2
• Spider 2.0 contains 632 real-world wrangling, transformation, and analysis tasks from industrial applications, averaging 812 schema columns and 144 tokens per ground-truth SQL query.Its databases include local systems and cloud warehouses, terabyte-scale data, and tasks grounded in project codebases, documents, and database interfaces.Source: §1 · p. 2
• The agentic benchmark requires navigating project codebases and databases with SQL and scripts to transform data or extract insights, while lite and snow provide preprocessed-schema, SQL-only variants.Spider 2.0-lite spans BigQuery, Snowflake, and SQLite, while Spider 2.0-snow is entirely hosted on Snowflake; these settings omit the codebase and database transformation, and provide less information such as execution feedback.Source: §1 · pp. 2–3
• 21.3% performance is achieved by the best o1-preview-based code-agent framework on Spider 2.0; Spider 2.0-lite’s top parser addresses 5.7% of questions, versus 91.2% execution accuracy on Spider 1.0 and 73.0% on BIRD.Analysis identifies large-database schema linking, dialect handling, nested-query planning, and effective use of documentation and project codebases as major obstacles.Source: §1 · p. 3

2.1 TASK DEFINITION

• Spider 2.0 is an iterative code-agent task: agents modify SQL or Python in a project codebase, execute it through a database interface, and use the final observation as the answer.Spider 2.0-lite and Spider 2.0-snow instead ask a parser to produce SQL from a question, schema, and documentation; Snow is entirely hosted on Snowflake.Source: §2.1 · p. 3

2.2 ANNOTATION PIPELINE

• Annotators curated real databases, complex SQL queries, and DBT projects, then rewrote SQL to prevent leakage and checked that rewrites ran successfully and returned non-empty results.The sources include cloud warehouses and locally hosted databases; the retained collection includes 547 SQL queries and 78 DBT projects.Source: §2.2 · pp. 3–4
• Annotators paired task instructions with project context, built focused execution-based evaluators, and reviewed instructions, gold SQL, and scripts through multiple validation rounds.After a second validator round, 5% of examples still contained errors; all were corrected, and a red-team assessment tested the automatic evaluation.Source: §2.2 · pp. 4–5

2.3 DATASET STATISTICS

• Spider 2.0 tasks span real data-pipeline work and use queries with more columns, tokens, and functions per query than prior SQL benchmarks.The examples cover data wrangling, transformation, and analysis, and were collected from real tutorials and forums.Source: Table 2 · p. 5, §2.3 · p. 5
• Spider 2.0 databases have large, enterprise-like schemas with complex structures such as nested schemas and partitioned tables, dynamic daily updates, and diverse data types, volumes, and scopes.The paper characterizes these databases as more diverse than those in previous datasets.Source: Table 2 · p. 5, Table 3 · p. 5, §2.3 · p. 5
• 85.98% of examples require specialized SQL-dialect functions, averaging 7.1 such functions per ground-truth SQL, across cloud, local, and lightweight database systems.The systems include BigQuery, Snowflake, Postgres, ClickHouse, SQLite, and DuckDB.Source: Table 3 · p. 5, Figure 3 · p. 5, §2.3 · p. 5
• Spider 2.0 tasks provide codebase context and documentation, and some add DBT-based project-level transformation workflows requiring codebase and database navigation, documentation understanding, and multi-step execution.The text-to-SQL variants also require external knowledge and SQL-dialect documentation.Source: §2.3 · pp. 5–6

3.1 EXPERIMENTAL SETUP

• Spider 2.0 uses Success Rate, while Spider 2.0-lite and Spider 2.0-snow use Execution Accuracy; evaluation scripts score each task 0 or 1 and focus on essential answer components.Difficulty is classified by gold-SQL token count: under 80 is Easy, 80–159 Medium, and at least 160 Hard. Experiments include open- and closed-source LLMs, several code-agent frameworks, and prompting and fine-tuned text-to-SQL methods.Source: §3.1 · p. 6

3.2 EVALUATION RESULTS

• 23.77% on Spider 2.0-snow and 23.40% on Spider 2.0-lite are the reported peak performances for o1-preview and o3-mini, which outperform GPT-4o and Claude-3.5-Sonnet across difficulty levels.DeepSeek-V3 achieves 8.78%; Spider-Agent results also indicate substantial room for improvement on enterprise SQL tasks.Source: §3.2 · p. 6, Table 4 · p. 7, Table 6 · p. 7
• 5.68% EX is the highest result for DAIL-SQL + GPT-4o on Spider 2.0-lite, versus 86.6% on Spider 1.0 and 57.4% on BIRD; the best Spider 2.0-snow method reaches 2.20% EX.SFT CodeS-15B scores only 0.73% EX on Spider 2.0-lite despite extensive fine-tuning; the passage attributes snow’s added challenge to SQL dialect differences.Source: Table 5 · p. 7, §3.2 · p. 7
• Spider-Agent frameworks struggle because tasks require codebase and documentation exploration, complex database navigation, intricate SQL generation, execution-feedback debugging, and schema exploration across database types.The paper identifies code grounding as a key demand and notes Snowflake as an example database system.Source: Table 6 · p. 7, §3.2 · p. 7

4.1 ANALYSIS OF DIFFERENT TASK TYPES

• Models often perform poorly on nested columns, where array- and dictionary-formatted data within a column makes schema linking and interpretation of internal field functions difficult.The analysis notes that humans can understand these schemas through multi-step reasoning and iteration.Source: §4.1 · p. 8, Table 7 · p. 8
• 11 of the external-document tasks were answered correctly (11.54%), with failures arising when models could not ground document requirements in SQL and schema information.Models often explored databases and used appropriate problem-solving strategies, but failed to translate complex document requirements into SQL.Source: Table 8 · p. 8, §4.1 · p. 8
• 12.82% of DBT project tasks were solved correctly (10 examples), with challenges spanning multi-query transformations, repository exploration, and simultaneous data exploration and SQL coding.A cited failure involved not exploring the mrr.md file needed for a monthly recurring revenue classification.Source: Table 9 · p. 8, §4.1 · p. 8

4.2 ERROR ANALYSIS OF SQL GENERATION

• 35.5% of sampled errors involved data analysis: dialect-function usage (10.3%), advanced calculation (7.5%), and intricate query planning (17.7%).The analysis examines errors in 300 randomly sampled examples; models struggled with complex functions, statistical calculations, nested queries, CTEs, and set operations.Source: §4.2 · pp. 8–9
• 27.6% of errors involved schema linking, including column linking (16.6%) and table linking (10.1%).Spider 2.0-lite databases average over 755 columns versus approximately 54 in BIRD; models also struggled to use BigQuery TABLE SUFFIX and wildcard features.Source: §4.2 · p. 9
• 8.3% of errors were JOIN errors, as missing explicit foreign keys in BigQuery forces models to infer relationships from column names and descriptions.This inference can lead to invalid JOINs.Source: §4.2 · p. 9

4.3 ANALYSIS OF DIFFERENT EXPERIMENTAL SETTINGS

• Oracle SQL-function documentation yields only slight improvement, while manually selected few-shot examples yield only marginal improvement.The analysis suggests models can select functions and understand basic syntax, but struggle to use functions to express user intent; extensive schema prompts may hinder learning from few-shot examples.Source: Table 10 · p. 9, Table 11 · p. 9, §4.3 · p. 10

Code generation and text-to-SQL benchmark

• Benchmarks span seq2seq code generation, data-science tasks, interactive environments, repository coding, and multimodal data science and engineering (Spider2-V, Cao et al., 2024).Examples include SQL-Spider, MLAgent-Bench (Huang et al., 2023), Intercode (Yang et al., 2024b), and SWE-Bench.Source: §5 · p. 10
• Existing text-to-SQL benchmarks target lightweight local databases and miss industrial schema and data scale and agentic SQL workflows across dialects; Spider 2.0 bridges this gap.Source: §5 · p. 10

Code agent framework and text-to-SQL methods

• Code-generation agents seek to improve language-model reasoning (Yao et al., 2022; Zhang et al., 2022; Chen et al., 2023; Wang et al., 2023b; Shinn et al., 2024; Zhang et al., 2024; Xia et al., 2024), with some standardizing agent actions.Source: §5 · p. 10
• Text-to-SQL fine-tuning and prompting methods perform strongly on previous benchmarks; Spider-Agent targets database tasks, and Spider 2.0-lite adapts several such methods.The cited methods include fine-tuning (Li et al., 2024a) and prompting (Dong et al., 2023; Wang et al., 2023a; Zhang et al., 2023; Talaei et al., 2024; Pourreza & Rafiei, 2024; Gao et al., 2024).Source: §5 · p. 10

6 CONCLUSION

• o1-preview achieves only 21.3% success on Spider 2.0, underscoring the benchmark’s high difficulty.The paper presents Spider 2.0 as a novel challenge for text-to-SQL research.Source: §6 · p. 10
• Spider 2.0 covers diverse database systems and SQL dialects, large complex schemas, and data-engineering tasks in real project scenarios.These scenarios include codebases and documentation.Source: §6 · p. 10
• The benchmark provides a direction toward more realistic and intelligent text-to-SQL solutions.Source: §6 · p. 10

A SPIDER 2.0 EVALUATION SCRIPTS

• Spider 2.0-lite uses execution accuracy, checking that every gold-result column appears in the prediction, while Spider 2.0 uses success rate judged by human-written, output-specific scripts.The column-focused check reduces false negatives without increasing false positives and reliably recognizes valid alternative outputs.Source: A SPIDER 2.0 EVALUATION SCRIPTS · p. 15

B.1 SQL ANNOTATION EXAMPLES

• SQL annotations rewrite source queries into richer tasks through semantic interpretation or external knowledge, and sometimes combine related queries into a cross-dataset workflow.Examples extend NYC bike/car comparisons to taxi suitability, add a patent-originality calculation using external documentation, redefine analytics page views as conversion, and merge Chicago weather-station and NYC bike-rain queries.Source: §B.1 · p. 17

B.2 DBT PROJECT ANNOTATION EXAMPLES

• DBT benchmark examples are built by validating a complete project and its DAG, using YAML/Markdown plans, deleting selected SQL flows, then prompting for the missing transformation.YAML model definitions let task instructions omit detailed output-column specifications.Source: §B.2 · pp. 17–18
• Solving DBT examples requires understanding the request, inspecting project YAML, Markdown, SQL, and source data, then locating where code should change.The database must be explored to identify available source data and missing models.Source: §B.2 · pp. 18–19
• The workflow also requires writing SQL models, running dbt, inspecting resulting tables, and debugging or repeating steps until the transformation meets requirements.Successful execution can add or remove database tables, so the result must be checked against the task.Source: §B.2 · p. 19
• Worked examples classify December sessions into 18 traffic channels from source, medium, and campaign rules, and compute patent originality from backward-citation IPC diversity.The traffic task counts sessions and calculates each channel’s share; the patent task selects the highest originality score.Source: §B.2 · pp. 20, 22
• GA4 examples extract January 2021 page-view events, categorize page paths as product-listing or product-detail pages, and calculate page transitions and conversion rates.The SQL unnests event parameters and uses page-location patterns to distinguish PLP from PDP pages.Source: §B.2 · p. 23
• A merged GSOD and NYC public-data example rewrites related SQLs into a workflow that classifies 2016 days by rainfall and compares average Citibike rentals on rainy versus non-rainy days.The annotation selects a weather station near New York City, uses its precipitation data to classify days, then joins those classifications with daily bike rentals.Source: §B.2 · p. 24, Table 17 · p. 24
• A separate Chicago query finds weather stations within 50 km of the city center and calculates their distances from its coordinates.The snippet sets Chicago's center to (-87.63, 41.88) and limits stations to the nearest 50 km search radius.Source: §B.2 · p. 24

B.3 SPIDER 2.0 DATABASE EXAMPLES

• Google Analytics 4 exports create property-specific BigQuery datasets with daily event tables, nested or repeated parameters, and up to three days of late-arriving updates.The item_params RECORD stores custom item parameters that vary by implementation.Source: §B.3 · p. 25, Figure 11 · p. 25
• Spider 2.0 examples include enterprise BigQuery environments with multiple schemas and varied tables, illustrating database structures more complex than those in other benchmarks.Figure 12 shows an example environment.Source: §B.3 · p. 25, Figure 12 · p. 25

B.4 EXAMPLES OF EXTERNAL DOCUMENTS

• The external documents include a traffic-channel categorization table and Google Page Category guidance, including Product List Page and Product Detail Page classifications.Source: §B.4 · p. 25, Figure 13 · p. 26, Figure 14 · p. 26

B.5 EXAMPLES OF CONTEXT SETUP

• For a bike-trip query example, query.py provides the cloud-database interface, while result.csv specifies the required answer format for the otherwise ambiguous request.The setup also includes a README task description and BigQuery credentials.Source: §B.5 · p. 27
• Some task setups provide demonstration answers to constrain output format and support agent self-debugging, such as checking a November query against supplied results.The paper describes this consistency check as a realistic possibility, not a task requirement.Source: §B.5 · p. 27

B.6 THE DIFFERENCE IN TASK INSTRUCTIONS BETWEEN SPIDER 2.0 AND SPIDER 2.0-LITE.

• Spider 2.0 emphasizes naturalness while Spider 2.0-Lite emphasizes unambiguity, reflecting different possible question forms in real-world text-to-SQL workflows.Source: §B.6 · p. 28

B.7 SQL DIALECT DOCUMENTS COLLECTION

• Spider 2.0 crawls and preprocesses official database-system function documentation for retrieval-augmented agents, reusing shared category descriptions when webpages do not describe functions individually.Detailed crawl and parsing statistics are presented in Table 18.Source: §B.7 · p. 29, Table 18 · p. 29

B.7.1 PROCESSED FUNCTIONS FOR DIFFERENT DATABASE SYSTEMS

• Parsed documentation chunks can be retrieved and inserted into prompts to compensate for agents’ deficiencies in SQL dialect knowledge.Source: §B.7.1 · p. 29

Document of DuckDB Functions; Document of SQLite Functions

• DuckDB’s datediff counts date-partition boundaries; SQLite’s group_concat concatenates non-NULL values and accepts an optional separator, with string_agg as an alias.Source: §B.7.1 · pp. 29, 31, 32

Document of Clickhouse Functions

• ClickHouse’s JSONHas returns 1 when a value exists in JSON and 0 otherwise, accepting keys or indices to access object members or array elements.Source: §B.7.1 · pp. 29, 32

Document of BigQuery Functions

• BigQuery’s ST INTERSECTS returns TRUE when two GEOGRAPHY inputs share at least one point, implying ST DISJOINT returns FALSE.Source: §B.7.1 · p. 29

Document of Postgres Functions

• PostgreSQL documents enum support functions as a way to program without hard-coding particular enum values.The passage points to Table 9.35 and illustrates the context with a rainbow enum type.Source: §B.7.1 · p. 30, Table 9.35 · p. 30

Document of Snowflake Functions

• Snowflake’s ATAN2(y, x) computes an inverse tangent, with a DOUBLE result in radians within [-pi, pi].The documentation notes that the first argument is the Y coordinate, not the X coordinate.Source: §B.7.1 · p. 31

B.8 EXTEND DATASET STATISTIC

• Spider 2.0 databases span a wide range of real-world domains and scenarios, as illustrated by Figure 15.Source: §B.8 · p. 32, Figure 15 · p. 33
• Spider 2.0 includes text, numeric, structured, time-related, and spatial data types, whose variability complicates SQL dialects and data handling.Examples include STRING, BOOLEAN, INTEGER, FLOAT, STRUCT, JSON, TIMESTAMP, and GEOGRAPHY in BigQuery datasets.Source: §B.8 · pp. 32–33, Figure 16 · p. 33
• Spider 2.0 queries contain more SQL keywords than prior datasets, and its databases contain more tables while queries require more joins on average.Figure 17 presents the comparative statistical analysis.Source: §B.8 · p. 33, Figure 17 · p. 33
• Spider 2.0 databases average 5273.42M rows, compared with 17 in WikiSQL, 2K in Spider 1.0, 280K in KaggleDBQA, and 549K in BIRD.Many Spider 2.0 databases reach TB-level sizes.Source: §B.8 · p. 33

C.1 SPIDER-AGENT FRAMEWORK

• Spider-Agent, inspired by React (Yao et al., 2022) and Intercode (Yang et al., 2023), focuses on database-related coding tasks and projects.Source: §C.1 · p. 33
• Spider-Agent supports multi-turn database interaction through specialized command-line actions, with temperature 1.0, top-p 0.9, and beginning-of-input truncation when needed.The agent automatically terminates after repeating a result three times or an action exceeds 120 seconds; tasks have a heuristic 30-step limit, sufficient for most tasks.Source: §C.1 · p. 35, Table 19 · p. 35
• No clear correlation was observed between the number of joins and model performance in the reported analysis.The authors speculate that complex SQL annotation made performance independent of the number of tables involved.Source: §C.1 · p. 35, Figure 19 · p. 35
• 9.0 steps on average (6 minimum, 17 maximum) were needed for correctly completed tasks; Figure 20 analyzes action frequency across solved trajectories.The figure caption specifies 286 solved trajectories for Spider-Agent with o1-Preview.Source: §C.1 · p. 35, Figure 20 · p. 35

C.2 DETAILS OF SPIDER 2.0-LITE EXPERIMENTS

• DIN-SQL (Pourreza & Rafiei, 2024) uses task decomposition and adaptive prompting strategies tailored to task complexity.Source: §C.2 · pp. 35–36
• DAILSQL (Gao et al., 2024) achieves the best EX on Spider through prompt optimization and in-context learning.Source: §C.2 · p. 36
• CHESS (Talaei et al., 2024) achieves the best EX on BIRD using schema filtering through entity and context retrieval, followed by SQL revision.Source: §C.2 · p. 36
• CodeS (Li et al., 2024a) fine-tunes open-source code-generation models on large text-to-SQL corpora, reaching performance comparable to LLM-prompting methods.Source: §C.2 · p. 36
• DAIL-SQL’s Spider 2.0-Lite prompt components are highlighted in Figure 21, and similar components are implemented for DIN-SQL, CHESS, and CodeS.Source: Figure 21 · p. 36

C.3 DETAILS OF ERROR ANALYSIS

• Figure 22 provides descriptions and examples for all error categories summarized in the error-analysis section.Source: §C.3 · p. 37, Figure 22 · p. 37

C.4 OTHER ANALYSIS

• DAIL-SQL + GPT-4o reaches 5.68% EX on Spider 2.0-lite, versus 86.6% on Spider 1.0 and 57.4% on BIRD; the best Spider 2.0-snow result is 2.20% EX.SFT CodeS-15B achieves only 0.73% EX on Spider 2.0-lite despite extensive fine-tuning.Source: §C.4 · p. 37
• 12.78% performance on BigQuery versus 6.6% on Snowflake for the same 180 questions shows that SQL dialect differences affect generation performance.Snowflake-type examples are the most challenging among the Spider 2.0 database types.Source: §C.4 · p. 37

C.5 EXPERIMENTS COST

• Average API-call cost per instance is summarized across methods in Table 21.Source: §C.5 · p. 37, Table 21 · p. 39

C.6 CASE STUDY OF SPIDER-AGENT

• In the Google Patents case, execution feedback lets Spider-Agent iteratively debug and refine SQL until it produces a bug-free result.The paper identifies automated iterative debugging as a key advantage of Spider-Agent.Source: §C.6 · p. 39, Figure 25 · p. 40
• For a DBT project, Spider-Agent explores the codebase and database, writes two SQL scripts, debugs, runs DBT, and completes the transformation.The case study describes locating where code needed to be added before writing the scripts.Source: §C.6 · p. 39, Figure 26 · p. 40
• Failures stem from missing codebase information about monthly recurring revenue, inability to translate complex document strings into SQL, and difficulty understanding nested columns.These cases respectively produced an incorrect query, inaccurate SQL, and repeated unsuccessful coding attempts.Source: §C.6 · p. 39

C.7 PROMPTS OF SPIDER-AGENT

• Spider-Agent follows ReAct (Yao et al., 2022), alternating reasoning, action, and observation so the model observes each action’s outcome before reasoning again.Source: §C.7 · p. 43
• The response format requires a task-and-environment analysis prefixed “Thought:” followed by an action-space command prefixed “Action:”.The system prompt casts the agent as a database, SQL, and DBT-proficient data scientist restricted to the provided actions.Source: §C.7 · p. 43
• The prompt directs agents to inspect files and database schemas, run and revise multiple SQL queries, validate results, and verify DBT models with dbt run.It rejects empty result files as incorrect and requires a final CSV or text answer rather than an intermediate SQL statement.Source: §C.7 · p. 43
• Available operations include shell commands and file creation or editing, plus BigQuery SQL execution, table and column metadata retrieval, and sample-row retrieval.BigQuery query results can be printed or saved to CSV, while table and schema information can be saved to files.Source: §C.7 · pp. 43–44