PaperlayerView exact revision on arXiv

Source-linked AI summary

ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration

Minghang Deng, Ashwin Ramachandran, Canwen Xu, Lanxiang Hu, Zhewei Yao, Anupam Datta, Hao Zhang

arXiv:2502.00675v5cs.CL

TL;DR

Real-world Text-to-SQL remains difficult because databases have complex schemas, multiple dialects, and ambiguous or sophisticated queries. ReFoRCE combines schema compression, iterative SQL refinement, consensus, and selective column exploration; it reports state-of-the-art performance on Spider 2.0’s Snow and Lite tasks while maintaining efficiency.

  • Problem

    Text-to-SQL methods often face large, complex schemas, multiple dialects, nested or under-documented data types, and ambiguous queries that complicate realistic database use.

    Source: §1 · p. 1

  • Method

    ReFoRCE compresses schema context, iteratively refines SQL, uses majority voting to identify high-confidence answers, and explores columns for unresolved cases.

    Source: §1 · p. 2, §3.2.2 · p. 5, §3.2.3 · p. 6

  • Results

    ReFoRCE achieves state-of-the-art execution accuracy on Spider 2.0 Snow and Lite, with 35.83% EX on Snow and 36.56% EX on Lite.

    Source: §4.2 · p. 7, §5.2 · p. 9

  • Takeaways & Limitations

    ReFoRCE’s structured workflow reports strong Spider 2.0 performance while maintaining efficiency and supporting smaller open-source models.

    Source: §5.2 · p. 9

    ReFoRCE generalizes less well to BIRD, does not include external schema-linking implementations, and relies on proprietary models such as o3 without training or fine-tuning its own model.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

We present ReFoRCE, a Text-to-SQL agent that tops the Spider 2.0 leaderboard--a challenging benchmark reflecting complex, real-world Text-to-SQL scenarios. While Text-to-SQL systems enable natural language queries over structured databases, deploying them in enterprise environments remains difficult due to large, complex schemas (with over 1,000 columns), diverse SQL dialects (e.g., BigQuery, Snowflake), and sophisticated query requirements (e.g., transformations and analytics). ReFoRCE addresses these challenges through: (a) database information compression via pattern-based table grouping and LLM-guided schema linking to alleviate long-context issues; (b) self-refinement to iteratively correct syntax and semantic errors across dialects; (c) majority-vote consensus to select high-confidence candidates while deferring ambiguous cases arising from sophisticated queries; and (d) iterative column exploration guided by execution feedback to resolve those deferred cases. ReFoRCE achieves new state-of-the-art results, with scores of 35.83 on Spider 2.0-Snow and 36.56 on Spider 2.0-Lite.

1 Introduction

ReFoRCE targets the gap between simplified Text-to-SQL benchmarks and Spider 2.0’s complex real-world settings with a modular agentic workflow. It reports state-of-the-art Spider 2.0 results and resource savings, with compatibility with smaller open-source models.

1 Introduction

  • Earlier methods were tested largely on small schemas, SQLite, short queries, and minimal ambiguity, unlike Spider 2.0’s large cross-domain databases, multiple dialects, nested data, and complex reasoning.These differences motivate agentic pipelines beyond single-shot SQL generation.Source: §1 · p. 1
  • ReFoRCE compresses schemas through pattern-based table grouping and LLM-guided schema linking, self-refines dialect-specific SQL, applies majority-vote consensus, and explores columns for unresolved cases.Column exploration is selectively triggered to balance accuracy and efficiency.Source: §1 · p. 2
  • ReFoRCE reports reduced resource overhead and improved compatibility with smaller open-source models in addition to its Spider 2.0 results.The introduction reports 3.52 LLM and 3.89 DB calls per example; disabling column exploration lowers this to 1.69 calls and 15K tokens per example with minimal accuracy loss.Source: §1 · pp. 2–3
  • Gold table annotations yield only marginal improvements, suggesting strong reasoning models can infer relevant schema structures without explicit supervision in these scenarios.The paper reports this finding for real-world Text-to-SQL settings [13].Source: §1 · p. 3

2 Related Work

Related work spans fine-tuning, LLM prompting, coding agents, and structured workflows, but prior Text-to-SQL methods struggle with complex benchmarks. ReFoRCE combines table compression, consensus enforcement, and iterative column exploration for enterprise-scale SQL generation.

Text-to-SQL Methods

  • Fine-tuning and LLM prompting perform well on simpler datasets but struggle on Spider 2.0 with schema understanding, ambiguity resolution, and dialect handling.Other lines include semantic matching, in-context learning, sub-query decomposition, self-refinement, calibration, and reinforcement learning.Source: §2 · p. 3

Coding Agents

  • Coding-agent frameworks support tool use and iterative problem-solving, while generalized agents may underperform domain-specific solutions on specialized tasks.Examples include ReAct [34], Self-Debugging [3], InterCode [33], and Spider Agent [8].Source: §2 · p. 3

Structured Workflows

  • Structured workflows decompose Text-to-SQL into subtasks, but most prior agents were evaluated mainly on simpler datasets; ReFoRCE combines compression, consensus, and column exploration for enterprise-scale generation.Workflow subtasks include question parsing, schema linking, value retrieval, candidate generation, and selection.Source: §2 · p. 3

3 Methodology

ReFoRCE structures Text-to-SQL as controllable subtasks, compressing database context before generating and refining candidates. It selects unique-vote answers directly and applies column exploration to unresolved, low-confidence cases.

3.1 Preliminaries

  • Spider 2.0 frames the task as iterative code modification using execution observations, while Spider 2.0-Snow and Lite are self-contained Text-to-SQL tasks.The supplied description identifies a question, database interface, and codebase as inputs to the broader Spider 2.0 task.Source: §3.1 · p. 3

3.2 ReFoRCE: Self-Refinement, Consensus Enforcement, and Column Exploration

  • ReFoRCE combines database compression, self-refinement, majority voting, and column exploration in a structured workflow for Text-to-SQL.Column exploration is triggered for low-confidence predictions, including difficult cases involving nested schemas or complex task descriptions.Source: §3.2 · p. 4

3.2.1 Database Information Compression

  • Spider 2.0 schemas can exceed 800 columns, and including descriptions and sample rows can expand schema context to 77MB, beyond current LLM context limits.ReFoRCE groups tables with similar prefixes or suffixes, retains one representative table in full, and uses table-level schema linking for prompts still longer than 50K tokens.Source: §3.2.1 · p. 4
  • For GA360, pattern-based compression reduces DDL files from over 50 MB to less than 2 MB, exceeding 96% compression without information loss.The example database contains a year of similarly named tables.Source: §3.2.1 · p. 5

3.2.2 Generation and Consensus Enforcement

  • A unique highest-vote candidate is selected as high-confidence; ties remain low-confidence and are retained for additional refinement.The method generates self-refined candidates and does not immediately resolve ambiguous cases through random selection.Source: §3.2.2 · p. 5

3.2.3 Column Exploration

  • Column exploration addresses nested-column and sample-value challenges by progressively querying relevant columns, refining faulty queries, and targeting unresolved low-confidence examples.It is deferred because of computational cost and may produce more diverse outputs that slightly reduce accuracy; self-refinement and voting are typically sufficient for simpler questions.Source: §3.2.3 · pp. 5–6

4 Experiments

Experiments evaluate ReFoRCE on Spider 2.0’s Snow and Lite tasks, then examine its efficiency, smaller-model combinations, and component contributions. ReFoRCE leads the reported comparisons, while ablations show differing benefits from exploration, compression, and schema information.

4.1 Experimental Setup

  • The evaluation covers 547 examples per task across more than 150 databases, averaging 800 columns and about 150 SQL tokens per query.Snow uses Snowflake; Lite supports BigQuery, Snowflake, and SQLite. Execution Accuracy is the primary metric, with EX Pass@k and API-call and token costs also reported.Source: §4.1 · p. 6

4.2 Evaluation Results

  • 35.83% EX on Snow and 36.56% EX on Lite; ReFoRCE with o3 also records the highest EX and EX@8 across both datasets.Spider-Agent is roughly 10% lower for each model.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • Without column exploration, ReFoRCE uses 1.69 LLM and database calls per example and about 15K tokens, with a roughly 2% performance drop.The full setup requires 3.52v LLM calls and 3.89v database calls per example, where v is the number of voting candidates.Source: §4.2 · p. 7
  • On Spider 2.0-SQLite, ReFoRCE + o3 + Arctic-Text2SQL-R1-7B reaches 29.63% EX (Greedy) and 36.30% EX@8, surpassing GPT-4o and matching o4-mini.OpenSearchSQL extraction adds minimal improvement for smaller models alone, while the ReFoRCE combination substantially improves their effectiveness.Source: §4.2 · pp. 7–8

4.3 Ablation Studies

  • Applying column exploration to all examples raises EX@8 by 5.75 on Snow and 4.93 on Lite, while slightly decreasing EX.Disabling exploration also decreases both EX and EX@8, particularly on Lite (-2.37 EX and -4.76 EX@8).Source: §4.3 · p. 8
  • Removing database information compression causes context-length failures and the largest reported drops: -3.65 EX and EX@8 on Snow and -3.48 EX on Lite.Removing random selection or majority voting yields smaller EX drops of -0.73 to -2.01.Source: §4.3 · pp. 8–9
  • Gold tables increase EX@8 by 6.95 on Snow and 4.93 on Lite, but improve EX by only 0.37 on each subset.The authors suggest strong reasoning models can infer relevant schema information without explicit supervision in this setting.Source: §4.3 · p. 9
  • Gold schema raises EX by 8.26% and EX@8 by 3.31% on Snow, and by 3.13% and 2.73%, respectively, on Lite.The evaluation uses only subsets with gold SQLs: 121 Snow examples and 256 Lite examples, with differing difficulty levels.Source: Table 4 · p. 8, §4.3 · p. 9

5 Discussion

ReFoRCE has limited generalization to simpler datasets such as BIRD, and the authors conclude that it achieves state-of-the-art execution accuracy on Spider 2.0 while maintaining efficiency. They also identify external retrieval integration, multi-dialect fine-tuning, and broader query support as future work.

5.1 Limitations

  • ReFoRCE generalizes less well to simpler datasets such as BIRD, which emphasize fine-grained schema understanding rather than long-context reasoning.The authors also note that their approach relies on proprietary models’ internal reasoning and uses no training or fine-tuning of their own.Source: §5.1 · p. 9

5.2 Conclusion

  • ReFoRCE achieves state-of-the-art execution accuracy on Spider 2.0’s Snow and Lite tasks while maintaining efficiency in LLM and database API usage.The conclusion highlights schema compression and column exploration as especially important for long contexts and complex queries, and reports generalization to smaller open-source models.Source: §5.2 · p. 9
  • Future work may integrate external schema linking and retrieval, fine-tune across SQL dialects, and support broader analytical and transformation-oriented queries.The authors did not incorporate external schema linking or value retrieval because publicly available, high-quality implementations were lacking.Source: §5.2 · p. 9, §5.1 · p. 9

A Table-level Schema Linking

Schema linking is applied only to long-context cases exceeding 50K tokens, with recall as the primary concern. After linking, only a small number of examples fail to achieve full recall.

A Table-level Schema Linking

  • Figure 3 shows table-level schema-linking prompts that ask an LLM to return structured JSON with reasoning and a final Y-or-N answer.Each table is queried individually.Source: A Table-level Schema Linking · p. 13
  • Only 15 Lite examples and 7 Snow examples failed to achieve full recall after schema linking.The method is applied only to cases exceeding 50K tokens, where recall is the primary concern; Table 5 reports recall and precision for long-context subsets and all examples.Source: A Table-level Schema Linking · p. 13

B API Cost

The cost analysis contrasts self-refinement alone with the full pipeline, where column exploration is triggered for unresolved cases. The weighted full-pipeline cost remains 3.52 LLM calls, 3.89 database calls, and 18.73K tokens per example.

B API Cost

  • 1.69 LLM/DB calls and 15.44K tokens are the weighted averages without column exploration.Most examples (407) succeed without self-refinement, using one LLM and DB call and 15K tokens; 50 edge cases reach the five-iteration cap.Source: B API Cost · p. 13, Table 6 · p. 14
  • The cost breakdown compares self-refinement usage without column exploration against costs when column exploration is added as a second step.The full-pipeline averages are scaled by the number of voting candidates to reflect per-example cost.Source: Table 7 · p. 14, Table 6 · p. 14, B API Cost · p. 14
  • 3.52 LLM calls, 3.89 DB calls, and 18.73K tokens per example are the weighted averages with column exploration in the full pipeline.Column exploration is used for cases unresolved after self-refinement; the 100 hard examples requiring it average 12 database calls and 30K tokens.Source: B API Cost · pp. 13–14, Table 7 · p. 14

C Column Exploration

Column exploration uses LLM-generated, executable queries to inspect relevant database columns, with invalid queries corrected and candidate answers consolidated before producing a final exploration output. Its prompts specify dialect-aware query forms and constrain exploration to the provided tables.

C Column Exploration

  • The prompt requests up to 10 diverse SELECT queries in the chosen API dialect and requires exploration to use provided tables, avoid schema inspection, and include LIMIT 20.Each query must include a --Description annotation.Source: C Column Exploration · p. 14
  • The column-exploration pipeline generates candidate SQL, executes and self-corrects invalid queries with bounded retries, then uses majority voting before producing final column-level output.When corrections are made, similar queries are refined to prevent repeated errors.Source: C Column Exploration · p. 14
  • For nested JSON columns, the prompts show dialect-specific extraction patterns and recommend inspecting the whole column first when its structure is unknown.The examples use LATERAL FLATTEN for Snowflake and UNNEST with JSON_EXTRACT_ARRAY for BigQuery.Source: C Column Exploration · p. 15
  • For uncertain string matches, the prompts recommend fuzzy matching, using ILIKE in Snowflake, LOWER with LIKE in BigQuery, and LIKE in SQLite.The examples replace spaces in target phrases with wildcard matches.Source: C Column Exploration · p. 15

D A Case Study for Column Exploration

The case study illustrates how iterative schema and data exploration helps resolve uncertainty and refine Snowflake SQL for a complex California road query, replacing a failing nested-FLATTEN approach with conditional aggregation to count overlapping road pairs that share no node IDs.

Step 1: Failure Without Column Exploration

  • Without column exploration, repeated self-refinement could not solve the task because Snowflake dialect constraints and limited schema visibility caused persistent SQL errors.Failures included unsupported lateral table-function joins and subqueries, and the model struggled with tag handling and EXISTS versus JOIN usage.Source: D A Case Study for Column Exploration · pp. 15, 19, 20

Step 2: Success With Column Exploration

  • The model iteratively explored sample tags, highway values, bridge encoding, and spatial and node metadata before constructing its final query.Exploratory SQL commands grounded the model’s understanding of PLANET_WAYS and its nested all_tags field.Source: D A Case Study for Column Exploration · pp. 16–17
  • Column exploration enabled a successful query counting 13 overlapping California road pairs that share no nodes and lack bridge tags.The model inspected PLANET_WAYS and its nested tags, then used aggregation and conditional logic such as MAX(IFF(...)) in HAVING clauses to avoid previously failing SQL patterns.Source: D A Case Study for Column Exploration · p. 17

D A Case Study for Column Exploration

  • After a nested-FLATTEN query failed with Snowflake error 002031, a single GROUP BY pass with conditional MAX checks avoided the unsupported subquery pattern.The revised logic flattened tags once and tested for a qualifying highway tag and absence of a bridge tag.Source: D A Case Study for Column Exploration · p. 24
  • The California candidate-roads query returned 15,429 records with its no-bridge-tag filter applied; the supplied passage does not support naming additional selection requirements.Source: D A Case Study for Column Exploration · p. 28
  • 13 qualifying overlapping California road pairs had no shared node IDs in the final count.The self-refined query also returned a total of 13.Source: D A Case Study for Column Exploration · pp. 31–32

E Broader Impact

The section describes ReFoRCE’s intended role in making database analysis more accessible, while noting deployment risks and recommending safeguards.

E Broader Impact

  • ReFoRCE aims to help programmers verify and refine generated SQL and broaden data-analysis access for non-technical users.The stated application areas include healthcare, education, public policy, and small-business analytics.Source: E Broader Impact · p. 32
  • Incorrect SQL in high-stakes domains may lead to erroneous conclusions or decisions, while automated database access raises privacy and security concerns.The section also notes potential misuse for unauthorized data extraction or inference attacks.Source: E Broader Impact · p. 32
  • The authors recommend deployment safeguards including access controls, human validation, query logging, and domain-tailored models.They also encourage further research on robustness, explainability, and error detection.Source: E Broader Impact · p. 32

F Spider2.0 Leaderboard Screenshot

The section identifies a Spider 2.0 leaderboard screenshot dated May 22, 2025.

F Spider2.0 Leaderboard Screenshot

  • The section presents a Spider 2.0 leaderboard screenshot dated May 22, 2025.The supplied passage identifies the screenshot and date but gives no leaderboard values.Source: Figure 5 · p. 33

Related papers

Loading 2502.00675v5…