PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment

Xiangjin Xie, Guangwei Xu, Lingyan Zhao, Ruijie Guo

arXiv:2502.14913v1cs.CLcs.AIcs.IR

TL;DR

LLM-based multi-agent Text-to-SQL systems remain limited by incomplete task frameworks, instruction-following failures, and hallucination, while multi-agent workflows can accumulate errors. OpenSearch-SQL addresses these issues with a four-stage framework, consistency alignment, SQL-Like reasoning, and self-taught Query-CoT-SQL few-shot examples. Without supervised fine-tuning or reinforcement learning, it reports first-place BIRD leaderboard results at submission across three metrics.

  • Problem

    Multi-agent Text-to-SQL methods have methodological gaps, instruction-following failures, and hallucination risks that can accumulate across agents.

  • Method

    OpenSearch-SQL combines Preprocessing, Extraction, Generation, and Refine stages with consistency alignment, SQL-Like reasoning, and self-taught Query-CoT-SQL few-shot examples.

  • Results

    69.3% EX on BIRD development, 72.28% EX on test, and 69.36% R-VES ranked first at submission without fine-tuning.

  • Takeaways & Limitations

    The method demonstrates reported effectiveness and efficiency while using base pretrained LLMs without SFT or RL.

  • Takeaways & Limitations

    The authors identify unoptimized prompt and schema-element selection, single-prompt candidate generation, and early-stage alignment research as remaining limitations.

Abstract

from arXiv · show

Although multi-agent collaborative Large Language Models (LLMs) have achieved significant breakthroughs in the Text-to-SQL task, their performance is still constrained by various factors. These factors include the incompleteness of the framework, failure to follow instructions, and model hallucination problems. To address these problems, we propose OpenSearch-SQL, which divides the Text-to-SQL task into four main modules: Preprocessing, Extraction, Generation, and Refinement, along with an Alignment module based on a consistency alignment mechanism. This architecture aligns the inputs and outputs of agents through the Alignment module, reducing failures in instruction following and hallucination. Additionally, we designed an intermediate language called SQL-Like and optimized the structured CoT based on SQL-Like. Meanwhile, we developed a dynamic few-shot strategy in the form of self-taught Query-CoT-SQL. These methods have significantly improved the performance of LLMs in the Text-to-SQL task. In terms of model selection, we directly applied the base LLMs without any post-training, thereby simplifying the task chain and enhancing the framework's portability. Experimental results show that OpenSearch-SQL achieves an execution accuracy(EX) of 69.3% on the BIRD development set, 72.28% on the test set, and a reward-based validity efficiency score (R-VES) of 69.36%, with all three metrics ranking first at the time of submission. These results demonstrate the comprehensive advantages of the proposed method in both effectiveness and efficiency.

1 INTRODUCTION

The paper identifies methodological gaps and hallucination risks in multi-agent Text-to-SQL systems, then proposes a structured, alignment-based framework with dynamic few-shot support and SQL-Like reasoning. OpenSearch-SQL uses these components without post-training and reports leading BIRD results at submission.

  • Motivation: Multi-agent Text-to-SQL systems face incomplete frameworks, insufficient verification and error correction, missing few-shot learning, and weak instruction construction.These gaps are presented as limitations of prior LLM-driven approaches.
  • Motivation: Instability and weak coupling between agents can cause downstream agents to underuse upstream outputs, accumulating hallucinations and performance loss.The paper links this problem to incoherent multi-agent workflows.
  • Framework: OpenSearch-SQL organizes Text-to-SQL into Preprocessing, Extraction, Generation, and Refine stages based on the human SQL-construction workflow.The framework prepares auxiliary information, selects relevant schema elements, generates SQL, and checks or optimizes results.
  • Contributions: Query-CoT-SQL dynamically augments few-shot examples with Chain-of-Thought information, while SQL-Like lets models generate SQL structure before specific syntax details.The paper presents both mechanisms as ways to improve LLM SQL generation.
  • Results: 69.3% EX on the BIRD development set, 72.28% EX on the test set, and 69.3% R-VES ranked first at submission without SFT or RL.The contribution summary reports R-VES as 69.36%, whereas the results paragraph states 69.3%.

2 PRELIMINARY

The preliminary section defines Text-to-SQL, situates OpenSearch-SQL among multi-agent approaches, and frames hallucination as a major reliability problem. The proposed framework uses dynamic few-shot methods and consistency alignment to organize stages and reduce hallucination-related failures.

  • Text-to-SQL: Text-to-SQL translates a Natural Language Query into an SQL query using database information, a model, and a prompt.The task aims to enable database access without requiring users to know SQL.
  • Existing approaches: Current Text-to-SQL systems commonly combine value retrieval, schema linking, few-shot driving, CoT prompting, SQL correction, and self-consistency across four stages.The stages are Preprocessing, Extraction, Generation, and Refinement.
  • OpenSearch-SQL: OpenSearch-SQL applies dynamic few-shot methods across task stages and uses an Alignment module to connect them and reduce LLM hallucinations in multi-agent workflows.The framework is described as simplifying complex instruction synchronization across stages.
  • Hallucination: Hallucination in Text-to-SQL includes nonexistent database information, instruction-following failures, typographical and syntactic errors, and randomness-related deviations.The paper treats these issues as architectural targets rather than relying only on post-training or post-processing.

3 METHODOLOGY

OpenSearch-SQL organizes Text-to-SQL around consistency alignment and dynamic few-shot generation. Its alignment mechanism connects agent inputs and outputs to reduce hallucinations and preserve logical consistency across the workflow.

  • 3.1 Alignments: LLM hallucinations can propagate through multi-agent Text-to-SQL workflows when downstream agents inherit incorrect upstream columns or other outputs.The paper describes cumulative errors as making total hallucination nearly monotonically non-decreasing.
  • 3.1 Alignments: Consistency alignment passes each agent’s aligned output to downstream agents to reduce instruction-following failures and hallucinations.The mechanism aligns an agent’s output with upstream information and is compared to residual connections across collaborative agents.
  • 3.2 Self-Taught Fewshot: Dynamic few-shot selection uses Masked Question Similarity to retrieve similar queries and self-taught Query-CoT-SQL examples to provide richer generation guidance.The method supplements Query-SQL pairs with chain-of-thought information and uses error-specific examples for refinement.

3.3 Preprocessing

OpenSearch-SQL preprocessing builds database-aware retrieval and prompting resources before query processing. The stage is automated and produces a vector database, Query-CoT-SQL few-shots, and the database schema.

  • 3.3 Preprocessing: Preprocessing indexes database values against the true database structure to help generated SQL avoid small character-discrepancy errors.Only string-type data is indexed to reduce retrieval-database storage requirements.
  • 3.3 Preprocessing: Dynamic few-shot construction adds chain-of-thought information and correction examples tailored to different error types.These resources are prepared during preprocessing for later generation and refinement.
  • 3.3 Preprocessing: The automated preprocessing stage takes database information and training data as input and outputs a vector database, Query-CoT-SQL few-shots, and database schema.The process is agent-driven and does not require human intervention.

3.4 Extraction

Extraction prepares query-specific schema, value, entity, and few-shot information before SQL generation. OpenSearch-SQL combines LLM selection, vector retrieval, filtering, and information alignment.

  • 3.4 Extraction: Extraction prepares schema links, stored database values, few-shot examples, and instructions required for a specific natural-language query.The framework treats extraction as decoupled from the particular database query language.
  • 3.4 Extraction: OpenSearch-SQL performs entity extraction, value extraction, and column filtering, then applies Info Alignment to the extracted information.Entities are extracted from the NLQ, while candidate columns and values are obtained through LLM-based processing.
  • 3.4 Extraction: Vector retrieval finds database values similar to extracted entities and uses split retrieval for phrases or longer texts to reduce recall failures from storage-format differences.Results below a similarity threshold are removed from the top-K retrieved entities.
  • 3.4 Extraction: Column filtering combines LLM-selected tables and columns with vector-retrieved columns whose similarity to NLQ entities exceeds a threshold.The combined results form a preliminary subset of schema information.
  • 3.4 Extraction: Info Alignment matches NLQ phrases or clauses to SELECT content and reintegrates primary keys and same-named columns to reduce table-column ambiguity.This alignment targets both the quantity and order of SELECT content.

3.5 Generation

Generation uses progressive structured reasoning, SQL-Like intermediate queries, and dynamically retrieved Query-CoT-SQL examples to produce candidate SQL queries. Alignment mechanisms then address mismatches in values, functions, and SQL style.

  • 3.5 Generation: Progressive generation asks the LLM to produce reasoning, columns, values, SELECT content, SQL-Like, and final SQL sequentially.The sequence is designed to emphasize SQL structure and facilitate error identification.
  • 3.5 Generation: SQL-Like omits selected syntax details, including JOIN formatting and function formatting, to focus generation on SQL logic.The final SQL is produced after the SQL-Like representation.
  • 3.5 Generation: Dynamic few-shot generation retrieves the top K_f similar queries using masked-question similarity and supplies their Query-CoT-SQL forms as examples.Multiple candidate SQL queries may be generated for an individual NLQ.
  • 3.5 Generation: Generation inputs combine rules, database schema, few-shot examples, similar values, and Info Alignment outputs before the model produces structured SQL components.The requested outputs include analysis, relevant columns, related values, SELECT content, SQL-Like, and final SQL.
  • 3.5 Generation: Alignment reduces generation discrepancies by correcting database-value mismatches, standardizing aggregate functions, and addressing SQL-style differences.The paper identifies agent, function, and style alignment as the three components.

3.6 Refinement

Refinement optimizes generated SQLs through execution-based correction and consistency-based selection. It first addresses execution errors, then chooses among viable candidates using answer consistency and execution time.

  • Correction: Correction executes candidate SQLs and fixes syntax errors, empty results, and other execution problems with error-specific instructions.The correction step uses different few-shots for different error types.
  • Self-consistency & vote: Self-consistency and voting exclude unfixable or empty-answer SQLs before selecting the candidate with the highest consistency.Candidates are represented by their SQL query and execution answer.
  • Self-consistency & vote: Among SQL queries producing the same answers, the refinement selects the one with the shortest execution time.The selection objective combines execution results with execution speed.
  • Cost consideration: Self-consistency often increases costs, although OpenSearch-SQL v2 still ranks first on BIRD when generating one SQL without self-consistency.This comparison concerns the evaluation setting without self-consistency.

3.7 Algorithm

Algorithm 1 summarizes OpenSearch-SQL from preprocessing through final SQL generation. It builds database indexes and schema information, processes the question, and applies refinement to select the final query.

  • Algorithm overview: Algorithm 1 presents the complete OpenSearch-SQL workflow, covering each step from preprocessing to final SQL generation.The algorithm takes a target database, user question, and training set as inputs.
  • Preprocessing: Preprocessing builds a value-and-column index database D_v and the original database schema S by processing tables and fields.The index supports later retrieval of database information.
  • Question processing: The main function processes the NLQ to obtain entity information E_Q and select relevant columns before later SQL generation and refinement.Training Query-SQL pairs are also used to construct the few-shot set F with CoT information.
  • Refinement: The refinement stage corrects execution errors and selects the SQL with the highest consistency and shortest execution time as the final answer.This selection follows the framework's correction and consistency-based refinement procedure.

3.8 Optimization

The paper identifies several optimization opportunities in OpenSearch-SQL, including prompt design, column and value selection, generation prompting, and alternative few-shot strategies. It also notes that alignment research remains at an early stage.

  • Current limitations: Prompt details and the precise selection of columns and values have not been fully optimized.The authors identify these as current areas for improvement.
  • Current limitations: Generation currently uses a single prompt to produce the SQL candidate set without additional optimization.This limits the degree of prompt-level optimization in the generation stage.
  • Future directions: SFT models are identified as having potential to improve Text-to-SQL performance, supported by prior work including CHESS, MCS-SQL, and distillery.The cited studies are presented as evidence that further optimization can pursue this direction.
  • Future directions: Few-shot approaches are not limited to Query-CoT-SQL pairs, and alignment research remains at an early stage with room for further development.The authors present both observations as opportunities for improving Text-to-SQL performance.

4 EXPERIMENTS

OpenSearch-SQL is evaluated on BIRD and Spider, with experiments measuring benchmark performance, difficulty effects, module contributions, few-shot strategies, candidate scaling, and structured CoT.

  • Main Results: 69.3% EX on BIRD development, 72.28% EX and 69.36% R-VES on the holdout test set ranked first at submission.The reported results used no fine-tuning; a single-generation SQL still achieved 67.8 EX on BIRD development.
  • Difficulty Analysis: 7.64% absolute improvement from Self-Consistency & Vote occurred on difficult problems, while easy and medium problems showed no notable difference.The authors associate increasing difficulty with greater susceptibility of large models to hallucinations.
  • Modular Ablation: Execution accuracy increased monotonically through the workflow, supporting distinct positive contributions from the evaluated modules.The ablation examines Extraction, Generation, Refinement, and Alignment across intermediate and final outputs.
  • Few-shot Analysis: Few-shot examples improved SQL performance across stages, with Query-CoT-SQL producing the greatest generation-phase improvement among the compared strategies.Few-shot also improved final performance during Refinement, although its Refinement-stage gain was smaller.
  • CoT and Voting: Structured CoT achieved the best results and a larger relative improvement when multiple answers were sampled for voting.Self-Consistency & Vote provided greater improvement than CoT alone, with structured CoT showing the higher relative enhancement.

5 RELATED WORK

Related work progresses from classical structured SQL construction to LLM-based agents and retrieval methods, while Text-to-SQL benchmarks become broader and more complex.

  • Task Background: Text-to-SQL converts natural-language queries into SQL, aiming to provide database access without requiring SQL knowledge.The task remains difficult because SQL structure is strongly tied to database structure and constrained expression formats.
  • Classical Methods: Classical systems construct SQL through component or sketch-based methods that separately model clauses and their dependencies.SQLNET, RYANSQL, and SyntaxSQLNet are cited as examples of structured decoding approaches.
  • LLM and Agent Methods: LLM-based agents and retrieval-augmented methods use specialized roles or retrieved documents to support complex question answering and SQL generation.The related work describes Modular RAG and Advanced RAG as retrieval techniques for richer and more accurate answers.
  • LLM-based Text-to-SQL: Recent Text-to-SQL research increasingly relies on multi-agent LLM algorithms, alongside a shift in benchmarks from WikiSQL toward Spider and BIRD.These methods provide agents with information intended to assist SQL prediction.
  • Emerging Direction: The field has converged toward LLMs plus agents, with Text-to-SQL methods increasingly positioned as tools for simplifying everyday database work.The passage characterizes this framework as increasingly standardized as model capabilities improve.

6 CONCLUSION

OpenSearch-SQL enhances Text-to-SQL with dynamic Few-shot and consistency alignment mechanisms, achieving the top position in all three BIRD leaderboard metrics at submission. The authors present the approach as a potential contribution to Text-to-SQL and other multi-agent collaborative tasks.

  • OpenSearch-SQL enhances Text-to-SQL through dynamic Few-shot and consistency alignment mechanisms.
  • The approach achieved the top position in all three BIRD leaderboard metrics at the time of submission.
  • The authors hope its Few-shot construction method and consistency alignment-based workflow will offer new perspectives for Text-to-SQL and other multi-agent collaborative tasks.
Loading 2502.14913v1…