PaperlayerView exact revision on arXiv

Source-linked AI summary

MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL

Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, LinZheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, Zhoujun Li

arXiv:2312.11242v6cs.CL

TL;DR

LLM-based Text-to-SQL methods degrade on huge databases and complex multi-step questions and often neglect external tools and model collaboration. MAC-SQL addresses these issues with collaborating agents for schema selection, question decomposition, and SQL refinement, achieving 59.59 execution accuracy with GPT-4 on BIRD and comparable performance from SQL-Llama.

  • Problem

    Recent LLM-based Text-to-SQL methods face performance degradation on huge databases and complex multi-step questions and often neglect external tools and model collaboration.

  • Method

    MAC-SQL uses a Decomposer with auxiliary Selector and Refiner agents to progressively reason, simplify large schemas, and correct SQL using external execution feedback.

  • Results

    59.59 execution accuracy is achieved by MAC-SQL+GPT-4 on the BIRD holdout test set, while SQL-Llama reaches 43.94 versus 46.35 for vanilla GPT-4.

  • Takeaways & Limitations

    MAC-SQL provides a multi-agent Text-to-SQL framework that integrates external tools and model collaboration and can be expanded to broader scenarios.

  • Takeaways & Limitations

    The authors report that prompts may not be optimal and that performance could improve with larger models than the fine-tuned 7B Code Llama model.

Abstract

from arXiv · show

Recent LLM-based Text-to-SQL methods usually suffer from significant performance degradation on "huge" databases and complex user questions that require multi-step reasoning. Moreover, most existing methods neglect the crucial significance of LLMs utilizing external tools and model collaboration. To address these challenges, we introduce MAC-SQL, a novel LLM-based multi-agent collaborative framework. Our framework comprises a core decomposer agent for Text-to-SQL generation with few-shot chain-of-thought reasoning, accompanied by two auxiliary agents that utilize external tools or models to acquire smaller sub-databases and refine erroneous SQL queries. The decomposer agent collaborates with auxiliary agents, which are activated as needed and can be expanded to accommodate new features or tools for effective Text-to-SQL parsing. In our framework, We initially leverage GPT-4 as the strong backbone LLM for all agent tasks to determine the upper bound of our framework. We then fine-tune an open-sourced instruction-followed model, SQL-Llama, by leveraging Code Llama 7B, to accomplish all tasks as GPT-4 does. Experiments show that SQL-Llama achieves a comparable execution accuracy of 43.94, compared to the baseline accuracy of 46.35 for vanilla GPT-4. At the time of writing, MAC-SQL+GPT-4 achieves an execution accuracy of 59.59 when evaluated on the BIRD benchmark, establishing a new state-of-the-art (SOTA) on its holdout test set (https://github.com/wbbeyourself/MAC-SQL).

1 Introduction

LLM-based Text-to-SQL methods struggle with huge databases and complex questions requiring multi-step reasoning, while often underusing external tools and model collaboration. MAC-SQL addresses these challenges through specialized collaborating agents and reports strong BIRD results with GPT-4 and SQL-Llama.

  • Text-to-SQL improves database accessibility by generating SQL queries from natural-language text.
  • Recent LLM-based methods suffer performance degradation on huge databases and complex questions requiring multi-step reasoning.
  • MAC-SQL combines a Decomposer, Selector, and Refiner in a multi-agent framework for Text-to-SQL parsing.The Decomposer handles progressive reasoning, the Selector reduces large schemas, and the Refiner executes and corrects faulty SQL.
  • The Selector reduces irrelevant schema interference by creating smaller sub-databases, while the Refiner uses execution feedback to correct erroneous SQL.
  • SQL-Llama is an instruction-tuned open-source model based on Code Llama 7B for database simplification, question decomposition, SQL generation, and correction.
  • 59.59 execution accuracy is achieved by MAC-SQL+GPT-4 on the BIRD holdout test set, establishing a new state-of-the-art at the time of writing.SQL-Llama reaches 43.94 execution accuracy versus 46.35 for vanilla GPT-4.

2 Preliminaries

The Text-to-SQL task maps a natural-language question and database schema, optionally with external knowledge, to a correct SQL query. LLMs perform this as token-by-token SQL generation conditioned on the input representation.

  • Text-to-SQL generates the correct SQL query Y from a natural-language question Q, database schema S, and optional external knowledge K.The schema consists of multiple tables and columns.
  • The database schema S is represented by tables T and columns C.
  • LLM-based Text-to-SQL formulates SQL generation as predicting the query token by token from prompts.
  • Each SQL token is conditioned on the preceding query prefix and the input triple X = (Q, S, K).

3 MAC-SQL Framework

MAC-SQL uses a core Decomposer agent with Selector and Refiner auxiliaries to handle complex questions, large schemas, and SQL errors through conditional collaboration.

  • MAC-SQL combines a Decomposer for SQL generation with Selector and Refiner agents for schema reduction, tool usage, and SQL refinement.The three agents collaborate within a multi-agent framework, with auxiliary agents activated as needed.
  • 3.2 Selector: The Selector identifies a minimal schema for answering the question and activates only when the schema prompt exceeds a length threshold.Otherwise, the original database schema is retained for subsequent processing.
  • 3.3 Decomposer: The Decomposer generates intermediate sub-questions and sub-SQL queries before producing the final SQL for a complex question.It uses previous sub-SQLs, filtered schema, and optional knowledge; simple questions can be answered directly.
  • 3.3 Decomposer: MAC-SQL adopts chain-of-thought prompting rather than iterative least-to-most prompting to avoid higher computational costs and stopping-criterion issues.Complex questions are progressively broken down from simpler sub-questions until the final SQL is obtained.
  • 3.4 Refiner: The Refiner diagnoses generated SQL for syntax, execution feasibility, and non-empty results, then uses external-tool error feedback to correct it.The paper reports reductions in syntax, schema-linking, and other simple errors.

4 SQL-Llama Model

SQL-Llama is an open-source Code Llama 7B model fine-tuned on curated multi-agent instructions, targeting GPT-4-like agent capabilities for Text-to-SQL.

  • The Agent-Instruct dataset contains 10,000 curated instruction examples covering three agent tasks and the BIRD and Spider data distributions.GPT-4 generated the multi-agent instruction data, which was filtered to remove incorrect SQL outputs.
  • SQL-Llama is based on Code Llama 7B and is supervised-fine-tuned with three intelligent-agent instruction datasets.The training targets capabilities including database simplification and question decomposition.
  • The supervised fine-tuning objective models task outputs conditioned on the question, selected schema, and task-specific context.The selected schema and intermediate SQL are identified for each task.
  • Balancing model complexity with performance and maintaining instruction-data quality were key challenges during training.The paper states that instruction-data quality directly affected model performance.
  • The authors open-source SQL-Llama and the instruction dataset to support further research on accessible database-querying models.The stated aim is to democratize access to high-performance language models for database-related tasks.

5 Experiments

Experiments evaluate MAC-SQL on Spider and the large-scale BIRD benchmark using execution-oriented metrics, baselines, ablations, few-shot settings, and error analysis. MAC-SQL achieves strong BIRD and Spider performance, while component removal and dataset errors reveal important dependencies and limitations.

  • Datasets: BIRD contains 95 large-scale databases spanning 37 domains, with up to 33.4GB of stored data, while Spider covers 200 databases and 138 domains.BIRD emphasizes massive real database content, external knowledge reasoning, and SQL efficiency; Spider tests adaptation to unfamiliar database structures.
  • Evaluation Metrics: Execution accuracy measures whether predicted and reference SQL queries produce identical results, whereas exact match compares all query clauses.Valid efficiency score evaluates the efficiency of valid SQL queries whose result sets align with ground truth.
  • Overall Performance: 59.59 execution accuracy makes MAC-SQL+GPT-4 state of the art on BIRD’s holdout test set, surpassing the second-best method by 2.18% on the test set.The method also surpasses all LLM-based baselines in execution accuracy and valid efficiency score on BIRD development and test sets.
  • Overall Performance: MAC-SQL+GPT-4 achieves the highest execution accuracy on the Spider development set, demonstrating generalization beyond BIRD.The experiments evaluate both development and test sets because Spider’s test set is openly available.
  • Ablation Study: 59.39% overall accuracy is achieved by the full MAC-SQL+GPT-4 model, including 65.73% on Simple, 52.69% on Moderate, and 40.28% on Challenging cases.Removing Selector reduces overall accuracy to 57.28%, including a 2.11% decrease, while removing Decomposer or Refiner also decreases accuracy across difficulty levels.
  • Discussion: Two-shot evaluation produces the highest performance across EX, VES, and EM on both BIRD and Spider, but cost constraints limit analysis to at most two shots.Full development-set testing consumes approximately 6 million Spider tokens and 10 million BIRD tokens through GPT-4.
  • Error Analysis: Gold Error is the most common error type at 30% in BIRD and 22% in Spider, while Semantic Correct accounts for 14% and 22%, respectively.Schema Linking Error occurs at 2% in BIRD and 8% in Spider, showing differing error distributions across datasets.

6 Related Work

Related work has advanced from prompt design and single-agent or domain-specific systems, but Text-to-SQL still lacks broadly generalizable multi-agent collaboration. Existing methods also show limitations when addressing complex datasets and diverse error types.

  • LLMs for Text-to-SQL: Recent LLM-based Text-to-SQL research emphasizes prompt design, in-context learning, supervised fine-tuning, and multi-stage refinement.These approaches aim to improve LLM reasoning and SQL generation through structured prompting and refinement.
  • Limitations of Existing Methods: Prior methods were evaluated mainly on Spider-family datasets, limiting evidence for generalization to more complex benchmarks such as BIRD.Some approaches also rely on Spider-specific difficulty classifiers and customized correction biases, restricting broader error coverage.
  • LLM-based Agents: LLM-agent research includes autonomous and customizable conversable agents that combine language models, human input, and tools.AutoGen is described as an open-source framework supporting agents operating in various modes with these combinations.
  • Research Gap: Multi-agent collaborative frameworks for Text-to-SQL remain under-explored compared with single-agent paradigms and domain-specific applications.The paper positions MAC-SQL as an approach in which specialized agents collectively interpret SQL queries.

7 Conclusion

The paper proposes MAC-SQL, a multi-agent framework for Text-to-SQL, together with the open-source SQL-Llama model. On BIRD, the framework reaches 59.59 execution accuracy and establishes a holdout-test-set SOTA at the time of writing.

  • Conclusion: MAC-SQL uses multi-agent collaboration to address challenges in Text-to-SQL tasks.The paper presents the framework as a novel approach and states that it can be expanded to support a broader spectrum of scenarios.
  • Conclusion: 59.59 execution accuracy is achieved on the BIRD benchmark’s holdout test set, establishing a new state of the art at the time of writing.The result is reported for the MAC-SQL framework together with the open-source SQL-Llama model.

Limitations

The paper identifies limitations in prompt optimization and in the scale of the fine-tuned open-source model. It suggests that larger models could further improve performance.

  • Agent prompts may not represent the most optimal choice and could benefit from further optimization.
  • The reported fine-tuning results focus on the 7B CodeLlama model.
  • Although SQL-Llama performs comparably, the authors believe larger models could improve its performance.

Ethics Statement

The paper reports no ethical concerns associated with its datasets, models, implementation, or resulting models.

  • The authors report no ethical concerns for the datasets used in the paper.
  • The models utilized in the paper are reported to have no associated ethical concerns.
  • The implementation and resulting models are also reported to have no associated ethical concerns.

A Implementation Details

The implementation combines conditional database selection, few-shot question decomposition, and iterative SQL refinement. Prompts use schema descriptions, evidence, constraints, and examples to guide JSON selection and valid SQLite generation.

  • Selector Agent: The Selector activates for large databases, determined by schema token count or database size criteria.For GPT-4-32k, a schema longer than 25k tokens is considered large.
  • Selector Agent: After filtering, each table retains at least six column names to reduce the risk of missing relevant columns.
  • Decomposer Agent: The Decomposer uses up to two few-shot examples and schema descriptions containing tables, columns, descriptions, and selected cell values.Numerical-only columns and unconventional values such as URLs and emails are ignored when examining cell values.
  • Refiner Agent: The Refiner corrects SQL syntax errors, schema illusions, and empty results, with at most three correction rounds.It does not further correct SQL that runs without errors and returns non-empty results, even when the query does not match the intended question.
  • Decomposer Agent: The Decomposer uses database schema, external evidence, and the question to break complex requests into subquestions for valid SQLite generation.Its examples show joins, aggregation, filtering, ordering, and subquestion-specific SQL generation.
  • Selector Agent: The Selector prompt discards unrelated schemas, prioritizes relevant columns, and returns JSON while preserving small tables and marking irrelevant tables for dropping.The instructions keep the top six columns per relevant table, include at least three tables, retain all tables with at most ten columns, and mark completely irrelevant tables as drop_all.

C Error Type Examples

Figure 6 presents eight major BIRD error types, with some cases simplified as shortcuts for clearer presentation. The examples are referenced as appearing on the next page.

  • The paper directs readers to Figure 6 on the next page for examples of error types.
  • Figure 6 presents eight major types of BIRD error cases.
  • Some Figure 6 cases use shortcuts to improve presentation.
Loading 2312.11242v6…