PaperlayerView exact revision on arXiv

Source-linked AI summary

FALCON: A Model and Dataset Agnostic Framework for Synthetic Data Generation for NL2SQL Pairs

Darian Lee, Shannon Rumsey, Jack St. Clair, Xinyi Tang, Aditya Bansal, Yuanming Shi

arXiv:2610.03625v1cs.CLcs.DBcs.LG

TL;DR

Existing synthetic NL-to-SQL data often overlooks ambiguity and produces queries simpler than real-world database requests require. FALCON combines seeded generation with filtering to produce complex, ambiguity-aware data using compact open models. Its models increasingly outperform Spider-trained models as query complexity rises, while the framework remains focused on complex queries and has limited ambiguity-detection recall.

  • Problem

    Existing synthetic NL-to-SQL methods often overlook ambiguity and produce oversimplified queries, while frontier-model APIs can make generation cost-prohibitive.

    Source: §1 · p. 1

  • Method

    FALCON combines reserved-word and persona seeding with ambiguity detection, alignment-based filtering, and prompt-based quality filtering to generate NL-to-SQL data using compact open models.

    Source: §1 · p. 1, §3.1.2 · p. 3, §3.1.3 · p. 4

  • Results

    Models trained on FALCON-generated data increasingly outperform Spider-trained models as query complexity increases, and the generated data exceeds existing benchmarks in SQL complexity and natural-language richness with human evaluation above 0.92 across models.

    Source: §7 · p. 8

  • Takeaways & Limitations

    FALCON’s local, model-agnostic design lets organizations generate high-complexity training data without relying on frontier APIs.

    Source: §1 · p. 1, §7 · p. 8

    FALCON targets complex queries, leaving simpler queries underrepresented and sole-data training at 0.8% EM on Spider; ambiguity detection has 28.6% recall.

    Source: §8 · p. 8

Abstract

from arXiv · show

Relational databases are among the most widely deployed forms of structured knowledge, and natural language access to them requires grounding language onto schema entities and relations while handling the ambiguity inherent in how people phrase requests. Existing synthetic NL-to-SQL data generation methods largely ignore this ambiguity and produce oversimplified queries that fail to prepare models for the complexity of real-world structured knowledge access. We present FALCON, a framework that generates realistic, ambiguity-aware NL-to-SQL data matching the complexity of challenging real-world benchmarks, at low cost using compact open models. Our approach combines reserved-word SQL seeding and persona-based prompting to generate structurally complex queries, while alignment-based filtering preserves difficulty by distinguishing genuinely incorrect examples from complex but valid queries. Human evaluation confirms consistent high quality across model sizes, and our generated data exceeds existing benchmarks in both SQL complexity and natural language richness. Difficulty-stratified analysis shows models trained on FALCON data increasingly outperform baseline-trained models as query complexity increases, validating our pipeline's success in generating challenging training data. When combined with a small proportion of existing benchmark data, mixed training recovers performance on simpler queries while preserving these advantages on complex ones. The model- and database-agnostic design enables organizations to generate high-complexity NL-to-SQL training data locally without external APIs.

1 Introduction

Natural-language access to databases requires grounding requests in schema entities and relations while handling diverse phrasing and ambiguity. FALCON targets gaps in existing synthetic data with complex, ambiguity-aware NL-to-SQL generation using compact open models.

1 Introduction

  • Real-world database requests combine diverse linguistic styles, ambiguity, and complex SQL operations that existing benchmarks underrepresent.NL-to-SQL therefore requires grounding language references in specific schema entities and relations.Source: §1 · p. 1
  • Synthetic generation faces a trade-off between costly frontier-model APIs and oversimplified queries that miss production-level complexity.Human-authored challenging datasets such as BIRD are limited in scalability and adaptability to new domains.Source: §1 · p. 1
  • FALCON targets BIRD-level query complexity, realistic linguistic diversity including ambiguity, and low-cost generation with compact open models.It is model- and database-agnostic and designed to generate domain-specific data for internal schemas without sending sensitive information to external APIs.Source: §1 · p. 1

2 Related Work

Prior NL-to-SQL work leaves gaps between academic benchmarks and production needs, including complex queries, diverse language, and ambiguity. Synthetic-data approaches also face cost, scale, and coverage constraints that motivate a unified, cost-efficient framework.

2.1 The Academic-Industry Gap in NL-to-SQL

  • WikiSQL focused on single-relation queries, while Spider and BIRD added cross-domain complexity and professional-domain examples; production gaps remain around ambiguity and enterprise-scale databases.Recent work identifies informal, underspecified queries as part of the benchmark-to-production gap.Source: §2.1 · p. 2

2.2 The Cost Barrier to Industry Deployment

  • Frontier-model API costs and human annotation undermine local deployment, while fine-tuning smaller models requires substantial high-quality training data.Smaller-model fine-tuning is described as offering cost and privacy advantages (Li et al., 2025).Source: §2.2 · p. 2

2.3 Synthetic Data Generation for NL-to-SQL

  • Source2Synth, OmniSQL, and SQLForge each address parts of synthetic NL-to-SQL generation but retain limits in query scope, ambiguity handling, model size, or API cost.Source2Synth focuses on single-table queries, OmniSQL lacks ambiguity handling, and SQLForge relies on closed-source GPT-4.Source: §2.3 · p. 2

2.4 Handling Query Ambiguity

  • Most synthetic datasets omit ambiguous pairs; existing ambiguity work either focuses on schema-level ambiguity or leaves broader linguistic ambiguity underrepresented.Ambrosia identifies scope, attachment, and vagueness as database-independent ambiguity types (Saparina and Lapata, 2024).Source: §2.4 · p. 2
  • The paper motivates a unified, cost-efficient framework that handles query complexity diversity and ambiguity.The stated gaps are those left by existing methods addressing specific facets of NL-to-SQL generation.Source: §2.4 · p. 2

3 Pipeline

FALCON generates SQL and natural-language questions through seeded prompting, then filters examples for ambiguity, alignment, and quality. The pipeline preserves complex valid pairs while adding alternative interpretations for genuinely ambiguous questions.

3 Pipeline

  • FALCON uses a multistage pipeline that generates SQL and NL, detects ambiguity, filters misaligned pairs, and applies a final quality check.Its components are evaluated for complexity, richness, and retention of high-quality examples.Source: §3 · p. 2, Figure 1 · p. 3

3.1 SQL Query and NL Question Generation

  • SQL-then-NL generation empirically increases both SQL and natural-language complexity.Queries are generated from a schema and executed with up to three validation attempts; the validated query and schema then condition four NL candidates, from which a judge selects one.Source: §3.1 · pp. 2–3

3.1.1 Seeding Techniques

  • Persona and reserved-word seeding encourage varied questions and SQL aligned with human-authored query distributions.Persona prompts specify user goals and examples; reserved-word counts are sampled from BIRD, with adherence r = 0.6 and median deviation of 3 words, while ablations find improved SQL complexity.Source: §3.1.1 · p. 3

3.1.2 Ambiguity Detection

  • A DistilBERT model fine-tuned on Ambrosia data assigns continuous ambiguity scores, filtering extremely ambiguous questions and prompting alternative SQL interpretations when the score exceeds 7.Ambiguity is defined as an NL question that can be correctly interpreted as several distinct SQL queries; prompt-based filters validate genuine ambiguity and alternatives.Source: §3.1.2 · p. 3

3.1.3 Alignment-Based Filtering

  • Alignment filtering uses a model trained on 9,000+ BIRD-derived examples, half systematically corrupted, to reject plausible but schema-ungrounded NL-SQL pairs.The hybrid metric combines semantic similarity (40%) and schema entity matching (60%) with corruption penalties, and correlates with human judgments at r = 0.70.Source: §3.1.3 · p. 4

3.2 Multi-Interpretation Generation and Filtering

  • Ambiguous examples can receive a validated alternative interpretation, and all pairs then undergo a prompt-based check for valid interpretation, sensible non-redundant SQL, and human-like NL.The ambiguity-specific alternative generation has three attempts; prompt-based filtering complements the reward model, whose specificity is 71%.Source: §3.2 · p. 4

4 Evaluation

The evaluation defines measures of natural-language and SQL difficulty, uses human annotations to assess quality and alignment, and tests fine-tuned models across datasets and query difficulty levels.

4.1 Difficulty Metrics

  • NL richness combines linguistic-complexity features, while SQL complexity weights parsed query structures including joins, subqueries, operators, and splits.The NL features include tokens before the main verb, constituents per word, subordinate clauses, and coordinate clauses.Source: §4.1 · p. 4

4.2 Human Evaluation

  • Four annotators independently scored disjoint examples for NL quality, SQL quality, and alignment, resolving disagreements through structured discussion.Each annotator scored 100 examples using a rubric that distinguishes failure modes such as inefficient queries and misaligned pairs.Source: §4.2 · p. 5

4.3 Downstream Task Evaluation

  • Downstream evaluation fine-tunes Qwen 2.5 3B and 1.5B on synthetic, Spider, or mixed data and evaluates on synthetic and Spider test sets.Performance is measured with Exact Match, Fuzzy Score, and Soft Execution Accuracy, including difficulty-stratified analysis.Source: §4.3 · p. 5

5 Ablations

The ablations select generation and filtering choices that balance SQL complexity, natural-language richness, and example quality. The ambiguity pipeline is deliberately conservative, favoring valid alternative queries over detecting every ambiguous question.

5.1 Generation Order and Component Ablations

  • SQL-then-NL outperforms the reverse order; reserved-word seeding improves SQL complexity, persona seeding improves NL richness, and the Compare Judge leads on NL richness and SQL complexity.The final pipeline uses SQL-then-NL generation, interleaved seeding, and the Compare Judge.Source: §5.1 · p. 5, Table 1 · p. 5

5.2 Filtering Ablation

  • 94% F1 and 71% specificity at threshold 0.6; the reward model reduces misaligned examples from 16% to 4% while retaining 92.81% of correct examples.Filtered SQL and NL complexity metrics match BIRD benchmarks, indicating that filtering preserves difficulty.Source: §5.2 · p. 5, Table 2 · p. 6

5.3 Ambiguity Ablation

  • The ambiguity pipeline reaches 94% accuracy and 98.9% specificity, but only 28.6% recall on human-annotated examples.This conservative behavior is intentional: false positives produce noisy multi-interpretation pairs, while false negatives omit optional enhancements.Source: §5.3 · p. 6, Table 3 · p. 6

6 Results

FALCON-generated data receives high human quality scores and exceeds original datasets in SQL complexity and natural-language richness. Models trained on FALCON data gain an increasing advantage on harder queries, while a small Spider-data mix benefits easier queries; error analysis and cost results qualify these findings.

6 Results

  • Human evaluation scores for SQL, natural language, and alignment all exceed 0.92 across models.Evaluation used four annotators, each scoring 100 examples.Source: §6 · p. 6, Table 4 · p. 6
  • FALCON-generated datasets outperform their original datasets in SQL complexity and natural-language richness, despite using the same table information.The exception is SQL complexity for the WikiSQL-seeded pipeline; FALCON data is otherwise comparable to or more complex than human-generated BIRD data.Source: §6 · p. 6, Table 5 · p. 6
  • FALCON-trained models outperform Spider-trained models increasingly as query complexity rises, with the advantage growing in harder quartiles.Figure 4 presents soft execution accuracy by complexity; Table 7 provides cross-dataset evaluation context.Source: §6 · p. 6, Figure 4 · p. 7, Table 7 · p. 7
  • With 5% Spider data, mixed training achieves 27% EM on Spider and 13.6% on FALCON.Figure 5 indicates that real-data seeding primarily benefits easier queries, while FALCON-only training retains advantages on harder queries.Source: §6 · p. 6, Figure 5 · p. 7

6.1 Error Analysis

  • About one third of flagged errors are acceptable variations, while spurious predicate hallucination and result-boundary errors are the most common genuine failures.The analysis attributes those genuine failures to complex-query overrepresentation and recommends seeding with 5% existing data to represent simpler queries.Source: §6.1 · p. 7
  • Schema-entity confusion is attributed more to context pressure from condensed schema information than to the NL-to-SQL pairs.The model must resolve table and column references under significant context pressure.Source: §6.1 · p. 7

6.2 Cost

  • Generating 5,500 examples is estimated to cost about $20 with an open model on a commodity GPU, versus about $326 at GPT-5.5 pricing.The estimate uses about five LLM calls per example across five pipeline stages; the paper notes API use also entails privacy implications.Source: §6.2 · p. 8

7 Conclusion

FALCON is a model- and database-agnostic framework for generating complex, linguistically rich NL-to-SQL data at low cost with compact open models. Its generated data exceeds existing benchmarks on complexity and richness while maintaining high human-evaluation quality, and models trained on it outperform Spider-trained models increasingly as query complexity rises.

7 Conclusion

  • FALCON combines reserved-word seeding, persona-based prompting, and alignment-based filtering to generate high-complexity NL-to-SQL data with compact open models.The framework is model- and database-agnostic.Source: §7 · p. 8
  • FALCON-generated data exceeds existing benchmarks in SQL complexity and natural-language richness while maintaining human-evaluation scores above 0.92 across models.The framework also targets ambiguity and diverse language reflecting real-world linguistic challenges.Source: §7 · p. 8
  • Models trained on FALCON-generated data outperform Spider-trained models as query complexity increases.The conclusion presents FALCON as a template for training data for natural-language interfaces to structured knowledge.Source: §7 · p. 8

8 Limitations

FALCON targets complex queries, leaving simpler queries underrepresented; this limits standalone Spider performance and makes real-data supplementation necessary for broad applicability.

8 Limitations

  • FALCON targets the complex end of the query distribution, underrepresenting simpler queries; models trained only on FALCON achieve 0.8% EM on Spider and need 5–10% real-data supplementation for broad applicability.Ambiguity detection has 98.9% specificity but 28.6% recall, missing many genuinely ambiguous queries.Source: §8 · p. 8
  • FALCON currently supports SQLite only; other SQL dialects require dialect-specific reserved-word distributions and syntax constraints.Evaluation is limited to public benchmarks, and production databases may contain complexity dimensions not captured academically; downstream evaluation was also limited to compact models.Source: §8 · p. 8

Ethical Considerations

Generated data inherits domain coverage from its seed databases, which may carry domain-specific biases.

Ethical Considerations

  • Generated data inherits domain coverage from Spider and WikiSQL and may therefore carry domain-specific biases.The database-agnostic design can generate data for organizations’ specific use cases.Source: Ethical Considerations · p. 8

A.1 Detailed Pipeline Overview

The pipeline seeds SQL generation for complexity and linguistic diversity, then generates and filters natural-language–SQL pairs through executability, ambiguity, alignment, and quality checks.

A.1 Detailed Pipeline Overview

  • For each schema, the pipeline chooses reserved-word or persona seeding with equal probability; reserved-word counts are sampled from BIRD’s training distribution to target query complexity.Persona seeding adds user goals and example questions, and influences both SQL and NL generation; reserved-word seeding influences SQL only.Source: §A.1 · p. 10
  • Generated SQL is executed against the database, and queries that fail execution are discarded.The walkthrough’s reserved-word example selects states by license-plate background color.Source: §A.1 · p. 11
  • The validated SQL and schema produce four candidate NL questions; persona prompts shape voice and terminology, and a judge selects for naturalness, accuracy, and completeness.Without persona seeding, the pipeline uses a standard chain-of-thought prompt.Source: §A.1 · p. 11
  • A DistilBERT ambiguity score sends questions scoring 7 or above to a prompt filter that checks Scope, Attachment, Entity Vagueness, or Broadness.Broadness captures underspecification, such as “best player” without specifying whether best means most points or highest rating.Source: §A.1 · p. 11
  • For confirmed ambiguity, the pipeline generates and validates a distinct alternative SQL interpretation, allowing up to three attempts before retaining the original as an unambiguous pair.A failed ambiguity confirmation causes the example to be discarded; confirmed alternatives must be executable and semantically distinct.Source: §A.1 · p. 11
  • Questions below the ambiguity threshold proceed to alignment filtering, which retains scores of 0.6 or higher; ambiguous examples bypass that model because it was not trained to handle ambiguity.Every path then passes a final prompt filter checking NL–SQL validity, SQL form, and whether the question sounds like something a real user would ask.Source: §A.1 · pp. 11–12

A.2 Error Analysis

The appendix analyzes model errors and describes how FALCON’s data, filtering, and evaluation behave across complexity, cost, alignment, ambiguity, and training settings.

A.2 Error Analysis

  • The cited taxonomy includes acceptable variation, NL misinterpretation, schema entity confusion, result boundary error, spurious predicate hallucination, and cross-row constraint flattening, with examples illustrating these categories.Source: §A.2 · p. 12

A.3 Complexity and Performance

  • Across Qwen-2.5-3B through Llama-3.3-70B, generated data has similar SQL complexity and NL richness; filtering largely preserves complexity while modestly improving NL richness.The alignment filter does not systematically shift complexity toward BIRD’s distribution despite training on BIRD-derived pairs.Source: §A.3 · p. 12, Table 9 · p. 13

A.4 Cost Analysis

  • At estimated costs, generating 5,500 kept examples takes about $326 through an API versus about $20 locally, a 16× cost reduction.The estimate assumes 5,265 input and 784 output tokens per example, an 84% retention rate, and local Qwen-2.5-3B inference on one commodity GPU.Source: §A.4 · p. 13

A.5 SQL Complexity Distribution Analysis

  • FALCON-generated data is more complex than Spider data, with bimodal peaks and substantial density above 0.5; Spider-trained models struggle on FALCON data while FALCON-trained models generalize less well to easier queries.The distributions differ most in the 0.4–0.6 complexity range, which is underrepresented in Spider.Source: §A.5 · p. 13, Figure 9 · p. 20

A.6 Alignment Model

  • On 101 examples, alignment scores correlated with human annotations at Pearson r = 0.70; ModernBERT trained on this dataset achieved MSE 0.0203 and MAE 0.1129.Corrupted examples incur an additional 10% penalty per corruption.Source: §A.6 · p. 13

A.7 Human Evaluation Across Seed Databases

  • Human evaluation used four annotators on disjoint subsets of 100 examples each, with disagreements resolved through group discussion.The scoring rubric maps ratings 1, 2, and 3 to 0, 0.5, and 1, respectively.Source: §A.7 · p. 14, §A.6 · p. 14

A.8 Ambiguity Model Training.

  • Ambiguity scores are mapped from the number of distinct interpretations, with unambiguous questions assigned 0 and ambiguous questions mapped to 5–10; scores above 7 trigger alternative-query generation.The scorer’s training data includes a canonical query, additional valid interpretations, and an ambiguity label.Source: §A.8 · p. 14

A.12 Qwen 2.5-1.5B Training and Results

  • Qwen 2.5-1.5B results compare Spider-only, FALCON-only, and mixed training on both Spider and FALCON test sets, showing training-distribution effects.Mixed variants use 5% or 10% Spider data with the remainder from the pipeline; the supplied table caption gives no individual metric values.Source: Table 14 · p. 16

Related papers

Loading 2610.03625v1…