PaperlayerView exact revision on arXiv

Source-linked AI summary

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation

Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe, Amine Mhedhbi

arXiv:2609.08950v1cs.DBcs.AI

TL;DR

Text-to-SQL evaluation lacks realistic, reproducible coverage of enterprise workloads and binary metrics obscure partial correctness. SQLMorph addresses these gaps with controlled query mutations and fine-grained execution metrics, revealing degradation and failure differences that standard accuracy obscures.

  • Problem

    Text-to-SQL evaluation lacks realistic, reproducible coverage of enterprise workloads and binary metrics obscure partial correctness and failure modes.

  • Method

    SQLMorph uses algorithmic Join Query Expansion, controlled Textual Query Augmentation, and Execution Precision/Recall with F1 to stress systems and diagnose result errors.

  • Results

    JQE causes an Execution Accuracy drop of up to 20%, while TQA causes a drop of up to 17%; fine-grained metrics distinguish errors that EX scores identically.

  • Takeaways & Limitations

    SQLMorph provides a reproducible and diagnostic evaluation framework better aligned with enterprise requirements.

Abstract

from arXiv · show

Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven by large language models (LLMs), evaluation remains a major bottleneck: public benchmarks fail to capture the complexity of enterprise schema, while building private evaluation sets is costly and nondeterministic, making evaluation results difficult to reproduce. To address this issue, we present SQLMorph, a framework for Text-to-SQL evaluation via query mutation. SQLMorph introduces two techniques to automatically generate and expand evaluation sets: Join Query Expansion (JQE), which systematically increases structural complexity through valid join additions, and Textual Query Augmentation (TQA), which generates controlled natural language perturbations to assess robustness to linguistic variation. JQE and TQA create targeted choke points to challenge specific system components. When applied to state-of-the-art systems, JQE increases query coverage and reveals accuracy degradation as the number of joins grows. Meanwhile, TQA shows that linguistic brittleness induced by heavy abbreviation can reduce accuracy by up to 17%. Beyond evaluation sets, SQLMorph introduces a family of execution-level metrics that address the limitations of current binary measures, such as Execution Accuracy. We define Execution Precision (EXP) and Execution Recall (EXR) to quantify the fraction of correct and recovered results, respectively, and combine them via F1 for unified scoring. Our experiments show that these relaxed metrics enable fine-grained analysis of over- and under-prediction, revealing differences across systems that binary metrics obscure. Together, SQLMorph's query mutation and fine-grained metrics support debugging and better align Text-to-SQL evaluation practices with real-world deployments.

I. INTRODUCTION

SQLMorph addresses static, costly, and coarse Text-to-SQL evaluation by mutating queries into targeted stress tests and adding fine-grained execution metrics. Its components expose weaknesses in join complexity, naming robustness, and over- or under-prediction.

  • Motivation: Public benchmarks underrepresent enterprise queries with multiple tables, intricate joins, and domain-specific terminology, while private evaluation is costly and difficult to reproduce.These limitations can produce overly optimistic capability assessments and fragmented evaluation practices.
  • Framework: SQLMorph reframes evaluation as a dynamic process that generates valid query variants to stress specific system components.The framework targets static NL–SQL pairs and coarse binary metrics through controlled query mutation.
  • Query mutation: JQE increases SQL join complexity through valid structural expansions, while TQA perturbs schema and natural-language naming to test linguistic robustness.JQE can reduce Execution Accuracy by up to 20%, while TQA can reduce it by up to 17%.
  • Fine-grained metrics: Execution Precision, Execution Recall, and F1 complement binary Execution Accuracy by separating correct predicted results, recovered ground-truth results, and their unified score.The metrics diagnose over-prediction and under-prediction that binary accuracy obscures.
  • Reproducibility: SQLMorph releases its framework and experimental scripts as open-source software to support reproducible evaluation.The framework is intended for offline development and evaluation where ground-truth queries and outputs are available.

III. EXPERIMENTAL SETUP

The experiments evaluate SQLMorph on BIRD’s development set using representative open-source Text-to-SQL systems and a standard binary execution metric. The setup emphasizes realistic, cost-conscious evaluation and reproducibility.

  • Evaluation setting: The experimental setup targets a realistic medium-token-budget setting and treats parameter defaults as cost-conscious rather than universally optimal.The appropriate trade-off depends on evaluation goals and available resources.
  • Dataset: The experiments use BIRD, containing 12,751 NL–SQL pairs across 95 databases and 37 domains, with all analyses conducted on its development split.The development split contains 1,534 queries over 11 databases.
  • Reproducibility: SQLMorph’s model manager supports pinned backends and local open-source models, while these experiments use GPT-4o and text-embedding-3-small.Pinning model versions is intended to support reproducibility and control model drift.
  • Metric: Execution Accuracy assigns 1 when predicted and ground-truth output relations exactly match and 0 otherwise.Later SQLMorph metrics relax this binary judgment to capture partial correctness and support diagnosis.
  • Systems: The evaluation uses three open-source systems that differ in architecture, retrieval, correction, and query-decomposition strategies.The systems are DIN-SQL, MAC-SQL, and CHESS, with CHESSIR+CG+UT selected as the strongest reported CHESS configuration.

IV. JOIN QUERY EXPANSION

Join Query Expansion deterministically creates harder Text-to-SQL queries by adding valid, nonredundant joins while preserving semantic intent. Its graph-based design supports structural diversity, user control, and reproducible query generation.

  • Motivation and goal: JQE addresses underrepresented enterprise join complexity by deterministically adding valid joins to existing NL–SQL pairs.Repeated expansion can generate queries with up to n additional joins from a small seed subset.
  • Design principles: JQE’s core expansion is algorithmic and graph-driven, with an LLM used only to regenerate natural language after SQL expansion.Users can control join complexity, table centrality, query budget, and structural pattern coverage.
  • Validity constraints: Added joins must introduce genuine structural complexity rather than being implied transitively by existing joins.This requirement ensures each expansion contributes at least one new explicit join condition.
  • Pipeline: JQE retains structurally diverse variants through a multi-stage process covering candidate identification, join-condition enumeration, pruning, SQL synthesis, and natural-language regeneration.The pipeline is illustrated as a multi-stage join query expansion process.
  • Inputs: The method uses a schema graph whose table edges carry admissible equi-join predicates derived from key compatibilities.Seed queries supply the existing join query graph, and user preferences bound the expansion process.
  • Expansion: JQE expands a seed query by adding an unseen table and admissible join predicates while retaining variants that satisfy semantic and diversity constraints.The corresponding natural-language query is rewritten to reflect the added table.

2) Output:

JQE constructs expanded SQL queries by enumerating valid join additions, removing redundant or structurally equivalent variants, and regenerating aligned natural-language queries.

  • Pipeline: JQE enumerates candidate tables and non-empty join-condition combinations before pruning redundant joins.Candidate tables are adjacent to existing query tables; redundant combinations are removed when transitive cycles make them unnecessary.
  • Pipeline: Diversity-aware pruning ranks expansions by user preferences and retains up to n queries per unique join-query-graph pattern.JQG isomorphism determines structural uniqueness, preserving variety while controlling cost.
  • Pipeline: Expanded SQL is synthesized programmatically by injecting tables and predicates, with optional predicates or projections.The generated SQL extends FROM/JOIN and ON/WHERE clauses.
  • Pipeline: A fixed few-shot LLM prompt regenerates natural-language queries to reflect added joins while preserving original intent.The pipeline keeps prompts fixed for reproducibility.
  • Output: The retained expansion set contains queries non-isomorphic to the input evaluation set or previously retained queries.The default setting retains one query per unique JQG.

F. Experimental Analysis

On BIRD dev, JQE generated a substantially larger executable query set with more connected and cyclic join structures, while most regenerated natural-language queries remained faithful.

  • JQE output: 6,873 queries were generated, producing a set 4.5× larger than BIRD dev, and all queries executed with non-empty results.Only 58 queries formed the final expansion set; 6,815 were pruned as isomorphic duplicates.
  • Alignment: 74.1% (43/58) of expanded natural-language–SQL pairs fully captured the intended meaning.13.8% added context or noise, while 12.1% lost the intent and required correction.
  • Structural distribution: Average JQG node degree rose from 0.82 in BIRD dev to 1.35 in the generated set and 1.80 in the expansion set.The increase reflects the default preference for greater join complexity.
  • Structural distribution: JQE generated 58 new join patterns while increasing average degree from 0.82 to 1.35 and cyclicity from 0.27% to 4.31%.The final expansion set showed even more pronounced structural effects.

2) Accuracy after JQE:

JQE exposes performance sensitivity to structural complexity: expanded queries degrade system accuracy, and the degradation becomes stronger as join counts increase.

  • Accuracy after JQE: 24.14% of DIN-SQL cases and 18.97% of CHESS cases degraded from correct originals to incorrect expanded queries.Degradations were more frequent than improvements for CHESS and DIN-SQL.
  • Join complexity vs. performance: Accuracy declined monotonically with join count, reaching ≤15% at higher join counts across the sampled query groups.The groups covered JQGs with 2–5 vertices and 1–4 edges.
  • Join complexity vs. performance: Manual inspection attributed most failures to join-structure changes, with fewer failures involving DISTINCT, projection, or GROUP BY mismatches.A fixed-context control experiment was used to separate join complexity from prompt noise.
  • Join complexity vs. performance: 54.90% of expanded-query failures recovered at njoin=1, falling to 24.05% at njoin=4 when the original NL query replaced the expanded one under fixed context.Recovery decreased as join count increased.
  • Accuracy after JQE: Across CHESS, DIN-SQL, and MAC-SQL, execution accuracy dropped as join complexity increased.The results identify join-heavy queries as a key failure mode.

B. Design Principles

TQA is designed to isolate lexical robustness by making schema and query references less natural while preserving SQL execution, semantics, values, and clauses.

  • Design principles: TQA transforms identifiers while keeping values, constants, and clauses fixed so performance changes reflect textual rather than semantic variation.The resulting schemas and queries remain executable and semantically faithful.
  • Design principles: TQA starts from an NL–SQL pair with accompanying evidence and relevant schema, then applies configurable naturalness transformations.Users can target NL, schema, or both, and choose the aggressiveness of denaturalization.
  • Design principles: A naturalness classifier assigns identifiers to regular (N1), low (N2), or least (N3) levels before transformation.TQA uses the SNAILS classifier and records a confidence score.
  • Design principles: The default setting transforms N1 and N2 identifiers into N3 using a temperature-0 LLM invocation with a fixed seed.An example transformation is WaterTemperature→WtTp.
  • Design principles: TQA alters schemas, ground-truth SQL, and NL queries by rewriting schema mentions while avoiding values, SQL keywords, functions, and constants.Schema renames are applied with ALTER, and NL rewriting is restricted to schema mentions.

D. Experimental Setup and Analysis

TQA evaluates robustness by making schema and natural-language references less natural while preserving semantics and executability. Across settings, these perturbations degrade system performance and expose retrieval and schema-linking failures.

  • Experimental setup: TQA evaluates four settings that independently or jointly rename natural-language and SQL-side schema references while preserving semantics.The settings are O/O, L/O, O/L, and L/L.
  • System-level analysis: All systems degrade under schema and natural-language de-naturalization, with DIN-SQL showing the largest degradation, followed by MAC-SQL and CHESS.The comparison uses queries that were correct under O/O.
  • Stage-level analysis: −69.1%: Full-Schema SLR drops under L/O, while SCSL and TCSL decline by −3.3% and −6.0%, respectively.FPR generally decreases, with a slight increase for TCSL.
  • Stage-level analysis: Under O/L, SLR decreases by −3.1%, −5.2%, and −2.0% for Full-Schema, SCSL, and TCSL, respectively.FPR rises for SCSL and TCSL by +4.2% and +2.7%.
  • Qualitative analysis: Schema retrieval frequently misses correct columns such as atom_id and connected.atom_id under O/L and L/O, causing downstream generation and execution errors.TQA therefore stress-tests retrieval, schema linking, and query reformulation.

VI. FINE-GRAINED EVALUATION METRICS

SQLMorph replaces binary Execution Accuracy with relaxed execution metrics that quantify partial result correctness. Execution Precision and Recall distinguish over-prediction from under-prediction, while F1 summarizes both.

  • Motivation: Execution Accuracy assigns 1 only to exact output-relation matches and 0 otherwise, obscuring partial correctness and error severity.Near-correct and irrelevant outputs receive the same score.
  • Metric definitions: Execution Precision measures the fraction of predicted cells that are correct, while Execution Recall measures the fraction of ground-truth cells recovered.Their harmonic mean, F1, provides a unified summary.
  • Matching regimes: The evaluator can use exact-column, semantic-column, or no-column matching before comparing rows and cells.Semantic-column matching uses descriptors, embeddings, type compatibility, and a similarity threshold.
  • Row/cell matching: Row and cell matching proceeds through exact matches followed by greedy partial matching based on the fraction of equal cells.Outputs are sorted before matching to make equivalent relations deterministic.
  • Extra-column handling: Extra predicted columns can either be penalized, reducing precision, or ignored when matched columns already contain the required data.Ignoring extras is unavailable under no-column matching.

C. Experimental Analysis

Controlled single-error SQL mutants test whether granular metrics distinguish different failure modes. Unlike EX, which falls to 0% for every mutant, EXP, EXR, and F1 reflect error severity and direction.

  • Controlled error sensitivity: Each benchmark mutant applies exactly one atomic SQL change to a BIRD development-set ground-truth query.The mutations target SQL components whose effects on rows, columns, values, and metrics are summarized in Table IX.
  • Controlled error sensitivity: For every mutant, EX drops to 0%, whereas EXP, EXR, and F1 spread errors across a wider range.The relaxed metrics therefore distinguish mild from severe errors.
  • Row-count mutants: Limit increases and less restrictive filters add rows, sharply lowering EXP while leaving EXR at 1; reductions in rows lower EXR while EXP remains nearly unchanged.These patterns separate over-prediction from under-prediction.
  • Schema mutants: projection_drop lowers EXR, add_star_wildcard mainly lowers EXP, and distinct_toggle lowers both metrics.Under semantic-column matching, the penalty for extra semantically related columns is smaller.
  • Join mutants: join_break causes one of the largest EXP drops, while join_type_to_left produces a smaller and more balanced degradation.Partial-cell matching helps mainly when predicted rows remain close but nonidentical, such as after join_break.

2) System-Level Comparison:

Fine-grained metrics preserve the system ranking on shared failures while revealing how closely each prediction approaches the correct answer. A qualitative example shows that binary EX can treat near-correct outputs as simply wrong.

  • System-level comparison: 410 of 1,534 BIRD examples are shared failures where all three systems have EX = 0.Binary EX provides no ranking or behavioral information for these predictions.
  • System-level comparison: The ranking CHESS > DIN-SQL > MAC-SQL remains unchanged under EX and the fine-grained metrics.EXP and EXR nevertheless reveal meaningful differences within shared failures.
  • System-level comparison: CHESS recovers nearly a quarter of ground-truth cells on shared failures, whereas MAC-SQL recovers about 15%.CHESS has higher EXR, while DIN-SQL shows a more balanced precision-recall trade-off.
  • Qualitative example: Both systems in the qualitative example recover the correct filtering condition but differ in projected attributes.DIN-SQL projects five attributes and MAC-SQL four, introducing extra projected columns.
  • Qualitative example: High EXR but low EXP indicates that both systems recover nearly all expected rows while being penalized for unnecessary columns.Binary EX treats these near-correct outputs as equally wrong.

VII. RELATED WORK AND CHALLENGES

Text-to-SQL evaluation is constrained by limited benchmark complexity, enterprise schema variation, and reproducibility challenges. SQLMorph addresses these gaps through deterministic query mutation and fine-grained execution metrics that expose partial correctness and failure behavior.

  • Benchmark and schema challenges: Public benchmarks have limited scale, label fidelity, compositional diversity, and join complexity, reducing their coverage of enterprise workloads.Recent studies also report sharp accuracy declines as join complexity increases.
  • Benchmark and schema challenges: Enterprise databases contain many interrelated tables, heterogeneous sources, and cryptic identifiers, creating ambiguity because multiple SQL queries can preserve the same natural-language intent.This schema complexity differs substantially from public datasets.
  • Reproducibility: Evaluation reproducibility is hindered by dataset splits, prompt design, sampling choices, and private settings; SQLMorph fixes prompts, seeds, and sampling parameters and validates variants through execution checks.The framework is also intended for private evaluation where schemas and workload traces cannot be shared.
  • Schema naming and linking: TQA complements schema-linking research by denaturalizing identifiers and mentions to stress robustness to lexical mismatch, schema linking, and retrieval.Prior work shows that schema naming quality affects Text-to-SQL performance.
  • Fine-grained metrics: Binary Execution Accuracy collapses failures into one bit, whereas EXP and EXR separate over-prediction from under-prediction and quantify partial correctness.System-level comparisons on shared-failed BIRD queries use EXP, EXR, and F1 to reveal differences in failure behavior that EX misses.
  • SQLMorph’s perspective: SQLMorph frames evaluation as reproducible, controlled stress testing through join expansion, naming variation, and metrics that expose execution errors obscured by binary EX.Its metrics also differ from proxy or composite rewards that do not explicitly decompose over- and under-prediction.
Loading 2609.08950v1…