PaperlayerView exact revision on arXiv

Source-linked AI summary

Why, Where, How: Taxonomy-guided Error Grounding for Code Repair in NL2SQL

Suchan Lee, Woomin Song, Hwanjo Yu, Sangwoo Mo

arXiv:2610.05060v1cs.AI

TL;DR

Execution alone does not establish that generated SQL is semantically correct, and an error taxonomy identifies why a query is wrong without indicating where or how to edit it. TEG grounds supplied diagnoses into masking and edit instructions; across tested model sizes and thinking modes, it outperforms evaluated baselines on single-error queries.

  • Problem

    An error taxonomy says why a SQL query is wrong but not where to look or how to change it, and executable queries can still return incorrect results.

    Source: §1 · p. 1

  • Method

    TEG maps each supplied error type to construct classes to reconsider and an edit instruction, using masks when applicable and sequential re-grounding for multiple annotations.

    Source: §1 · p. 2

  • Results

    Across evaluated model sizes and thinking modes, TEG beats all evaluated baselines on single-error queries; on Qwen2.5-7B, it reaches 47.3 single-error execution accuracy and 37.0 overall.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support using supplied error types to construct correction inputs through masking and edit instructions.

    Source: §5 · p. 9

    Evaluation uses one SQL benchmark, predicted-type evaluation covers only single-error queries, and TEG requires diagnoses and database access while adding inference cost.

    Source: §5 · p. 9

Abstract

from arXiv · show

SQL queries that large language models write from natural language questions can execute successfully yet produce incorrect results, so execution alone does not reveal what to fix. An error taxonomy says why the query is wrong, but not where to look or how to change it. Existing methods can guide SQL correction through feedback, error reports, or generated plans alongside an unmasked query. We introduce TEG(Taxonomy-guided Error Grounding), which turns a supplied diagnosis into a structured correction input for natural language-to-SQL (NL2SQL) correction. Type-specific rules map each error type to construct classes to reconsider and an edit operation to request. TEG masks the selected constructs in the query when applicable and states that operation in an edit instruction. TEG generates candidate corrections from this input, uses execution feedback to guide candidate selection, and repeats the process one annotation at a time for queries with several errors. On NL2SQL-BUGs, TEG reaches 47.3 single-error execution accuracy and 37.0 overall with Qwen2.5-7B-Instruct. Across the model sizes and thinking modes evaluated in the main comparison, TEG outperforms all evaluated baselines on single-error queries, even when the baselines receive the same error-type annotations. With predicted types, TEG stays above direct LLM correction and ErrorLLM on single-error queries.

1 Introduction

Execution can leave semantic SQL errors unresolved, while a taxonomy diagnosis says why a query is wrong but not where or how to edit it. TEG turns that diagnosis into a structured correction input and improves correction on NL2SQL-BUGs.

1 Introduction

  • TEG generates multiple grounded candidates and uses execution feedback to guide selection, processing errors one annotation at a time.The process edits existing constructs before inserting new ones and re-grounds the updated query.Source: §1 · p. 2
  • 47.3 single-error execution accuracy and 37.0 overall on Qwen2.5-7B-Instruct; TEG also beats evaluated baselines across tested model sizes and thinking modes.The single-error lead holds when baselines receive the same error-type annotations; TEG also leads every baseline on multi-error queries.Source: §1 · p. 2
  • TEG maps a supplied diagnosis to construct classes to reconsider and an edit instruction, without requiring exact fault localization.The mapping is called error grounding; examples include selecting reference-carrying expressions for replacement or selecting no existing element for insertion.Source: §1 · pp. 1–2
  • Type-specific rules cover 29 evaluated NL2SQL-BUGs sub-types, with masking when applicable and sequential re-grounding for multiple annotations.Each later step applies its rule to the query updated by the preceding correction.Source: §1 · p. 2

2 Related Work

Prior work addresses code and SQL correction through self-correction, execution feedback, error reports, and taxonomy-guided plans. TEG differs by translating a supplied diagnosis into construct-class masks and edit instructions.

Reasoning and self-correction

  • Self-correction methods use reasoning, critiques, explanations, or execution feedback, but provide no error-type diagnosis and leave the model to infer why, where, and how to correct.The passage notes that improvement through intrinsic feedback alone is not guaranteed.Source: §2 · p. 3

SQL-specific refinement

  • SQL refinement spans prompting, training, structure-guided generation, test-time scaling, and pipelines that use self-correction or execution feedback.DIN-SQL, MAC-SQL, and SQLFixAgent exemplify decomposed correction, coordinated schema selection and refinement, and agent-separated review and candidate refinement, respectively.Source: §2 · p. 3

Program repair and code infilling

  • Program repair methods localize faulty lines or tokens and use infilling, whereas TEG selects construct classes by error type and generates full queries without training an infilling model.TEG does not require exact fault localization, and its LLM may also edit unmasked constructs.Source: §2 · p. 3

Error taxonomies and detection

  • SQL error work includes taxonomy-based detection, type-specific correction feedback, clause-level reports, and sequential refinement.SQLens combines database and LLM signals into clause-level error reports with correction instructions.Source: §2 · p. 3

Taxonomy-guided refinement

  • SQL-of-Thought uses taxonomy-guided plans, while ErrorLLM localizes relevant syntax-tree nodes and schema elements for joint refinement; TEG instead masks selected constructs and requests edits.The paper adapts ErrorLLM's refinement stage as a principal baseline using the same supplied annotations as TEG.Source: §2 · p. 3

3 TEG: Taxonomy-guided Error Grounding

TEG converts each supplied SQL error annotation into a grounded correction step. Its type-specific masks and edit instructions shape the input, while candidate selection and sequential re-grounding support correction of single- and multi-error queries.

3 TEG: Taxonomy-guided Error Grounding

  • TEG applies type-selected masks and edit instructions, then selects a corrected query from candidates by execution; multiple annotations are processed one step at a time.Each updated query becomes the input to the next step.Source: §3 · p. 3, §3.2 · p. 5

3.1 Grounding error types into masks and edit instructions

  • The taxonomy maps nine categories and 29 observed sub-types into four edit groups: Mismatch, Redundancy, Missing, and Rewrite.Mismatch and Redundancy mask matching existing constructs; Missing and Rewrite leave the query unmasked, while the edit instructions are replace, remove, insert, and restructure, respectively.Source: §3.1 · p. 4

3.2 Grounded correction with candidate selection

  • For each annotation, TEG prompts with the possibly masked query, question, schema, and requested edit, then uses execution filtering and LLM selection or refinement across candidates.The original question and up to three paraphrases yield at most four candidates per step; the unmasked query is not included when masking is applied.Source: §3.2 · pp. 4–5

3.3 Sequential re-grounding for multiple errors

  • For multiple errors, TEG re-grounds the updated query in Mismatch, Redundancy, Missing, Rewrite order, editing existing constructs before insertions or restructuring.Later edits can undo an earlier repair, and TEG retains supplied annotations rather than predicting new types.Source: §3.3 · p. 5

4 Experiments

TEG leads the evaluated correction baselines on single-error queries, including when all methods receive the same error types, and also leads on multi-error queries. Experiments test this against baselines, predicted types, and component variants on NL2SQL-BUGs.

4.2 Main results

  • TEG reaches 47.3 single-error execution accuracy with Qwen2.5-7B-Instruct, 11.8 points above CoT, and leads across the evaluated model sizes and thinking modes.The margin over the strongest baseline ranges from 0.9 to 11.8 points.Source: Table 1 · p. 6, §4.2 · p. 7

4.1 Setup

  • The evaluation uses incorrect NL2SQL-BUGs queries spanning 29 error sub-types across nine categories, with execution accuracy based on matching result tuples.Execution errors and queries exceeding the 5-second limit count as incorrect; TEG does not see reference SQL or its result during correction.Source: §4.1 · p. 6

4.2 Main results

  • With Qwen2.5-7B-Instruct, TEG reaches 37.0 overall, ahead of SQLFixAgent at 26.2 and ErrorLLM at 24.0, and leads every multi-error group.Its accuracy declines as the number of errors increases.Source: Table 2 · p. 7, §4.2 · p. 7
  • TEG stays ahead when all methods receive the same error types, reaching 37.0 overall versus 26.0 for Self-Debug (Simple).The comparison controls diagnostic information but does not equalize candidate-generation and refinement procedures.Source: §4.2 · p. 7

4.3 End-to-end correction with predicted error types

  • With Opus 4.8-predicted types, TEG outperforms Base and ErrorLLM in every evaluated corrector configuration despite only 52.7% exact type matches.In the Opus 4.8 corrector setting, TEG scores 69.1, versus 64.5 for ErrorLLM and 63.6 for Base.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 Ablation study and analysis

  • Removing masking while retaining annotations and candidate procedures lowers Qwen2.5-7B-Instruct accuracy from 47.3 to 40.9 on single-error queries and from 37.0 to 33.9 overall.The ablation also disables the mask-failure retry branch, and mask-dependent edit instructions may change, so it does not isolate masking alone.Source: §4.4 · p. 8, Table 4 · p. 8
  • One-pass correction with cumulative masks scores 32.3 on two-error queries and 23.9 on queries with three or more errors, below sequential re-grounding at 36.7 and 29.6.The one-pass variant combines all annotations and selects one prompt template using the first annotation’s category.Source: §4.4 · p. 8
  • Without execution feedback, the multi-candidate variant scores 40.9 on single-error queries and 27.9 overall; TEG’s candidate-selection pipeline averages 42.9 seconds per query.Compared with the single-candidate variant, TEG gains 10.8 points on two-error queries and 9.2 on queries with three or more errors; single-error accuracy is equal.Source: §4.4 · pp. 8–9

5 Conclusion and Discussion

TEG maps error diagnoses to construct classes and edit instructions, and the paper reports improved SQL correction on NL2SQL-BUGs. Its broader-code extensions remain untested, and the evaluation has important scope and cost limits.

5 Conclusion and Discussion

  • TEG leads the main NL2SQL-BUGs comparisons, including with shared annotations, and beats direct correction and ErrorLLM on single-error queries with predicted types.Without masking, single-error accuracy falls by 6.4 points with Qwen2.5-7B-Instruct.Source: §5 · p. 9
  • Beyond SQL, error grounding would need domain-specific constructs, grounding rules, correction order, and candidate checks; these extensions remain untested.The paper gives API-misuse and configuration-file diagnoses as possible examples.Source: §5 · p. 9
  • The evaluation uses one SQL benchmark, uses it to select correction order, and tests predicted types only on single-error queries.TEG also requires diagnoses and database access, adds inference cost, and execution accuracy does not guarantee semantic equivalence.Source: §5 · p. 9
  • The paper reports that AI tools assisted drafting, editing, consistency checks, experiment scripts, and interpretation, with the authors reviewing the work and taking responsibility.Source: AI use statement · p. 10
  • The implementation, evaluation dataset, BIRD development schemas, and predicted error types are available in the project repository, with appendix documentation of methods and hardware.Locally hosted models use greedy decoding; execution accuracy is set agreement of row tuples with the reference result.Source: Reproducibility statement · p. 10

A A Closer Look at TEG

  • Appendix A expands TEG’s grounding rules, masking algorithms, prompt templates, and candidate-selection process.Candidate selection executes candidates generated under the question and its paraphrases, then keeps or refines one.Source: A A Closer Look at TEG · p. 15

A.1 Why: Error taxonomy and edit groups

TEG maps taxonomy diagnoses to construct classes and edit operations, then uses type-specific masking and prompts to guide correction. Candidate selection filters and refines generated SQL, while multi-error correction reapplies grounding one annotation at a time.

A.1 Why: Error taxonomy and edit groups

  • Four edit groups cover 29 observed NL2SQL-BUGs sub-types; Mismatch is the largest group at 64.7% of annotations, followed by Missing at 21.3%, Redundancy at 13.5%, and Rewrite at 0.5%.The rules support 31 sub-types across nine categories; Join Type Mismatch and Window Functions are absent from the evaluation subset.Source: §A.1 · p. 15, Table 6 · p. 15

A.2 Where: Masking rules and algorithms

  • Masking rules map error types to SQLGlot node classes and replace matching constructs, but the annotation identifies a type, not a unique faulty node.Missing and Rewrite errors leave the query unmasked; special rules target aggregate predicates, comparison scopes, and outermost logical expressions.Source: §A.2 · pp. 16–17, Table 7 · p. 16

A.3 How: Prompt templates for each edit

  • Correction prompts combine the masked or current SQL with the question, schema, and error labels, then request a type-specific replacement, deletion, insertion, completion, or rewrite.The base template omits error information and asks the model to regenerate a semantically misaligned query.Source: §A.3 · p. 20

A.4 Execution-based candidate selection

  • TEG generates up to four candidate queries per correction step, filters by execution, and uses an LLM to select or refine candidates, while treating execution heuristics as imperfect.Candidates that run, return rows, and contain no NULL in the first five rows are preferred; if none passes, all remain eligible.Source: §A.4 · pp. 20–21

B Experimental Details

  • Appendix B expands experimental details on Qwen3-8B thinking modes, baseline implementations, ablation variants, and computing resources.Source: B Experimental Details · p. 22

B.1 Dataset statistics and examples

The appendix documents the benchmark subset, model settings, baseline implementations, ablations, and computing resources. It also extends the main results with category-level single- and multi-error results and oracle-type comparisons.

B.1 Dataset statistics and examples

  • The evaluation subset contains 549 incorrect NL2SQL-BUGs queries across 11 SQLite databases, with 1,160 annotations covering nine categories and 29 observed sub-types.Single-error queries make up 20.0% of the subset, while 54.1% contain exactly two errors.Source: §B.1 · p. 22, Table 8 · p. 22

B.2 Qwen3-8B thinking modes

  • Qwen3-8B thinking OFF and ON use the same correction procedures but different generation budgets, and every method improves with thinking ON while TEG leads in both modes.The appendix also shows representative cases, including one requiring an insertion and a replacement in two sequential steps.Source: §B.2 · p. 22, Table 9 · p. 23

B.3 Baseline implementations

  • Baselines share LLM and decoding settings within each comparison; the appendix documents adapted CoT, DIN-SQL, Self-Debug, MAC-SQL, SQLFixAgent, and ErrorLLM implementations.ErrorLLM receives the same annotations as TEG, while TEG reuses SQLFixAgent prompts for its crafter and refiner.Source: §B.3 · pp. 23–24

B.4 Ablation variants

  • The ablations separately remove error annotations, masking, database execution, candidate selection, or sequential correction, with the latter applying all annotations in one prompt.Source: §B.4 · p. 24

B.5 Computing resources

  • Locally hosted Qwen2.5 experiments used one dedicated GPU per run; Opus 4.8 was accessed through an API on uncontrolled hardware.Source: §B.5 · p. 24

C Additional Main Results

  • Section C extends the main results with per-category single- and multi-error results and full results for baselines given oracle error types.Source: C Additional Main Results · p. 25

C.1 Single-error results by error category

Category-level results vary across error types even though TEG leads or ties on Attribute errors across the reported Qwen settings. Some competitors lead on particular categories, and small category subsets make rankings sensitive to a few queries.

C.1 Single-error results by error category

  • TEG leads or ties on Attribute errors across all five Qwen settings, while other methods lead on some Value, Condition, and Table categories.DIN-SQL is stronger on Value errors at Qwen2.5-14B and 32B; ErrorLLM leads on Condition and Table errors at 14B.Source: §C.1 · p. 25, Table 10 · p. 25, Table 11 · p. 26
  • At Qwen2.5-14B, ErrorLLM scores 70.6 on Condition and 73.3 on Table errors, versus TEG’s 58.8 and 66.7.The Table and Others subsets contain only 15 and 12 queries, so one or two queries can change category rankings.Source: §C.1 · p. 25

C.2 Multi-error results by error category

On 439 multi-error queries, TEG leads every reported error category and reaches 34.4 overall. It also remains most accurate in every Qwen setting when all baselines receive oracle error types.

C.2 Multi-error results by error category

  • 34.4 overall execution accuracy, with TEG leading every reported category across 439 multi-error queries.The largest lead is on Table errors: 43.8 versus 28.4 for the next-best method.Source: Table 12 · p. 26, §C.2 · p. 26

C.3 Baselines given oracle error types

  • With oracle error types supplied to every baseline, TEG remains the most accurate method in every Qwen setting.Source: §C.3 · p. 27

C.3.1 Single-error results

TEG leads single-error correction in all five Qwen settings when every method receives oracle error types. The comparisons preserve each baseline’s correction procedure while withholding TEG’s type-specific masking and sequential correction from those baselines.

C.3.1 Single-error results

  • All methods receive oracle error types on the same 110 queries and five Qwen settings; baseline prompts are augmented without adding TEG’s masking or sequential correction.Correction procedures, iteration limits, and decoding settings are retained.Source: §C.3.1 · p. 27
  • 0.9 to 11.8 points: TEG’s single-error margin over the strongest same-type baseline across the evaluated Qwen settings.TEG leads in all five settings.Source: Table 13 · p. 27, §C.3.1 · p. 27
  • 46.4 and 57.3: TEG’s single-error accuracy on Qwen3-8B with thinking OFF and ON, versus strongest same-type baselines at 45.5 and 52.7.Source: §C.3.1 · p. 27

C.3.2 Results by error count

With oracle error types supplied to all methods, TEG remains ahead across error-count groups and achieves the highest single-error accuracy overall. The section also points to follow-up evaluation of predicted types and category-level correction.

C.3.2 Results by error count

  • 37.0 overall execution accuracy, versus 26.0 for the strongest type-augmented baseline, Self-Debug (Simple).TEG stays above every such baseline in each error-count group.Source: Table 14 · p. 28, §C.3.2 · p. 28

C.3.3 Single-error results by error category

  • 47.3 single-error accuracy: TEG’s highest overall, versus 35.5 for the best baseline given oracle types.Oracle types improve two of six reasoning and SQL-specific baselines, reduce three, and leave one unchanged; Base falls from 30.0 to 28.2.Source: §C.3.3 · p. 28, Table 15 · p. 29

D Additional Results on End-to-end Correction

D.1 Accuracy of predicted error types

Predicted error types match oracle labels much more often on single-error queries than on multi-error queries. With predicted types, TEG’s category-level accuracy also declines in some settings, especially for Attribute and Condition errors.

D.1 Accuracy of predicted error types

  • One Opus 4.8 call per query receives the question, issued SQL, schema DDL, and definitions of all 31 taxonomy sub-types, but not the reference SQL or oracle annotations.Predicted labels are normalized to the taxonomy, unrecognized pairs are discarded, and the predictions are shared across correctors.Source: §D.1 · p. 30
  • Opus 4.8 predicts 726 errors against 1,160 oracle annotations, under-counting on 58.7% of queries; 16.6% receive no predicted error.These counts cover all 549 queries, while predicted-type correction is evaluated only on the single-error subset; empty predictions use TEG’s unmasked fallback.Source: §D.1 · p. 30
  • 52.7 exact match on single-error queries falls to 8.8 on two-error queries and 3.5 on queries with three or more errors.Ignoring sub-types raises overall exact match from 16.2 to 20.0.Source: Table 16 · p. 30, §D.1 · p. 30

D.2 Single-error results by error category with predicted types

  • Predicted types reduce TEG’s Attribute accuracy by 13.5 points at 14B and 13.6 at 32B, and reduce Condition accuracy at all three Qwen2.5 scales.Table and Others results vary in direction on only 15 and 12 queries, limiting interpretation; the oracle row is not a strict upper bound for every category.Source: §D.2 · p. 30, Table 17 · p. 31

E Extended Discussion

  • The extended discussion considers what the results say about diagnosis, the origins of error grounding, and what carrying the idea beyond SQL would require.Source: E Extended Discussion · p. 32

E.1 Using error diagnoses to construct correction inputs

The discussion finds that taxonomy diagnoses can structure correction inputs, while extensions beyond SQL remain prospective. It relates TEG’s masking to visual grounding and inpainting and identifies conditions for transfer to other domains.

E.1 Using error diagnoses to construct correction inputs

  • With shared error types, TEG retains its lead over baselines, while removing masking lowers accuracy; these comparisons do not isolate masking alone.The 7B gains suggest potential for on-premises correction if annotations are also supplied locally; benefit depends on the LLM and whether types are oracle or predicted. Under oracle types, Opus 4.8 nearly ties ErrorLLM.Source: §E.1 · p. 32

E.2 Relation to visual grounding and inpainting

  • TEG borrows visual grounding [Yu et al., 2018b] and inpainting [Yu et al., 2018a; Avrahami et al., 2022] as analogies, but can change unmasked SQL too.Unlike diffusion or fixed-boundary editing, an LLM generates a complete query.Source: §E.2 · p. 32

E.3 Carrying error grounding beyond SQL

  • Beyond SQL, transfer remains prospective: domains need recurring error categories, editable elements, constraint-preserving instructions, and dependency-aware correction order.Tests, type checks, or schema validation may guide selection but do not establish semantic correctness; extension effectiveness and cost remain unevaluated.Source: §E.3 · p. 32

Related papers

Loading 2610.05060v1…