PaperlayerView exact revision on arXiv

Source-linked AI summary

Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards

Tengjun Jin, Yoojin Choi, Yuxuan Zhu, Daniel Kang

arXiv:2601.08778v3cs.AIcs.DB

TL;DR

Text-to-SQL benchmarks can be unreliable because annotation errors affect benchmark examples and leaderboard evaluations. This paper measures those errors in BIRD and Spider 2.0-Snow, corrects a BIRD Dev subset, and finds substantial changes in agent performance and rankings.

  • Problem

    Annotation errors make current text-to-SQL benchmarks unreliable and can distort reported agent performance and rankings.

  • Method

    The study quantifies annotation errors in BIRD and Spider 2.0-Snow, manually verifies diagnostic reports, and corrects a BIRD Dev subset for evaluating 16 leaderboard agents.

  • Results

    52.8% of BIRD Mini-Dev and 62.8% of Spider 2.0-Snow examples contain annotation errors, while agent performance changes from -7% to +31% and rankings shift by -9 to +9 positions.

  • Takeaways & Limitations

    Annotation errors can significantly distort reported text-to-SQL performance and leaderboard rankings, potentially misleading researchers and practitioners.

  • Takeaways & Limitations

    The evaluation used Spider 2.0-Snow annotations available on Aug. 20, 2025, before later gold-query updates, and corrected-query accuracy was not evaluated.

Abstract

from arXiv · show

Researchers have proposed numerous text-to-SQL techniques to streamline data analytics and accelerate the development of data-driven applications. To compare these techniques and select the best one for deployment, the community depends on public benchmarks and their leaderboards. Since these benchmarks heavily rely on human annotations during question construction and answer evaluation, the validity of the annotations is crucial. In this paper, we conduct an empirical study that (i) benchmarks annotation error rates for two widely used text-to-SQL benchmarks, BIRD and Spider 2.0-Snow, and (ii) corrects a subset of the BIRD development (Dev) set to measure the impact of annotation errors on text-to-SQL agent performance and leaderboard rankings. Through expert analysis, we show that BIRD Mini-Dev and Spider 2.0-Snow have error rates of 52.8% and 62.8%, respectively. We re-evaluate all 16 open-source agents from the BIRD leaderboard on both the original and the corrected BIRD Dev subsets. We show that performance changes range from -7% to 31% (in relative terms) and rank changes range from $-9$ to $+9$ positions. We further assess whether these impacts generalize to the full BIRD Dev set. We find that the rankings of agents on the uncorrected subset correlate strongly with those on the full Dev set (Spearman's $r_s$=0.85, $p$=3.26e-5), whereas they correlate weakly with those on the corrected subset (Spearman's $r_s$=0.32, $p$=0.23). These findings show that annotation errors can significantly distort reported performance and rankings, potentially misguiding research directions or deployment choices. Our code and data are available at https://github.com/uiuc-kang-lab/text_to_sql_benchmarks.

1 INTRODUCTION

Public text-to-SQL benchmarks guide technique development and deployment choices, but annotation errors undermine their reliability. This study audits two benchmarks, corrects BIRD examples, and measures consequences for agent performance and rankings.

  • Audit and correction: The audit uses SAR-Agent diagnostic reports, expert adjudication, and additional manual review guided by recurring error patterns.The study also uses SAPAR to support manual correction of a sampled BIRD Dev subset.
  • High annotation error rates: 52.8% of BIRD Mini-Dev examples and 62.8% of Spider 2.0-Snow examples contain annotation errors.The BIRD rate is 20.5% higher than a previously reported 32.3% rate, while Spider 2.0-Snow is examined comprehensively here.
  • Leaderboard impact: −7% to 31% relative performance changes and −9 to +9 ranking changes occur when 16 BIRD agents are re-evaluated after corrections.CHESS rises from 62% to 81% and from 7th to 1st place after corrections.
  • Leaderboard reliability: Original-subset rankings correlate strongly with the original full Dev rankings (r_s=0.85, p=3.26e-5), but weakly with corrected-subset rankings (r_s=0.32, p=0.23).This comparison tests whether the subset rankings generalize to the full BIRD Dev set.
  • Implications: Annotation errors can distort reported performance and rankings, potentially misleading research and deployment choices.The study frames benchmark reliability as important because benchmarks guide technique improvement and agent selection.

2 EXPERIMENT DESIGN

The study selects BIRD and Spider 2.0-Snow based on leaderboard usage and SQL complexity, then audits annotations and evaluates corrected BIRD examples. Its evaluation uses the Dev set because it supports local assessment and error analysis, while the hidden Test set is unavailable.

  • Benchmark selection: BIRD and Spider 2.0-Snow are selected using leaderboard usage and ground-truth SQL complexity across six benchmarks.BIRD has 100 evaluated leaderboard methods, while Spider 2.0-Snow has the highest average SQL length at 161.8 tokens per query.
  • Annotation audit: SAR-Agent generates per-example diagnostic reports, which SQL experts review to assign final correctness labels.SAPAR supports manual correction of identified errors in a sampled BIRD Dev subset.
  • Evaluation design: The experiments quantify error rates, correct 100 BIRD Dev examples, and re-evaluate 16 open-source leaderboard agents on original and corrected subsets.These procedures address the study’s questions about error rates, leaderboard effects, and SAR-Agent effectiveness.
  • Evaluation scope: The study focuses on BIRD Dev because Train is used for fine-tuning, Test is hidden, and researchers use Dev for local evaluation and error analysis.Spider 2.0-Snow annotations were taken from the version available on August 20, 2025, before later gold-query updates.

3 TOOLKIT

The toolkit combines SAR-Agent, which incrementally verifies annotations through database queries, with SAPAR, which integrates agent diagnostics into an expert-reviewed correction workflow.

  • SAR-Agent: SAR-Agent incrementally verifies each annotation by formulating targeted queries, executing them against the database, and retaining prior results to guide later iterations.It synthesizes the accumulated findings into a diagnostic report covering correctness and potential ambiguity.
  • SAR-Agent: SAR-Agent takes the natural-language question, ground-truth SQL query, database schema, and database access environment as inputs.For Spider 2.0-Snow, the schema is restricted to gold tables because databases average 812.1 columns.
  • SAR-Agent: SAR-Agent outputs verification-query logs and a final diagnostic report containing correctness judgments, ambiguity assessment, explanations, and a proposed corrected query.The report is intended to support annotation review rather than replace expert decisions.
  • SAPAR: SAPAR integrates SAR-Agent into annotation workflows so annotators revise flagged annotations and consult SQL experts when disagreements remain.The workflow proceeds through automated verification, manual review and adjudication, and annotation revision.
  • SAPAR: SAPAR is designed to provide diagnoses within minutes and reduce routine SQL-expert review effort in cross-domain benchmarks.The paper states that annotators can correct issues immediately after initial annotation rather than waiting until batch review.

4 BIRD AND SPIDER 2.0 HAVE PERVASIVE ANNOTATION ERRORS

The audit finds pervasive annotation errors in BIRD Mini-Dev and Spider 2.0-Snow, using a three-step examination that combines agent detection with human verification and additional manual review.

  • 4.2 Three-step Examination: The examination combines automatic SAR-Agent detection, manual verification by SQL experts, and additional manual detection guided by recurring error patterns.Experts verify flagged cases, consult authoritative sources when needed, execute verification queries, and inspect related examples in the same database.
  • 4.3 Evaluation Results: BIRD Mini-Dev contains annotation errors in 263 of 498 unique examples, yielding an error rate of 52.8%.The identified errors include 161 examples reported in prior work and additional errors uncovered by the examination.
  • 4.3 Evaluation Results: Spider 2.0-Snow contains annotation errors in 76 of 121 examples with publicly released gold queries, yielding an error rate of 62.8%.The evaluation covers the 121 examples for which gold queries are available.
  • 4.4 Analysis of Annotation Errors: The audit identifies errors involving query-question mismatches, data or schema misunderstandings, incorrect domain knowledge, and ambiguous or underspecified questions.Examples include incorrect comparison operators, missing aggregation, misuse of domain terms, and questions admitting non-equivalent SQL interpretations.
  • 4.4 Analysis of Annotation Errors: E2 is the most common error pattern in both benchmarks, occurring in 57.79% of erroneous BIRD Mini-Dev examples and 57.89% of erroneous Spider 2.0-Snow examples.E2 concerns limited understanding of the data or database schema, and examples may contain multiple error patterns.

5 ANNOTATION ERRORS BREAK THE BIRD LEADERBOARD

The study corrects annotation errors in a sampled BIRD Dev subset and re-evaluates 16 agents, revealing substantial changes in execution accuracy and leaderboard rankings. Original-subset rankings align strongly with the full Dev set, whereas corrected-subset rankings align weakly.

  • Benchmark Correction: 48% of the sampled examples required correction, including questions, external knowledge, SQL queries, schema, or database contents.Questions were corrected in 19 examples, external knowledge in 17, SQL queries in 41, schema in one, and database data in six.
  • Execution Accuracy: −7% to 31% relative changes in execution accuracy occurred across the 16 agents after correction.12 agents improved; CHESS rose from 62% to 81%, while SFT CodeS-15B fell from 56% to 52%.
  • Rankings: −9 to +9 ranking-position changes occurred across agents, with CHESS and GenaSQL moving from middle ranks to first place.MAC-SQL moved from 14th to 5th, while GSR fell from 4th to 13th.
  • Causes of Performance Changes: CHESS’s execution accuracy rose because 91.7% of its generations in 12 analyzed cases matched revised ground-truth queries.The analysis identified false-negative labels in the original Dev set, including a recurring error pattern in 5 of 12 cases.

6 SAR-AGENT HELPS EXPERTS DETECT 42% MORE INCORRECT EXAMPLES

SAR-Agent combines diagnostic reports with human verification to detect annotation errors in BIRD Mini-Dev and Spider 2.0-Snow. It shows high precision, detects more errors than the Arcwise team, and incurs higher cost on the more complex Spider benchmark.

  • Dataset Versions: The released Arcwise-Plat-SQL and Arcwise-Plat versions separate SQL-only corrections from corrections that also resolve question and evidence ambiguities.Both versions are available in the project repository.
  • Evaluation Method: The evaluation counts a detection as successful when manual review confirms at least one reported error reason.The study collected diagnostic reports rather than relying only on binary YES/NO judgments.
  • Precision: 83% precision on BIRD Mini-Dev and 89% on Spider 2.0-Snow were achieved after manual verification.SAR-Agent flagged 274 BIRD examples and 79 Spider examples as incorrect; 228 and 70, respectively, were verified as true positives.
  • Comparison with Arcwise: 85.7% of Arcwise’s 161 error examples were detected by SAR-Agent, which identified 41.6% more error examples overall.SAR-Agent successfully detected 138 of the 161 Arcwise examples.
  • Cost and Execution Steps: $0.44 per example and 5.1 average steps were required on BIRD Mini-Dev, versus $1.11 and 7.6 steps on Spider 2.0-Snow.Spider 2.0-Snow’s more complex queries and schemas required more annotation-diagnosis steps.

7 ABLATION STUDY

The ablation study compares database modification and SQL-only correction settings against the fully corrected Dev subset. Database modification has little ranking impact, while SQL-only settings align less closely with full correction.

  • Database Modification: r_s=0.95 indicates that database modification has minimal impact on agent rankings.This compares corrected subsets with and without database modification.
  • SQL-Only Corrections: r_s=0.73 for SAR-Agent’s SQL-only corrections exceeds r_s=0.59 for human SQL-only corrections against the fully corrected subset.The paper attributes the gap to residual issues in human-repaired SQL that remain aligned with problematic questions or external knowledge.

8 RELATED WORK

Related work spans text-to-SQL benchmarks, text-to-SQL methods, benchmark annotation-error studies, and LLM-based evaluation. This paper combines agent-powered automation with manual verification for text-to-SQL annotation auditing.

  • Text-to-SQL Benchmarks: Text-to-SQL benchmarks have expanded from single-domain datasets toward cross-domain datasets that assess generalization across diverse schemas.Recent benchmark development has also been driven by advances in LLMs and text-to-SQL techniques.
  • Text-to-SQL Methods: Recent text-to-SQL methods include agents that decompose the task into schema linking, SQL generation, and SQL refinement, alongside LLM fine-tuning.These directions reflect the field’s evolution with advances in LLMs.
  • Annotation Errors in Benchmarks: Prior studies identify annotation errors in benchmarks such as Spider and BIRD, often combining automatic correction with manual verification.The present work extends this pattern to text-to-SQL annotation auditing.
  • LLM-as-a-Judge: SAR-Agent is introduced as an AI co-reviewer for text-to-SQL annotation requiring database expertise and domain knowledge.The paper frames this as an Agent-as-a-Judge application for benchmark quality improvement.

9 CONCLUSION

The study finds substantial annotation errors in BIRD and Spider 2.0-Snow and shows that correcting them can change text-to-SQL agent performance and leaderboard rankings. It therefore highlights the need for higher-quality benchmark construction and evaluation.

  • 52.8% and 62.8% are the annotation error rates found in BIRD Mini-Dev and Spider 2.0-Snow, respectively.
  • -7% to +31% are the relative performance changes observed after re-evaluating 16 open-source BIRD leaderboard agents on corrected data.
  • -9 to +9 positions are the leaderboard shifts observed after correcting the sampled BIRD Dev set.
  • The authors advocate using SAR-Agent and SAPAR to develop higher-quality text-to-SQL benchmarks.

A.1 Example

The paper illustrates SAR-Agent’s annotation-review process with BIRD Mini-Dev Example 416, including its diagnostic report and reasoning trajectory. Figure 13 depicts the agent’s trajectory for that example.

  • SAR-Agent takes a user query, relevant external knowledge, ground-truth SQL query, and database schema as inputs.
  • The resulting diagnostic report and reasoning trajectory are presented for BIRD Mini-Dev Example 416.

A.2 Configurations of Text-to-SQL Agents

The study evaluates text-to-SQL agents and benchmark corrections through multiple configuration settings, with particular attention to how annotation updates affect evaluation outcomes. It reports that corrected evaluation can substantially alter rankings, while updating Spider 2.0-Snow questions alone leaves many errors unresolved.

  • The 16 evaluated agents are characterized by six modules: fine-tuning, question preprocessing, schema linking, SQL generation, SQL refinement, and SQL selection.
  • The BIRD correction study uses four settings: fully corrected data, corrected data without database modification, SAR-Agent SQL-only correction, and human SQL-only correction.
  • CHESS moves from 7th on the original BIRD Dev subset to 1st across all four corrected subsets.
  • Spider 2.0-Snow updates user questions in 71 of 121 examples with released gold queries.
  • 66.1% to 62.8% is the reduction in Spider 2.0-Snow’s error rate after the team’s correction, fully resolving errors in only 3.3% of examples.
  • The Spider 2.0-Snow fix introduces additional errors in 6 examples, showing that updating user questions alone is insufficient.
Loading 2601.08778v3…