PaperlayerView exact revision on arXiv

Source-linked AI summary

PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation

Wei Zhou, Guoliang Li, Haoyu Wang, Yuxing Han, Xufei Wu, Fan Wu, Xuanhe Zhou

arXiv:2509.23338v1cs.DBcs.AIcs.CLcs.IRcs.LG

TL;DR

Cross-System SQL translation is important for heterogeneous database environments, but existing benchmarks often lack system diversity and system-specific dialect coverage. PARROT provides curated translation data and an evaluation suite; state-of-the-art LLM performance across dialects remains around 17%–60%.

  • Problem

    Existing SQL benchmarks are poorly suited to cross-system translation because they cover few database systems and often omit system-specific dialect challenges.

    Source: §1 · p. 1

  • Method

    PARROT curates real-world translation pairs, challenge cases, and an augmented SQL pool, with an evaluation protocol that prioritizes execution-based semantic correctness.

    Source: §1 · pp. 2–3, §3 · p. 3

  • Results

    State-of-the-art LLMs achieve around 17%–60% performance across dialects, indicating substantial headroom for future research.

    Source: §1 · p. 3

  • Takeaways & Limitations

    PARROT provides a practical assessment of system-specific translation, while the reported performance range leaves substantial headroom for future research.

    Source: §1 · p. 3

    The paper prioritizes translation accuracy; improving execution efficiency may require subsequent use of external tools.

    Source: §4 · p. 7

Abstract

from arXiv · show

Large language models (LLMS) have shown increasing effectiveness in Text-to-SQL tasks. However, another closely related problem, Cross-System SQL Translation (a.k.a., SQL-to-SQL), which adapts a query written for one database system (e.g., MySQL) into its equivalent one for another system (e.g., ClickHouse), is of great practical importance but remains underexplored. Existing SQL benchmarks are not well-suited for SQL-to-SQL evaluation, which (1) focus on a limited set of database systems (often just SQLite) and (2) cannot capture many system-specific SQL dialects (e.g., customized functions, data types, and syntax rules). Thus, in this paper, we introduce PARROT, a Practical And Realistic BenchmaRk for CrOss-System SQL Translation. PARROT comprises 598 translation pairs from 38 open-source benchmarks and real-world business services, specifically prepared to challenge system-specific SQL understanding (e.g., LLMS achieve lower than 38.53% accuracy on average). We also provide multiple benchmark variants, including PARROT-Diverse with 28,003 translations (for extensive syntax testing) and PARROT-Simple with 5,306 representative samples (for focused stress testing), covering 22 production-grade database systems. To promote future research, we release a public leaderboard and source code at: https://code4db.github.io/parrot-bench/.

1 Introduction

Cross-System SQL translation matters in heterogeneous database environments, but existing benchmarks often lack system diversity and system-specific dialect coverage. PARROT addresses this gap with translation data and tests, while evaluations show substantial performance headroom for current LLMs.

1 Introduction

  • Existing benchmarks are poorly suited to SQL-to-SQL evaluation because they cover few systems and rarely represent system-specific dialect characteristics.The paper reports that most benchmarks focus on SQLite, over 89% of BIRD queries are system-agnostic, and fewer than 13% of BIRD-mini PostgreSQL and MySQL queries exhibit system-specific syntax.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Examples show GPT-4o missing a MySQL division-by-zero safeguard and failing to adapt nested ROLLUP syntax amid unrelated clauses.A further example describes an incorrect assumption about CTE alias visibility across query scopes in MySQL.Source: §1 · pp. 1–2
  • PARROT combines 598 manually verified query pairs, a 5,306-case challenge set across 22 systems, and a 28,003-statement augmented training pool.The challenge set isolates system-specific constructs such as window-function variants, geo-types, and bitmap operations.Source: §1 · p. 2
  • State-of-the-art LLMs achieve around 17%–60% performance across dialects, leaving substantial headroom for future research.The introduction describes these results as empirical analysis of PARROT.Source: §1 · p. 3

2 Problem Formulation

Cross-System SQL Translation maps a query from a source database system to a target system while preserving its functionality. Functional equivalence requires target-dialect compatibility and semantic consistency.

2 Problem Formulation

  • A translated query must conform to the target system’s SQL syntax and preserve the source query’s semantics and equivalent functionality.The target query must be syntactically compatible and semantically consistent with the source.Source: §2 · p. 3
  • Functional equivalence requires both syntactic compatibility and semantic consistency between query operations.The target operation must follow the target syntax standards and produce the same results or have the same effect as the source operation.Source: §2 · p. 3
  • CURRENT_TIMESTAMP in PostgreSQL and NOW() in MySQL illustrate different dialect syntax for operations that return the same current timestamp.The paper presents these functions as functionally equivalent despite their different syntax.Source: §2 · p. 3

3 Collection and Curation of PARROT

PARROT collects real-world SQL from public and private sources, then curates and annotates it through a multi-step workflow. The workflow filters redundant or unsuitable queries and checks translations for syntax and semantic consistency.

3 Collection and Curation of PARROT

  • PARROT uses real-world SQL because assembling representative workloads requires expertise, while synthesized queries may miss production workload structure.The collection pipeline selects representative queries that meet benchmark design principles.Source: §3 · p. 3

3.1 SQL Source Collection

  • Open sources include SQL from 38 benchmarks and 1,041 test cases from public code repositories; private sources include 343 manually rewritten and expert-verified ByteHouse SQL pairs.The private workload covers 102 tables and translates Postgres-variant SQL into ClickHouse syntax.Source: §3.1 · pp. 3–4

3.2 SQL Curation Workflow

  • The curation workflow addresses repetitive, simple queries and single-dialect coverage through preprocessing, filtering, clustering, selection, annotation, and validation.Prefix-based clustering groups normalized queries, then parser code coverage guides selection of representative SQLs.Source: §3.2 · pp. 4–5
  • Multiple rule-based tools and an LLM-based candidate annotator generate target-dialect translations, with rule-based tools prioritized for efficiency and effectiveness.SQLGlot and jOOQ are among the tools used to accumulate annotation results.Source: §3.2 · p. 5
  • Dialect parsers check syntax, experts revise unresolved parsing errors, and generated test cases compare execution results to assess semantic equivalence.Queries with unresolved syntax errors or inconsistent execution results are excluded; queries already translated by a small LLM are also removed to increase difficulty.Source: §3.2 · pp. 5–6

4 PARROT Benchmark Analysis

PARROT is characterized as a broad, scalable benchmark spanning open-source and enterprise SQL workloads, with expanded variants and representative-query curation. Its assessment covers syntax and semantic correctness, while the paper primarily focuses on translation accuracy rather than execution efficiency.

  • PARROT collects 33,952 SQL pairs across 22 data systems, including 38 benchmarks and ByteHouse enterprise workloads.Source: §4 · p. 6
  • PARROT-DIVERSE contains 28,003 SQL pairs across 22 dialects, while PARROT-SIMPLE contains 5,306 single-translation-type snippets for testing whether models internalize translations.Source: §4 · p. 6
  • PARROT reduces 9,912,231 raw SQL pairs to 28,003 representative queries by retaining structurally and semantically diverse queries within defined groups.Source: §4 · p. 7
  • PARROT supports syntax and semantic assessment through AccEX executable-query rate and AccRES strict result consistency, including format, precision, and display order.Source: §4 · p. 7
  • Although business SQL execution efficiency matters, the paper focuses primarily on translation accuracy and notes efficiency can be improved with external tools.Source: §4 · p. 7

5 Experiments

Experiments show that translation accuracy varies across dialects, model sizes, and query lengths. A ByteHouse case study identifies failures involving datetime conversion and NULL handling.

5.1 Experimental Setup

  • The evaluation considers open-source and proprietary LLMs, varied parameter scales, and general-purpose and code-specialized models, using AccEX and AccRES.The setup specifies two Intel Xeon E5-2678 v3 CPUs, 256 GB memory, and four GeForce RTX 3080 and H100 Ti graphics cards.Source: §5.1 · p. 7

5.2 Comparative Analysis

  • All evaluated LLMs lose accuracy as SQL length increases from 0–402 to 1214–2182 tokens.The authors attribute the regression to more operations in longer queries and increased risk of hallucination and lost-in-the-middle errors.Source: Table 3 · p. 8, §5.2 · p. 8
  • GPT-4o reaches 58.62% accuracy translating to PostgreSQL but 50.00% translating to MySQL, showing uneven performance across dialects.The authors relate this pattern to the stringent syntax standards of different dialects.Source: §5.2 · p. 8
  • DeepSeek-R1 32B outperforms DeepSeek-R1 671B on PostgreSQL, MySQL, and SQL Server, while the authors identify SQL understanding and syntax matching as needed abilities.The respective accuracies are 58.62%, 58.82%, and 42.11% for 32B, versus 48.28%, 44.12%, and 36.84% for 671B.Source: §5.2 · p. 8

5.3 Case Study

  • In a ByteHouse customer-workload case, o3-mini’s translation drops the datetime input conversion and format and changes an IS NOT NULL check to != '', causing runtime errors.The reported errors include an invalid Int64 argument to parseDateTimeBestEffort and an attempt to convert an empty string to Float64.Source: §5.3 · pp. 8–9, Table 4 · p. 9

6 Related Work

Prior work includes rule-based dialect translators and NL2SQL benchmarks, but the latter generally target one dialect rather than cross-system translation.

  • SQLGlot, SQLines, and jOOQ use handcrafted rules or pattern-based templates to translate common syntax across dialects.Source: §6 · p. 9
  • Spider, BIRD, and WikiSQL primarily target a single dialect, commonly SQLite, and lack dialect-specific syntax annotations needed to assess cross-system syntax fidelity and semantic correctness.Source: §6 · p. 9

7 Conclusion

PARROT combines curated data, diagnostic cases, and an evaluation protocol to assess cross-system SQL translation and support reproducible research.

  • PARROT provides a diverse dataset, specialized diagnostic cases, and an evaluation protocol for practical assessment of LLMs translating across systems.The authors state that it facilitates reproducible research and supports development of more robust, accurate, and generalizable SQL translation methods across database systems.Source: §7 · p. 9

A Technical Appendices and Supplementary Material

The supplementary material classifies cross-dialect translation challenges, catalogs benchmark details, and describes PARROT’s SQL annotation interface and prompt design.

A Technical Appendices and Supplementary Material

Related papers

Loading 2509.23338v1…