ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines

1 INTRODUCTION

• ELT pipelines load data directly into warehouses before transformation, and cloud infrastructure supports scalable processing and flexible transformations.Their use is growing as cloud warehouses become more accessible and storage prices fall.Source: §1 · p. 1
• Pipeline development requires substantial manual work to integrate disparate sources and understand their schemas for transformation queries.Prior studies estimate data engineers spend over 60% of their time on pipeline building in data-warehousing projects.Source: §1 · p. 1
• Existing data-domain benchmarks assess isolated tasks, but none evaluates agents building end-to-end ELT pipelines.The paper notes strong text-to-SQL benchmark results but identifies the missing end-to-end ELT evaluation.Source: §1 · p. 1
• Creating a benchmark also requires extensive setup, ground-truth labeling, and manual execution and verification of pipelines.Ground truth includes complex relational transformations such as joins, aggregation, and ranking.Source: §1 · p. 1
• ELT-Bench contains 100 pipelines, 835 source tables, and 203 data models across various domains.The benchmark’s ground truth averages 187 lines of code per configuration file and 200 SQL tokens per data model.Source: §1 · pp. 1–2
• The benchmark covers extraction and loading followed by transformation, requiring agents to build data models through a complete workflow from scratch.Agents interact with databases and data tools, write code and SQL, and orchestrate the pipeline stages.Source: §1 · p. 2, Figure 1 · p. 2
• 3.9% success in data transformation versus 57% in data extraction & loading for Spider-Agent Claude-3.7-Sonnet with extended thinking.The agent averages $4.30 and 89.3 execution steps per task.Source: §1 · p. 2

2 ELT-BENCH

• The section proceeds from benchmark data and statistics through its overview and annotation pipeline to a worked task example.Source: §2 · p. 2

2.1 Data Collection

• Sources come from Bird databases and Fivetran; the authors manually checked Bird questions and SQL, correcting errors, and sampled 22 Fivetran databases.Source: §2.1 · p. 2

2.2 Benchmark Statistics

• ELT-Bench is the first end-to-end benchmark for building complete ELT pipelines, unlike Spider2-V’s data-tool focus and Spider 2.0’s text-to-SQL focus.Source: §2.2 · p. 2, Table 1 · p. 3
• The benchmark spans 100 pipelines, 835 source tables, and 203 data models; 60 pipelines use five source categories, while 73 require over 200 SQL tokens per data model.Source: §2.2 · pp. 2–3

2.3 ELT-Bench Overview

• Tasks begin with an existing project base and require extracting and loading data before transforming it into target models, using Airbyte and DBT with Snowflake.Source: §2.3 · pp. 3–4

2.4 Annotation Pipeline

• The annotation pipeline diversifies source formats, sets up data environments, records connection configurations, and defines models with descriptive attributes and quantitative metrics.Source: §2.4 · p. 4
• Annotators derive, aggregate, categorize, and rank model columns, then validate or write transformation SQL and manually verify each pipeline’s configurations and annotations.Source: §2.4 · pp. 4–5

2.5 Task Example

• The example builds a retail customers model by configuring Airbyte extraction and loading, then creating and checking a Snowflake DBT transformation.Source: §2.5 · p. 5

3.3 Evaluation Results

• 57% SRDEL and 3.9% SRDT: Spider-Agent Claude-3.7-Sonnet with extended thinking leads overall, improving 54.1% in extraction and loading and 290% in transformation over SWE-Agent Claude-3.5-Sonnet.Spider-Agent GPT-4o averages 43.7 steps and $2.03 per ELT-Bench task, compared with 30 steps and $0.30 per instance on Spider 2.0.Source: Table 4 · p. 6, §3.3 · p. 7

3.1 Evaluation Metrics

• SRDEL counts pipelines that load data from every source successfully; SRDT measures correctly generated models, compared with ground truth using stable unique-key ordering and allowing extra columns.A generated model must contain all ground-truth columns to count as correct.Source: §3.1 · p. 6

3.2 AI Agent Frameworks

• Spider-Agent supports CLI-based multi-turn database access and ReAct actions, whereas SWE-Agent interacts with the filesystem using function-calling or thought-action modes.The evaluation uses six LLMs; Spider-Agent allows up to 100 steps, SWE-Agent retains 25 observations, and both frameworks receive a $6 cost budget.Source: §3.2 · p. 7

4.1 Error Analysis of Open-Sourced LLMs

• Open-source LLMs achieve 0% success in data extraction and loading, with failures linked to project-base interaction, configuration use, documentation, and prompt length.Spider-Agent DeepSeek-R1 rarely references config.yaml, while Qwen2.5-Coder-32B-Instruct exceeds its 32,767-token context limit in 15 Spider-Agent tasks and 87 SWE-Agent tasks.Source: §4.1 · p. 8

4.2 Error Analysis of Data Extraction & Loading

• 78.8% of tasks succeed when Spider-Agent GPT-4o reads documentation before writing configuration, versus 9.6% when it writes configuration first.The documentation-first strategy is used in 27% of tasks; the configuration-first strategy in 73%.Source: Figure 7 · p. 8, §4.2 · p. 10
• 47% of Spider-Agent Claude-3.5-Sonnet tasks terminate in Stage 1 because of unparsable actions, compared with 7% for Spider-Agent GPT-4o.SWE-Agent avoids parsing errors through function calling, but misformatted code occurs in 4% of Claude-3.5-Sonnet tasks and 28% of GPT-4o tasks.Source: §4.2 · p. 9
• SWE-Agent GPT-4o fails to configure the Snowflake password field in up to 98% of tasks, compared with 12% for SWE-Agent Claude-3.5-Sonnet.Incorrect password fields are one of the configuration problems examined in the extraction and loading analysis.Source: Figure 6 · p. 8, §4.2 · p. 10
• 12 Spider-Agent Claude-3.5-Sonnet tasks repeat synchronization jobs, while SWE-Agent GPT-4o fails on all 24 multiple-flat-file cases.Repeated jobs inflate loaded table sizes; even SWE-Agent Claude-3.5-Sonnet fails on 70.8% of multiple-flat-file cases.Source: Figure 9 · p. 9, Figure 10 · p. 9, §4.2 · p. 10

4.3 Error Analysis of Data Transformation

• Only SWE-Agent Claude-3.5-Sonnet generates two correct data models; Stage 2 errors include runtime failures, DBT compilation errors, and SQL semantic errors.The evaluated agents entered Stage 2 with 35, 47, and 63 data models, respectively.Source: Figure 11 · p. 9, §4.3 · p. 10
• 84.1% of failed data model generations involve agent runtime errors, including premature termination and incorrect synchronization-status checks.SWE-Agent Claude-3.5-Sonnet fails to create 42 data models within its $6 maximum cost constraint.Source: Figure 11 · p. 9, §4.3 · p. 10
• 11 Spider-Agent GPT-4o data models have SQL semantic errors, while Spider-Agent Claude-3.5-Sonnet has 26 and SWE-Agent Claude-3.5-Sonnet has 8.Errors include incorrect schemas, missing columns, incorrect model size, and flawed SQL logic.Source: Figure 11 · p. 9, §4.3 · p. 10

5 CASE STUDY

• In first-stage analysis, Spider-Agent Claude-3.7-Sonnet reduced error frequencies across four issue categories versus Spider-Agent Claude-3.5-Sonnet by up to 95.8%, while failing on all data sources for nine tasks.Source: §5 · p. 11
• Spider-Agent Claude-3.7-Sonnet improves second-stage performance by 290% over SWE-Agent Claude-3.5-Sonnet, but its leading issues include excessive iterations, incorrect SQL logic, and invalid actions.These issues account for 28.7%, 24.3%, and 21.7%, respectively.Source: §5 · p. 11
• 83.6 steps are executed on average for each successful case, with most steps spent understanding the project base and generating the data model.Those phases average 20.6 and 17 steps, respectively.Source: §5 · p. 11

6.1 Multiple Runs Improve Agent Performance

• pass^k for Spider-Agent GPT-4o falls to 0 when k equals 5, indicating low consistency across repeated trials.The study uses pass^k to evaluate consistency and robustness.Source: §6.1 · p. 11, Figure 17 · p. 12

6.2 Using Documentation Improves Agent Performance

• Without documentation, Spider-Agent Claude-3.5-Sonnet succeeds in one task and Spider-Agent GPT-4o fails in all tasks during data extraction and loading.The reported success rates decrease from 21% to 1% for Claude-3.5-Sonnet and from 15% to 0% for GPT-4o.Source: §6.2 · p. 11, Figure 18 · p. 12

6.1 Multiple Runs Improve Agent Performance

• 57% of Spider-Agent GPT-4o tasks succeed in Stage 1 with five attempts, a 3.8× improvement over its pass@1 performance, but it still fails to build a correct Stage 2 model.The Stage 1 result means at least one of five attempts extracts data from multiple sources and loads it into the warehouse.Source: §6.1 · p. 11, Figure 16 · p. 12

ELT and ETL data pipelines

• Prior ETL and ELT automation work includes attribute mapping, automatic data loading, and machine-learning-based data integration; ELT-Bench targets agent automation of pipeline construction.Cloud data warehousing has driven increasing adoption of ELT pipelines [16] [36].Source: §7 · p. 12

Text-to-SQL benchmarks and methods

• Text-to-SQL research has moved from single-database datasets to cross-domain and large-scale benchmarks, while ELT-Bench evaluates complex transformation SQL based on column names and descriptions.The queries can involve nested subqueries and multi-table joins.Source: §7 · p. 12

AI agent benchmarks

• Existing data-agent benchmarks focus on code generation, analysis, or individual data-tool use; ELT-Bench is presented as the first benchmark for real-world, end-to-end ELT pipeline construction.Spider 2-V evaluates data tools individually, while Spider 2.0 assesses enterprise-focused text-to-SQL workflows.Source: §7 · p. 12

• The top-performing Spider-Agent Claude-3.7-Sonnet correctly generates data models in only 3.9% of cases.Source: §8 · p. 12
• ELT-Bench provides realistic ELT pipeline environments with diverse data sources and widely adopted data tools.Source: §8 · p. 12
• The low data-model accuracy highlights opportunities to develop more capable agents for complex ELT workflows.Source: §8 · p. 12