ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time

1 Introduction

• Production tickets rarely specify tolerance to redelivery, leaving this operational knowledge with the engineer rather than in the prompt.The paper argues that evaluations must deliberately probe such unstated requirements.Source: §1 · p. 1
• Existing evaluations run stateful artifacts once against a fixed snapshot, so they cannot test whether a pipeline remains correct when records arrive again or out of order.Of nine benchmarks audited, seven produce stateful artifacts; a second run on the same data is identified as the cheapest test that would catch this gap.Source: §1 · p. 1
• The oracle compares each scheduled run’s final state with a batch recomputation of the complete logical log, giving crashes and silently wrong tables distinct verdicts.The recomputed target is schedule-independent, so discrepancies indicate temporal defects.Source: §1 · p. 2
• Snapshot evaluation certifies 86–100% of pipelines, while temporal evaluation separates the same artifacts across an elevenfold range; the snapshot repair loop does not produce the gap.The benchmark contains 40 tasks and 42 negative controls; the replay-invariance oracle has verdicts that reproduce exactly.Source: §1 · p. 2

2 Related work

• Seven audited stateful-artifact benchmarks grade once; ADE-Bench varies database or project, while DataAgentBench’s Pass@1 varies agent runs on byte-identical data.The two text-to-SQL benchmarks are treated as appropriately using single execution [Lei et al., 2025, Chen et al., 2024].Source: §2 · p. 2
• ArrivalBench extends work on prompting defaults to data pipelines; prior reports include safe-code recovery with explicit instructions [Xu et al., 2026] and degradation from narrowing instructions [Litvak, 2026].The paper frames its displacement finding as a measured mechanism, with the mediating variable observable in generated SQL.Source: §2 · p. 3
• Unlike safe/unsafe classification, batch recomputation gives crashes and silently wrong answers different verdicts.The paper says this distinction enables its intervention analysis.Source: §2 · p. 3
• Temporal non-interference forbids information flowing backward in time, whereas ArrivalBench requires identical final state under any delivery order; neither property implies the other [Fonseca, 2026].The cited work statically verifies hand-written pipelines, while ArrivalBench executes generated ones.Source: §2 · p. 3
• Streaming systems use exactly-once state, log replay, and algebraic properties such as idempotence, while declarative tests check outputs; ArrivalBench asks whether generated code provides these guarantees unprompted.The cited examples include Flink, Kappa, and declarative data tests [dbt Labs, 2026].Source: §2 · p. 3
• ArrivalBench’s replay-invariance oracle and separate FAIL/ERROR verdicts distinguish a 37.7-point improvement from a 7.0-point one.The paper identifies the oracle and verdict separation as what is new, rather than the elicitation result itself.Source: §2 · p. 3

3 Method

• Each schedule partitions the same logical event log into batches, and correctness requires the pipeline’s final state to equal the batch recomputation after the last batch.Schedules can duplicate, reorder, withhold and later release events, or abort mid-batch.Source: §3 · p. 3
• The schedule changes delivery, not logical input content, making this metamorphic test an environmental transformation.The design adapts fault injection and deterministic simulation testing to an artifact an agent leaves behind [Kingsbury and Alvaro, 2020].Source: §3 · pp. 3–4
• Six documentation-grounded hazards cover late arrival, out-of-order changes, duplicates, crash-and-retry, backfill, and additive schema evolution; two in-order controls test ordinary incrementality.Schema evolution raises an error rather than silently corrupting state and is reported separately from the other five hazards.Source: §3 · p. 4
• Agents receive schemas and a persistent-database, once-per-batch execution contract, but no hazard information; a word-boundary scan checks for leakage.The contract specifies no cumulative table of past arrivals.Source: §3 · p. 4
• The snapshot-only repair loop allows at most three turns, ends when the script passes, and grades only the final script without naming hazards.Snapshot-pass is the incumbent metric’s best case here: 79.5% of cells pass on the first turn, and the loop lifts the per-arm figure by up to 32.5 points.Source: §3 · p. 4
• The admissibility gate checks that gold pipelines pass declared hazards and controls, negative controls fail only where declared while passing snapshots, and hazards actually perturb delivery.It rejected five vacuous negative controls, two over-declared hazard sets, and one under-powered generator; duplication controls must reflect task aggregation semantics [Budiu et al., 2022].Source: §3 · p. 4
• Rates are task means; the primary conditional gap is temporal failure among snapshot-pass artifacts, while per-hazard pass and ERROR-inclusive failure use different denominators.A loud failure on one hazard can mask a silent failure on another in the same cell.Source: §3 · p. 4
• The same task, schedule, seed, and artifact produce byte-identical verdicts; 28,350 verdicts were stored, of which 28,041 were gradeable.The remaining 309 verdicts belong to 39 cells where no pipeline artifact was emitted.Source: §3 · p. 4

4.2 Snapshot evaluation cannot separate these models

• Single-execution grading gives eleven models near-identical, near-perfect snapshot scores despite an elevenfold temporal-correctness spread; excluding Codestral, the gap remains 7.0–73.7%.Table 1 contrasts the incumbent methodology with the failure signature; Figure 2 plots snapshot-pass against silent-failure rate.Source: Table 1 · p. 5, Figure 2 · p. 5, §4.2 · p. 6

4.1 Verifying the oracle

• 28,041 gradeable verdicts re-graded with zero mismatches, and four hand-derived oracle checks included a known failure.Two harness defects were found by audit; the larger changed one arm’s gap by 2.0 points.Source: §4.1 · p. 5

4.2 Snapshot evaluation cannot separate these models

• Within the same arm and task, repaired and first-turn snapshot passes differed by only +0.2pp in replay failure, with an interval spanning zero.Independent reruns moved rates by at most 5.9 points; restricting the gap to cells passing both benign controls leaves a 7.0–57.6% range across the same ten arms.Source: §4.2 · p. 6

4.3 Exactly-once is the weaker axis in every arm

• +4.0 to +39.4pp: ordering hazards had higher mean pass rates than idempotency hazards in all eleven unhinted arms, though a shared anti-join idiom may shape the pattern.The group-mean comparison avoids the weakest-link statistic’s built-in bias toward a positive margin.Source: §4.3 · pp. 6–7

4.4 A small human anchor

• −12.7pp: on five tasks, the human arm’s weakest ordering hazard passed less often than its weakest idempotency hazard; four people and nineteen cells cannot separate participant from task effects.The inversion also holds under two alternative comparisons, but this small arm is an anchor rather than a matched control.Source: §4.4 · pp. 7–8

4.5 Where prompting converts silent failure into loud failure

• 48.2%→10.5% silent failure under Sonnet hazard hinting came with crashes rising 9.0%→37.0%, leaving all-in failure only 51.0%→44.0%.For frontier models, the same warning reduced the measured gap to 0.0%; the small-model architecture intervention instead reduced all-in failure from 67.5% to 32.0%.Source: §4.5 · p. 8

4.6 A partial specification can displace a model’s own defences

• 72.2% of unguarded cells silently failed duplicate delivery, versus 0% of guarded cells; guarded cells instead crashed 40.3% of the time.Across arms, guarded pipelines silently failed less often in 12 of 13 comparisons, an exploratory association; the prompt specified history-based deduplication but not within-batch duplicates.Source: §4.6 · p. 9

• Across seven intervention arms, five lower all-in failure, one converts silent failures into crashes, and one performs worse than no prompt.Error composition also differs by arm.Source: §5 · p. 9
• The 40 tasks, golds, and 42 negative controls were written by one author, and results come from one engine and dialect.Every gold uses one anti-join staging idiom, so the idempotency findings generalize less far; recalculating over 35 tasks declaring both hazard classes keeps the rule positive in eleven of eleven.Source: §5 · p. 9
• 40.3% of 67 guarded cells crash on duplicate delivery with no silent failures, while 72.2% of 133 unguarded cells fail silently with no crashes.Dropping the guard raises silent failure in 12 of 13 arms; the exception is a tie.Source: Figure 5 · p. 10
• Schema evolution is declared by only 2 of 40 tasks, and the four-person, 19-cell human anchor carries no claim.The human anchor leaves open whether the signature reflects task design, shared staging idiom, or dialect.Source: §5 · p. 10

• Re-executing artifacts separates models across an elevenfold range, while snapshot evaluation certifies ten of eleven models at 97.9% or better.In every unhinted arm, failures concentrate on exactly-once semantics rather than event-time ordering.Source: §6 · p. 10
• Whether the failure pattern reflects the models, task design, shared staging idiom, or dialect remains open.Source: §6 · p. 10
• The conclusion recommends rerunning pipelines against the same data and treating prompts as incomplete specifications.Whether an invariant constraint beats a query that assumes it remains open; a uniform template baseline keyed to each task’s primary key could test it without model calls.Source: §6 · p. 10

• Generative AI tools helped implement the evaluation harness and analysis scripts, list candidate hypotheses and failure modes, and edit the prose.Source: AI use statement · p. 10
• The LLM assistant drafted 19 of 40 benchmark tasks, and 11 more were generated from templates; every task passed the admissibility gate.Source: AI use statement · p. 10
• The authors state that the research questions, design, result interpretation, claims, and artifacts are their responsibility.Evaluated model outputs are experimental data, not evidence about the work itself.Source: AI use statement · p. 10

• Four practising data engineers participated voluntarily and unpaid by invitation among professional contacts.Source: Ethics statement · p. 10
• Participants received consent information, could withdraw after seeing their result, and were told submissions would be graded and reported in aggregate.Source: Ethics statement · pp. 10–11
• No personal, demographic, or employer-identifying data was collected; the exercise was not institutionally reviewed and was judged to pose no foreseeable risk beyond ordinary professional work.The tasks are synthetic and the benchmark does not target a deployed system.Source: Ethics statement · p. 11

A How the audited benchmarks grade

• Seven of nine audited benchmarks produce stateful pipelines, and all seven grade them with one execution; the other two are text-to-SQL.For stateful artifacts, a single run leaves temporal correctness unobserved; a single run is inherent to text-to-SQL query artifacts.Source: A How the audited benchmarks grade · p. 13, Table 2 · p. 13
• None of the audited benchmarks re-executes an artifact under varied delivery, so none can observe the reported temporal failure mode.The authors characterize this as a narrower scope distinction, not a claim that the benchmarks are wrong or undemanding on other axes.Source: A How the audited benchmarks grade · p. 13
• Gaia2 evaluates an agent’s trajectory, while DataAgentBench’s Pass@1 resamples generation against identical data rather than replaying one artifact.The table contrasts execution count and whether a wrong result and a crash receive different verdicts; Gaia2 grades a trajectory, not an artifact.Source: Table 2 · p. 13, A How the audited benchmarks grade · p. 14

Scope of the evaluation

• 40 tasks were authored by one person, the work uses DuckDB and SQL, and all eleven models were accessed through hosted APIs.Seven open-weights arms used one third-party aggregator; intervention arms cover only commercial models, and the evaluation materials are not publicly available with this version.Source: B Full limitations and disclosures · p. 14

The two harness defects in full

• The 8,000-token cap truncated fourteen cells on the longest tasks; headline results retain that uniform cap for like-for-like comparison.Thirteen of those cells produced a pipeline when re-run at a doubled cap, and the resulting gap shifts were within the reported intervals.Source: B Full limitations and disclosures · p. 14

A third defect, found by re-running an arm

• A NULL-key sort bug aborted 18 Codestral tasks, but completed verdicts were unaffected; no corpus cell exercises the repaired path.Regression tests cover the correction, so its behavior rests on tests rather than corpus data; re-executing an arm exposed the defect.Source: B Full limitations and disclosures · pp. 14–15

Replication in full

• 5.9 points was the largest rate movement across independent reruns; the failing pipelines largely recurred despite rate changes.The broader replication re-ran all eleven unhinted arms from scratch, with rates moving by at most 5.9 points.Source: B Full limitations and disclosures · p. 16, Table 3 · p. 16

Idiom diversity across the gold pipelines

• Every gold pipeline uses anti-join staging, limiting how far the idempotency result generalizes despite 11 distinct tracked signatures across the suite.The shared staging idiom is also why Figure 4 is described as representative rather than anecdotal.Source: B Full limitations and disclosures · p. 16

The architecture arms and the guard

• Three architecture arms across model tiers confound guard rate with capability, so the authors do not treat them as evidence.They also report that a second engine is the most valuable extension, since the DuckDB dialect may shape the observed signature.Source: B Full limitations and disclosures · pp. 17–18

Watermarks and the late-arrival hazard

• The late-arrival verdict assumes convergence to full batch recomputation; a bounded-state system could instead choose to score watermark drops differently.The schedule delays 5% of eligible events to the final batch, beyond any threshold short of the whole run.Source: B Full limitations and disclosures · p. 18

Human baseline, per-cell detail

• The four-person human baseline spans only five tasks, and a different task sample could substantially change its counts.The authors treat it as an anchor rather than a matched control; the by-hand verification of schema evolution also covered only four cells and omitted crash-and-retry semantics.Source: B Full limitations and disclosures · pp. 15, 18

The agent

• 79.5% of cells pass the snapshot on the first turn, and the repair loop raises per-arm snapshot-pass by up to 32.5 points.The loop tests the script on the snapshot path, returns an execution error or typed diff after failure, and ends when it passes or reaches three turns.Source: C The agent, the prompts, and the… · pp. 18–19, Table 6 · p. 19
• Snapshot scores span 64.0–100.0%, while temporal evaluation spans 7.0–79.2% and leaves the arms in a different order.The reported post-loop comparison is the headline because it reflects the condition agents actually ran under.Source: C The agent, the prompts, and the… · p. 19

The prompts

• The snapshot test description says to load the complete log as one batch, although the execution contract runs the script once per delivery batch.Every arm received the same test paragraph, so the passage says the between-arm comparison is unaffected, though models could interpret it as the deployment setting.Source: C The agent, the prompts, and the… · p. 19
• The architecture prompt asks agents to persist delivered-event history with event_id deduplication, then derive the target as a pure function of that history.It specifies stored-history deduplication but says nothing about duplicates arriving together.Source: C The agent, the prompts, and the… · p. 20

The human arm

• Four of five invited practicing data engineers returned work, producing 19 cells across five tasks, with one task declined.They worked untimed without a repair loop; their 68.4% snapshot-pass rate was below every model arm, and the authors say the lack of prior assistant use was not enforced.Source: C The agent, the prompts, and the… · p. 20
• The task selection covered all six delivery hazards, three task modes, and three difficulty bands using a deterministic rule fixed before submissions arrived.The same oracle grades benign controls and hazards, making the in-order stream result interpretable as ordinary incrementality failure.Source: C The agent, the prompts, and the… · p. 20