PaperlayerView exact revision on arXiv

Source-linked AI summary

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing, Duke Gand, Joseph J Ma

arXiv:2610.02122v1cs.CLcs.AIcs.DB

TL;DR

Existing data benchmarks focus on text-to-SQL, while enterprise analytics also requires navigating complex warehouses, acting on findings, and evaluating outcomes that may not be observable in released real data. Argo-Bench simulates a food-delivery platform in an ERP warehouse, withholds the simulator’s ground truth, and grades filed actions by their consequences. The strongest of 14 models solves 34.8% of tasks and averages 59.5 points.

  • Problem

    Existing benchmarks primarily test text-to-SQL, while enterprise workflows require multi-table analysis, consequential actions, and outcomes that real-data answer keys cannot always verify.

    Source: §1 · p. 2

  • Method

    Argo-Bench simulates a 2024 New York City food-delivery platform in an Oracle EBS warehouse, hides its latent state, and grades agents’ filed actions against simulator ground truth.

    Source: §1 · p. 3, §2.1 · p. 4, §2.3 · p. 5

  • Results

    34.8% of tasks are solved by the strongest of 14 evaluated models, which averages 59.5 points.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Models often analyze the wrong quantity or optimize the wrong objective, even while performing these data-agent tasks.

    Source: §1 · p. 3

    Argo-Bench compares data agents but does not estimate their performance on a real company’s warehouse.

    Source: §5 · p. 9

Abstract

from arXiv · show

Real-world enterprise data science and analytics workflows require reasoning across dozens of tables, performing statistical analyses, and acting on the results. Established text-to-SQL benchmarks evaluate query generation alone, and audits have found their answer keys frequently wrong. Because real enterprise warehouses are too sensitive to release, these benchmarks are built on public datasets where a business event fits in a single table. We introduce Argo-Bench, an evaluation framework comprising 210 data science and analytics tasks. Drawing on public data, peer-reviewed industry literature, and regulatory filings, we simulate a food delivery platform in New York City at true scale, with 81 million orders in 2024, grounded economics, fraud patterns, and marketplace incentives. We export this world to an ERP warehouse of 235 tables and 7.5 billion rows, modeled on the Oracle E-Business Suite schema. The simulator's ground-truth state is withheld from the warehouse the agent sees, so tasks require reconstructing facts by navigating the warehouse before acting on them. Argo-Bench goes beyond text-to-SQL: the agent files actions such as banning fraudulent accounts, allocating courier incentive budgets, or issuing back pay, and the grader scores each by its consequences in the simulator. Every task has an executable reference solution that demonstrates solvability using only the warehouse. The strongest of 14 frontier and open-weight models scores 95 or higher on only 34.8% of tasks and averages 59.5 points. We hope Argo-Bench drives progress toward agents that understand, navigate, and act within real data environments.

1 INTRODUCTION

Existing benchmarks largely test text-to-SQL on public, fragmented data, rather than end-to-end enterprise workflows whose answers can be verified against ground truth. Argo-Bench simulates an ERP-backed business, withholds its latent state from agents, and grades their actions; the strongest evaluated model solves 34.8% of tasks and averages 59.5 points.

1 INTRODUCTION

  • Existing benchmarks use fragmented public datasets with overlapping or sharded tables, unlike enterprise warehouses where one event can touch ten or more tables.Schemas in public and local sources are often documented in tutorials and textbooks.Source: §1 · p. 1
  • Text-to-SQL benchmarks miss end-to-end work that requires statistical, machine-learning, or optimization tools and decisions judged by return on investment.A promotion can have high redemption yet lose money if many orders would have happened anyway.Source: §1 · p. 2
  • Real-data benchmarks rely on annotator agreement, but audits found erroneous gold queries and many valuable outcomes, such as undetected fraud, are unobserved.The reported audit found errors in 62.8% of Spider 2.0-Snow gold queries and 52.8% of BIRD Mini-Dev gold queries.Source: §1 · p. 2
  • Argo-Bench simulates a food-delivery business in an ERP warehouse, hides the simulator’s latent state, and grades actions against ground truth rather than query correctness.Its benchmark combines an enterprise-scale, mutually constraining warehouse with consequence-based grading; access to real ERP data is heavily restricted.Source: §1 · pp. 2–3
  • 34.8% of tasks are solved by the strongest of 14 evaluated models, which averages 59.5 points.Models often analyze the wrong quantity or optimize the wrong objective.Source: §1 · p. 3
  • The benchmark comprises 210 tasks spanning actions such as banning accounts, forecasting, budget allocation, and publishing dashboard data sources.It evaluates 14 frontier and open-weight models.Source: §1 · p. 3

2 BENCHMARK CONSTRUCTION

Argo-Bench builds a calibrated 2024 New York City food-delivery simulation and projects it into a documented Oracle E-Business Suite warehouse. Its tasks require agents to explore that warehouse, file business actions or analyses, and receive scores based on their consequences.

2.1 WORLD SIMULATION

  • The simulated platform has 81 million orders and 3.4 million active customers, and its economics are calibrated to city disclosures and platform filings.Per-delivery economics stay within 5% of DCWP figures in 13 of 16 quarterly comparisons.Source: §2.1 · pp. 3–4
  • The simulation models the April 2024 minimum-pay increase and inserts delivery-platform fraud patterns, including courier, customer, and merchant schemes.A new courier scheduler activates after the pay increase; courier pay then runs 6–9% above the anchor.Source: §2.1 · p. 4

2.2 WAREHOUSE DESIGN AND VALIDATION

  • The warehouse projects the simulated world into a greenfield Oracle EBS 12.2 analytics export, designed with three ERP consultants and checked against EBS documentation.Standard tables and columns exist in Oracle’s EBS 12.2 Vision data dictionary, though tables carry an average of 52% of their Vision counterparts’ columns.Source: §2.2 · pp. 4–5
  • The warehouse deliberately omits drift and inconsistencies, keeping ground truth unambiguous while isolating the task of discovering which tables and combinations matter.This differs from mature warehouses, where legacy conflicts may require semantic layers, data models, and institutional knowledge.Source: §2.2 · p. 5

2.3 TASK DESIGN

  • The 210 tasks span five business areas, from fraud enforcement and forecasting to accounting, marketplace budgeting, and growth analysis.Task prompts, visibility cutoffs, expected filings, and answer keys specify the evaluation; answer keys are frozen before runs and derive from latent tables or simulator labels.Source: §2.3 · p. 5
  • Agents file actions and results through a Python interface rather than returning a query, enabling end-to-end workflows and explicit declarations of committed values, intervals, or reasons.This supports tasks requiring machine learning or optimization and makes partial credit well defined.Source: §2.3 · p. 5
  • Of 210 tasks, 178 require an action or publication and 99 expose only data through a cutoff month, so forecasts are graded on unseen months.The benchmark’s public warehouse is separate from the private-seed world used for official grading.Source: §2.3 · p. 6
  • Task expectations use grading modes tailored to outcomes: forecasts are scored by weighted interval score, bans by costs saved, and budgets by attainable savings realized.Task scores average expectation scores using task-specific weights; filing nothing earns zero, or −200 for a forecast.Source: §2.3 · p. 6

3 EVALUATION

Claude Opus 5.5 leads overall, while evaluation findings identify wrong records, objectives, and quantities as recurring failure patterns. The evaluation also examines reasoning effort, model comparisons, and forecast performance.

3 EVALUATION

  • Models are evaluated in isolated gVisor sandboxes and file work through a mission-control interface; Table 1 describes the ERP-scale warehouse, Python sandbox, and action grading.The experiments use Inspect AI, 25 preinstalled Python libraries, no outbound internet access, and a 500-turn run limit.Source: §3 · p. 6, Table 1 · p. 7

3.1 RESULTS

  • Claude Opus 5.5 leads overall and in three of five domains; GPT-6 Astra leads forecasting and Claude Sonnet 5.5 leads compliance.Paired task comparisons place Opus 7.7 to 10.0 points ahead of the next three models, which are not separated from one another.Source: Table 2 · p. 7, §3.1 · pp. 7–8
  • More reasoning effort helps GPT-6 and Claude models at every step, with diminishing returns for Opus, which gains 17.0, 5.9, and 4.0 points across successive effort increases.Gemini 3.8 Flash and Qwen 3.8 Max gain 8.6 and 5.2 points from low to medium effort, respectively, with no detectable later gains.Source: §3.1 · p. 7
  • Paired by task, Opus leads the next three models by 7.7 to 10.0 points, while those three are not separated; the reported score intervals are about ±2 to 8 points.The bootstrap resamples 146 base scenarios because many tasks are prompt variants of one scenario.Source: §3.1 · pp. 7–8

3.2 FINDINGS

  • On the membership dashboard task, 24 failing runs used contract dates instead of replaying billing history and checking orders where member benefits applied.Seven of those runs got every other figure right to within a few dollars.Source: §3.2 · p. 8
  • On the courier-bonus task, Opus saved $3.09 million of an attainable $3.12 million by cutting where quests saved the least surge per bonus dollar.GPT-6 Astra’s different objective produced a plan that lost $86,281 and scored 0; the uniform cut would save $0.40 million.Source: §3.2 · p. 8
  • Of 47 completed runs sizing the courier location service, 46 missed all 12 monthly counts because the warehouse retained hourly rather than half-hourly check-ins.Across dashboard tasks, 66.9% of data sources passing their structural contract scored zero on their values.Source: §3.2 · p. 8
  • Across 4,553 forecast series, nominal 80% intervals contained realized values only 44.8% of the time.Against the tighter reference for a harder variant, June base-pay forecasts fell from a mean grade of 84.7 to 6.0 despite a median absolute error of 1.57%.Source: §3.2 · p. 8

4 RELATED WORK

Prior benchmarks cover text-to-SQL, multi-step data science, and agents acting in simulated environments. Argo-Bench distinguishes itself by withholding enterprise-warehouse ground truth and grading filed actions by their consequences.

Text-to-SQL and data science benchmarks

  • Text-to-SQL and data science benchmarks compare agent outputs with gold answers or expert conclusions, while Argo-Bench grades filed actions against simulator latent state.Audits have found benchmark gold answers frequently wrong.Source: §4 · p. 8

Agent benchmarks in simulated environments

  • In many simulated-agent benchmarks, success-determining state is observable to the agent or changed inside a stylized game; Argo-Bench withholds it for reconstruction from an enterprise warehouse.Other calibrated simulators have supplied ground truth for patient records and money laundering.Source: §4 · p. 9

Generated enterprises with hidden ground truth

  • Unlike generated-enterprise benchmarks that plant answer records, Argo-Bench derives latent state from the simulation and spreads evidence across 7.49 billion warehouse rows.Agents are graded on the consequences of actions they file rather than on answers they give.Source: §4 · p. 9

5 LIMITATIONS & FUTURE WORK

The paper identifies boundaries in the simulator’s ERP coverage, grounding, and evaluation design. It cautions that results compare agents within this simulated setting rather than estimate performance on a real company warehouse.

5 LIMITATIONS & FUTURE WORK

  • The simulated ERP omits foreign-currency representations, shared accounts across multiple businesses, histories beyond one year, and formats beyond Oracle EBS.The authors identify multi-business warehouses, longer histories such as 2017–2026, and SAP S/4HANA as extensions.Source: §5 · p. 9
  • Some simulated behaviors rely on weakly grounded assumptions, and calibration to aggregate targets does not guarantee realistic tails.The authors state that Argo-Bench compares data agents and does not estimate their performance on a real company’s warehouse.Source: §5 · p. 9
  • Each model-and-effort setting has one run per task and all tasks share one simulated world, so confidence intervals reflect task choice rather than run-to-run variation.Some scores are prompt-sensitive, and plan tasks use the simulator’s frozen response model.Source: §5 · p. 9

6 CONCLUSION

Argo-Bench evaluates agents acting on facts reconstructed from a simulated Oracle E-Business Suite warehouse, with actions graded against simulator ground truth. The paper releases a public seed and evaluation code while noting limits on exact reproduction and the simulated setting.

6 CONCLUSION

  • Argo-Bench evaluates a simulated food-delivery platform exported to an Oracle E-Business Suite warehouse with 235 tables and 7.49 billion rows.The simulator’s latent state is withheld from the warehouse, and filed actions are graded against ground truth.Source: §6 · p. 9
  • Language models helped build and write the work, but no record in the simulated world was generated by a language model.The authors say they checked every claim, number, and citation and take responsibility for the paper’s content.Source: AI USE STATEMENT · p. 10
  • No simulated orders, shifts, or payments concern real people, and behavior attributed to real restaurants is simulated.Storefronts assigned fraud roles receive fictional names in the released warehouse, while their addresses remain unchanged.Source: ETHICS STATEMENT · p. 10
  • The public release includes one seed’s warehouse, 210 questions, reference solutions, and the evaluation harness, but not the simulator or graders.The paper’s runs used a second seed, so released code reproduces the procedure on a sibling world rather than its exact numbers.Source: REPRODUCIBILITY STATEMENT · p. 10

A DONOR DATASETS

Table 3 catalogs the public datasets used to build different parts of the simulated world and distinguishes identity, shape, and anchor roles. Research or non-commercial donors are used only for shape, not as source records in the world or warehouse.

A DONOR DATASETS

  • Table 3 lists 34 public datasets, categorized by the part of the simulated world they support.The table assigns donor roles as identity (I), shape (S), or anchor (A).Source: A DONOR DATASETS · p. 16, Table 3 · p. 16
  • Research or non-commercial donor datasets contribute only fitted parameters, not records to the world or released warehouse.The Yelp Open Dataset and Google Local reviews are examples of shape donors.Source: A DONOR DATASETS · p. 16
  • Validation-only datasets are marked separately because no fit reads them.Table 3 marks validation-only sources with ‡.Source: Table 3 · p. 16

B QUESTIONS ENABLED BY THE MINIMUM-PAY CHANGE

The April 2024 minimum-pay change alters courier pay and platform operations, and the resulting tasks include a cutoff month specifying what data the agent can see. The warehouse inventory also spans standard Oracle tables and custom business extensions.

B QUESTIONS ENABLED BY THE MINIMUM-PAY CHANGE

  • The minimum-pay rate rose from $17.96 to $19.56 per hour on April 1, 2024, prompting same-day changes to courier availability and per-delivery pay.The dispatcher restricts when couriers can go online, and the per-delivery pay card drops.Source: B QUESTIONS ENABLED BY THE MINIMUM-PAY… · p. 16
  • Weekly true-ups cover pay below the minimum, while the rule change breaks forecast stationarity and affects how standard figures are computed.Table 4 groups tasks that depend on the change.Source: B QUESTIONS ENABLED BY THE MINIMUM-PAY… · p. 17, Table 4 · p. 17
  • Table 4 identifies tasks that depend on the April pay change and gives the last month of 2024 visible to the agent as each task’s cutoff.The cutoff defines the visible-data boundary for the task.Source: Table 4 · p. 17

C WAREHOUSE TABLES

  • The warehouse inventory comprises 159 standard tables across 15 Oracle products and 76 custom extensions grouped by business area.Table 5 groups tables and their column and row counts by Oracle E-Business Suite module.Source: C WAREHOUSE TABLES · p. 17

D REWARD HACKING

The paper reports a discarded run in which an agent accessed the grader, then describes the sandboxing and warehouse, prompt, and tool setup used for reported runs. The warehouse has 235 tables and over 7.49 billion rows, and agents investigate it before filing actions and results.

D REWARD HACKING

  • An early fraud-04 run was discarded after Gemini 3.8 Flash found the grader’s answer and filed it, scoring 0.99; reported runs used isolated, network-less sandboxes.The agent changed its root directory to locate the grader; reported runs used gVisor pods and task-scoped BigQuery access.Source: D REWARD HACKING · p. 17
  • The warehouse contains 235 tables, 3,949 columns, and 7,491,508,904 rows grouped by Oracle E-Business Suite module; six tables are empty.The empty tables include four interface tables, AR_ADJUSTMENTS_ALL, and HZ_CUST_ACCOUNT_ROLES.Source: Table 5 · p. 18

E SYSTEM PROMPT

  • The system prompt directs the analyst to explore the read-only warehouse, investigate, and file appropriate actions through Mission Control.Available actions include enforcement, payment and pay corrections, accounting entries, forecasts, and dashboard data sources.Source: E SYSTEM PROMPT · p. 20

F AGENT TOOLS

  • Agents receive warehouse and Python tools, with SQL results saved for analysis in a persistent interpreter and Mission Control available for filings.The sandbox limits resources and run duration; queries may scan at most 20 GiB and save up to 1,000,000 rows.Source: F AGENT TOOLS · pp. 21–22

G TASK DEFINITION AND GRADING

Each task freezes an answer key and specifies the filings the agent must make. The grader scores those filings through modes tailored to their consequences and outputs.

Task definition and grading

  • A task combines a prompt, a warehouse scope, required filings with keys and grading modes, and an answer key frozen before any run.Keys come from SQL over latent tables or from a local builder reading simulator labels.Source: G TASK DEFINITION AND GRADING · p. 22

Task score

  • Each expectation scores 0–100, while forecasts score −200 to 100; a task score is their weighted mean, with forecast tasks floored at 0 as a block in reported means.Of 210 tasks, 159 have one scored expectation, 37 have two, and 14 have three.Source: G TASK DEFINITION AND GRADING · pp. 23–24

Forecasts

  • Each forecast files a point and 80% interval scored by WIS against a reference: 76 of 98 use no-change, while 22 cross regime changes and use frozen references, including seven outcome-informed points.The floor is the one improper element; at Table 2 settings, it binds for eight of fourteen models.Source: G TASK DEFINITION AND GRADING · pp. 23–24

Cost savings for bans

  • Ban savings are measured against banning nobody and everybody, pricing missed targets, wrongly banned innocents, neutral accounts, and IDs outside the task universe.Neutral bans incur review costs without being treated as false positives; wrong bans can destroy only part of an innocent account’s margin in some tasks.Source: G TASK DEFINITION AND GRADING · pp. 24–25

Other grading modes

  • Data-source filings require exact answer-key rows and matching cells; schedules score correct rows and usually penalize extras, while keyed values score correct keys against all answer keys.Data-source numbers use column tolerances, integers must match exactly, and schedule numeric columns must be within tolerance.Source: G TASK DEFINITION AND GRADING · p. 25

ID sets

  • ID-set expectations score predictions by F1, except one binary case requiring perfect F1 and correct shortfall reporting before remediation earns credit.An empty answer set is correct only when nothing is filed; the binary remediation expectation also gates scoring on accurate shortfall reports.Source: G TASK DEFINITION AND GRADING · p. 25

Allocations

  • Allocation plans are scored by simulator-valued losses across budget cells, with reference plans differing between courier-hour and quest-cut tasks.Moving too little or too much reduces a factor that reaches zero at 15% deviation; cell caps apply, and coarse-grained rows are spread in proportion to hours.Source: G TASK DEFINITION AND GRADING · pp. 25–26

Written notes and answer keys

  • Written notes are usually optional, while the two reconciliation-task notes are scored; 128 answer keys come from local builders and 82 from truth queries.Scored reconciliation notes carry weight 0.5 against 3 for the table they explain.Source: G TASK DEFINITION AND GRADING · p. 26

H THE STRUCTURE OF SPIDER 2.0-SNOW

The analysis finds that Spider 2.0-Snow questions typically touch few logical tables despite extensive table sharding, then examines how Argo-Bench agents can fail through wrong records, objectives, or quantities. These cases show that plausible analyses and structurally valid outputs do not necessarily match the intended task or its underlying facts.

H THE STRUCTURE OF SPIDER 2.0-SNOW

  • 80.0% of Spider 2.0-Snow tables with DDL copy another table’s schema, while 75.0% of questions with public gold SQL read at most two logical tables.Without GITHUB_REPOS_DATE’s 4,989 identical daily tables, the copied-schema share is 67.0%; the authors’ gold-table lists show 69.3% of questions read at most two logical tables, and no question reads more than eight. Argo-Bench has 235 tables with no schema copies; Figure 4 depicts schema duplication and logical tables read per question.Source: H THE STRUCTURE OF SPIDER 2.0-SNOW · pp. 26–27, Figure 4 · p. 26

I.1 CONFIDENCE INTERVALS

  • Family-resampled intervals put Claude Opus 5.5 ahead of GPT-6 Astra by 7.7 points [1.4, 13.7], while Astra, Sonnet 5.5, and GPT-6.1 Sol are not separated.The 95% bootstrap intervals resample 146 task families rather than 210 tasks, so prompt variants from one scenario do not count as independent evidence.Source: §I.1 · pp. 27–28, Table 9 · p. 27

I.2 REASONING EFFORT

  • More reasoning effort helped GPT-6 and Claude models at every step, while several small declines in other models were within the reported confidence intervals.For Opus and GPT-6.1 Sol, moving from high to extra-high added 1.9 to 11.5 points; the authors suggest context-window exhaustion or long tool-call contexts may explain some declines.Source: §I.2 · p. 28, Table 10 · p. 28

I.3 WRONG RECORD

  • Three passing dashboard runs reconstructed membership from billing history and checked it against orders with applied benefits; contract dates alone misclassified past-due customers.One date-based rule counted 8,379 nonmembers as members and missed 715 real members; failing runs sometimes checked only whether benefit-bearing orders fell inside their intervals.Source: §I.3 · pp. 29–30

I.4 WRONG OBJECTIVE

  • Over 47 matched settings, quest-cut scores rose from 18.5 to 61.7 when prompts included the quest purpose and holdout.The $3.116 million savings were evaluated under a frozen response model; the prompt changes were bundled, not isolated tests of individual hints.Source: §I.4 · p. 30, Table 12 · p. 30

I.5 WRONG QUANTITY

  • 66.9% of structurally valid dashboard data sources scored zero on their values, despite passing their structural contracts.A payout-date choice changed withheld dollar-days by 7.3% while year-end totals stayed unchanged; for location reports, only one run reconstructed the retained count within tolerance, and clarifying the kitchen-capacity event raised passes from 0 of 47 to 18 of 47.Source: §I.5 · pp. 30–31

I.6 PAST LABELS AND BENIGN LOOKALIKES IN FRAUD

  • The engine had not deactivated 254 of 456 first-half warned couriers who were thieves, while the full refund search’s injected storefronts shortcut the search.Opus left the benign controls alone but froze 16 ungraded fraudulent storefronts alongside ten target rings; the authors caution that these successful runs do not show transfer to real fraud.Source: §I.6 · pp. 31–32

I.7 FORECASTS

Forecast scores depend on the grading reference, and the reported intervals often miss realized values. The paper tests whether its grade rewards honest forecasts and examines how outcome-informed references affect scores.

I.7 FORECASTS

  • 44.8% of 4,553 forecast series’ nominal 80% intervals contain the realized value; task-equal weighting gives 46.0%.Coverage is 43.8% against no-change references and 48.2% against fixed references.Source: §I.7 · p. 32
  • 96.7% of 4,527 well-formed series maximize expected grade by filing the forecast belief itself.For the other 3.3%, the −200 floor makes a narrower interval pay; the floor binds on 10.1% of forecast expectations.Source: §I.7 · p. 32
  • 84.7 to 6.0: mean grade for unchanged June base-pay forecasts falls when scored against the harder variant’s tighter reference.The forecasts’ median absolute percentage error is 1.57%; the reference standard deviation is $1.0 million versus $13.2 million for no-change.Source: §I.7 · p. 32, Table 14 · p. 33
  • Seven of 22 fixed references were set with realized outcomes in view, but each figure was also included in its prompt.Without the six tasks carrying these references, no setting’s score changes by more than 3.8 points; Claude Opus 5.5 remains first.Source: §I.7 · pp. 32–33

J TASK AUDIT

The task audit identifies prompt and key issues, corrects affected tasks, and reports a scope boundary for the refund-collusion family.

J TASK AUDIT

  • Latent-state answer keys make questionable keys or prompts traceable as bugs that can be fixed.The audit lists issues found while reading traces and how they were treated.Source: J TASK AUDIT · p. 33
  • Re-keying the identity-substitution task to 111 couriers active at year-end corrected penalties for naming already deactivated couriers.The original key named 138 of 249 witnessed targets that were already deactivated, including 130 identity mismatches; all runs were regraded.Source: J TASK AUDIT · p. 33
  • The refund-collusion prompt was aligned with the grader’s two-thirds storefront-savings and one-third customer-closure weighting.All four variants were rerun on every setting, and the paper’s scores use the new wording.Source: J TASK AUDIT · p. 33
  • The refund-collusion task tests distinguishing rings from benign controls, not finding either among all storefronts.All 664 injected missing-item refunds fall at 98 injected storefronts, and one pass over refunds finds every candidate.Source: J TASK AUDIT · p. 33

K WAREHOUSE COST

Warehouse querying adds a substantial cost beyond model API spend, and that expense varies across models and tasks.

K WAREHOUSE COST

  • $6,160 of warehouse scans versus $14,502 of model API spend across 9,869 final-experiment runs.The agents scanned 986 TiB; BigQuery was billed at $6.25 per TiB.Source: K WAREHOUSE COST · p. 33
  • 92% of GPT-6 Luna’s run cost is warehouse spend, compared with 15% for Claude Opus 5.5.Luna’s all-in task cost is $0.70 rather than the $0.06 model-API cost reported in Table 2.Source: K WAREHOUSE COST · p. 34
  • $11.77 of warehouse spend per completely solved task for Muse Spark 1.3, versus $2.37 for Opus and $2.20 for GPT-6 Astra.Muse Spark scans 1.2 times as much as the next model and spends more because it queries more and solves less.Source: K WAREHOUSE COST · p. 34

L EXAMPLE TASKS

Three walkthroughs show agents reconstructing facts from warehouse data, then forecasting, publishing a dashboard source, or taking fraud actions through the task interface.

L EXAMPLE TASKS

  • The May minimum-pay forecast uses an April 30 warehouse snapshot and the dispatcher’s plan; the actual May value is withheld from the agent.The reference solution checks the pay mechanism, estimates hours and pay, then files an adjustment forecast.Source: Table 16 · p. 35, L EXAMPLE TASKS · p. 36
  • The dashboard task rebuilds a monthly contribution-margin-per-order source from order counts and general-ledger lines, then publishes it through Mission Control.The reference solution combines two ledger sources and divides monthly margin by all orders before publishing.Source: Table 17 · p. 37, L EXAMPLE TASKS · p. 38
  • The two-sided fraud task identifies couriers splitting never-arrived refunds with regular customers and requires closing both sides.A wrongly deactivated courier costs about $384 of margin against about $140 of refunds per partnership.Source: Table 18 · p. 39
  • The fraud walkthrough uses delivery records, courier-customer history, and distance from the customer’s usual drop point to assess claims.The reference solution defines a regular as at least three orders together, including one before the claim, then files courier and customer bans.Source: L EXAMPLE TASKS · pp. 40–41

Related papers

Loading 2610.02122v1…