PaperlayerView exact revision on arXiv

Source-linked AI summary

ERPBench: Evaluating LLM Agents for Enterprise Decision-Making Across Competitive Market Ecologies

Xinran Zhang, Pengrui Lu, Lyumanshan Ye, Pengfei Liu

arXiv:2609.04667v1cs.AI

TL;DR

Enterprise-agent evaluations rarely test whether business-decision conclusions transfer across competitive market ecologies. ERPBench compares the same 100 ERP problems in matched Solo and Arena settings, finding the same task-level winner on only 21 problems.

  • Problem

    Existing evaluations rarely test whether enterprise-agent business-decision conclusions transfer across competitive market ecologies, where agents’ decisions are strategically coupled through shared markets.

    Source: §1 · p. 1

  • Method

    ERPBench evaluates the same 100 ERP problems in matched Solo markets with fixed rule-based opponents and Arena markets shared by six evaluated LLM agents.

    Source: §3.2 · p. 4

  • Results

    Only 21 of 100 problems had the same task-level winner across ecologies, while aggregate leadership shifted from DeepSeek in Solo to Gemini in Arena.

    Source: §4.2 · p. 5, §4.5 · p. 6

  • Takeaways & Limitations

    A single-ecology leaderboard can give an incomplete conclusion because model-selection conclusions may not transfer across specified competitive market ecologies.

    Source: §5 · pp. 7–8

    Logged execution interventions are process diagnostics rather than causal decompositions of terminal valuation, and capacity skips can reflect environment constraints rather than malformed decisions.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly proposed for enterprise workflows, yet existing evaluations rarely test whether business-decision conclusions transfer across competitive market ecologies. We introduce ERPBench, an execution-instrumented benchmark for enterprise decision agents in a six-round Enterprise Resource Planning (ERP) simulation with coupled pricing, production, procurement, inventory, finance, and shared-market competition. ERPBench evaluates the same 100 fixed problems in two matched competitive market ecologies: Solo, where each evaluated LLM agent competes against fixed rule-based opponents, and Arena, where six evaluated LLM agents compete in a shared market. Across six model families, this yields 1,200 model-level trajectories spanning 7,200 decision rounds. Under the observed service configuration, the leading model differs between ecologies: DeepSeek leads in Solo (252.29M mean valuation; mean rank 1.67), whereas Gemini leads in Arena (263.95M; 1.76). The two ecologies identify the same task-level winner on only 21 of 100 problems, and Gemini's bottom-rank rate falls from 22 % to 0 % in Arena. ERPBench supports paired evaluation of whether enterprise-agent rankings transfer across competitive market ecologies, supplemented by aggregate execution-intervention analysis. Code and benchmark resources are available in our https://github.com/GAIR-NLP/erp-bench.

1 Introduction

ERPBench tests whether enterprise-agent performance conclusions transfer across matched competitive market ecologies, pairing the same ERP problems across conditions and recording execution-audit evidence alongside outcomes.

Introduction

  • Terminal valuation uses a six-round dividend discount model, complemented by within-problem rank, bottom-rank rate, normalized regret, and execution-audit records.The audit links parsed decisions and feasibility checks with clamping, fallback, and runtime recovery where available.Source: §1 · p. 1
  • Opponent behavior changes prices, demand allocation, inventory pressure, market share, and cash flows, so competitive ecology should be a controlled measurement condition.An agent strong against stable rule-based competitors may lose its advantage when evaluated LLM agents shape the same market.Source: §1 · pp. 1–2
  • ERPBench records parsed decisions, feasibility checks, clamping, fallback, simulation updates, and terminal outcomes because plausible actions may violate constraints or require repair.This process-level record distinguishes how a score was produced, not just the terminal score itself.Source: §1 · p. 2
  • Across the same 100 ERP problems, paired Solo and Arena evaluations yield 1,200 model-level trajectories and 7,200 decision rounds across six model families.Solo uses fixed rule-based opponents, whereas Arena places six evaluated LLM agents in a shared market.Source: §1 · p. 2, Figure 1 · p. 2
  • 21 of 100 problems share the same task-level winner across ecologies; DeepSeek leads Solo, Gemini leads Arena, and tail-risk profiles differ.The benchmark reports these as aggregate findings from its ecology comparison.Source: §1 · p. 2
  • ERPBench supplies an ERP simulation jointly constraining pricing, production, procurement, inventory, and finance decisions over six rounds.The independently constructed simulation also covers supply chain, market competition, and carbon costs.Source: §1 · p. 2

2 Related Work

ERPBench situates enterprise-agent evaluation across stateful, professional-workflow, economic-agent, and benchmark-validity research. Its distinguishing focus is whether enterprise decision conclusions transfer across competitive market ecologies, with execution audits as part of measurement.

Stateful executable agent benchmarks

  • Stateful executable benchmarks establish reproducible, stateful, outcome-verifiable agent evaluation, while ERPBench asks whether conclusions remain stable when competitive market ecology changes.Prior examples include AgentBench, SWE-bench, WebArena, OSWorld, and 𝜏-bench (Liu et al., 2024; Zhou et al., 2024; Xie et al., 2024).Source: §2 · p. 2

Enterprise and professional agent benchmarks

  • Workplace and CRM benchmarks test agents in professional workflows, whereas ERPBench measures executable, jointly constrained ERP decisions rather than office-task or CRM completion.The cited examples are TheAgentCompany, WorkArena++, CRMArena, and CRMArena-Pro (Xu et al., 2025; Boisvert et al., 2024; Huang et al., 2025, 2026); ERPBench couples pricing, production, procurement, inventory, capacity, finance, and valuation.Source: §2 · pp. 2–3

Economic and competitive agents

  • ERPBench makes competitive ecology an experimental factor by comparing whether the same model–problem pair yields the same conclusion in fixed-opponent Solo and shared-market Arena.It builds on economic-agent benchmarks including CompeteAI, EconAgent, InvestorBench, RetailBench, and Market-Bench (Zhao et al., 2024; Li et al., 2024, 2025; Zhang et al., 2026; Zheng et al., 2026).Source: §2 · p. 3

Benchmark validity, trajectories, and protocol sensitivity

  • ERPBench addresses benchmark validity and protocol sensitivity with round-level decision records, execution-audit information, and paired outcomes across controlled competitive ecologies.Audits record whether proposed actions could be parsed, repaired, clamped, or executed under ERP constraints; related work includes 𝜏2-Bench (Barres et al., 2025), protocol-sensitivity analyses (Alzahrani et al., 2024), and BetterBench (Reuel et al., 2024).Source: §2 · p. 3

3 ERPBench

ERPBench evaluates executable, operationally coupled ERP decisions in a six-round cereal-manufacturing simulation. It compares matched model-family–problem trajectories across Solo and Arena ecologies using terminal valuation and structured reliability, ranking, and task-family measures.

3.1 Operational Simulation and Executable Interface

  • Six rounds of coupled pricing, production, procurement, inventory, capacity, cash, and valuation make terminal performance depend on both strategic choices and operational feasibility.The simulation covers 12 SKUs across three distribution channels, with batch and lot-size constraints, supplier lead times, carbon costs, and competitive demand.Source: §3.1 · p. 3
  • Agents receive ERP state and tool outputs, return structured JSON actions, and face parsing, feasibility checks, and repairs before each decision is executed.Prices, stock, procurement, and finance are interdependent, so early errors can propagate through inventory, capacity, and cash-flow constraints before affecting valuation.Source: §3.1 · p. 4

3.2 Paired Ecologies and Measurement Protocol

  • The same 100 problems are evaluated in Solo against five fixed rule-based opponents and in Arena against five other evaluated LLM agents, with problem conditions and measurement held matched.Each trajectory spans six rounds; the primary outcome is sixth-round terminal company valuation, alongside within-problem rank, winner agreement, normalized regret, bottom-rank rate, and intervention categories.Source: §3.2 · p. 4
  • Model-level effects use 95% percentile-bootstrap confidence intervals, while paired valuation tests use Wilcoxon signed-rank tests with Benjamini–Hochberg correction across six model-wise tests.Intervention categories are non-exclusive diagnostics, and the Decision Quality Pyramid organizes structural validity, operational feasibility, rank-derived performance, hard-task performance, and rank improvement as descriptive proxies; task strata test heterogeneity.Source: §3.2 · p. 4

4 Experiments

ERPBench compares six LLM families across two matched competitive ecologies and finds that aggregate and task-level rankings change between Solo and Arena. Execution audits and difficulty-stratified results qualify how terminal valuations should be interpreted.

4.1 Setup

  • The paired protocol evaluates six model families on the same 100 problems in both ecologies, yielding 1,200 trajectories and 7,200 decision rounds.Coverage checks found all expected records with no duplicates or omissions; Arena–Solo contrasts therefore pair the same problem, model family, and seed under the observed service configuration.Source: §4.1 · pp. 4–5

4.2 Ecology Reorders Aggregate Performance

  • DeepSeek leads Solo (252.29M mean valuation; mean rank 1.67), while Gemini leads Arena (263.95M; 1.76) on the same 100 problems.Gemini gains +94.54M in Arena, while DeepSeek and Qwen decrease by 17.58M and 18.74M, respectively.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 Terminal Scores Need Trace Qualification

  • Execution-audit records link model outputs and parsing to feasibility checks, repairs, fallbacks, executed actions, and resulting ERP states.This audit layer distinguishes valuation, transfer across ecologies, and intervention burden, since identical terminal scores can arise through clean execution or repeated recovery and repair.Source: §4.3 · p. 5

4.4 Execution Audit Exposes Execution Fragility

  • Production-capacity skips account for 3,761 logged interventions (82.5%), followed by unavailable investment choices (355, 7.8%), fallbacks (275, 6.0%), clamping (118, 2.6%), and tracebacks (48, 1.1%).Capacity skips may reflect production constraints rather than malformed decisions, whereas fallbacks and tracebacks more directly indicate output-pipeline or runtime fragility; the displayed shares omit five run-level failures.Source: §4.4 · pp. 5–6, Figure 3 · p. 5

4.5 Task-Level Rankings Disagree

  • The ecologies select the same task-level winner on only 21 of 100 problems, with mean within-problem Spearman rank correlation 0.182 (bootstrap CI [0.090, 0.273]).DeepSeek-to-Gemini is the largest transition: Gemini wins in Arena 35 problems DeepSeek won in Solo; Figure 4 shows the broad disagreement across matched tasks.Source: §4.5 · p. 6, Figure 4 · p. 6

4.6 Reliability Qualifies Terminal Scores

  • Gemini’s bottom-rank rate falls from 22% in Solo to 0% in Arena, while its P90 normalized regret falls from 0.73 to 0.27 and DeepSeek’s rises from 0.16 to 0.39.In Arena, fewer logged interventions tend to coincide with higher valuation, but the six-model relationship is descriptive, not inferential; Solo uses stable opponents, whereas Arena agents compete against one another.Source: §4.6 · p. 6, Figure 5 · p. 7

4.7 Difficulty Conditions the Ecology Effect

  • Arena–Solo effects vary across difficulty strata: Gemini benefits consistently, while other model families show smaller, mixed-direction shifts.Uncertainty varies with stratum size and operational difficulty, and the scale of the ecology effect depends on task pressure rather than acting as a uniform offset.Source: §4.7 · p. 7, Figure 6 · p. 7

5 Discussion and Limitations

ERPBench shows that single-ecology leaderboards can miss whether enterprise-agent conclusions transfer across competitive settings: DeepSeek appears strongest against fixed opponents, while Gemini leads in Arena. The findings are benchmark contrasts under the observed service configuration, not universal rankings or route-invariant causal effects.

5 Discussion and Limitations

  • DeepSeek appears strongest as an independent optimizer in Solo, while Gemini becomes the aggregate leader and avoids bottom-rank outcomes in Arena.Solo tests performance against stable rule-based competitors; Arena lets evaluated LLM agents shape prices, demand allocation, inventory, and market outcomes. Neither ecology is universally superior or uniquely correct.Source: §5 · pp. 7–8
  • Execution audits characterize parsing recovery, feasibility repair, clamping, and runtime recovery, but logged interventions are not causal mechanisms or interchangeable error units.Production-capacity skips may reflect binding environment constraints, whereas fallbacks and tracebacks provide stronger evidence of output-pipeline fragility.Source: §5 · p. 8
  • Ecology effects vary by problem: models with near-neutral aggregate changes can improve on some tasks and worsen on others, indicating a problem-dependent interaction.Terminal valuation and rank establish the benchmark comparison, while aggregate audit summaries support process-level analysis.Source: §5 · p. 8
  • Results are limited to a controlled ERP-inspired simulation with a fixed 100-problem slice, six-round horizon, and one six-model Arena composition.Additional ecologies, fixed-opponent sets, longer horizons, or information regimes may yield different results; service-side routing and endpoints may change, so model-family labels do not guarantee immutable or route-identical snapshots. Future work should test candidate mechanisms involving price following, capacity, inventory, and market share.Source: §5 · p. 8

6 Conclusion

Across paired ERPBench evaluations, the leading model changes from DeepSeek in Solo to Gemini in Arena, while task-level winners, difficulty effects, and tail-risk profiles also change. The findings support evaluating whether enterprise-agent conclusions transfer across competitive market ecologies and how executable decisions are produced.

6 Conclusion

  • The leading model changes from DeepSeek in Solo to Gemini in Arena, while task-level winners, difficulty effects, and tail-risk profiles also change.The comparison covers 100 matched ERP problems across two competitive market ecologies.Source: §6 · p. 8
  • ERPBench evaluates six model families across 100 matched ERP problems and 1,200 model-level trajectories, with logged intervention evidence.Source: §6 · p. 8
  • Enterprise-agent evaluation should report whether model-selection conclusions transfer across competitive market ecologies and how executable decisions are produced.Source: §6 · p. 8

A Problem Set and Coverage Audit

The benchmark uses matched Solo100 and Arena100 panels with the same 100 problem identifiers, enabling direct ecology comparisons for each model–problem pair. Coverage checks verified expected records, while logged round-level data supported analysis of decisions, feasibility events, simulation states, and execution interventions.

A Problem Set and Coverage Audit

  • 100 problems use the same identifiers in Solo100 and Arena100, giving each model–problem pair a direct ecology comparison.Each problem specifies a seed, difficulty stratum, scenario category, initial operational state, and market parameters.Source: A Problem Set and Coverage Audit · p. 10
  • 100/100 problems were covered, with zero duplicate model–problem records and zero missing expected records in each ecology.Each ecology contains 600 terminal outcome records: 100 problems times six evaluated model families.Source: A Problem Set and Coverage Audit · p. 10
  • Terminal valuations use sixth-round records; ordered per-round records associate model, problem, round, parsed decisions, feasibility events, and resulting simulation states.Execution-intervention summaries were computed from logged events where available.Source: A Problem Set and Coverage Audit · p. 10

B Metrics and Statistical Protocol

ERPBench defines valuation and within-problem comparative metrics, then uses paired problem-level procedures to assess ecology sensitivity in its controlled 100-problem panel.

B Metrics and Statistical Protocol

  • Metrics report valuation in millions, within-problem mean rank, normalized regret relative to the same-ecology best model, and each model’s fraction of last-place finishes.Lower mean rank indicates better terminal valuation; normalized regret scales the gap to the same-ecology best model by the observed within-problem valuation range.Source: B Metrics and Statistical Protocol · p. 10
  • Table 2 reports paired ecology effects with percentile-bootstrap 95% confidence intervals over problems and Benjamini–Hochberg-adjusted q-values; tests use paired, two-sided Wilcoxon signed-rank tests.The problem is the resampling unit, so each resampled problem retains the corresponding Solo and Arena outcomes for the same model.Source: Table 2 · p. 10, B Metrics and Statistical Protocol · p. 10
  • These tests, pairwise win rates, and winner-disagreement summaries describe ecology sensitivity in the controlled 100-problem panel, not universal model superiority.The authors frame them as benchmark statistics for a defined measurement condition, rather than estimates for all possible ERP markets.Source: B Metrics and Statistical Protocol · p. 10

C Trace Taxonomy and Audit Scope

Trace interventions are logged in six categories spanning production constraints, unavailable investments, output fallbacks, quantity adjustments, runtime recovery, and run failures. These categories are aggregate process diagnostics, not a single error rate or causal explanation of terminal outcomes.

C Trace Taxonomy and Audit Scope

  • The six logged categories are production-capacity skip, investment unavailable, fallback decision, production quantity clamped, traceback, and experiment failed.Capacity skips may be normal consequences of the ERP environment, while fallbacks and tracebacks are closer to interface robustness failures.Source: C Trace Taxonomy and Audit Scope · p. 10
  • Figure 3 displays the five most frequent categories, omitting five experiment-failed events and excluding them from the displayed-share denominator.Source: C Trace Taxonomy and Audit Scope · p. 10
  • Intervention categories may overlap and occur at different pipeline layers, so they qualify—but do not replace or causally explain—the terminal outcome analysis.Source: C Trace Taxonomy and Audit Scope · p. 10

Related papers

Loading 2609.04667v1…