Source-linked AI summary
ERPBench: Evaluating LLM Agents for Enterprise Decision-Making Across Competitive Market Ecologies
Xinran Zhang, Pengrui Lu, Lyumanshan Ye, Pengfei Liu
TL;DR
Enterprise-agent evaluations rarely test whether business-decision conclusions transfer across competitive market ecologies. ERPBench addresses this gap with an execution-instrumented, six-round ERP simulation that compares matched Solo and Arena settings. The leading model changes from DeepSeek in Solo to Gemini in Arena, while task-level winners and lower-tail reliability also shift.
Problem
Existing evaluations rarely test whether enterprise-agent conclusions transfer across competitive market ecologies, despite competitive opponents changing prices, demand, inventory pressure, and cash-flow outcomes.
Method
ERPBench evaluates six model families on 100 matched ERP problems in Solo and Arena, using executable structured actions, terminal valuation, reliability metrics, and logged intervention diagnostics.
Results
The same task-level winner appears in only 21 of 100 problems, while the aggregate leader changes from DeepSeek in Solo to Gemini in Arena.
Takeaways & Limitations
Enterprise-agent evaluation should report whether model-selection conclusions transfer across competitive market ecologies, not only terminal scores.
Takeaways & Limitations
Logged interventions are process diagnostics rather than causal decompositions of terminal valuation, and capacity skips may reflect binding environment constraints rather than malformed decisions.
Abstract
from arXiv · showhide
Large language model (LLM) agents are increasingly proposed for enterprise workflows, yet existing evaluations rarely test whether business-decision conclusions transfer across competitive market ecologies. We introduce ERPBench, an execution-instrumented benchmark for enterprise decision agents in a six-round Enterprise Resource Planning (ERP) simulation with coupled pricing, production, procurement, inventory, finance, and shared-market competition. ERPBench evaluates the same 100 fixed problems in two matched competitive market ecologies: Solo, where each evaluated LLM agent competes against fixed rule-based opponents, and Arena, where six evaluated LLM agents compete in a shared market. Across six model families, this yields 1,200 model-level trajectories spanning 7,200 decision rounds. Under the observed service configuration, the leading model differs between ecologies: DeepSeek leads in Solo (252.29M mean valuation; mean rank 1.67), whereas Gemini leads in Arena (263.95M; 1.76). The two ecologies identify the same task-level winner on only 21 of 100 problems, and Gemini's bottom-rank rate falls from 22 % to 0 % in Arena. ERPBench supports paired evaluation of whether enterprise-agent rankings transfer across competitive market ecologies, supplemented by aggregate execution-intervention analysis. Code and benchmark resources are available in our https://github.com/GAIR-NLP/erp-bench.
1 Introduction
ERPBench addresses whether enterprise-agent conclusions remain stable when the same business problem is evaluated under different competitive market ecologies. It combines matched ecological evaluation with terminal outcomes and execution auditing.
- Enterprise decisions compound across pricing, production, procurement, inventory, and finance while one agent’s policies alter competitors’ demand.
- The benchmark asks whether the selected model differs across matched competitive market ecologies for the same enterprise task.
- ERPBench evaluates the same 100 ERP problems in matched Solo and Arena ecologies while changing opponent composition.
- Execution auditing complements terminal valuation because syntactically valid plans may violate capacity, investment, cash, or parsing constraints.
2 Related Work
Prior agent benchmarks evaluate interactive and executable tasks across digital and professional environments. ERPBench extends this direction to coupled enterprise resource-planning decisions and competitive market outcomes.
- Recent benchmarks move beyond static question answering toward interactive, stateful, and executable tasks.
- CRM-focused benchmarks test agents in professional workflows with realistic enterprise data, roles, and interactions.
- ERPBench differs by evaluating coupled pricing, production, procurement, inventory, capacity, finance, and final valuation rather than CRM or office-task completion.
- Aggregate outcome figures compare Solo and Arena mean valuations on the same matched problems, with mean rank reported above each bar.
3 ERPBench
ERPBench is designed as an operationally coupled, executable benchmark that compares model families across matched competitive ecologies. Agents act through a structured interface, while the simulator executes, repairs, and audits their decisions across six rounds.
- The benchmark couples pricing, production, procurement, inventory, capacity, cash, and valuation rather than treating them as isolated subtasks.
- Each agent receives ERP state and tool outputs, returns structured JSON actions, and undergoes parsing, feasibility checks, repairs or clamping, execution, and state recording.
- ERPBench compares six model families on the same 100 problems in Solo against five deterministic opponents and in Arena against six evaluated LLM agents.
- The primary outcome is sixth-round terminal company valuation, supplemented by ranks, winner agreement, normalized regret, bottom-rank rate, difficulty effects, and intervention categories.
- Execution interventions are non-exclusive process diagnostics, while the Decision Quality Pyramid organizes descriptive proxies rather than forming a composite score.
4 Experiments
ERPBench compares six model families on 100 matched ERP problems across Solo and Arena ecologies, revealing that aggregate rankings, task-level winners, reliability patterns, and ecology effects vary with competitive context and execution conditions.
- Experimental setup: 1,200 model-level trajectories and 7,200 decision rounds evaluate six model families across the same 100 problems under Solo and Arena.The protocol uses six-round ERP simulations and matched problem identifiers across ecologies.
- Aggregate performance: DeepSeek leads Solo with 252.29M mean valuation and mean rank 1.67, whereas Gemini leads Arena with 263.95M and mean rank 1.76.The paired comparison changes opponent composition while preserving the underlying problem instances and evaluation protocol.
- Execution audit: 3,761 production-capacity skips account for 82.5% of displayed intervention events, followed by unavailable investment choices, fallbacks, clamping, and tracebacks.The displayed shares pool Solo and Arena trajectories and exclude five omitted run-level failures.
- Task-level rankings: Only 21 of 100 problems have the same task-level winner across ecologies, with mean within-problem Spearman rank correlation 0.182 [0.090, 0.273].The largest transition is from DeepSeek in Solo to Gemini in Arena on 35 problems.
- Reliability: Gemini’s bottom-rank rate falls from 22% in Solo to 0% in Arena, while its P90 normalized regret falls from 0.73 to 0.27.DeepSeek shows the opposite tail pattern, with P90 regret rising from 0.16 to 0.39.
- Difficulty-conditioned effects: Arena–Solo effects vary across difficulty strata rather than forming a uniform offset, with Gemini benefiting consistently and other model families showing smaller mixed-direction shifts.The stratified analysis treats ERPBench as a structured problem-family benchmark; intervention categories remain process diagnostics rather than causal decompositions of valuation.
5 Discussion and Limitations
ERPBench shows that model rankings and competence profiles can change between Solo and Arena, so single-ecology leaderboards may incompletely characterize enterprise agents. The benchmark also limits interpretation to its controlled simulation and treats execution interventions as process evidence rather than causal explanations.
- Ecology comparison: Solo evaluates agents against fixed rule-based competitors, whereas Arena evaluates them against other LLM agents that jointly shape market outcomes.The paired design changes opponent composition while retaining the underlying benchmark task structure.
- Ecology comparison: DeepSeek appears strongest in Solo, while Gemini becomes the aggregate leader in Arena and avoids bottom-rank outcomes there.The paper characterizes these as independent-optimizer and competitive-responder profiles, respectively.
- Execution audit: Execution audits complement terminal valuation and rank by recording parsing recovery, feasibility repair, clamping, and runtime recovery.Logged interventions constrain interpretation but should not be treated as interchangeable model-error units or causal mechanisms.
- Heterogeneity and scope: The ecology effect varies by problem, with models improving on some tasks and worsening on others rather than experiencing a constant shift.This makes the observed ecology effect a problem-dependent interaction.
- Heterogeneity and scope: ERPBench remains bounded by a fixed 100-problem slice, six-round horizon, one six-model Arena composition, and potentially changing service-side routing and endpoints.Results may differ under additional ecologies, opponent sets, longer horizons, or information regimes.
6 Conclusion
ERPBench uses paired competitive ecologies and logged intervention evidence to test whether enterprise-agent conclusions transfer across market conditions. Across the benchmark, the leading model, task-level winners, difficulty effects, and tail-risk profiles change between Solo and Arena.
- ERPBench evaluates enterprise decision agents under paired competitive market ecologies with logged intervention evidence.
- Across 100 matched ERP problems and 1,200 model-level trajectories, the leading model changes from DeepSeek in Solo to Gemini in Arena.
- Task-level winners, difficulty effects, and tail-risk profiles also change across the two ecologies.
- The benchmark supports reporting terminal scores together with transfer of model-selection conclusions and evidence about how executable decisions are produced.
A Problem Set and Coverage Audit
The benchmark uses a matched 100-problem panel rather than independent ad hoc runs, preserving direct Solo–Arena comparisons for each model–problem pair. Each problem is specified by operational and market conditions, and each ecology contains 600 terminal outcome records.
- The 100-problem slice is assembled as a matched benchmark panel rather than as independent ad hoc runs.
- Each problem specifies a seed, difficulty stratum, scenario category, initial operational state, and market parameters.
- Solo100 and Arena100 use the same problem identifiers, enabling a direct ecology comparison for every model–problem pair.
- Each ecology contains 600 terminal outcome records, corresponding to 100 problems times six evaluated model families.
B Metrics and Statistical Protocol
ERPBench combines valuation, rank, regret, and bottom-rank metrics with paired statistical testing over matched problems. The protocol uses bootstrap intervals and corrected Wilcoxon tests while treating the resulting comparisons as descriptive evidence for this controlled panel.
- Metrics: Valuation is reported in millions, while mean rank is computed within each problem and ecology among six model families.Lower mean rank indicates better terminal valuation.
- Metrics: Normalized regret measures each model’s gap to the best same-ecology model, scaled by the observed within-problem valuation range.
- Metrics: Bottom-rank rate is the fraction of problems on which a model ranks last among the evaluated families.
- Statistical protocol: Paired effects resample problems rather than individual rows, preserving corresponding Solo and Arena outcomes for each model.
- Statistical protocol: Paired valuation differences use two-sided Wilcoxon signed-rank tests with Benjamini–Hochberg correction across six model-wise tests.Reported intervals are 95% percentile-bootstrap confidence intervals over problems.
C Trace Taxonomy and Audit Scope
ERPBench logs six trace-intervention categories to audit execution behavior across different pipeline layers. These diagnostics qualify benchmark results rather than replacing them or causally explaining outcomes.
- Trace taxonomy: Six categories organize execution interventions: production-capacity skip, investment unavailable, fallback decision, production quantity clamped, traceback, and experiment failed.The categories span scheduling constraints, unavailable actions, output recovery, numeric repair, runtime recovery, and run-level failure.
- Audit scope: Figure 3 displays the five most frequent categories, omitting five experiment-failed events from the displayed-share denominator.
- Audit scope: The categories are not summed into a single error rate because they occur at different pipeline layers and may overlap.
- Interpretation: Capacity skips can be normal ERP-environment consequences, whereas fallback and traceback events more closely indicate interface robustness failures.
- Interpretation: Trace interventions serve as aggregate process diagnostics that qualify benchmark results rather than replacing them or causally explaining them.