PaperlayerView exact revision on arXiv

Source-linked AI summary

EnterpriseBench: Benchmarking LLM Agents on Enterprise-Level Strategic Reasoning and Decision-Making

Min Yang, Yichen Pan, Jinghua Piao, Dandan Song, Yongshun Gong, Yong Li

arXiv:2609.37658v1cs.AI

TL;DR

Existing enterprise and financial benchmarks mainly assess static capabilities, leaving interactive and long-horizon decision-making underexplored. EnterpriseBench combines reorganized QA datasets with three interactive enterprise tasks. Experiments find that agents do not show stable, comprehensive reliability across these settings.

  • Problem

    Existing enterprise and financial benchmarks mainly evaluate fixed-input capabilities, leaving limited evidence about LLM agents’ interactive and strategic decision-making in realistic workflows.

    Source: §1 · p. 1

  • Method

    EnterpriseBench reorganizes enterprise and financial QA datasets into a foundational suite and adds Consulting, Beer Game, and Enterprise Digital Twin as interactive decision-making tasks.

    Source: §1 · pp. 1–2

  • Results

    Strong QA performance does not reliably transfer to interactive tasks, and current agents have not achieved stable, comprehensive, cross-task reliability in enterprise scenarios.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    EnterpriseBench provides a benchmark for assessing LLM agents’ strengths and limitations in enterprise strategic reasoning and decision-making.

    Source: §6 · p. 9

    The controlled interactive environments do not fully capture real enterprises’ social, organizational, and institutional factors, and the benchmark remains text-based and limited in task and experiment coverage.

    Source: Limitations · p. 9

Abstract

from arXiv · show

LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-term trade-offs. However, existing enterprise and financial benchmarks mainly test static capabilities such as information extraction, numerical calculation, domain knowledge, and financial QA, leaving interactive and long-horizon decision-making underexplored. To bridge this gap, we introduce EnterpriseBench, a benchmark that evaluates LLM agents across this spectrum, from static question answering to dynamic decision-making. Specifically, EnterpriseBench reorganizes existing enterprise and financial QA datasets into a unified foundational suite annotated by capability and difficulty, and introduces three professional interactive settings: Consulting, based on management-consulting-style business cases for client problem diagnosis through multi-turn information seeking; the Beer Game, adapted from a classic supply-chain management simulation for inventory control under delayed feedback; and Enterprise Digital Twin, a project-based business simulator for workforce, risk, and project planning. Experiments with nine agent methods under four backbone models show that current agents have not yet achieved stable, comprehensive, and cross-task reliability in enterprise scenarios. These results show that EnterpriseBench provides a practical benchmark for evaluating LLM agents in realistic enterprise strategic reasoning and decision-making.

1 Introduction

EnterpriseBench addresses the gap between static enterprise QA and decision-making that requires information gathering, adaptation to feedback, and long-horizon planning. It combines foundational evaluation with interactive tasks and finds that current agents do not perform reliably across enterprise settings.

1 Introduction

  • Existing enterprise and financial benchmarks mainly test fixed-input capabilities, providing limited evidence about interactive strategic decision-making in realistic workflows.Real enterprise scenarios can require agents to acquire missing information, respond to delayed feedback, and plan under uncertainty.Source: §1 · p. 1
  • EnterpriseBench combines reorganized enterprise and financial QA datasets with three interactive tasks: Consulting, Beer Game, and Enterprise Digital Twin.Its two layers cover foundational capabilities and interactive decision-making in scenarios involving missing information, uncertainty, feedback, and long-term trade-offs.Source: §1 · pp. 1–2
  • Strong QA performance does not reliably transfer to Consulting, Beer Game, or Enterprise Digital Twin, and no method consistently performs best across tasks and backbones.A proof-of-concept dynamic method assembly improves overall QA from 0.725 to 0.729 over the best fixed method.Source: §1 · p. 2
  • The evaluation covers nine agent methods under four backbones; audits and robustness checks support the reliability of the benchmark’s evaluation and annotation processes.Consulting scores varied by at most 0.12 across prompt variants, and difficulty annotations aligned with human ratings at Pearson r = 0.96.Source: §1 · p. 2
  • Consulting introduces simulated professional interviews in which agents clarify objectives, acquire missing information, structure analyses, and communicate recommendations over multiple turns.The cases are based on consulting materials, with an LLM acting as interviewer.Source: §1 · p. 2

2 EnterpriseBench

EnterpriseBench uses two layers: foundational enterprise reasoning and interactive decision-making. Its tasks span evidence extraction and professional reasoning, simulated consulting interviews, supply-chain control, and project-based enterprise planning.

2.1 Overview and Design Principles

  • The two-layer design tests whether agents can apply foundational capabilities when information is incomplete, feedback is delayed, and actions affect future outcomes.The interactive layer includes Consulting, Beer Game, and Enterprise Digital Twin.Source: §2.1 · p. 2

2.2 Foundational Enterprise Reasoning Tasks

  • The foundational suite organizes enterprise reasoning into information extraction, numerical calculation, domain knowledge, and complex reasoning, with samples annotated by capability and difficulty.Difficulty scores range from 0–10 and are grouped as Easy (0–3), Middle (4–6), and Hard (7–10); human audits validate the annotations.Source: §2.2 · p. 3

2.3 Interactive Decision-Making Tasks

  • Consulting uses 423 simulated case interviews with visible problem statements, hidden facts, and reference solutions; agents seek information and deliver recommendations scored on four dimensions.The dimensions are structure, quantitative reasoning, business sense, and communication.Source: §2.3 · p. 4
  • Beer Game tests sequential ordering by a retailer in a four-stage supply chain, with local observations and delayed consequences shaping inventory and backlog costs.Episodes last 25 time steps, and lower accumulated cost indicates better decisions.Source: §2.3 · p. 4
  • Enterprise Digital Twin tests scenario-level planning: agents set workforce, revenue risk, and project schedules, balancing revenue opportunities against labor, capacity, deadlines, and uncertainty.The simulator executes the configured strategy for a full episode; accumulated earnings are the primary evaluation measure.Source: §2.3 · p. 4

3 Experiment Setup

The experiments compare varied agent methods under shared task interfaces while preserving methods’ native adaptation and inference configurations. Evaluation protocols and metrics differ by task, spanning fixed-input QA, simulated interviews, and business simulations.

3.1 Evaluated Agent Methods

  • The nine evaluated methods cover single-agent reasoning, iterative refinement, multi-agent collaboration, retrieval-augmented adaptation, prompt optimization, and memory-based reasoning.Examples include CoT, Self-Refine, Reflexion, Debate, Discussion, DC, GEPA, ACE, and AMEM.Source: §3.1 · pp. 4–5

3.2 Evaluation Details

  • Tables 1 and 2 compare single-agent and multi-agent methods across domains and tasks using DeepSeek-V3 and GPT-4.1 backbones, respectively.Beer Game reports total costs, where lower is better, while EDT reports earnings, where higher is better.Source: Table 1 · p. 5, Table 2 · p. 6

3.1 Evaluated Agent Methods

  • Methods retain their native inference and adaptation configurations rather than being forced into identical resource budgets, and use only information generated within the same benchmark run.The paper reports detailed call, token, history-access, and prompt-evolution statistics in Appendix A.11.Source: §3.1 · p. 5

3.2 Evaluation Details

  • Evaluation uses fixed-input metrics for foundational tasks, transcript-and-recommendation rubric scoring for Consulting, accumulated cost for Beer Game, and accumulated earnings for EDT.Beer Game evaluates 25-step replenishment decisions, with lower cost better; EDT uses a scenario configuration followed by a rollout, with higher earnings better.Source: §3.2 · p. 5

4 Experiment Results

Across foundational and interactive tasks, agent strengths vary by capability, task, and backbone. Human audits, prompt checks, and adaptive-routing results provide additional evidence about evaluation reliability and method selection.

4.1 Main Result

  • Information Extraction varies relatively little across methods, while Numerical Calculation and Domain Knowledge drop sharply on harder examples under both backbones.The foundational suite distinguishes relatively stable document-level reasoning from more fragile numerical and specialized-knowledge reasoning.Source: §4.1 · p. 6
  • Leaders shift across tasks and backbones: DeepSeek-V3 favors GEPA on Consulting, Discussion on Beer Game, and AMEM on EDT; GPT-4.1 favors DC on Consulting and GEPA on Beer Game and EDT.Strong static QA does not reliably transfer to interactive tasks, and no single method consistently performs best.Source: §4.1 · p. 6

4.2 Analysis of Interactive Tasks

  • GEPA leads Consulting under DeepSeek-V3 with 8.28 overall, while DC leads under GPT-4.1 with 8.08 overall.Self-Refine and Reflexion remain below CoT in overall score under both backbones.Source: §4.2 · pp. 6–7
  • Discussion achieves the lowest Beer Game cost under DeepSeek-V3 at 1.94×10^4, whereas GEPA achieves the lowest cost under GPT-4.1.The reversal indicates that performance depends on the fit between agent design and backbone; additional interaction is not always beneficial for delayed-feedback control.Source: §4.2 · p. 7
  • AMEM earns the most on EDT under DeepSeek-V3 with 6.76M, while GEPA leads under GPT-4.1 with 8.16M.EDT evaluates scenario-level choices about workforce, risk, and project schedules before a full episode runs.Source: §4.2 · p. 7

4.3 Validity and Reliability Analysis

  • Human and LLM Consulting scores have similar relative profiles, with r = 0.71 (p < 0.001) and an 8.5% average gap relative to human scores.Across 500 transcripts from 100 cases, the expanded audit reports r = 0.887, 85.0% within-one-point agreement, and human ICC(2, 3) = 0.892.Source: Figure 2 · p. 7, §4.3 · p. 7
  • Consulting prompt variants change the overall score by at most 0.12 points, while LLM difficulty annotations align with expert ratings at r = 0.96.The difficulty audit also reports a 0.47-point mean absolute difference and 94% of samples within one point of the human score.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

4.4 Adaptive Routing: A Proof of Concept

  • AOA maps sample characteristics to a selected base method and scores 0.729 on QA, slightly above ACE at 0.725.Its routing rules are derived from execution traces without using held-out test labels; the result is presented as a proof of concept.Source: Figure 3 · p. 8, §4.4 · pp. 8–9

5 Related Work

EnterpriseBench complements financial and business benchmarks focused on static capabilities with enterprise-oriented QA and interactive tasks for strategic reasoning.

5 Related Work

  • Financial and business benchmarks cover static report understanding, numerical reasoning, domain knowledge, financial QA, and information structuring [Chen et al., 2021; Reddy et al., 2024; Krumdick et al., 2024; Xie et al., 2024; Zhang et al., 2025c], while EnterpriseBench adds information-seeking dialogue, delayed-feedback control, and long-horizon planning.General agent benchmarks study tool use, web interaction, embodied environments, and simulations [Qin et al., 2024; Zhou et al., 2024; Shridhar et al., 2020], but are not designed around enterprise strategic reasoning.Source: §5 · p. 9

6 Conclusion

EnterpriseBench combines reorganized enterprise and financial QA with three interactive enterprise tasks. Experiments find that static QA performance does not reliably predict interactive decision-making, and that agents still face distinct challenges across enterprise task types.

6 Conclusion

  • EnterpriseBench evaluates LLM agents from foundational enterprise reasoning to interactive decision-making by combining reorganized QA datasets with Consulting, Beer Game, and Enterprise Digital Twin.Experiments with nine agent methods and four backbone models show that static QA performance does not reliably predict interactive decision-making.Source: §6 · p. 9

Limitations

The benchmark uses controlled, text-based environments and a limited set of tasks, methods, and backbones; its results have a bounded scope for real-world enterprise decisions.

Limitations

  • Controlled benchmark environments enable reproducible evaluation but do not fully capture social, organizational, and institutional factors in real enterprise decision-making.Source: Limitations · p. 9
  • The current benchmark is text-based and does not cover multimodal evidence such as slides, spreadsheets, emails, dashboards, or meeting transcripts.Source: Limitations · p. 9
  • Consulting, Beer Game, and EDT cover representative forms but not all industries, organizational scales, or decision types; experiments cover nine methods and four backbones.The results should not be interpreted as evidence that LLM agents are ready for autonomous high-stakes business, financial, or operational decisions.Source: Limitations · p. 9

A.1 Detailed Task Descriptions and Metrics

The benchmark evaluates foundational financial and logical reasoning alongside three interactive enterprise settings, each with task-specific metrics. The interactive tasks measure consulting recommendations, inventory decisions, and enterprise earnings.

A.1 Detailed Task Descriptions and Metrics

  • The foundational suite covers Information Extraction, Numerical Calculation, Domain Knowledge, and Complex Reasoning using task-specific evaluation protocols.These tasks span financial and logical reasoning.Source: §A.1 · p. 12
  • Program synthesis uses a sandboxed Python environment, accepting absolute error of 10−6 near zero and relative error of 0.01 for larger magnitudes.Source: §A.1 · p. 12
  • Strict quantity extraction requires an absolute epsilon of 10−6 or an exact match to the normalized text span, with zero relative tolerance.Source: §A.1 · p. 12
  • Numerical formula evaluation checks extracted final answers using exact or standardized floating-point comparison, while knowledge tagging uses regex option extraction and exact entity-label matching.Source: §A.1 · p. 12
  • Consulting cases assess structured recommendations across Structure, Quantitative Reasoning, Business Sense, and Communication, with Overall computed as their average.Source: §A.1 · p. 12
  • The Beer Game tests multi-tier replenishment decisions under dynamic demand and lead times, scoring Total Cost from inventory holding costs and backlog penalties.Lower Total Cost indicates superior strategic planning.Source: §A.1 · p. 12
  • Enterprise Digital Twin evaluates joint project selection and workforce allocation through BPTK simulation, using Accumulated Earnings as total profit over the simulation period.Source: §A.1 · p. 12

A.2 Agent-based Classification Prompts

The classification prompt assigns each question one primary capability and a difficulty score, with rules distinguishing direct extraction from computation and guidance for challenging labels. It also specifies the required output and uses deepseek-v3 for consistent classification.

A.2 Agent-based Classification Prompts

  • Information Extraction covers locating, extracting, or restating information explicitly in the input without external knowledge or calculations.Source: §A.2 · p. 12
  • Numerical Calculation includes arithmetic, numerical reasoning, formula computation, or code writing, even when the input values must first be extracted.Source: §A.2 · pp. 12–13
  • Domain Knowledge tests specialized finance, accounting, XBRL, standards, CFA, or business-subject knowledge and recall.Source: §A.2 · pp. 12–13
  • Complex Reasoning involves judgment or decision-making under constraints, including scenario-based CFA and business-subject questions asking what should be done.Source: §A.2 · p. 13
  • The decision rules classify verbatim single-value lookup without computation as Information Extraction, but net change, ratios, sums, and other calculations as Numerical Calculation.Source: §A.2 · p. 13
  • Difficulty scoring uses the 0–10 range, reserving 0–3 for trivial lookups or straightforward recall and assigning higher scores to complex contexts, multi-step computation, ambiguity, or decision-making.Source: §A.2 · p. 13
  • Classification difficulty rises for large, overlapping label sets, multiple subquestions, or labels requiring semantic disambiguation; the prompt requests one capability, a score, and a brief justification.The classification is performed using deepseek-v3 for consistency across the benchmark.Source: §A.2 · p. 13

A.3 AOA Experience Extraction Prompts

AOA uses specialized prompts to learn from execution traces and synthesize routing policies.

A.3 AOA Experience Extraction Prompts

  • AOA uses two specialized prompts: one for learning from execution traces and another for synthesizing routing policies.Source: §A.3 · p. 13

Trace-based Experience Extraction Prompt

The trace-based prompt extracts one reusable experience item from an agent’s reasoning trace and output, to improve future use or route similar samples elsewhere. It constrains the result to an actionable JSON entry with diagnosis or success factors and routing guidance.

Trace-based Experience Extraction Prompt

Meta-level Experience Synthesizer Prompt

This section describes the prompts, evaluation protocols, and simulation mechanics used across the Consulting, Beer Game, and Enterprise Digital Twin settings, including a routing playbook synthesized from experience across tasks and agent methods. It also reports supplemental benchmark evidence, including additional-backbone results, Consulting judge validation, and resource-use comparisons.

Meta-level Experience Synthesizer Prompt

A.4 Consulting Prompts

  • The interviewer paraphrases a predefined consulting case, reveals gated facts only when clearly requested and one piece at a time, and withholds solution and internal guidance.Source: §A.4 · p. 15
  • The interview proceeds through focused questions and case-length-dependent pacing, redirects data requests toward analysis, and closes only after a supported recommendation and feedback.Source: §A.4 · pp. 15–16
  • The judge evaluates candidate behavior independently of the interviewer across structure, quantitative reasoning, business sense, communication, and overall consulting suitability.Source: §A.4 · p. 16
  • The consulting judge assesses only the candidate from the case and chronological interview transcript, returning 0–10 dimension scores and concise feedback in a fixed JSON object.Guidelines discourage inflated ratings and penalize verbosity, hallucinated facts, and unsupported conclusions.Source: §A.4 · pp. 16–17
  • The consulting judge scores candidates on structure, quantitative reasoning, business sense, and communication, alongside a holistic overall score.The overall score is not simply an arithmetic average; it reflects whether the candidate merits a consulting-role recommendation.Source: §A.4 · pp. 16–17

A.5 Beer Game Configuration

  • The Beer Game runs for 25 steps as demand rises from 100 to 400 units starting in week 2, with one-week order-information and two-week shipment delays.Holding costs 0.5 per unit per step, backorders cost 1.0 per unit, and a fixed minimum inventory cost of 200 applies.Source: §A.5 · pp. 17–18
  • The tested Beer Game agent controls the retailer, while other roles use fixed equation-based ordering policies, including inventory-and-backlog correction and four-step demand smoothing.The typical policy uses α = 1 and β = 1; the smoothing policy substitutes a four-step moving average for observed demand.Source: §A.5 · p. 18

A.6 Enterprise Digital Twin (EDT) Implementation Details

  • The EDT implementation appendix outlines scenario specification, stepwise dynamics, stochastic project extensions and follow-ons, firm accounting and metrics, and evaluation protocol, with a brief scenario example.The scenario is represented as JSON with runspecs, firm-level properties, and instantiated consultant and project agents.Source: §A.6 · p. 18, §A.6.1 · p. 19

A.6.1 Scenario specification

  • The EDT agent chooses consultant count, firm-wide revenue risk, and project acceptance and timing windows; the simulator applies these choices to a template and runs the full horizon.Other template fields, including salaries, workplace costs, and baseline project parameters, remain unchanged.Source: §A.6.1 · p. 19

A.6.2 Consultant dynamics and capacity constraints

  • EDT consultants incur salary and workplace costs each step, can serve only one project at a time, and remain assigned until its base and any extended workload finish.Project staffing requirements cap concurrent workers, so starting a long project can delay higher-margin work.Source: §A.6.2 · p. 19

A.6.2 Consultant dynamics and capacity constraints; A.6.3 Project state

  • Projects track workload, contracted effort, billing rate, and staffing cap; delivery is limited to the project’s start-to-deadline window, and unfinished work cannot earn revenue after the deadline.For each project, delivered effort is bounded by the number of consultants working on it and the effort delivered per step.Source: §A.6.3 · pp. 19–20, §A.6.2 · p. 19

A.6.4 Extensions and follow-on projects

  • Higher revenue risk increases the chance of extensions and follow-on projects, potentially raising earnings while tying up capacity and increasing uncertainty; lower risk yields more predictable revenue but less upside.Extensions may trigger after base-scope completion before a deadline, while follow-ons may be instantiated at the deadline; both are risk-gated, and follow-ons cannot chain further.Source: §A.6.4 · p. 20

A.6.5 Firm-level accounting and returned metrics

  • The firm aggregates step revenue and expenses into cumulative accounting metrics, and reports utilization as the fraction of consultant capacity engaged in project delivery, both per step and on average.Utilization is zero when the consultant pool is empty.Source: §A.6.5 · p. 20

A.6.6 Evaluation protocol in the benchmark

  • Evaluation applies each model’s {C, R, P} schema to fixed template scenarios, executes them in BPTK, and records metrics; repeated runs can use prior feedback to adapt configurations within a fixed action space.Source: §A.6.6 · p. 21

A.7 Dataset Statistics; A.8 Extended Related Work

  • Online evaluation uses only the test split with sequential adaptation, while the Beer Game and Enterprise Digital Twin are simulation-based tasks; EnterpriseBench adds consulting cases and serious games beyond static QA for strategic enterprise decisions.Prior financial and business benchmarks largely use static QA, and general reasoning benchmarks are not designed for financial or enterprise strategic decision-making.Source: §A.7 · p. 21, §A.8 · p. 21

A.9 Additional Backbone Results

  • Under DeepSeek-V4-Pro, GEPA led Consulting overall and across all five dimensions, while under GLM-5.2, AMEM led across all five dimensions.The reversal shows that the best-performing method depends on the backbone; foundational QA performance also showed capability and difficulty effects, without uniformly transferring to interactive decision-making.Source: §A.9 · p. 22

A.10 Expanded Human Validation of the Consulting Judge

  • The Consulting judge’s overall Pearson correlation was 0.887, Spearman correlation 0.736, MAE 0.679, and within-one-point agreement 85.0%.Across rubric dimensions, Pearson correlations ranged from 0.840 to 0.899; ICC(2, 3) ranged from 0.782 to 0.892, while quantitative reasoning and communication had comparatively lower within-one-point agreement.Source: §A.10 · p. 22

A.11 Resource and Adaptation Budgets

  • CoT and AMEM achieved 0.98 accuracy with approximately one call and 1.1K tokens per sample; ACE used 6.12 calls and 21.4K tokens, and GEPA used 13.84 calls and 48.9K tokens, without higher accuracy.Resource use was measured on 50 evaluation instances, including adaptation overhead.Source: §A.11 · p. 22

Related papers

Loading 2609.37658v1…