PaperlayerView exact revision on arXiv

Source-linked AI summary

EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair, Sagar Davasam, Aman Tiwari, Sathwik Tejaswi Madhusudhan, Sridhar Krishna Nemala, Srinivas Sunkara, Sai Rajeswar

arXiv:2603.13594v1cs.AIcs.LG

TL;DR

Enterprise AI agents require long-horizon planning amid persistent state changes and strict access protocols, but existing benchmarks inadequately capture these enterprise conditions. EnterpriseOps-Gym introduces an interactive, high-fidelity benchmark with expert-authored tasks and outcome-based verification, and its evaluation shows that current agents remain unreliable, with strategic planning and safe refusal as major weaknesses.

  • Problem

    Existing benchmarks inadequately capture enterprise workflows requiring long-horizon planning, persistent state changes, cross-system coordination, and strict access protocols.

  • Method

    EnterpriseOps-Gym combines a containerized enterprise sandbox with 1,150 expert-curated tasks across eight domains, infeasible scenarios, and SQL-based outcome verification.

  • Results

    The best model achieves only 37.4% task success, while human-authored plans improve performance by 14–35 percentage points and the best infeasible-task refusal rate is 53.9%.

  • Takeaways & Limitations

    The results indicate that current agents are not ready for unsupervised enterprise deployment and that constraint-aware plan generation is the central challenge.

  • Takeaways & Limitations

    The benchmark’s conclusions concern enterprise workflows represented by its sandbox, domains, tasks, and policies rather than all possible agent deployments.

Abstract

from arXiv · show

Large language models are shifting from passive information providers to active agents intended for complex workflows. However, their deployment as reliable AI workers in enterprise is stalled by benchmarks that fail to capture the intricacies of professional environments, specifically, the need for long-horizon planning amidst persistent state changes and strict access protocols. In this work, we introduce EnterpriseOps-Gym, a benchmark designed to evaluate agentic planning in realistic enterprise settings. Specifically, EnterpriseOps-Gym features a containerized sandbox with 164 database tables and 512 functional tools to mimic real-world search friction. Within this environment, agents are evaluated on 1,150 expert-curated tasks across eight mission-critical verticals (including Customer Service, HR, and IT). Our evaluation of 14 frontier models reveals critical limitations in state-of-the-art models: the top-performing Claude Opus 4.5 achieves only 37.4% success. Further analysis shows that providing oracle human plans improves performance by 14-35 percentage points, pinpointing strategic reasoning as the primary bottleneck. Additionally, agents frequently fail to refuse infeasible tasks (best model achieves 53.9%), leading to unintended and potentially harmful side effects. Our findings underscore that current agents are not yet ready for autonomous enterprise deployment. More broadly, EnterpriseOps-Gym provides a concrete testbed to advance the robustness of agentic planning in professional workflows.

1. Introduction

EnterpriseOps-Gym targets the gap between general tool-use benchmarks and enterprise workflows requiring long-horizon planning, persistent state management, cross-domain coordination, and policy adherence. It introduces a large, expert-authored sandbox and evaluates frontier models, finding substantial performance limitations and a pronounced strategic-planning bottleneck.

  • Motivation and benchmark scope: EnterpriseOps-Gym evaluates multi-step planning, policy adherence, state-driven tool calling, and cross-domain orchestration across eight enterprise domains.Its sandbox is designed for realistic professional workflows rather than isolated tool calls.
  • Motivation and benchmark scope: Existing benchmarks often use stateless atomic tool calls or shallow single-vendor environments with fewer than 25 database tables, under 50 tools, short horizons, and limited policy constraints.These limitations leave state dependencies and cross-system coordination insufficiently represented.
  • Benchmark construction: 1,150 expert-authored tasks span eight interconnected ecosystems, including productivity, HR, ITSM, CSM, and Hybrid cross-domain workflows.The benchmark includes 30 infeasible scenarios that test whether agents refuse unsatisfiable requests without side effects.
  • Benchmark construction: 164 relational database tables and 512 functional tools create an interactive environment for long-horizon planning, cross-system state management, and policy-constrained execution.Tasks are evaluated with outcome-based SQL verification covering goal completion, state integrity, policy compliance, and unintended side effects.
  • Evaluation findings: 37.4% is the overall success rate achieved by the best model, Claude Opus 4.5, across the evaluation of 14 frontier models.Performance falls to 28.5% on ITSM and 30.7% on Hybrid tasks, while the best collaboration-domain results reach 51–52%.
  • Evaluation findings: 14–35 percentage points are gained when models receive human-authored plans, identifying strategic planning rather than tool use as the primary bottleneck.The best model refuses policy-violating infeasible tasks cleanly only 53.9% of the time.

2. Related Works

Related benchmarks cover general tool use, enterprise platform simulation, and agentic planning or computer use. EnterpriseOps-Gym differs by combining interconnected enterprise domains with policy-constrained, stateful workflows and broader environment complexity.

  • API and tool-use benchmarks: General API benchmarks measure tool-calling accuracy across open-domain APIs but do not model multi-system, policy-constrained, stateful enterprise ecosystems.Examples include ToolLLM, API-Bank, ACEBench, τ-bench, and τ 2-bench.
  • Enterprise benchmarks: Enterprise benchmarks commonly evaluate single-vendor workflows such as ServiceNow, Salesforce, or IT incident resolution, while multi-domain efforts broaden scope with limited depth.The supplied comparison frames EnterpriseOps-Gym against these platform-specific environments.
  • Comparison dimensions: Table 1 compares benchmarks using dimensions including database tables, average foreign keys, execution steps, human plans, and task templates.Avg. FK indicates relational density, while average steps refer to ideal human-authored trajectories.
  • Agentic planning and computer use: Agentic-planning and computer-use benchmarks study sequential decision-making in travel, mobile, service, desktop, and web environments rather than policy-governed enterprise workflows.These settings address sophisticated planning but differ from enterprise systems in domain and constraint structure.

3. ENTERPRISEOPS-GYM

ENTERPRISEOPS-GYM is constructed as a realistic, multi-domain enterprise benchmark emphasizing policy-constrained, stateful, cross-system workflows. Its containerized environment, expert-authored tasks, outcome-based verification, and extensive task and environment statistics support evaluation of complex agentic planning.

  • Domain selection: Domain selection prioritizes industry relevance, policy and data-sensitivity diversity, and access to experts who can author and validate authentic tasks.These principles motivated the division between operational enterprise functions and universal collaboration tools.
  • Domain selection: The benchmark spans eight complementary domains, combining mission-critical operations, collaboration tools, and Hybrid cross-domain orchestration.The domains cover CSM, HR, ITSM, Email, Calendar, Teams, Drive, and coordinated execution across fragmented systems.
  • Sandbox environment: The containerized Docker sandbox combines domain-specific databases, APIs, tools, seed data, and SME-informed schemas to provide a reproducible enterprise evaluation ground.Annotators extend the fixed environment with tables, schemas, and records as tasks require.
  • Task construction: Tasks use multi-step scenarios with tool-count, verification, state-dependency, access, and policy-conflict requirements, while allowing multiple valid paths to a unique final state.Action ordering can be crucial, and tasks are designed around realistic workflow complexity.
  • Task construction: Outcome-based verification evaluates final goal achievement, integrity constraints, permission compliance, and side effects rather than enforcing rigid action sequences.Annotators author executable SQL scripts to verify the completed environment state.
  • Dataset and environment statistics: 1,150 tasks include 30 infeasible scenarios, use 512 tools across 164 database tables, and average 5.3 verification conditions over an average 24.9-table task subgraph.Expert trajectories average 9.15 steps, while Hybrid scenarios can involve up to 73 tables.

4. Experiments

Experiments evaluate frontier agents under unified enterprise conditions, revealing weaknesses in planning, policy compliance, long-horizon execution, and safe refusal. Controlled analyses show that externalizing planning substantially improves execution, while tool overload is not the primary bottleneck.

  • Overall performance: 37.4% was Claude Opus 4.5’s best overall task-completion rate, while ITSM and Hybrid were the hardest domains at 28.5% and 30.7%.Models performed better on collaboration workflows but struggled on policy-governed and cross-domain tasks.
  • Tool robustness: Performance remained stable with added distractor tools, indicating that tool discovery was less limiting than task planning and policy adherence.The augmentation increased output tokens by an average of 4–9%.
  • Safe refusal: 30 infeasible tasks tested whether models could abstain without side effects, but the best refusal score was only 53.9%.Infeasibility arose from unavailable tools, policy violations, or unavailable resources, often under compound constraints.
  • Task horizon: Performance declined consistently as task horizons increased, with long-horizon error accumulation remaining a barrier to autonomous reliability.Closed-source models were more resilient, while open-weight performance fell more sharply with horizon length.
  • Thinking budget: Higher thinking budgets improved GPT-OSS-120B performance substantially in several domains, including Drive from 8.6% to 41.0%.Scaling was not universally monotonic: Email peaked at 45.2% under the medium budget, while ITSM plateaued at 6.1%.
  • Planning ablations: 6-13% gains from Claude-generated plans and 14-35 percentage-point gains from human-authored plans show that planning quality is a meaningful bottleneck.The larger human-plan gains were observed when the same executor models followed externally supplied strategic plans.

5. Discussion and Conclusion

EnterpriseOps-GYM exposes a substantial gap between current agents and enterprise requirements, while experiments identify planning—not tool execution—as the dominant bottleneck. The results motivate constraint-aware planning, long-horizon state management, and adaptive reasoning as priorities.

  • Discussion: Current agents remain far from enterprise-ready: the best model reaches only 37.4% task success, while infeasible-task refusal reaches only 53.9%.Performance is especially weak under policy and process constraints, with failures involving cascading state transitions and compliance.
  • Discussion: Human-authored plans yield 14–35 percentage point gains across models and domains, identifying strategic planning as the dominant bottleneck.The reported gains exceed those from automated planning or more complex multi-agent orchestration.
  • Discussion: Test-time compute improves performance in most domains but plateaus early in some, indicating domain-specific ceilings tied to domain knowledge or policy understanding.The authors propose adaptive compute allocation and targeted training for constraint-heavy domains.
  • Conclusion: Future work prioritizes constraint-aware plan generation and long-horizon state management to address policy constraints, prerequisites, and error accumulation.The benchmark is intended to support community research on planning, safety, and policy compliance in realistic workflows.

A. Data Collection and Human Annotation

The benchmark’s data collection combines domain expertise, simulated enterprise environments, complexity thresholds, and layered verification. Its seed data is designed for realistic, structurally consistent, and reproducible task scenarios.

  • Data Collection: Subject-matter experts and software engineers supported task authoring and sandbox construction across technical enterprise domains.The broader annotation workforce included annotators, reviewers, team leads, and more than 160 contributors.
  • Task Generation: Tasks were authored within domain-specific databases and tool environments, with complexity thresholds based on required tools and verification steps.More tools and verifiers directly correlated with higher task complexity, while action trajectories were captured by internal tooling.
  • Verification: Verification retained challenging tasks through multi-layered checks, including draft execution against GPT-5, Gemini, and Claude to identify invalid paths and missing tools.The pipeline analyzed trajectories for task-definition errors, missing tools, and invalid database entries.
  • Seed Data: Seed data modeling used official API documentation, data models, usage examples, and SME guidance to reproduce entity structures, field constraints, and API behavior.The resulting environment remained platform-agnostic and was tested for database consistency.
  • Seed Data: Each task uses a distinct high-level dataset, with databases dynamically expanded as annotators designed and vetted new scenarios.The datasets are intentionally unique rather than simple variations of a shared seed.
  • Environment: The Docker environment initializes a fresh database for each task run, preventing state leakage and prior-execution side effects.Containerization also standardizes tool invocation and response handling for reproducible evaluation.

B. Task Categories and Complete Examples

EnterpriseOps-GYM spans administrative, collaboration, and customer-support workflows that require sequential reasoning, verified lookups, policy compliance, and multi-step orchestration. The examples illustrate how domain-specific constraints shape tool execution.

  • Calendar: The Calendar category requires managing time, access, resources, audit listeners, and calendar metadata beyond ordinary meeting booking.Tasks include administrative restructuring and enforcement of access-control policies.
  • Calendar: Calendar tasks require conditional checks, temporal reasoning, timezone conversion, recurrence handling, and translation of high-level intent into policy-compliant actions.Agents may need to verify calendar existence or attendee status before modifying events.
  • Calendar: The Calendar example’s oracle sequence includes calendar lookup, free/busy querying, event creation, access-control updates, and event watching.The listed tools reflect a multi-step workflow rather than a single scheduling action.
  • Customer Service Management: Customer Service Management models the lifecycle of customer issues, including entitlement verification, asset management, SLA handling, escalation, and case-state transitions.The workflow combines entity resolution with business-rule enforcement in a technical support setting.
  • Customer Service Management: CSM workflows require identifying entities from vague descriptions and coordinating actions such as onboarding agents, reassigning cases, and applying priority rules.Examples include transferring cases by priority and deactivating an agent.

CSM Example Task

The CSM example encodes a tightly constrained case-management workflow in which agents must validate identities, relationships, entitlements, products, permissions, lifecycle transitions, SLAs, and knowledge links before closure.

  • Case Creation: Case registration begins by identifying the reporting contact, verifying account association, collecting issue details, and confirming product ownership.Missing priority defaults to moderate and the initial case state is new.
  • Assignment: Case assignment requires an active agent or manager who belongs to the selected assignment group.Cases may be assigned to a group or to a user within that group.
  • Lifecycle: Cases follow constrained lifecycle transitions and cannot be closed directly from an earlier state; resolution must precede closure.Each transition requires validating the acting user and recording applicable timestamps.
  • Validation: Entitlements, installed products, and knowledge articles must match account, product, status, publication, visibility, and ownership requirements.Invalid entitlements, retired products, or unsuitable knowledge records must not be applied.
  • Escalation: Escalation requires a supported trigger, an escalation flag, a reason, and justification recorded in case notes.Triggers include SLA-breach risk, explicit customer requests, or high business impact.
  • Permissions: Access controls restrict users to cases they own, are assigned to, or can access through their assigned group, while administrators have broader privileges.The policy also prohibits disclosure of case details outside those permissions.
  • Service Levels: SLA targets depend on linked entitlements, products, priorities, support levels, and coverage hours, with applicable SLAs paused while cases are pending.If no entitlement or SLA is associated, no service-level commitment is in scope.
  • Example Failure: The hidden incident example shows that accepting an unvalidated incident and relying on a default published state can produce an incorrect knowledge link.The agent failed to list incidents, created the article as published, and linked it to the wrong incident.

C.6. Failure Analysis

The agent made two policy-sensitive errors: it selected the wrong incident and used the wrong knowledge-base state. Both failures arose from accepting plausible defaults or matches without contextual cross-validation.

  • The agent linked the wrong incident after treating a number match as identity confirmation without checking status, description, or assignee.The verifier expected INC 011, but the agent used INC 004.
  • The agent used the default published knowledge-base state even though the task required a draft.The verifier therefore rejected the new knowledge-base entry.
  • The missing disambiguation call prevented the agent from resolving the incident identity before linking it.The relevant list-incidents step was never called.
  • Both failures reflect accepting the first plausible result without cross-validating it against task context or policy constraints.
  • Claude Sonnet 4.6 passed 4/5 verifiers in this case.

C.10. Hidden Challenges

The case-management task required several policy-inferred operations beyond the user’s wording, including knowledge-base remediation, group creation, membership, and a lifecycle transition. The agent completed nearly all operations but omitted the mandatory pending-state update.

  • Hidden Challenges: The knowledge article had to be updated from retired to published before linking, while the nonexistent Case Management group had to be created as a support group.
  • Hidden Challenges: Knowledge-base linkage unconditionally required the case to transition to pending.
  • Hidden Challenges: The agent completed a near-perfect five-turn trajectory but omitted state="pending" when updating the case after linking the knowledge article.It correctly found and published the retired article, created the support group, added Joanne, linked the article, and assigned the case.
  • Hidden Challenges: The verifier passed article remediation, suggested linking, group creation, and membership, but failed the omitted pending-state transition.
  • Hidden Challenges: The failure was lifecycle truncation: reasoning focused on assignment and did not revisit the lifecycle rule after linkage.

C.17. Hidden Challenge: Two Simultaneous Closure Prerequisites

“Wrap up” required an ordered closure sequence rather than merely creating a survey: deactivate active tasks, close the case, and approve the request. All three steps were necessary for verification.

  • Two Simultaneous Closure Prerequisites: The task’s hidden difficulty was translating a natural-language wrap-up request into multiple policy-ordered state changes.
  • Two Simultaneous Closure Prerequisites: The closure sequence required deactivating active tasks, setting status to closed complete, and approving the request.
  • Two Simultaneous Closure Prerequisites: The case began awaiting approval with a requested approval and two active tasks, making each prerequisite operationally relevant.
  • Two Simultaneous Closure Prerequisites: All three closure steps had to complete; failing any one caused the verifier to return COUNT=0.
  • Two Simultaneous Closure Prerequisites: The agent correctly created the survey instance with the required survey, task, and assignee parameters.

C.20. Failure Analysis

The agent misread lifecycle language as a content-editing instruction and stopped after survey creation. It neither deactivated the active task nor executed the required case closure and approval updates.

  • Failure Analysis: The agent converted task 6 into a survey instead of setting active=false, leaving the mandatory closure task active.The verifier required task deactivation under the closure constraint.
  • Failure Analysis: The agent never called update hr case, so the case remained unclosed and its approval request remained unapproved.The required transition was awaiting approval to closed complete, with requested changing to approved.
  • Failure Analysis: The survey instance itself was created correctly, but that success did not satisfy the closure prerequisites.
  • Failure Analysis: Both failures came from reading “wrap up” and “using the first task” as surface content directives instead of lifecycle commands.

D. Additional Analysis and Results

The paper presents additional analysis on task complexity and reports full evaluation results in dedicated figures and tables.

  • Additional task-complexity analysis is presented in Figure 7 and Tables 6 and 7.The paper also directs readers to Table 2 for full results.
  • Figure 7 provides the task-complexity analysis.
  • Tables 6 and 7 contain additional analysis, while Table 2 reports full results.

E. Impact Statement

EnterpriseOps-Gym targets reliable planning and execution over persistent-state tools, while emphasizing reproducible, safety-conscious evaluation. Its analyses and stated limitations frame both the benchmark’s diagnostic value and the risks of overfitting, over-reliance, and computational cost.

  • EnterpriseOps-Gym provides a reproducible testbed for studying agentic planning, tool selection, and error recovery.
  • Task-complexity distributions vary substantially across domains, with CSM and HR showing longer, heavier-tailed execution traces.Email and Calendar tasks are shorter and more tightly distributed, while Hybrid and ITSM workflows show moderate to high variance.
  • Models show substantial degradation on long-horizon and cross-domain workflows, with policy compliance receiving the lowest verifier pass rates.
  • Outcome-based verification and safety-critical constraints encourage agents that are reliable and auditable.The benchmark emphasizes verification of system state, task completion, and policy compliance.
  • Benchmark-optimized systems may overfit to specific task patterns or verification criteria.
  • Operational automation may increase over-reliance on AI, while agentic training and evaluation also incur computational and environmental costs.
Loading 2603.13594v1…