EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

Introduction

• Enterprise agents must maintain state across interleaved tool calls, plan through dozens of actions, and follow workplace access policies and procedures.Because actions can modify live databases, trigger workflows, and affect users, errors can propagate silently or cause irreversible side effects; agents must also stop when a task violates policy.Source: §1 · pp. 1–2
• Existing general tool-use evaluations treat calls as atomic and stateless, while enterprise benchmarks often cover one vendor, fewer than 25 tables, under 50 tools, short horizons, and limited policy constraints.The cited general evaluations include Li et al. (2023), Qin et al. (2024), and Chen et al. (2025); enterprise benchmark examples include Drouin et al. (2024), Boisvert et al. (2024), Huang et al. (2025a;b), Jha et al. (2025), and Vishwakarma et al. (2025).Source: §1 · p. 2
• The benchmark covers 1,150 expert-authored tasks across eight enterprise domains, including 30 infeasible scenarios, with SQL verification of completion, state integrity, policy compliance, and side effects.It spans productivity and mission-critical domains and includes a Hybrid category for coordinated cross-domain execution.Source: §1 · pp. 2–3
• 164 relational database tables and 512 functional tools form a containerized, interactive environment with expert trajectories averaging 9 steps and reaching 34.The environment is designed to test long-horizon planning, cross-system state management, and policy-constrained execution.Source: §1 · p. 3
• 37.4% is Claude Opus 4.5’s overall success rate, leaving current agents short of autonomous enterprise readiness.The best model cleanly refuses policy-violating tasks only 53.9% of the time.Source: §1 · p. 3
• 51–52% on Email, Teams, and Drive contrasts with 28.5% on ITSM and 30.7% on Hybrid tasks.Performance drops in policy-governed and cross-domain work, where constraint-aware reasoning is unavoidable.Source: §1 · p. 3
• 14–35 percentage points of improvement from human-authored plans identifies strategic planning, rather than tool use, as the primary bottleneck.Adding distractor tools has negligible impact, while decomposing tasks for multi-agent orchestration can regress performance because of sequential state dependencies.Source: §1 · p. 3

API and Tool-Use Benchmarks

• General API and tool-use benchmarks measure tool-calling accuracy, scale, dynamic turns, user simulation, and refusal, but do not model enterprise multi-system, policy-constrained, stateful ecosystems.Source: §2 · p. 3

Enterprise Benchmarks

• Existing enterprise benchmarks have fewer than 25 tables and often lack multi-vendor scope, expert grounding, or refusal evaluation; EnterpriseOps-Gym spans eight domains, 164 tables, 512 tools, and 1,150 tasks.The tasks include policy-constrained infeasible scenarios.Source: §2 · pp. 3–4

Agentic Planning and Computer-Use

• Planning and computer-use benchmarks test sophisticated or extended planning in everyday services and GUI environments, rather than policy-governed enterprise contexts.EnterpriseOps-Gym shares their emphasis on extended planning while combining it with enterprise domain fidelity.Source: §2 · p. 4

3.1. EnterpriseOps-GymConstruction

• Eight domains combine policy-heavy operational systems—Customer Service Management, HR, and ITSM—with collaboration tools—Email, Calendar, Teams, and Drive—and Hybrid cross-domain orchestration.Domain selection considered real-world relevance, diversity in policy complexity and data sensitivity, and expert availability; together, the domains cover internal operations, customer-facing processes, and collaboration infrastructure.Source: §3.1 · pp. 4–5
• A containerized Docker sandbox provides domain-specific databases, APIs, and tools, built with input from more than 160 contributors, including engineers and subject-matter experts.It starts from fixed seed data and tools, then annotators extend tables, schemas, and records as tasks require, using product documentation and SME knowledge to model realistic structures and constraints.Source: §3.1 · p. 5
• Tasks are designed as challenging multi-step scenarios with ordering dependencies, access constraints, and policy conflicts, and have a unique final state despite potentially multiple valid paths.The 30 infeasible tasks are impossible because of insufficient tools, explicit policy violations, or unavailable resources; annotators also author detailed execution trajectories and natural-language plans grounded in system constraints and available tools.Source: §3.1 · p. 5
• Executable SQL scripts verify task outcomes, including goal achievement, integrity, permission compliance, and side effects, while quality assurance checks feasibility, clarity, and verification coverage.Evaluation is outcome-based, allowing alternative valid solution paths rather than requiring a fixed action sequence.Source: §3.1 · p. 5

3.2. Dataset Statistics

• 1,150 tasks and 512 tools span eight enterprise domains, including 30 infeasible tasks designed to test refusal of unsatisfiable requests.Expert trajectories average 9.15 steps, ranging from 6.2 in Email to 12.1 in CSM and reaching 34 in HR; tasks average 5.3 verification conditions, with a maximum of 44.Source: §3.2 · pp. 5–6
• The sandbox models 164 database tables, with each task interacting with an average of 24.9 tables and up to 73 in Hybrid scenarios.Tasks operate over an average of 3,443 rows, exceeding 10,000 in data-heavy domains such as CSM; FK density averages about 1.7 per table, ranging from 1.1 in Calendar to 2.4 in HR.Source: §3.2 · p. 6

4.1. Baselines

• Agents are compared under identical instructions, tools, sandboxes, and protocols, generally with oracle tool retrieval; the primary metric is average pass@1 across three runs, verified by SME-written SQL checks.Pass@1 requires all task requirements and constraints to be satisfied; verifier-level success can overstate utility by counting trivial trajectory checks despite failures in core logic, compliance, or side effects. Baselines include closed- and open-source models using a ReAct-style loop.Source: §4.1 · p. 6, §4.2 · p. 6

How do models perform across different domains?

• Claude Sonnet 4.5’s average completion rates remain stable with +5, +10, or +15 distractor tools, increasing by +0.07%, +2.35%, and +0.64%, respectively; the authors identify planning and policy adherence, not tool discovery, as the primary bottleneck.Output tokens increase by an average of 4–9%; the distractors are retrieved as tools appearing relevant to the task.Source: How do models perform across different… · pp. 6–7
• 37.4% average task completion makes Claude Opus 4.5 the top model; Gemini-3-Flash reaches 31.9%, while ITSM and Hybrid remain the hardest domains.Gemini-3-Flash offers a stronger closed-source cost–performance tradeoff than pricier GPT-5 (29.8% at 0.16 USD) and Claude Sonnet 4.5 (30.9% at 0.26 USD), at 0.03 USD per task; Claude Opus 4.5 costs 0.36 USD per task. DeepSeek-V3.2 (High) is the strongest open-source model at 24.5%.Source: How do models perform across different… · pp. 6–7, Table 2 · p. 7
• 53.9% is GPT-5.2 (Low)’s infeasible-task abstention rate with no side effects; DeepSeek-V3.2 (High) reaches 53.8%, but all models remain below safe production applicability.The 30 infeasible tasks combine insufficient tools, policy violations, or unavailable resources with an average of 3 to 4 constraints per request.Source: How do models perform across different… · p. 7
• Pass@1 falls monotonically as task horizons lengthen, more sharply for open-weight models; higher GPT-OSS-120B thinking budgets raise Drive from 8.6% to 41.0%, Calendar from 8.7% to 35.6%, and Teams from 4.0% to 32.0%.At low budget, GPT-OSS-120B scores 1.1% on CSM and ITSM and 0.0% on HR; Email peaks at the medium budget (45.2%) and ITSM plateaus at 6.1% from medium to high.Source: How do models perform across different… · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 9

4.4. Further Analysis

• 14–35 percentage-point gains from human-authored plans across executor models and domains exceed the 6–13% gains from Claude-generated plans, indicating strategic planning is a major bottleneck.The evaluations use weaker executors across CSM, ITSM, and HR; Qwen3-4B with human-authored or Claude plans is competitive with or outperforms larger models under the same condition.Source: §4.4 · pp. 8–9, Table 3 · p. 8
• Planner+Executor improves Claude Sonnet 4.5 over ReAct by 10.7% in CSM and 8.8% in HR, but decomposition regresses in CSM and HR, consistent with disruption of sequential state dependencies.The Planner+Decompose+Subtask Executor architecture gives only a minor lift in ITSM and drops below ReAct in CSM (16.2% vs. 16.7%).Source: §4.4 · p. 9, Figure 6 · p. 9
• Models often fail through missing prerequisite lookups, missed cascading policy actions, incorrect ID resolution, and premature completion; they struggle most with permission and process compliance.The paper categorizes verifier checks as task completion, integrity constraints, and permission and process compliance; the latter is described as a critical deployment gap.Source: §4.4 · pp. 9–10

5. Discussion and Conclusion

• EnterpriseOps-Gym spans 1,150 expert-curated tasks across eight enterprise productivity domains, with 512 tools and SQL-based verifiers authored by subject matter experts.It is a benchmark and sandboxed evaluation platform.Source: §5 · p. 10
• 37.4% task success was the best model’s result even under oracle tool access, while the best infeasible-task refusal rate was 53.9%.Performance also declines with longer horizons and cross-domain coupling, and agents struggle particularly with Permission and Process Compliance.Source: §5 · p. 10
• 14–35 percentage point gains across models and domains from human-authored plans identify planning, rather than tool execution, as the dominant bottleneck.Qwen3-4B conditioned on human plans is competitive with much larger models under the same condition; distractor tools do not meaningfully hurt performance.Source: §5 · p. 10
• Test-time compute yields substantial gains in most domains, but scaling is not universally monotonic and some domains plateau early.Additional reasoning tokens may not overcome gaps in domain knowledge or policy understanding; future work should investigate adaptive compute allocation and targeted training on constraint-heavy domains.Source: §5 · p. 10
• Future research priorities are constraint-aware plan generation, long-horizon state management, and safe refusal and escalation.The proposed mechanisms include reasoning over policy constraints, side-effect dependencies, and prerequisites before acting; episodic memory or structured state representations; and reliable detection of infeasible or policy-violating requests.Source: §5 · p. 10
• The authors will release EnterpriseOps-Gym, its sandbox environment, and evaluation tooling for open, community-driven research.The modular sandbox allows new domains, tools, and workflows to be added as enterprise practices evolve.Source: §5 · p. 10

A.1. Demographics and Recruitment

• A multi-tiered team of annotators, reviewers, and leads included contributors across Asia, North America, Latin America, and Africa, all with bachelor’s degrees and prior data-labeling and UI-research experience.Contributors were 22–37 years old and held degrees in engineering, computer science, or related disciplines.Source: §A.1 · p. 14
• Over 160 contributors supported the campaign, averaging 20 annotators and 6 reviewers per domain, with four months of collection and approximately $100 per task.Recruitment prioritized technical writing, English, computer science, and prompt-engineering skills, with ITSM and CSM subject-matter experts and engineers building the sandbox; the campaign began with a one-month pilot.Source: §A.1 · p. 14

A.2. Annotation Process

• Contributors were assigned domains and taxonomies and used simulated databases and functions; task complexity was tied to required tools and verification steps, with trajectories captured internally.Annotators used domain expertise to design scenarios within their taxonomies.Source: §A.2 · p. 14

A.3. Quality Assurance and Verification

• Draft tasks were tested with GPT-5, Gemini, and Claude, then reviewed by humans for language, tool logic, execution, prompt clarity, trajectory quality, and verifier robustness.Automated testing exposed flawed task definitions, missing tools, invalid entries, access conflicts, overly simple reasoning paths, and grounding issues; annotators refined scenarios and discarded trivial tasks.Source: §A.3 · p. 14

A.4. Sandbox Environment

• SMEs grounded seed data in official API documentation, data models, and usage examples; engineers modeled platform-agnostic entity, field, and API behavior and checked database consistency.The research examined typical entity structures and relationships, field semantics and constraints, and expected API behavior and response patterns.Source: §A.4 · pp. 14–15
• 164 unique tables, with a mean foreign-key degree of 1.7, formed a densely connected data ecosystem expanded as annotators added scenario requirements.Task databases represented distinct high-level use cases rather than simple variations, even though individual field values could sometimes repeat.Source: §A.4 · p. 15
• Each task ran in a fully containerized Docker environment with a fresh database instance, preventing state leakage and enabling reproducible execution and deterministic verification.The environment was designed to be extensible, with containers and guidance for adding tasks and verification scripts slated for release.Source: §A.4 · p. 15

B.1. Calendar

• Calendar tasks require ecosystem-level administration of access controls, audit listeners, and resource metadata, alongside conditional action, timezone, recurrence, and fuzzy-search reasoning.The example begins with checking calendar availability and lists oracle tools including free/busy queries, event creation, ACL changes, and event watches.Source: §B.1 · p. 15, Calender Example Task · pp. 15–16

B.2. Customer Service Management (CSM)

• Customer Service Management tasks span case intake through resolution, requiring entitlement verification, asset and case-state management, SLA compliance, escalation, and policy-aware routing.Cases must be tied to verified account contacts and matching products; priority rules and organizational hierarchy shape assignment and escalation.Source: §B.2 · p. 16, CSM Example Task · p. 17

CSM Example Task

• CSM procedures constrain case work through valid lifecycle transitions, active matching entitlements, published knowledge links, and logged escalation decisions.Knowledge-search and article-creation policies further specify how findings should be linked to cases and labeled as suggested, applied, or resolution.Source: CSM Example Task · pp. 17–18

B.3. Drive

• Drive tasks emphasize secure file-system governance, including permission inheritance, retention metadata, audit trails, graph traversal, precise search, and version-aware organization.Example operations include excluding all but a specified external domain and finding older large video files with compound filters.Source: §B.3 · p. 19

Drive Example Task

• Drive policies require permission checks before operations, scope-limited access, valid file identifiers, and appropriate confirmation for irreversible deletion or public sharing.They also favor atomic validated actions, require Editor access to both folders for moves, prefer trash over deletion, and require exact roles when creating permissions.Source: §1 · p. 19, §2 · p. 20, §3 · p. 20

Permission Modification and Deletion

• Permission operations require a permission ID and new role for modification, Editor access for deletion, and current-Owner status for ownership transfer.These rules constrain changes to file access and ownership.Source: Permission Modification and Deletion · p. 20

5. Monitoring and Synchronization Watch Subscriptions (Webhooks)

• Webhook subscriptions provide real-time change notifications and require a page token, unique channel ID, verified address, and renewal mechanism; incremental synchronization uses start-page tokens and change listings.Source: §5 · p. 20, Changes Feed · p. 21

6. Validation, Error Handling, and Quotas

• File operations require pre-validation, quota and filename checks, exponential backoff for 429 errors, and clear error responses; an example task combines copying files with labels, descriptions, sharing, and comments.The example requests a new compliance folder and copies three files into it, with access, label, description, sharing, and comment updates.Source: §6 · p. 21

B.4. Email

• Email tasks combine mailbox administration and security with pattern recognition and temporal logic, including precise search queries and conditional workflows.Example capabilities include verifying a send-as alias before sending a draft and managing automated filters or out-of-office responses.Source: §B.4 · p. 21, Email Example Task · p. 21

B.5. Human Resources (HR)

• HR tasks model sensitive employee lifecycle work under strict SOPs, role-based access, and privacy constraints; agents must not assume data or violate employee privacy or security.An example requests an urgent medical-benefits case for Travis Wood, assigned to the HR Service Desk with a specified account number and description.Source: §B.5 · p. 22, HR Example Task · p. 22, §6 · p. 23

B.6. IT Service Management (ITSM)

• ITSM tasks require relational and causal reasoning across incidents, problems, changes, and configuration items, including priority and SLA assessment.An example asks the agent to link a completed switch replacement change to related records, check blockers, close it with notes, and notify the caller.Source: §B.6 · p. 23, ITSM Example Task · p. 24

B.7. Teams

• Teams tasks require structural and organizational reasoning about collaboration spaces, security boundaries, events, and targeted communication.The example combines channel creation, membership, app tabs, a welcome message, and an organization-only townhall with specified co-organizers.Source: §B.7 · p. 24, Teams Example Task · p. 25

B.8. Hybrid

• Hybrid tasks span two domains and average 40 tables, versus a mean of 25 for single-domain tasks, requiring planning across systems whose actions trigger one another.They represent the benchmark’s most complex task class and require state tracking across disparate APIs.Source: §B.8 · p. 25
• Hybrid tasks can require checking warranty data, logging a customer interaction, and scheduling a calendar event while keeping state and entity mappings consistent across systems.The example conditions the write and scheduling actions on whether the product warranty extends beyond 2025.Source: §B.8 · p. 25

Hybrid Example Task

• The hybrid example asks the agent to check a product warranty, log an open email interaction if it extends beyond 2025, and schedule a January 2026 event using the product name.The prompt specifies the interaction start time as 2025-12-20 10:00 and the event time as January 25, 2026 at 10:00 AM in the default time zone.Source: Hybrid Example Task · pp. 25–26

C. Rollout examples (concise)

• Abridged Claude-Sonnet-4.5 rollout examples summarize user tasks, system policy, and tools from the ITSM, CSM, and HR domains.The cited passage identifies Claude-Sonnet-4.5 as Anthropic, 2025b.Source: C. Rollout examples (concise) · p. 26

C.2. Task (condensed)

• The task required Kenji Tanaka to draft a new internal article titled “VPN Connection Failure Guide” and link it to the incident.Policy required creating and linking a knowledge draft before final closure.Source: §C.2 · p. 26
• The policy prohibited fabricated IDs or assumed optional and default arguments, requiring reliance on verified API results.Source: §C.2 · p. 26

C.3. Hidden Challenge: Duplicate Incident Numbers

• Two incidents in different tenants shared external ID INC0000004: INC 004 was a new TechCorp issue, while INC 011 was Acme’s resolved VPN incident.INC 011 was assigned to Kenji Tanaka.Source: §C.3 · p. 26
• Searching by number returned INC 004 first; filtering for the resolved incident assigned to USER 009 recovered INC 011.The gold trajectory detects the mismatch before using the filtered incident list.Source: §C.3 · p. 27, §C.4 · p. 27

C.4. Gold Trajectory

• The gold trajectory checked the incident’s knowledge links, found none, and linked KB 006 to INC 011 as the resolution.Source: §C.4 · p. 27

C.5. Agent Behavior

• The agent received INC 004 from find incident by number and accepted it without validation; it never called list incidents to explore the difference, then created a published-by-default KB article linked to INC 004.Source: §C.5 · p. 27

C.6. Failure Analysis

• The agent selected the wrong incident without cross-validation and accepted the published default instead of the required draft state.INC 004 had the wrong status, description, and assignee; the verifier required incident ID INC 011 and state "draft".Source: §C.6 · p. 27

C.7. Summary

• The incident summary identifies INC 011 and INC 004 alongside a KB link-verifier failure.Source: §C.7 · p. 27
• The KB was draft then published, but the new KB verifier still failed.Source: §C.7 · p. 27
• The disambiguation step that lists incidents was called.Source: §C.7 · p. 27
• The incident-selection root cause was that the disambiguation step was never called.Source: §C.7 · p. 27
• Both failures shared a pattern: accepting the first plausible result without cross-validating task context or policy constraints.Source: §C.7 · p. 27

C.8. Case Study: CSM — “KB Remediation and Case Setup”

• 4/5 verifiers passed for Claude Sonnet 4.6 in the CSM case study.Source: §C.8 · p. 27

C.9. Task (condensed) / C.10. Hidden Challenges

• For case CS-0000002, the agent had to link a published article and assign Joanne Simpson within a support group; policy required a pending-state transition.The hidden work included publishing retired KB-0000197 and creating the nonexistent Case Management group as type="support".Source: §C.9 · pp. 27–28, §C.10 · p. 28

C.11. Gold Trajectory

• The gold trajectory created the new active Case Management support group with group id 81.Source: §C.11 · p. 28

C.12. Agent Behavior / C.14. Summary

• In the Service case, the near-perfect five-turn trajectory omitted state="pending" and left the case new; in HR, the agent mis-updated a task and never closed the case.The Service agent otherwise linked the KB and completed assignment steps; the HR agent did create the survey instance.Source: §C.12 · p. 28, §C.14 · p. 29

HR Example

• Karen Watkins was asked to wrap up James Hill’s resolved HR portal case and add a follow-up technical issue survey to gather feedback.Source: HR Example · p. 29

C.17. Hidden Challenge: Two Simultaneous Closure Prerequisites

• Closing the HR case required deactivating both active tasks, setting status="closed complete", and approving the request; failing any step leaves verifier COUNT=0.The seeded case was awaiting approval, the request was requested, and tasks 6 and 7 were active.Source: §C.17 · p. 29

C.18. Gold Trajectory

• The HR gold trajectory found case 3 awaiting approval with two active tasks, closed and approved it, and created a survey instance for task 6.Source: §C.18 · pp. 29–30

C.19. Agent Behavior

• The HR agent completed four lookups and created the survey, but set task type="survey" instead of deactivating task 6 and never updated the case.It declared completion after six turns, describing the survey as something James Hill would complete later.Source: §C.19 · p. 30

C.20. Failure Analysis

• The agent converted task 6 to a survey-type task instead of deactivating it, so the verifier found active=true rather than active=false.It interpreted “using the first task” as a directive to change the task type and called update hr case task with type="survey" and a short description.Source: §C.20 · p. 30
• The Closure Constraint required task deactivation, not type conversion, and update hr case task’s presence in the tool set signalled that lifecycle action.The verifier checked for active=false, so the task-type update did not satisfy the requirement.Source: §C.20 · p. 30
• The agent treated “wrap up his case” as James Hill completing a survey later, stopped after survey creation, and declared success without calling update hr case.The case remained in awaiting approval, so it did not reach the required closed complete status.Source: §C.20 · p. 30
• The valid transition was awaiting approval to closed complete; the visible request status was requested, while verification required closed complete and approved.The update hr case tool was present as a planning signal, but the required closure action was not performed.Source: §C.20 · p. 30

C.21. Summary

• The summary checklist records the task’s active=false requirement as failed because of wrong parameters, and the case’s closed complete requirement as failed because the tool was never called.It also marks request status=approved as failed because the tool was never called.Source: §C.21 · p. 30
• The checklist marks the survey instance as passing, with correct parameters, while the task deactivation and case-closure checks fail.It labels the survey result V3 passes and the failed checks V1 fails and V2 fails.Source: §C.21 · p. 30
• Both failures stemmed from misreading “wrap up” and “using the first task” as content directives rather than lifecycle commands, thereby ignoring the policy-defined closure sequence.The agent acted on a plausible surface reading instead of the required sequence.Source: §C.21 · p. 30

• Task-complexity analyses are presented in Figure 7 and Tables 6–7, with full results in Table 2.Source: D. Additional Analysis and Results · p. 30

Positive Impacts

• EnterpriseOps-Gym offers a rigorous, reproducible testbed for agentic planning, tool selection, and error recovery, supporting safer, more reliable, and auditable automation.The benchmark emphasizes outcome-based verification and safety-critical constraints; it may reduce human burden and help practitioners focus on higher-level decisions.Source: E. Impact Statement · pp. 30–31

Potential Risks and Limitations

• Benchmark overfitting, over-reliance on AI when failures are poorly understood or monitored, and the environmental costs of training and evaluation are key risks.Systems optimized exclusively for benchmark performance may overfit to specific task patterns or verification criteria.Source: E. Impact Statement · pp. 31–32

Risk Mitigation

• The authors mitigate risks by positioning EnterpriseOps-Gym as a diagnostic benchmark, using synthetic sandboxed environments without proprietary or sensitive data.They emphasize analysis over leaderboard ranking.Source: E. Impact Statement · p. 32