PaperlayerView exact revision on arXiv

Source-linked AI summary

CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments

Monojit Banerjee, Zhujun Lan, Defu Cao

arXiv:2610.32004v1cs.HCcs.CL

TL;DR

Delegated CRM agents must execute multi-step workflows correctly against mutable records, but existing evaluation often emphasizes retrieval and answer correctness over state changes and policy. CRM-Agent-Bench evaluates 555 stateful tasks with task-specific conjunctive contracts across sixteen systems; strict success spans 64.32% to 83.60%, while leading systems also differ in cost and per-domain reliability.

  • Problem

    Existing CRM benchmarks bring realistic data and multi-turn interaction but largely evaluate retrieval and answer correctness rather than correct, policy-compliant state changes.

    Source: §1 · p. 2

  • Method

    CRM-Agent-Bench evaluates stateful CRM tasks with task-specific contracts that conjunctively check applicable actions, arguments, ordering, responses, target state, and policy constraints.

    Source: §1 · p. 2, §2.3 · p. 5

  • Results

    Strict success ranges from 64.32% to 83.60% across sixteen systems, while closely grouped leading systems diverge in cost and per-domain reliability.

    Source: §1 · p. 3, §6 · p. 13

  • Takeaways & Limitations

    The results support weighing execution cost, per-domain reliability, and policy-constrained completion alongside strict accuracy when selecting enterprise agents.

    Source: §6 · p. 13

    Call counts are system-level descriptives and do not isolate provider load, model architecture, or gateway implementation.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

Deploying language agents in real applications demands more than the right answer: an agent must execute a multi-step business process correctly, follow domain-specific rules, and interact with a user across a dialogue, where the gap between saying and doing is the gap between a satisfied customer and a real loss. We argue that a deployable-agent benchmark should jointly assess four facets: the outcome an agent reaches, the trajectory it takes, its operational viability, and its policy compliance. Customer relationship management (CRM), where mutable records meet operational rules and consequential effects, exercises all four. We introduce CRM-Agent-Bench, an execution-grounded benchmark of 555 stateful, multi-turn tasks across nine business domains and 36 capability tags. Each task supplies a task-specific executable contract drawn from checks over actions, arguments, order, response, target state, and protected state; success is the strict conjunction of the checks declared for that task. We evaluate the complete agent system through these contracts and report scored coverage, latency, and cost. Across sixteen systems evaluated on the audited final release, strict success spans 64.32% to 83.60%; the closely grouped leading systems nonetheless diverge sharply in cost and per-domain reliability. CRM-Agent-Bench offers an auditable framework for measuring contract-grounded business-transaction integrity and adherence to task-declared benchmark policies, with a portable structure designed for extension as models improve. Github repository: https://github.com/salesforce-misc/crm-agent-bench

1 Introduction

Delegated CRM agents must do more than answer correctly: they must execute permitted workflows against mutable records and preserve intended state. CRM-Agent-Bench evaluates this execution through task-specific contracts and reports outcome, trajectory, operational viability, and policy compliance.

From answers to delegated execution

  • Delegated execution can fail through an incorrect transaction, continuation after backend rejection, or a claim that an operation succeeded when it did not.CRM workflows couple mutable records and operational rules to downstream business processes, making preservation of records outside the user's intent important.Source: §1 · pp. 1–2

Evaluating enterprise execution

  • A complete assessment must jointly inspect outcome, trajectory, operational viability, and policy compliance, because a system can score well on one facet while failing another.In the duplicated-checkout example, response, action, and order checks pass while the state check detects a doubled loyalty balance and fails the strict contract.Source: §1 · p. 2, Figure 1 · p. 3

CRM-Agent-Bench

  • Task-specific contracts combine applicable checks, and strict success requires every declared check to pass; undeclared dimensions are absent rather than counted as passes.Negative checks cover forbidden actions, protected state, and arguments containing another party's identifiers; latency and cost are reported alongside the pass criterion.Source: §1 · pp. 2–3

Empirical evaluation

  • 64.32% to 83.60%: strict success spans this range across sixteen systems; response-only grading overestimates strict success by 10.5 to 11.0 percentage points.On tasks with explicit policy constraints, systems pass applicable negative checks on 91.2% of trials but satisfy the full contract on only 66.8%, versus 82.1% on tasks without negative assertions.Source: §1 · p. 3

Our Contribution

  • CRM-Agent-Bench contributes 555 stateful, multi-turn tasks across nine business domains and 36 capability tags, runnable without a live CRM or vendor backend.Its contracts grade applicable outcome, trajectory, and policy requirements, while latency and cost report operational viability separately.Source: §1 · p. 3

2 The CRM-Agent-Bench Benchmark

The benchmark evaluates complete agent systems through resettable CRM tasks, an on-policy user, and task-specific executable contracts. It grades declared trajectory, outcome, and policy checks conjunctively, while reporting operational measures separately.

2.1 Evaluation Target

  • The evaluation target is a complete agent system operating through tools against mutable enterprise state, with each task's declared checks determining strict success.The benchmark separates its task artifact from the system under test; Figure 2 depicts the routed agent, fresh backend, and conjunctive contract checks.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 Tasks and Stateful Environments

  • Each task specification induces a finite-horizon partially observable process: the agent sees dialogue, tool results, and exposed instructions, but not the raw CRM state, private user instruction, or contract.Later user turns depend on earlier visible agent replies rather than a fixed transcript.Source: §2.2 · p. 4

2.3 Trial Execution and Grading

  • The on-policy simulator generates each next turn from the agent's user-visible reply and continues until it stops, the 30-turn cap, or the 1,200-second timeout.It does not receive private reasoning, tool calls, tool results, or database state.Source: §2.3 · p. 5
  • Strict task success is the conjunction of declared checks for actions, arguments, order, response, target state, and forbidden behavior; undeclared checks are absent.Only response-content criteria go to a fixed LLM judge; programmatic checks cannot be overridden by that judge.Source: §2.3 · p. 5
  • The benchmark reports scored coverage alongside conditional success, latency, cost, and per-check evidence rather than collapsing the four facets into one composite index.The static capability-exposure audit measures graph-wide reachability of forbidden tools; turn-level exposure depends on routing.Source: §2.3 · p. 5

3 Benchmark Construction

Benchmark construction turns CRM workflows into resettable synthetic environments, architecture-neutral task definitions, and a reference-agent scaffold. Its coverage hierarchy spans multiple domains and capability requirements without prescribing an agent's route.

3.1 Task and Reference-Agent Construction

  • Source workflows are ported into synthetic environments, architecture-neutral task definitions with executable contracts, and a graph-structured reference agent.Synthetic records and transactions let the benchmark run without a live vendor CRM system.Source: §3.1 · p. 6
  • The 555 tasks draw on five CRM applications reviewed with practitioners and domain experts, plus 22 tool-using workflow families from a public collection of agent recipes.Every task undergoes schema, tool-reference, and fresh-environment gold-path validation.Source: §3.1 · p. 6

3.2 Workflow Taxonomy and Coverage

  • Coverage is organized from domain to capability, scenario, and executable contract; this hierarchy describes coverage without predetermining the agent's route.Grading consumes the runtime trace and final state rather than a post-hoc label.Source: §3.2 · p. 6
  • 133 tasks carry multiple capability tags, 235 declare at least two expected-action entries, and 164 impose explicit ordering constraints.The nine domains carry 36 capability tags; categories overlap and describe declared task requirements.Source: Figure 3 · p. 6, §3.2 · p. 7

4 Results and Analysis

Across sixteen systems and 555 audited tasks, the results compare strict success, execution trajectories, operational costs, and policy-constrained performance. The analyses show that aggregate success alone does not capture domain variation, resource trade-offs, or whether systems complete all required components.

4.1 Outcome: Task Success and Where It Concentrates

  • 83.60% for Gemini 3.8 Flash leads strict success, while Nemotron 3.5 Lightning 30B and GPT-4.1 Nano tie at 64.32%.The eight top-ranked models fall within a 2.88-point band; Figure 4 ranks the sixteen runs and annotates estimated tested-agent costs.Source: §4.1 · p. 7, Figure 4 · p. 8
  • Three-round results show 434/555 tasks succeed in every round for Gemini 3.8 Flash, versus its mean success rate of 85.53%.Its task outcomes also vary between executions, including 30 switches between rounds 2 and 3.Source: §4.1 · p. 7
  • Customer support averages 56.35% strict success across models, while onboarding averages 90.24%; domain leaders also differ from the overall leader.Gemini 3.8 Flash leads retail, while Muse Glimmer 30B leads customer support despite ranking 12th overall.Source: §4.1 · p. 8, Figure 5 · p. 9

4.2 Trajectory: A Valid Process, Not Just a Valid Answer

  • Response-only grading overestimates strict success by 10.5 to 11.0 percentage points for the four systems shown in Figure 6.Adding the action check alone removes 45 to 50 of 555 tasks; action mismatch is the largest non-success category across all sixteen runs.Source: §4.2 · pp. 8–9, Figure 6 · p. 10

4.3 Operational Viability: Cost and Latency

  • Gemini 3.8 Flash combines 83.60% strict success with the lowest mean token count, 24.0k per task, while latency and tool-call leaders differ.Nemotron 3.5 Lightning 30B has the lowest mean latency at 5.86 seconds, and tool calls range from 2.09 to 7.68 per task; these are system-level descriptives.Source: §4.3 · pp. 9–10, Figure 7 · p. 11
  • Among priced runs, estimated tested-agent cost spans 89.3×, from $2.14 for GPT-4.1 Nano to $191.00 for Claude Opus 5.Gemini 3.8 Flash scores 83.60% at $16.12, the fifth-cheapest priced run; three self-hosted runs have no metered per-token price.Source: §4.3 · p. 10

4.4 Policy-Constrained Tasks: Compliance, Completion, and Exposure

  • On policy-constrained trials, 91.2% pass all applicable negative checks, but only 66.8% pass the full contract.Of 876 strict failures on these tasks, 73.5% pass the negative checks but fail another required component.Source: §4.4 · pp. 10–11

5 Related Work

Prior benchmarks cover pieces of agent execution, from tool selection and stateful interaction to enterprise and computer-use tasks. CRM-Agent-Bench is distinguished by grading structured-tool execution through per-task contracts that jointly capture required behavior and prohibited effects.

Related benchmarks

  • Prior work spans tool selection, mutable-database dialogue, stateful environments, enterprise workflows, computer use, and safety, but each benchmark covers only a subset of delegated execution demands.CRM-Agent-Bench addresses the complementary structured-tool setting, removing visual grounding as a confound so workflow reasoning, routing, and capability exposure can be measured directly.Source: §5 · p. 12

CRM-Agent-Bench’s distinction

  • CRM-Agent-Bench grades executed state changes and policy compliance through per-task conjunctive contracts, rather than relying primarily on retrieval, answer correctness, or a single success signal.Its contracts jointly gate required actions, ordering, resulting state, and prohibited effects.Source: §5 · pp. 12–13

Evaluation challenges

  • Tool-selection and safety evaluations can conflate a forbidden action’s absence with model refusal, failure to act, or routing that never exposed the tool.Trajectory evaluators can also disagree, while simulated-user behavior adds a measurement variable examined by separate simulator studies.Source: §5 · pp. 12–13

6 Conclusion

CRM-Agent-Bench evaluates enterprise tasks with contracts that jointly measure declared outcomes, trajectories, and policy requirements, alongside operational viability. Its results support considering cost, domain reliability, and policy-constrained completion alongside strict accuracy, while the described setting prioritizes controlled, reproducible headless workflows.

6 Conclusion

  • Across sixteen systems, strict success ranges from 64.32% to 83.60%, while systems with similar accuracy diverge sharply in cost, per-domain reliability, and difficulty scaling.The authors support weighing execution cost, per-domain reliability, and policy-constrained completion alongside strict accuracy when selecting enterprise SaaS agents.Source: §6 · p. 13

Appendix

  • The benchmark prioritizes controlled, reproducible evaluation of headless, structured-tool workflows, with further repetitions, expert validation, and document-rich, visual, voice, and live-service workflows identified as extensions.These extensions are intended to further test robustness, measurement sensitivity, and external validity while preserving executable, auditable grading.Source: Appendix · p. 19

A.1 Formal Problem Definition and Evidence Boundary

  • Each task specification parameterizes a finite-horizon POMDP whose latent state includes mutable CRM state, private instruction, interaction history, and simulator and executor state.The agent sees dialogue, tool results, and exposed instructions and schemas, but not raw CRM state or the private instruction; later user turns depend on preceding visible replies.Source: §A.1 · p. 20
  • An orchestration condition determines an admissible tool mask from visible history, and the reference scaffold obtains that mask through a model-based per-turn router.Changing the router or exposed catalog changes access policy, not task initialization or contract; no intermediate reward is exposed to the agent.Source: §A.1 · p. 20

A Evaluation Protocol and Benchmark Construction Details

  • The evaluation protocol separates the task contract, on-policy user, graph-routed agent, reset CRM environment, and conjunctive grader, with the appendix detailing the frozen specification, access policy, evidence, checks, and readouts.The appendix also documents execution runs, audited-set construction, versioned releases, set composition, the reference scaffold, exposure analysis, and domain examples.Source: A Evaluation Protocol and Benchmark… · p. 20, Figure 9 · p. 21

A.2 Executable Contracts and Grading Semantics

  • Each task declares applicable checks across actions, arguments, ordering, semantic response, final state, and forbidden actions, arguments, or state; success requires every declared check to pass.Ordering can use subsequence or exact filtered-sequence matching, state checks support sparse paths and selected value constraints, and forbidden checks reject prohibited attempts or protected-state changes.Source: §A.2 · p. 22
  • Tool traces record attempts and outcomes separately from persistent state changes, while semantic judges assess only user-visible messages and cannot override structured action, ordering, or state failures.The audited set has state assertions in 200 of 555 tasks, including protected-state assertions in 45; task counts overlap because contracts can combine requirements.Source: §A.2 · pp. 22–23

A.3 Frozen Evaluation Protocol and Reporting

The appendix documents the frozen evaluation protocol, task construction and extension workflow, reference routing scaffold, and audited evaluation set. Across 16 runs, it reports full-suite and diagnostic results by failure type, domain, and difficulty.

A.3 Frozen Evaluation Protocol and Reporting

  • The frozen protocol evaluates all 16 runs on the same 555-task set; each received valid grades on all tasks, and model-attributable failures remain in the denominator.Latency and cache-neutral estimated cost cover the tested agent model, excluding simulator and judge usage; the leaderboard is a one-trial snapshot.Source: §A.3 · p. 24

A.4 Task Construction, Provenance, and Validation

  • Tasks from two construction lineages—original CRM applications and adapted workflow recipes—were admitted when their requirements could be represented in the portable harness.Table 4 labels domains Seed, Recipe, or Hybrid; these describe construction lineage, not evaluation splits or required architectures.Source: §A.4 · pp. 24–25, Table 4 · p. 25

A.5 Extension Interface and Release Workflow

  • The extension workflow supports new task records, agent graph configurations, and registered tool environments without requiring a new runner or grader.Canonical-release admission includes specification, evidence binding, fresh-world validation, gold-path execution, and release freezing.Source: §A.5 · pp. 25–26

A.6 Evaluation-Set Composition and Coverage

  • At each turn, a classifier routes the utterance and dialogue context to one capability node or refusal; selected nodes receive only assigned tools, and all 206 forbidden-action declarations across 107 tasks matched paired-graph tools, with zero unmatched.Source: §A.7 · pp. 26, 29

B.1 Full-Suite Leaderboard

B.2 Failure Composition

  • Action mismatch is the largest non-success category for every run, ranging from 54 tasks for Gemini 3.8 Flash to 157 for Nemotron Super 3 120B.Total non-success counts range from 91 for Gemini 3.8 Flash to 198 for Nemotron 3.5 Lightning 30B and GPT-4.1 Nano.Source: §B.2 · p. 32

B.4 Success by Domain

  • All 16 systems score lower on customer support than any other domain on average: 56.35%, versus 90.24% for onboarding.Within-model gaps between best and worst domain range from 23.3 points for Muse Glimmer 30B to 53.7 points for GPT-4.1 Nano.Source: §B.4 · p. 33

B.5 Success by Difficulty Stratum

  • Strict success declines monotonically from easy to hard for all systems; the easy-to-hard drop ranges from 5.37 points for Claude Sonnet 5 to 27.39 for GPT-4.1 Nano.Gemini 3.8 Flash leads all three strata, while Claude Sonnet 5 ranks second on hard tasks at 79.05%.Source: §B.5 · pp. 33–34

B.6 Per-Metric Standing

Headline ties in strict success do not imply equivalent performance across other metrics or costs.

B.6 Per-Metric Standing

  • Three runs tie at 81.26% strict success, yet the 13 priced runs span a 69.4× range in cache-neutral cost per success, from $0.00599 for GPT-4.1 Nano to $0.416 for Claude Opus 5.Runs sharing headline strict-success values also separate on domain macro and easy- and hard-task success.Source: Figure 15 · p. 34

B.7 Metric-Dependent Ranking

System rankings change across the six higher-is-better performance metrics, and no system leads every axis.

B.7 Metric-Dependent Ranking

  • Gemini 3.8 Flash leads five of six metrics but ranks third on customer support, where Muse Glimmer 30B ranks first despite placing twelfth on scored success.Claude Sonnet 5 rises from joint-fifth overall to second on hard tasks; Figure 16 shows the rank changes across metrics.Source: Figure 16 · p. 35

B.8 Policy-Constrained Tasks: Compliance versus Completion

Policy-constrained tasks separate negative-check compliance from full workflow completion. Every model has lower strict success on those tasks than on tasks without explicit negative assertions.

B.8 Policy-Constrained Tasks: Compliance versus Completion

  • Negative-check compliance counts policy-constrained tasks where all applicable negative checks held, regardless of other contract components.The reported rates do not isolate intrinsic model safety or establish a causal safety-and-utility trade-off.Source: §B.8 · pp. 35–36
  • Every model’s strict success is 8.8–20.0 points lower on policy-constrained tasks, while negative-check compliance ranges from 84.8% to 95.2%.Nemotron Super 3 120B has the highest observed negative-check compliance, 95.2%, but ranks 14th overall.Source: §B.8 · p. 35, Table 8 · p. 36

C.1 Repeated Evaluation and Aggregation

Three evaluation rounds preserve the leading systems’ ordering overall, while close systems and individual task outcomes still vary. The section reports how rounds are aggregated and compares ranking stability across runs.

C.1 Repeated Evaluation and Aggregation

  • All three rounds use the same 555 task contracts and strict-success definition, with fresh environments; round 1 is the formal result used in the main leaderboard.The reported table compares all 16 systems across the three rounds.Source: §C.1 · p. 36, Table 9 · p. 37
  • Three-round means give each round equal weight, and sample standard deviations describe variation across executions in percentage points.The mean is computed from round-level rates rather than pooling attempts across rounds.Source: §C.1 · p. 36

C.2 Aggregate and Ranking Stability

  • 85.53% mean strict success (SD 1.66 points) puts Gemini 3.8 Flash first across all three rounds; Claude Opus 5 averages 83.29% (SD 0.62 points).Gemini’s mean lead over Opus is 2.23 points.Source: §C.2 · p. 37, Table 9 · p. 37
  • 0.922, 0.869, and 0.934 are the pairwise Spearman rank correlations for rounds 1–2, 1–3, and 2–3, although nearby systems change positions.GPT-5.5, Sonnet 4.6, Sol, Terra, Inkling, and Muse move relative to nearby systems across rounds.Source: §C.2 · p. 37

C.3 Consistency on Individual Tasks

Three-run results show that stable aggregate scores can conceal task-level changes, while selected traces illustrate how contracts distinguish successful recovery, state failure, and non-convergence. The appendix also explains how to read and reconstruct those trace demonstrations.

C.3 Consistency on Individual Tasks

  • 434/555 tasks pass in all three Gemini rounds (78.20%), below its 85.53% mean strict-success rate, and 30 task outcomes switch between rounds 2 and 3.For Opus, 407/554 jointly scored tasks pass all three rounds (73.47%).Source: §C.3 · pp. 37–38

What is a “demo” in this appendix?

  • A trace demo connects a task’s executable contract, runtime route and tool observations, visible assistant behavior, and task-declared grader decisions.The appendix presents three diagnostic traces: strict success, graded failure, and timeout.Source: What is a “demo” in this appendix? · p. 38

Demo A at a glance

  • A rejected $1,000 transfer is recovered by validating and executing the user-corrected $50 amount, satisfying the declared checks and strict success.The contract forbids executing the rejected amount.Source: Demo A at a glance · p. 39

Demo B at a glance

  • A repeated checkout raises the loyalty balance from the expected 155 to 310; the state check catches the duplicate although action, order, and response checks pass.The semantic judge also missed an intermediate denial of the already recorded tool execution.Source: Demo B at a glance · p. 40

Demo C at a glance

  • After retrieving the delivered order status, the agent continues an unnecessary authentication loop and times out without reporting the result.The simulator introduced a PIN absent from the task specification, and authentication calls repeatedly returned 403 errors.Source: Demo C at a glance · pp. 40–41

D.4 Trace Artifact Mapping and Provenance

  • The appendix selects traces across strict outcomes and evaluator behaviors; a task ID joins each run artifact to its released contract for reconstruction.The artifacts preserve the full per-record trace, including dialogue, tool observations, declared checks, and grader fields.Source: §D.4 · p. 41

E Construct Validity and Benchmark Scope

The benchmark’s scope combines multi-step tool execution, stateful business workflows, and on-policy interaction, with task contracts grading only declared requirements. Its domain atlas illustrates how journeys pass through tools, policy gates, state transitions, and evaluation.

E Construct Validity and Benchmark Scope

  • CRM-Agent-Bench evaluates task-declared execution requirements over multi-step interactions with mutable state, combining dependent tool use, business workflows, and on-policy interaction.Its full-suite score measures executable-contract satisfaction, while a separate table reports constituent-check coverage.Source: E Construct Validity and Benchmark Scope · p. 41

E.1 Long-Horizon, Dependency-Laden Execution

  • 235 of 555 tasks declare at least two expected actions, and 164 impose explicit ordering constraints on dependent tool-call chains.Examples include validate_transfer → execute_transfer and process_payment → send_receipt → award_loyalty_points → log_transaction; an early error can fail the whole task.Source: §E.1 · pp. 41–42

E.2 Stateful Business Workflows

  • Workflows span nine domains and 36 capability tags, including transfers, refunds, case escalation, scheduling, guarded deletion, onboarding, and campaign creation.Synthetic, portable environments support controlled operations without a live CRM or vendor backend.Source: §E.2 · p. 42

E.3 Probabilistic, On-Policy Dynamics

  • Deterministic business rules meet variable on-policy dialogue: each simulated user turn depends on prior agent replies, so the same task may be solved or missed across trials.The paper reports multi-run means, standard deviations, ranking stability, and all-three-success rates to characterize run-to-run robustness.Source: §E.3 · p. 42

F Per-Domain System Flow Atlas

  • Figures 19–27 illustrate representative domain journeys through interaction, orchestration, capability, environment, policy gates, and contract evaluation.Action, argument, and order checks assess trajectory; response and state assess outcome; forbidden checks assess policy compliance, while operational viability is reported separately.Source: F Per-Domain System Flow Atlas · pp. 42–43, Figure 20 · p. 45
Loading 2610.32004v1…