PaperlayerView exact revision on arXiv

Source-linked AI summary

CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments

Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, Chien-Sheng Wu

arXiv:2411.02305v2cs.CLcs.AI

TL;DR

Professional CRM agents lack benchmarks that faithfully represent complex, privacy-constrained work environments. CRMArena builds an expert-validated Salesforce-based benchmark with realistic tasks and data, and finds that even state-of-the-art agents achieve limited success. The benchmark exposes a gap between current LLM capabilities and real-world CRM requirements.

  • Problem

    Existing benchmarks do not adequately capture the complexity of real-world CRM tasks and interconnected data, partly because enterprise data raises privacy concerns.

    Source: §1 · p. 1

  • Method

    CRMArena combines an expert-validated Salesforce-based sandbox, interconnected CRM objects, latent variables, and nine tasks spanning three customer-service personas.

    Source: §1 · p. 1, §2.2 · p. 4

  • Results

    State-of-the-art agents complete less than 40% of tasks with general-purpose tools and up to 55% with manually designed tools.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CRMArena highlights the gap between current LLM capabilities and the requirements of realistic CRM work environments.

    Source: §5 · p. 9

    The benchmark covers three CRM personas and does not include other common roles such as sales representatives.

    Source: §7 · p. 10

Abstract

from arXiv · show

Customer Relationship Management (CRM) systems are vital for modern enterprises, providing a foundation for managing customer interactions and data. Integrating AI agents into CRM systems can automate routine processes and enhance personalized service. However, deploying and evaluating these agents is challenging due to the lack of realistic benchmarks that reflect the complexity of real-world CRM tasks. To address this issue, we introduce CRMArena, a novel benchmark designed to evaluate AI agents on realistic tasks grounded in professional work environments. Following guidance from CRM experts and industry best practices, we designed CRMArena with nine customer service tasks distributed across three personas: service agent, analyst, and manager. The benchmark includes 16 commonly used industrial objects (e.g., account, order, knowledge article, case) with high interconnectivity, along with latent variables (e.g., complaint habits, policy violations) to simulate realistic data distributions. Experimental results reveal that state-of-the-art LLM agents succeed in less than 40% of the tasks with ReAct prompting, and less than 55% even with function-calling abilities. Our findings highlight the need for enhanced agent capabilities in function-calling and rule-following to be deployed in real-world work environments. CRMArena is an open challenge to the community: systems that can reliably complete tasks showcase direct business value in a popular work environment.

1 Introduction

CRMArena addresses the lack of realistic benchmarks for professional CRM tasks by combining expert-validated Salesforce-based data, interconnected objects, latent variables, and representative customer-service tasks. Experiments show that current LLM agents complete fewer than 40% of tasks with general-purpose tools and up to 55% with manually designed tools.

1 Introduction

  • CRMArena fills a benchmark gap by modeling realistic CRM complexity, including interconnected objects and work tasks that prior benchmarks simplify or omit.Prior benchmarks often use simple object dependencies and straightforward tasks such as webpage navigation or list filtering.Source: §1 · p. 1, Table 1 · p. 3
  • The benchmark covers nine customer-service tasks across Service Manager, Service Agent, and Service Analyst personas.These personas represent strategic resource allocation, customer inquiries, and service-performance analysis.Source: §1 · p. 2
  • The benchmark generates Salesforce Service Cloud data with rigorous validation and expert assessment, with 90% of domain experts rating the environment Realistic or better.The generated organization is uploaded to Salesforce and supports interaction through both the user interface and API.Source: §1 · p. 2, Figure 1 · p. 2
  • Top LLM agents complete less than 40% of tasks with general-purpose tools but solve up to 55% when manually designed tools are incorporated.Weaker LLMs often do not benefit from manually crafted tools because of weaker function-calling capabilities.Source: §1 · p. 2
  • CRMArena introduces a realistic CRM agent benchmark whose tasks are validated by domain experts for real-world business scenarios.The benchmark is designed to evaluate LLM agents in professional CRM environments.Source: §1 · pp. 1–2
  • Its data-generation strategy uses a real-world CRM schema, latent variables, deduplication, and rigorous validation to promote diversity and quality.The contribution explicitly combines these mechanisms to construct realistic benchmark data.Source: §1 · pp. 1–2

2 CRMArena

CRMArena benchmarks realistic CRM work through expert-validated tasks, interconnected Salesforce-shaped data, latent variables, and a reproducible query and tool pipeline. Its sandbox was judged realistic by most domain experts, while preserving a simplified setting for current agent capabilities.

2.1 Tasks

  • CRMArena covers nine expert-validated CRM tasks across Service Manager, Service Agent, and Service Analyst responsibilities.Source: §2 · p. 3, §2.1 · p. 3

2.2 Sandbox Environment

  • The sandbox models 16 business objects with Salesforce-based connectivity, including links among CASE, ACCOUNT, CONTACT, and USER.Source: §2.2 · p. 4
  • Latent variables such as SHOPPINGHABIT and SKILL generate purchasing patterns and agent-transfer scenarios needed for realistic task cases.Source: §2.2 · p. 4, Figure 3 · p. 5
  • The data pipeline uses mini-batch generation, deduplication, schema verification, and task-focused content verification to improve data quality and diversity.Source: §2.2 · pp. 4–5

2.3 Query Instance Generation

  • CRMArena generates query instances through seed construction, ground-truth computation, Salesforce ID mapping, and LLM paraphrasing.Source: §2.3 · pp. 5–6
  • Non-answerable false-presupposition queries comprise 30% of instances in each of five specified tasks, and the benchmark totals 1,170 queries.Source: §2.3 · p. 6

2.4 Tools: APIs and Functions

  • The evaluation provides general-purpose SOQL/SOSL access alongside 27 manually defined Python wrapper functions for task-specific operations.Source: §2.4 · p. 6

2.5 Expert Study

  • In an expert study, 90% rated the populated Org Realistic or Very Realistic, while more than 77% gave similarly high ratings to individual objects.Source: Figure 4 · p. 6, §2.5 · p. 7

3 Benchmarking Experiments

CRMArena evaluates diverse LLM agents on realistic CRM tasks through three agentic frameworks, revealing persistent performance, tool-use, cost, and consistency challenges.

3.1 Experimental Settings

  • The evaluation covers proprietary, open-source, and reasoning models under Act, ReAct, and Function Calling frameworks.The tested models include GPT, Claude, Llama, o1, and deepseek-r1; Llama models use prompt mode because native function calling performed poorly.Source: §3.1 · p. 7

3.2 Results

  • 57.7% overall with ReAct and 64.3% with human-crafted functions show that realistic CRM tasks remain difficult for top LLM agents.The benchmark’s ALL score averages performance across tasks, using F1 for KQA and exact match for other tasks.Source: §3.2 · p. 7, Table 2 · p. 8
  • Open-source Llama models sometimes match or exceed GPT and Claude, with greater error-recovery scope from execution feedback.This pattern appears across all three settings and is reported from Figure 6.Source: §3.2 · p. 8

3.3 Discussions

  • gpt-4o is the most cost-effective non-reasoning choice, performing best in two-thirds of frameworks with the lowest cost and fewest turns under function calling.The cost comparison averages agents across queries and tasks.Source: §3.3 · p. 8, Table 3 · p. 8
  • Human-crafted functions improve gpt-4o but can hurt weaker models, especially when functions require calculations or dependent multi-step calls.Functions are categorized by query versus calculation capability and dependent versus independent execution; weaker models may struggle to use them effectively.Source: §3.3 · pp. 8–9, Table 4 · p. 9
  • Across ten trials, pass^k declines at nearly the same rate for all three frameworks, indicating similar consistency and unreliable task completion.The consistency evaluation excludes KQA because its reward is not binary.Source: §3.3 · p. 9

4 Related Work

Related benchmarks evaluate web or work-oriented agents, but CRMArena’s context is situated among broader agent benchmarks and professional task datasets.

Agent Benchmark

  • Agent benchmarks commonly evaluate web navigation, everyday website actions, safety, or general agent behavior.Examples include web agents operating in ecommerce and social-discussion scenarios.Source: §4 · p. 9

Work-oriented Datasets

  • Work-oriented datasets assess business text generation, simple database tasks, user-authorized interactions, or visually mediated web work.The cited datasets include the CRM Benchmark, WorkBench, τ-Bench, and WorkArena.Source: §4 · p. 9

5 Conclusion

CRMArena introduces an expert-validated benchmark for realistic CRM tasks with intricate data interconnections, showing that current LLM agents still struggle in professional environments.

5 Conclusion

  • CRMArena combines expert-validated CRM tasks with intricate data interconnections to evaluate LLM agents in professional work environments.The benchmark is presented as a foundational step toward more sophisticated realistic-agent evaluations.Source: §5 · p. 9
  • Even state-of-the-art LLMs achieve limited success on realistic tasks despite function-calling capabilities.The conclusion frames this result as a gap between current LLM capabilities and real-world CRM requirements.Source: §5 · p. 9

6 Ethical Considerations

CRMArena uses synthetic data modeled on real-world CRM structures and tasks, raising ethical considerations around bias and privacy. Manual inspections found no observed bias patterns or personally identifiable information.

  • The benchmark is modeled after real-world CRM data structures and tasks, so ethical considerations include data bias and privacy.Source: §6 · p. 10
  • Synthetic CRM data may inherit biases from the real-world data used to train its generating models, although manual inspection found no such patterns.Source: §6 · p. 10
  • The benchmark uses no real customer data, but its CRM-like structure and sensitive synthetic information still motivate privacy review.Manual inspection verified the absence of personally identifiable information and private-information inference risks.Source: §6 · p. 10

7 Limitations

CRMArena covers nine tasks for service managers, agents, and analysts, while leaving other CRM personas outside its current scope. Further analyses examine turn distributions and non-answerable queries, and the appendix documents seed queries.

7 Limitations

  • CRMArena evaluates nine tasks across service manager, service agent, and service analyst roles, but excludes other common CRM personas such as sales representatives.The authors plan to incorporate additional roles in future studies.Source: §7 · p. 10

A Further Discussions

  • Figure 6 reports the distribution of turns required for agents to successfully complete user queries under ReAct.Source: A Further Discussions · p. 12
  • Function calling benefits stronger LLMs on non-answerable queries, while weaker models such as claude-3-sonnet and gpt-4o perform worse with it.Overall, agents handle non-answerable queries better than standard queries in the reported analysis.Source: A Further Discussions · p. 12

B Query Generation Details

C Action Space Details

Text-based agents act through SOSL, SOQL, and answer submission, whereas function-calling agents use carefully designed functions. The sandbox models densely connected CRM objects and tracks ReAct turn distributions.

C Action Space Details

  • Text-based settings use SOSL queries, SOQL queries, and answer submission, while function-calling settings use a list of designed functions.Source: C Action Space Details · p. 12
  • Figure 6 shows how many turns ReAct agents take to reach the goal.Source: Figure 6 · p. 12

D Sandbox Environment Details

  • Sandbox objects are densely connected, except Knowledge__kav, reflecting complex real-world work environments.The object-entry counts are reported in Table 8, and the data-generation flow in Figure 8.Source: D Sandbox Environment Details · p. 12

D.1 Object Details

The appendix defines CRMArena’s Salesforce-oriented objects, relationships, queries, functions, prompts, and expert-study materials. These details document the benchmark’s data schema and interaction setup.

D.1 Object Details

  • Product and pricing objects represent products, categories, category associations, price books, pricebook entries, orders, and order items.Source: §D.1 · pp. 12–13
  • Tables 6–9 document seed queries, available functions, object-entry counts, and query-generation examples.Source: Table 6 · p. 13, Table 7 · p. 13, Table 8 · p. 13
  • Knowledge articles provide documentation or information accessible to users or customers.Source: §D.1 · p. 13
  • The object appendix defines CRM entities including contacts, issues, accounts, users, cases, and CaseHistory, with CaseHistory recording changes and updates made to cases over time.Source: §D.1 · p. 13
  • Figure 7 depicts the objects and their dependencies in the sandbox environment.Source: Figure 7 · p. 14

E Prompts

  • Tables 10–12 provide system prompts for Act, ReAct, and function-calling settings.Source: E Prompts · p. 14

F Expert Study Details

F.1 Recruitment Criteria

The expert study recruited Salesforce users from customer-service, technical-support, and account-management roles, requiring frequent CRM use.

  • Participants were recruited from customer-service, technical-support, and account-management or analyst job titles.Source: §F.1 · pp. 14–15
  • Eligible candidates had to report using Salesforce CRM several times a day in the screener survey.Source: §F.1 · p. 15

F.2 The study

The expert study combined organizational familiarization, timed CRMArena task completion, and realism ratings with written rationales. Qualitative feedback assessed both the sandbox organization and its Salesforce objects.

F.2 The study

  • The study comprised five minutes of familiarization, 45 minutes to complete five query instances, and ten minutes of quality rating.Source: §F.2 · p. 15
  • Participants first reviewed task-relevant objects in a sandbox Org, then completed CRMArena tasks using the provided Salesforce interface.Source: §F.2 · p. 15
  • Participants rated organizational and object realism using ordered categories from Very Unrealistic to Very Realistic, with an unfamiliar-object option.Source: §F.2 · pp. 15–16
  • The study instructed participants to make one tool call at a time and submit only the final answer through the submit tool.Source: §F.2 · p. 18
  • Case routing prioritized issue expertise, then product expertise, and finally the agent with the fewest non-Closed cases.Source: §F.2 · p. 18

F.3 Qualitative Feedback

  • Expert feedback described familiar Salesforce navigation, realistic case-management elements, and authentic-looking structures, while one critique cited sparse customer data and empty fields.Source: §F.3 · p. 18

G Implementation Details

The implementation used multiple hosted language-model APIs and fixed experimental settings, while the expert-study materials included example queries, prompts, ratings, and qualitative feedback.

Loading 2411.02305v2…