PaperlayerView exact revision on arXiv

Source-linked AI summary

SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

P N Bhargav Teja, Velpula Lokesh Reddy, Madhu Basumata, Susen Sarkar, Rajasekhara Reddy Duvvuru Muni

arXiv:2610.12975v1cs.AIcs.CL

TL;DR

Existing dialogue benchmarks largely assume cooperative users, leaving resistant buyers in multi-turn sales conversations comparatively understudied. SmartSalesBench evaluates LLM sales agents against psychologically grounded resistant personas through controlled experiments and a tiered metric framework. Across strategy and length manipulations, flat role-inversion rates indicate stable persona character maintenance.

  • Problem

    Existing benchmarks assume cooperative users and measure task completion, while resistant multi-turn sales conversations and resistance as an evaluation dimension remain comparatively understudied.

    Source: §I · p. 1

  • Method

    SmartSalesBench evaluates sales agents against resistant buyer personas using controlled experiments and a three-tier framework of primary, secondary, and exploratory metrics.

    Source: §I · p. 1

  • Results

    Across strategy and length manipulations, flat RIR (p = 0.999) indicates stable persona character maintenance.

    Source: B. Four-Experiment Comparison · p. 6

  • Takeaways & Limitations

    The benchmark treats resistance as a measurable signal of simulation quality and uses controlled strategy conditions to distinguish agent effectiveness from simulation artifacts.

    Source: §I · p. 1

    Three local Ollama judges reduce dependence on one scorer but do not eliminate correlated local-model bias.

    Source: §VIII · p. 8

Abstract

from arXiv · show

We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas. Existing task-oriented dialogue benchmarks assume cooperative users and measure task completion; SmartSalesBench inverts this assumption, placing agents against personas programmed to resist and challenge across conversations of up to twenty turns. The benchmark includes five agent profiles, twelve buyer personas, fifteen smartphone products across six price tiers, and four controlled experiments with 3 iterations totaling 13,680 sessions. Both sales-agent and buyer persona generations are grounded through retrieval-augmented generation (RAG). We introduce nine evaluation metrics grouped into three tiers: primary outcome metrics (CR, BRS, RIR), secondary quality metrics (SMS, PFS, ETI), and exploratory metrics planned for future reporting (ITS, ARI, SPI). Our evaluation combines automated LLM-as-judge scoring with internal human assessment (n = 50, \k{appa} = 0.71, Pearson r = 0.81 with BRS).

I. INTRODUCTION

SmartSalesBench addresses adversarial, multi-turn sales dialogue, where agents must adapt to evolving resistance, a setting existing cooperative task-oriented benchmarks do not evaluate. It treats resistance as a measurable signal and combines controlled experiments with a tiered evaluation framework.

I. INTRODUCTION

  • Adversarial sales dialogue is comparatively understudied because agents must adapt persuasion to evolving resistance, emotional state, and brand loyalty across turns.Success requires more than task completion: the agent faces a resistant interlocutor and a concrete commercial outcome.Source: §I · p. 1

A. Task-Oriented Dialogue Benchmarks

  • Existing task-oriented benchmarks assume cooperative users, while SmartSalesBench makes buyer resistance a continuously measured evaluation dimension.MultiWOZ and Schema-Guided Dialogue measure task completion; SalesLLM [3] uses scripted configurations without a resistance-quality metric, RetailSim [4] focuses on economic regularities, and PersonaGym studies static question-answering.Source: §I · p. 1, A. Task-Oriented Dialogue Benchmarks · p. 2

I. INTRODUCTION

  • Resistance is a signal of simulation quality, purchase outcome alone is insufficient, and controlled ablations help distinguish agent effectiveness from simulation artifacts.A session nearing conversion with a highly resistant customer can be more informative than an easy conversion, although binary conversion metrics treat them alike.Source: §I · p. 1
  • SmartSalesBench covers 5 agent profiles, 12 buyer personas, 15 product configurations, and 4 controlled experiments totaling 13,680 sessions.The benchmark is grounded in Samsung smartphone retail and Indian retail objection patterns.Source: §I · pp. 1–2
  • The three-tier framework separates primary outcomes CR, BRS, and RIR from secondary quality metrics and exploratory metrics reserved for future full reporting.The framework also uses three local Ollama judges and independent human validation.Source: §I · p. 2
  • The four controlled experiments compare baseline, escalation, extended-turn, and short-turn conditions to assess sales-strategy variables.The authors describe this as the first direct comparison of sales strategy variables in a multi-agent LLM benchmark.Source: §I · p. 2
  • Generator screening identifies persona-holding failure modes and motivates selecting Llama3.1-8B for controlled experiments, alongside analysis of agent–persona tactic gaps.Source: §I · p. 2
  • A sample-size convergence analysis reports that stable estimates are achievable with fewer sessions than the full sample, informing future replication.Source: §I · p. 2

B. Persona Simulation and Fidelity

SmartSalesBench extends persona evaluation into sustained commercial pressure and goal-asymmetric dialogue. It specifies sales-agent profiles and resistant buyer personas, with RAG grounding applied to both sides of the conversation.

B. Persona Simulation and Fidelity

  • Persona evaluation extends to adversarial commercial pressure of up to twenty turns, where maintaining resistance requires strategic coherence beyond stylistic consistency.PersonaGym evaluates persona fidelity in static tasks, while PersonaChat, CharacterEval [8], and RoleInteract [9] study neutral or cooperative settings.Source: B. Persona Simulation and Fidelity · p. 2

C. Sales, Persuasion, and Negotiation

  • SmartSalesBench adds continuous resistance scoring, a four-experiment controlled ablation, and less than 8% RIR through conversation-structure design without specialized training.SalesLLM [3] reports reducing role inversion from 17.44% to 8.8%.Source: C. Sales, Persuasion, and Negotiation · p. 2

D. Multi-Agent Social Simulation

  • The benchmark adapts social-simulation evaluation to strictly goal-asymmetric settings, with agent and customer objectives explicitly opposed throughout.SOTOPIA evaluates social intelligence across seven dimensions; SOTOPIA-π adds interactive learning.Source: D. Multi-Agent Social Simulation · p. 2

A. Agent Profiles

  • Five sales-agent profiles specify a primary tactic, authority level, discount budget, and phase-based prompt schedule to prevent repetitive tactics across turns.The schedule covers discovery, pitch, objection handling, soft close, and final offer; lifestyle identity framing links Samsung ownership to aspirational identity rather than technical features.Source: A. Agent Profiles · p. 2

B. Buyer Persona Design

  • Twelve buyer personas specify resistance type, hard budget ceiling, core fear, persuasion arc, and English and Hinglish soft-yes phrases; hard-no phrases trigger early exit.Source: B. Buyer Persona Design · p. 2
  • The five-stage LangGraph architecture applies RAG grounding to both sales-agent and buyer-persona turns, then evaluates sessions with local judges and an independent human check.The evaluation layer also includes session logging and statistical analysis.Source: Figure 1 · p. 3

C. Product Catalogue and RAG Grounding

SmartSalesBench grounds its products and both buyer and agent generations with retrieval, while its simulation uses explicit purchase detection and turn-based control.

C. Product Catalogue and RAG Grounding

  • Fifteen Samsung models span six price tiers, and phonematching maps budget and stated needs to an opening-pitch tier while supporting downgrades after explicit price objections.Prices come from one canonical JSON file.Source: C. Product Catalogue and RAG Grounding · p. 3
  • Both buyer and agent generations use RAG; each buyer retrieves two relevant objection facts, while agent generation uses retrieved product and dialogue context.A RAG-ablation experiment is planned to quantify the grounding layer’s contribution.Source: C. Product Catalogue and RAG Grounding · pp. 3–4

D. Simulation Engine and Purchase Detection

  • Simulations include a manager escalation after turn five when mean intent across three turns is at most 4.0; purchase detection requires an explicit phrase or timely payment-link affirmation.Intent-score threshold detection is excluded to avoid spurious conversion-rate inflation.Source: D. Simulation Engine and Purchase… · p. 4

IV. EVALUATION FRAMEWORK

The framework separates benchmark outcomes from quality and exploratory measures, and defines conversion and buyer resistance as primary outcomes.

A. Metric Organisation

  • Primary metrics underpin benchmark results and cross-experiment comparisons, secondary metrics aid interpretation, and exploratory metrics are reserved for future full reporting.The framework groups nine metrics into these three tiers.Source: A. Metric Organisation · p. 4

1) Conversion Rate (CR):

  • Conversion Rate (CR) is the percentage of sessions ending in a purchase, counted only when an explicit purchase agreement occurs rather than inferred intent.Agreements include a direct purchase phrase or positive payment-link affirmation.Source: 1) Conversion Rate (CR) · p. 4

2) Buyer Resistance Score (BRS):

  • Buyer Resistance Score (BRS) averages resistance ratings across customer turns and five dimensions; BRS ≥ 6.5 indicates a realistic empirical challenge, while BRS < 4.0 signals critical persona-integrity breakdown.The dimensions are price, specification, emotional, character, and objection.Source: 2) Buyer Resistance Score (BRS) · p. 4

3) Role Inversion Rate (RIR):

The framework measures customer role inversion and defines Sales Motivation Score as a composite of resistance and intent dynamics across a session.

3) Role Inversion Rate (RIR):

  • Role Inversion Rate (RIR) measures customer turns that improperly assist the agent, such as repeating agent language approvingly or capitulating without narrative justification.The measure treats these behaviors as narrative collapse.Source: 3) Role Inversion Rate (RIR) · p. 4

1) Sales Motivation Score (SMS):

  • SMS is a weighted linear combination of Resistance Decay Function (PDF), Resistance Elasticity (RES), Tipping Point Detection (TPD), and BRS; the cited passages do not specify the component weights.The decay function models latent intent over continuous session time using exponential decay, with its session-specific constant fitted via ordinary least squares. Elasticity is the first derivative of intent in response to tactic switches, and the tipping point is the first turn at which intent surpasses 6.0.Source: 1) Sales Motivation Score (SMS) · pp. 4–5
  • The current SMS weights achieved rank-order correlation r = 0.81 with human quality judgments, versus r = 0.74 for equal weighting.All schemes use the same component scores; only aggregation weights differ.Source: 1) Sales Motivation Score (SMS) · pp. 4–5, Table IV · p. 5

2) Persona Fidelity Score (PFS) and Emotional Trajectory Index (ETI):

Persona fidelity and emotional trajectory capture distinct aspects of buyer behavior, alongside human validation and controlled experiments with fixed conditions.

2) Persona Fidelity Score (PFS) and Emotional Trajectory Index (ETI):

  • Persona Fidelity Score (PFS) measures linguistic and behavioral stationarity, while ETI tracks valence volatility, trajectory, and dominant valence; high BRS can coexist with low PFS.Gemma2-2B’s PFS collapse (3.97) is consistent with high ETI volatility; Phi3.5-mini raw PFS/ETI outputs remain excluded pending transcript verification.Source: 2) Persona Fidelity Score (PFS) and… · p. 5

D. Multi-Judge and Human Evaluation Protocol

  • Three internal annotators rated 50 EXP-1 sessions; human character-stability ratings correlated with BRS at r = 0.81, but external retail-sales experts are identified as stronger future validation.The reported agreement was κ = 0.71 for character stability and κ = 0.68 for agent realism; tables retain the validated Llama3.2:3B pass pending full three-judge re-scoring.Source: D. Multi-Judge and Human Evaluation… · p. 5

A. Controlled Experiment Design

  • Each of four experiments manipulates one variable while holding others identical; EXP-2 has 2,280 sessions, exceeding its empirical convergence point of n = 130.The design is presented as enabling direct causal attribution of differences.Source: A. Controlled Experiment Design · p. 5, Table V · p. 5

B. Multi-LLM Comparison

The validated model comparison selects Llama3.1-8B for controlled experiments based on multi-metric simulation quality. The paper reports local generation and independent judging, alongside baseline intent trajectories by purchase outcome.

B. Multi-LLM Comparison

  • Llama3.1-8B was selected for EXP-1–EXP-4 for its balance of resistance preservation, persona fidelity, and low role inversion, rather than conversion rate alone.Qwen2.5-3B had elevated role inversion, Gemma2-2B borderline inversion and weaker resistance, and Phi3.5-mini unrealistically high conversion and the highest inversion.Source: B. Multi-LLM Comparison · p. 5

C. Implementation Details

  • Simulations used four local generator models, while three independent local LLM judges scored sessions and rule-based conversion rates came directly from logs.The 13,680 sessions were logged and aggregated for analysis.Source: C. Implementation Details · pp. 5–6

A. EXP-1 Baseline (n = 3,600)

B. Four-Experiment Comparison

Across the four experiments, conversion differed significantly while BRS remained stable; both tested models showed a sharp conversion drop under the five-turn condition. Generator-model comparisons also favored Llama3.1-8B for stability across session volumes.

B. Four-Experiment Comparison

  • Across both models, EXP-4 (5 turns) had the lowest CR: 10.1% for Llama3.1-8B and 16.4% for Phi 3.5-mini-3.8B.The highest CR was 25.4% for Llama3.1-8B in EXP-2 and 72.6% for Phi 3.5-mini-3.8B in EXP-1.Source: Figure 3 · p. 6, Figure 4 · p. 6
  • Across models with different score distributions, EXP-1–EXP-3 remained comparatively stable while EXP-4 sharply reduced conversion and increased SMS; BRS stayed relatively stable.RIR was also flat across strategy and length manipulations (p = 0.999).Source: B. Four-Experiment Comparison · p. 6
  • CR differences across experiments were significant (χ2 = 68.74, df = 3, p < 0.001; Cramér’s V = 0.18), while BRS stayed within 7.51–6.72.The authors interpret stable BRS as evidence that conversion differences reflect agent effectiveness rather than simulation artifacts.Source: Table VII · p. 6, Figure 3 · p. 6, Figure 4 · p. 6

C. Multi-Model Comparison

  • Across four session-volume tiers, Llama3.1-8B had the lowest CR drift and the most consistent BRS, SMS, and fidelity profile.At 900 sessions, its CR was 26.2%, versus 39.2% for Gemma2-2B and 70.7% for Phi3.5-mini; the reduced tiers also supported its selection.Source: Table IX · p. 7, C. Multi-Model Comparison · p. 7
  • Llama3.1-8B had the smallest variation in CR (0.4 pp), BRS (0.05), and fidelity (0.05), while Qwen2.5-3B had the largest CR variation (3.4 pp).The reported behavioral pattern remained similar across all four models.Source: Table X · p. 7, C. Multi-Model Comparison · p. 7

D. Agent × Persona Interaction

Agent–persona outcomes varied substantially: Dinesh (Business Owner) appeared in every bottom-five pairing. The statistical tests also report a negative association between SMS and purchase outcome.

D. Agent × Persona Interaction

E. Statistical Tests and Effect Sizes

F. Sample-Size Convergence

Conversion-rate estimates stabilized with roughly 150–230 sessions per model in multi-LLM comparisons. The findings also distinguish conversion, conversation quality, and character maintenance across turn limits and persona pairings.

F. Sample-Size Convergence

  • In multi-LLM comparisons, CR stabilized within n = 148–228 sessions per model; EXP-4 stabilized at n = 109, versus n = 130–193 for the other controlled experiments.The reported range suggests approximately 150–230 sessions per model for stable multi-LLM CR estimates.Source: Figure 7 · p. 8, F. Sample-Size Convergence · p. 8

A. The Tech Advisor × Apple Loyal Persona Finding

  • Sanjay × Priya had the highest CR at 51.8%, followed by Neha × Priya at 51.3%; Dinesh appeared in all five lowest pairings, with CR from 16.8% to 21.7%.The authors suggest ecosystem-related objections were comparatively responsive, while productivity-downtime concerns remained difficult.Source: A. The Tech Advisor × Apple Loyal… · p. 8

B. SMS as a Complementary Metric to CR

  • EXP-4’s five-turn limit coincided with the highest SMS (6.43) and lowest CR (16.1%), while EXP-3 reached 42.6% CR with moderate SMS (5.70).The authors say SMS rewards sustained, high-quality persuasion arcs, so short conversations can earn fewer turn-level quality contributions.Source: B. SMS as a Complementary Metric to CR · p. 8
  • SMS provides a quality signal distinct from purchase outcome: hard-persona sessions can score well without converting, while quick conversions may score lower.The authors describe this divergence as especially relevant under EXP-4’s turn limit.Source: B. SMS as a Complementary Metric to CR · p. 8

C. Flat RIR as a Benchmark Validity Property

  • RIR remained within a narrow range across the four conditions: 0.44%–0.59%, supporting the interpretation that CR differences reflect agent effectiveness rather than simulation instability.The reported range spans less than 0.15 percentage points.Source: C. Flat RIR as a Benchmark Validity… · p. 8

D. Generator Selection and Model Capability

SmartSalesBench measures buyer resistance across multi-turn sales-agent evaluation, with Llama3.1-8B selected for controlled experiments based on a verified multi-metric balance. Its conclusions remain bounded by evaluator bias, generator coverage, language and market scope, and preliminary fine-tuning evidence.

D. Generator Selection and Model Capability

  • Llama3.1-8B was selected for controlled experiments because it provides the strongest verified balance across resistance, persona fidelity, and role-inversion behavior.Source: D. Generator Selection and Model… · p. 8

VIII. LIMITATIONS AND FUTURE WORK

  • Three local Ollama judges reduce dependence on a single scorer but do not eliminate correlated local-model bias; generator comparison covers only four locally hosted LLMs.Source: §VIII · pp. 8–9
  • The benchmark is Samsung-specific and English/Hinglish, leaving generalization to other product categories, languages, or markets open; four-prompt LoRA results are too small for definitive claims.Source: §VIII · p. 9

IX. CONCLUSION

  • Across 13,680 simulations, SmartSalesBench measures buyer resistance continuously, organizes nine metrics into three tiers, and reports tactic-gap and sample-size-convergence analyses.Source: §IX · p. 9

Related papers

Loading 2610.12975v1…