PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-Salesman: Towards Reliable Large Language Model Driven Telemarketing

Qingyu Zhang, Chunlei Xin, Xuanang Chen, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, Qing Ye, Qianlong Xie, Xingxing Wang

arXiv:2511.12133v1cs.CL

TL;DR

Goal-driven telemarketing challenges LLMs to plan persuasive multi-turn dialogues while remaining faithful to product constraints. The paper introduces TeleSalesCorpus and AI-Salesman, combining reasoning-aware training with dynamic script-guided inference and a sales-focused evaluation framework. The authors report that AI-Salesman significantly outperforms baselines in automated and human evaluations of persuasive, business-compliant dialogue.

  • Problem

    Telemarketing requires strategic multi-turn planning, factual faithfulness, and customer-specific responses, while LLMs struggle with these demands.

    Source: §1 · p. 1

  • Method

    The paper constructs TeleSalesCorpus and proposes AI-Salesman, combining Bayesian-supervised reinforcement learning, DOGA’s dynamic script guidance, and a fine-grained LLM-as-a-Judge evaluation framework.

    Source: §1 · pp. 1–2

  • Results

    AI-Salesman significantly outperforms baseline models in automated and human evaluations of persuasive, business-compliant dialogue.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors present AI-Salesman as an approach for generating persuasive, business-compliant dialogue in professional telemarketing scenarios.

    Source: §5 · pp. 7–8

    DOGA’s precise template injection can sound formulaic, and static prompts perform marginally better on role-playing and idle-chat rejection; static rules are more efficient for simple tasks.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Goal-driven persuasive dialogue, exemplified by applications like telemarketing, requires sophisticated multi-turn planning and strict factual faithfulness, which remains a significant challenge for even state-of-the-art Large Language Models (LLMs). A lack of task-specific data often limits previous works, and direct LLM application suffers from strategic brittleness and factual hallucination. In this paper, we first construct and release TeleSalesCorpus, the first real-world-grounded dialogue dataset for this domain. We then propose AI-Salesman, a novel framework featuring a dual-stage architecture. For the training stage, we design a Bayesian-supervised reinforcement learning algorithm that learns robust sales strategies from noisy dialogues. For the inference stage, we introduce the Dynamic Outline-Guided Agent (DOGA), which leverages a pre-built script library to provide dynamic, turn-by-turn strategic guidance. Moreover, we design a comprehensive evaluation framework that combines fine-grained metrics for key sales skills with the LLM-as-a-Judge paradigm. Experimental results demonstrate that our proposed AI-Salesman significantly outperforms baseline models in both automatic metrics and comprehensive human evaluations, showcasing its effectiveness in complex persuasive scenarios.

1 Introduction

The introduction frames telemarketing as a goal-driven persuasive dialogue problem that challenges LLMs’ planning, factual faithfulness, and customization. It presents AI-Salesman, TeleSalesCorpus, and a sales-focused evaluation framework as responses to these challenges.

1 Introduction

  • Telemarketing requires long-horizon strategic planning, factual faithfulness to product constraints, and tailored responses to customer needs, areas where LLMs struggle.The introduction describes risks of misleading claims and generic responses, alongside difficulty handling complex marketing strategies and objections.Source: §1 · p. 1
  • AI-Salesman combines Bayesian-supervised rewards for coherent, strategically useful reasoning with DOGA’s dynamic, turn-specific guidance from a pre-verified script library.DOGA uses customer profiles, real-time intent, and dialogue history to retrieve relevant strategies.Source: §1 · pp. 1–2
  • TeleSalesCorpus provides high-fidelity, real-world-grounded sales dialogues, while the evaluation framework assesses six capabilities using seven-metric rubrics and LLM-as-a-Judge.The framework is presented as a scalable offline alternative to resource-intensive online A/B tests.Source: §1 · p. 2
  • The paper presents AI-Salesman as the first LLM-based framework specifically designed for real-world telemarketing to address satisfaction, faithfulness, and customization.It also describes TeleSalesCorpus as the first large-scale, high-fidelity dataset grounded in real-world sales conversations for training and evaluation.Source: §1 · p. 2
  • The proposed offline evaluation framework covers six core sales capabilities for assessing practical proficiency across diverse scenarios.Its stated purpose is efficient and rigorous assessment.Source: §1 · p. 2

2 Telemarketing Scenarios

This section formalizes telemarketing dialogue as constrained, goal-oriented generation and describes an evaluation framework for sales capabilities and response quality.

2.1 Task Definition

  • At each turn, the model generates a response from the system prompt and dialogue history, maximizing conditional probability while meeting constraints and pursuing task goals.The prompt specifies global goals and constraints; responses must satisfy every predefined rule.Source: §2.1 · p. 2

2.2 Evaluation Framework

  • The evaluation framework combines six sales capabilities with seven qualitative metrics and uses GPT-4 to assign holistic dialogue-quality scores from 1–10.The capabilities span the sales-call lifecycle, while the metrics assess each response.Source: §2.2 · p. 3

3 End-to-End Intelligent Sales System

This section describes TeleSalesCorpus construction and AI-Salesman’s training and inference components for goal-oriented telemarketing. Training uses GRPO guided by a Bayesian reasoning reward, while DOGA assembles turn-specific prompts from a script library.

3.1 Data Construction

  • 2,000 high-fidelity conversations were produced from real-dialogue assets and diverse LLM-authored scenarios through a state-aware, three-agent simulation and quality assurance.The Dialogue Manager adjudicates conversational state and retrieves strategically relevant real-world examples to guide the Sales Agent.Source: §3.1 · p. 3

3.2 Stage-1: GRPO Training

  • GRPO explores sales strategies from noisy data, guided by a Bayesian reward that scores intermediate reasoning for logical soundness and factual grounding.The training objective is to produce responses that are persuasive and factually accurate.Source: §3.2 · p. 3

Reward Function Design

  • The total reward is a weighted sum of four components evaluating generated sequences against a ground-truth reference.The weights are hyperparameters.Source: Reward Function Design · p. 3

Core Reward

  • The Bayesian-Supervised Reasoning reward maximizes the joint probability of the reasoning chain and reference answer, decomposing into reasoning fluency and utility.The paper defines the reward as the log-joint probability, with prior and likelihood terms estimated by the model.Source: Core Reward · pp. 3–4

Auxiliary Reward

  • Auxiliary rewards promote format adherence, reference-aligned output length, and semantic similarity to the reference answer.Format adherence checks the required output schema; semantic similarity uses sentence embeddings and a baseline-normalized score.Source: Auxiliary Reward · p. 4

3.3 Stage-2: Inference With Dynamic Prompt

  • DOGA retrieves turn-specific guidance from a pre-built library of sales scripts and templates, combining it with the system prompt and dialogue history.The offline library distills strategies from successful historical dialogues and indexes them by dialogue intent.Source: §3.3 · p. 4, Online Stage: Real-time Dialogue… · pp. 4–5

4 Experiments

Experiments compare AI-Salesman with baseline and SFT variants, assess its reward components and DOGA, and test scaling and human preferences. Direct reinforcement learning without SFT yields the strongest reported overall mean score.

4.1 Experimental Setup

4.3 Ablation Studies

  • DOGA improved Business Analysis by 4.9%, Objection Handling by 11.1%, and Operational Guidance by 14.7% against a static prompt.The authors attribute the gains to dynamic use of expert templates for contextual guidance.Source: Figure 4 · p. 6, §4.3 · p. 7

4.4 Scalability Analysis

  • The 32B model peaked at 7.17, while scaling to 72B led to a slight performance drop.Scaling from 7B to 14B yielded only a minor improvement.Source: Figure 4 · p. 6, §4.4 · p. 7

4.2 Main Results

  • SFT improved Business Analysis from 6.49 to 6.78 and Objection Handling from 4.77 to 5.24; GRPO with SFT initialization offered negligible overall-mean gain over SFT alone.With SFT, Role-playing grew minimally from 5.54 to 5.66.Source: §4.2 · p. 6
  • Direct RL optimization of a base model without SFT using the GRPO reward signal raised the overall mean score from the Baseline’s 5.46 to 6.49, a significant 18.9% increase.It achieved high performance across Richness and User Satisfaction while maintaining strong Guideline Adherence.Source: §4.2 · p. 6

4.3 Ablation Studies

  • Removing Rbayes lowered the mean score by 5.2%, while removing Rsem lowered it by 1.5%.The authors describe Rsem as optimizing output quality and Rbayes as evaluating reasoning soundness.Source: Table 3 · p. 7, §4.3 · p. 7
  • The static prompt performed marginally better in Role-playing and Idle-chat Rejection; DOGA could sound formulaic, and direct rules were more efficient for simple tasks.The paper characterizes DOGA as specialized rather than a universal upgrade.Source: §4.3 · p. 7

4.5 Human Evaluation (A/B Test)

  • Human evaluators preferred Ours in 88.5% of matchups against the Baseline and 75.1% against the SFT-only model.The preference ranking aligned with the offline GPT-4-judge results.Source: Table 4 · p. 7, §4.5 · p. 7

5 Conclusion

AI-Salesman combines reinforcement learning with dynamic outline-guided inference, and the paper reports stronger persuasive, business-compliant dialogue than baseline models. Related work describes unresolved challenges in adapting dialogue systems and evaluating sales interactions.

5 Conclusion

  • AI-Salesman combines Bayesian-supervised reinforcement learning for sales strategies with DOGA for flexible, real-time conversation management.Source: §5 · p. 7
  • Extensive automated and human evaluations show AI-Salesman significantly outperforms baseline models in persuasive and business-compliant dialogue.Source: §5 · p. 8
  • The paper presents its methodology and resources as a systematic approach to building more effective and reliable goal-oriented persuasive AI.Source: §5 · p. 8

A Related Work

  • Traditional telephone sales face high labor costs, employee turnover, and conversion-efficiency bottlenecks, motivating conversational AI applications.Source: A Related Work · p. 11
  • Task-oriented dialogue systems handle fixed-goal processes reliably, but their rigidity, domain-extension costs, and vulnerability to unexpected inputs limit suitability for persuasive sales.Source: A Related Work · p. 11
  • Fine-tuning and in-context learning trade domain adaptation against cost, forgetting, context limits, and consistency, while reinforcement learning for persuasive multi-turn strategies remains underexplored.The passage cites (Luo et al. 2025) for fine-tuning risks, (Brown et al. 2020; Liu et al. 2024; Agarwal et al. 2024) for in-context learning, and (OpenAI 2024; DeepSeek-AI et al. 2025; Chen et al. 2025b) for RL applications.Source: A Related Work · p. 11
  • Word-overlap metrics correlate poorly with human judgments, while LLM-as-a-Judge raises concerns about bias, stability, and agreement with human evaluations (Thakur et al. 2025).Source: A Related Work · p. 11

B Detailed Evaluation Framework Components

The evaluation framework assesses six capabilities required for telemarketing interactions and uses qualitative metrics to evaluate model responses.

B Detailed Evaluation Framework Components

B.1 Core Sales Capabilities

  • The framework evaluates six core telemarketing capabilities, spanning role consistency, personalized business analysis, promotion explanation, focus, objection handling, and actionable guidance.The capabilities are Role-playing, Business Analysis, Activity Introduction, Idle-chat Rejection, Objection Handling, and Operational Guidance.Source: §B.1 · p. 11

B.2 Multi-dimensional Evaluation Metrics

Each model response is assessed using seven qualitative metrics intended to provide a fine-grained evaluation across sales capabilities.

B.2 Multi-dimensional Evaluation Metrics

  • The seven response metrics are Guideline Adherence, Factual Correctness, Logical Coherence, User Need Fulfillment, Response Richness, Safety, and Completeness.Source: §B.2 · p. 12

C Detailed Data Construction Methodology

TeleSalesCorpus is constructed through scenario design, state-aware dialogue simulation, and quality checks; the section also describes GRPO's group-normalized advantage formulation.

C.1 Stage 1: Asset Distillation and Scenario Synthesis

  • Real-world sales conversations inform reusable assets, including annotated dialogue-flow chunks indexed by dialogue state and diverse promotional scenarios with generated knowledge bases.The scenarios sample one to three campaigns from a pool of five; GPT-4 authors a knowledge base for each promotion using structured templates.Source: §C.1 · p. 12

C.2 Stage 2: State-Aware Three-Agent Dialogue Simulation

  • A three-agent GPT-4 simulation uses a Dialogue Manager to adjudicate conversation state, retrieve state-matched examples, assemble prompts, and guide Sales Agent responses turn by turn.Source: §C.2 · pp. 12–13

C.3 Stage 3: Quality Assurance and Refinement

  • After expert review and automated filtering, manual review sampled 20 dialogues per promotion type, yielding a final dataset of 2,000 multi-turn conversations.Filtering removed dialogues with fewer than four turns, high consecutive-agent n-gram overlap, unreplaced placeholders, or missing promotion keywords.Source: §C.3 · p. 13

D Detailed Formulation of GRPO

  • GRPO standardizes each rollout's reward within its group to form an advantage, then uses it in a PPO-clipped objective with a KL-divergence penalty.The empirical mean and variance are computed over the group of rollout rewards, and a small constant supports numerical stability.Source: D Detailed Formulation of GRPO · p. 13

E Theoretical Derivation of the Bayesian-Supervised Reasoning Reward

The section applies Bayes’ theorem to reasoning generation, treating a model-generated reasoning chain as a hypothesis evaluated against a reference answer. It defines Rbayes as the log-joint probability used to optimize that chain.

E.1 Bayes’ Theorem: Theoretical Background

  • Bayes’ theorem updates a hypothesis’s probability using evidence, with the posterior determined by the likelihood and prior relative to the evidence’s marginal likelihood.The posterior is proportional to the likelihood times the prior.Source: §E.1 · p. 14

E.2 Applying Bayes’ Theorem to Reasoning Generation

  • The method treats the model-generated reasoning chain as the hypothesis and the reference answer as evidence, seeking the chain with the highest posterior probability.The reference-answer probability is constant across candidate reasoning chains, so it can be omitted when optimizing.Source: §E.2 · p. 14
  • The objective is optimized in log-space to avoid underflow from multiplying many small probabilities while preserving the maximizing solution.Maximizing a positive value is equivalent to maximizing its logarithm.Source: §E.2 · p. 14
  • Rbayes is defined as the log-joint probability of a reasoning chain and reference answer, transforming the maximum a posteriori objective into a computable reward.The language model estimates its components, which are autoregressively decomposed as in Equation 3 of the main text.Source: §E.2 · p. 14

F DOGA Framework Implementation Details

DOGA builds an intent-indexed library of reusable scripts offline, then assembles turn-specific prompts online using dialogue intent and user information.

F.1 Offline Stage: Structured Script Library Construction

  • The offline stage extracts persuasive scripts from high-conversion historical dialogues and clusters and summarizes them into reusable templates indexed by dialogue intent.GPT-4 annotates and extracts scripts; Qwen3-Embedding-0.6B vectors and cosine similarity above 0.8 support clustering before GPT-4 summarizes templates with standardized placeholders.Source: §F.1 · p. 15

F.2 Online Stage: Real-time Dialogue Management Details

  • Online, a constrained intent classifier uses conversation history and prior intents, with finite-state transition rules restricting predictions to valid intents.The classifier is a fine-tuned Qwen2.5-7B model; the transition rules are described as improving accuracy and coherence.Source: §F.2 · p. 15
  • DOGA concatenates a base prompt containing persona and dialogue history with a dynamic component determined by predicted intent and user profile.The dynamic component retrieves intent-matched templates and fills their placeholders with user-specific profile information before passing the prompt to the model.Source: §F.2 · p. 15

G Performance on Real-World Data

On the held-out real-world tele-sales dataset, Rbayes again yields a more stable learning curve and higher final convergence point than the comparison in the ablation study. The appendix also includes examples of agent prompts.

G Performance on Real-World Data

  • Rbayes produces a more stable learning curve and higher final convergence point on noisier, more complex held-out real-world tele-sales data.The passage reports lower absolute reward scores than on synthetic data but says the stabilizing effect persists.Source: G Performance on Real-World Data · p. 15, Figure 6 · p. 16

H Agent Prompts

I Experimental Settings

The section points to a table of hyperparameters and resource use for scalability experiments.

I Experimental Settings

  • Table 6 reports detailed hyperparameters and resource utilization for scalability experiments.Source: Table 6 · p. 18

Related papers

Loading 2511.12133v1…