PaperlayerView exact revision on arXiv

Source-linked AI summary

PersuaRL: Reinforcement Learning-Driven Multi-Expert Selection for Persuasive Dialogue Generation in Insurance

Rohan Kirti, Akash Ghosh, Aryan Vats, Niladri Ghosh, Shipra Shriparn, Roshni Ramnani, Anutosh Maitra, Sriparna Saha

arXiv:2609.01188v1cs.CL

TL;DR

Persuasive insurance dialogue demands context-sensitive capabilities beyond factual generation, but existing approaches provide limited adaptive expert coordination. The paper introduces InsureDial and PersuaRL, which uses reward-guided reinforcement learning to select and coordinate experts with alternating selector–generator training. PersuaRL consistently outperforms baselines across evaluated settings and produces fluent, persuasive, emotionally aligned responses.

  • Problem

    Persuasive insurance agents must handle intent, emotion, policy benefits, and constructive decision guidance, while existing tool-use and dialogue approaches provide limited flexible coordination.

  • Method

    PersuaRL uses a reinforcement-learning selector to coordinate task-specific experts, with alternating optimization between selector updates and generator fine-tuning.

  • Results

    PersuaRL consistently outperforms strong baselines across single-shot, supervised fine-tuning, and reinforcement-learning settings, including larger models.

  • Takeaways & Limitations

    InsureDial and PersuaRL support modular, goal-driven persuasive dialogue generation across LLM backbones and show robust generalization beyond insurance.

  • Takeaways & Limitations

    InsureDial relies on GPT-4o-generated dialogues filtered by humans, which may introduce synthetic artifacts and incompletely represent real user behavior.

Abstract

from arXiv · show

Large Language Models (LLMs) are revolutionizing digital communication by powering conversational agents deployed across domains such as customer service, digital sales, and insurance. These agents, built on LLMs, can understand user input, retrieve relevant information, and generate coherent responses. However, while they excel at factual communication, they often lack the ability to engage in truly persuasive, context-sensitive dialogue, especially in domains like insurance, where trust and clarity are critical. Building on this need within the insurance domain, our work focuses on improving the persuasiveness of digital agents, aka LLMs. To support this, we introduce InsureDial, a Persuasive Insurance Dialogue dataset, designed to capture the nuances of persuasive communication specific to motor insurance interactions. We introduce PersuaRL, a reinforcement learning-based framework that equips LLM-driven dialogue agents with the ability to adaptively explore, select, and coordinate strategies across multiple expert modules, guided by the evolving dialogue context, to achieve more effective persuasion. We conduct extensive automatic human and qualitative evaluations on two benchmark persuasion dialogue datasets, including our InsureDial. Our evaluations consistently demonstrate that PersuaRL outperforms baseline, generating contextually appropriate and highly persuasive responses.

1 Introduction

Persuasive insurance dialogue requires context-sensitive coordination beyond factual generation, motivating a learnable multi-expert framework and a domain-specific benchmark. PersuaRL combines adaptive expert selection, reward design, and alternating optimization, with evaluations spanning in-domain and out-of-domain settings.

  • Motivation: Insurance persuasion requires agents to infer intent, adapt to emotional tone, highlight policy benefits, and guide decisions constructively.
  • Research gap: Prior tool-augmented methods use rigid or task-limited tool invocation, while persuasive dialogue systems often rely on static prompting or heuristic coordination.
  • Framework: PersuaRL treats expert coordination as a context-conditioned decision problem and dynamically coordinates task-specific experts for persuasive, context-aware responses.
  • Benchmark: InsureDial is a motor-insurance dialogue dataset annotated for intent, sentiment, key terms, and engagement strategy through a human-in-the-loop and LLM generation pipeline.
  • Reward design: The framework uses domain-specific rewards covering strategy alignment, intent consistency, contextual coherence, response diversity, and judge-based assessment without intermediate supervision.
  • Evaluation: Evaluations on insurance and tourism benchmarks combine automated, human, and qualitative analyses and report strong in-domain performance with robust out-of-domain generalization.

3 Development of InsureDial Dataset

InsureDial is a human-refined motor-insurance dialogue resource built around realistic terminology, multi-turn scenarios, and four aligned annotations. PersuaRL then uses these contexts with modular experts, a selector, and an alternating selector–generator training process.

  • Dataset construction: InsureDial contains 1,931 multi-turn conversations and more than 26,000 utterances covering policy quotes, coverage details, price inquiries, and strategic persuasion.
  • Dataset construction: The dataset uses insurance websites to capture terminology and interaction flows involving discounts, value-added services, coverage types, and add-ons.
  • Dataset construction: Seed dialogues were authored by trained annotators, designed across persuasive intents, and validated by domain experts for factual accuracy and persuasive quality.
  • Dataset construction: GPT-4o generated candidate dialogues from tested prompts, which expert annotators manually rated for persuasive quality.
  • Dataset annotation: Annotations cover persuasion strategy, key domain terms, user intent, and sentiment to support context-aware persuasive response generation.
  • Model architecture: The selector activates a subset of task-specific experts, whose outputs are fused with dialogue context before the generator produces the response.
  • Training: Alternating optimization updates the selector with GRPO using sampled-selection rewards, then fine-tunes the generator on the highest-reward expert-augmented input.

5 Experimental Results and Analysis

The experiments evaluate PersuaRL against open-weight, proprietary, and supervised baselines across insurance and out-of-domain dialogue settings. Assessment combines automatic metrics with human judgments of fluency, engagement, persuasion, strategy, and resistance handling.

  • Dataset: InsureDial is curated through LLM generation with human refinement across diverse motor-insurance scenarios and multi-turn user–agent contexts.
  • Baselines: Baselines include closed-source and open-weight models, an SFT baseline, and PersuaRL instantiated on Llama, Phi, Qwen, and Mistral backbones.
  • Baselines: The study compares single-shot, SFT, and PersuaRL regimes to assess performance across model families and training approaches.
  • Evaluation: Automatic evaluation uses ROUGE-1, BLEU-2, METEOR, BERT-F1, DISTINCT-2, and LLM-as-a-Judge metrics.
  • Evaluation: Human evaluation measures fluency, engagingness, persuasive effectiveness, strategy appropriateness, and resistance handling on a 5-point scale.

6 Results and Findings

PersuaRL outperforms the evaluated baselines on InsureDial, learns adaptive reward-guided expert routing, and transfers strongly to the out-of-domain DEAL dataset. Ablations indicate that jointly using all experts is best, with Engagement and Intent most influential.

  • InsureDial results: BF1 ≈0.771, B2 ≈0.398, and R1 ≈0.631 are achieved by Llama 3.2–3B PersuaRL, surpassing much larger vanilla baselines.The reported comparison includes roughly 16% higher BERT-F1 than Phi-3 Medium 14B and over 29% higher than Qwen3 32B for PersuaRL Phi-3 Mini.
  • Expert routing: PersuaRL surpasses prompt-based routing by learning expert combinations through GRPO and reward-driven credit assignment.
  • Transferability: Across backbone configurations, PersuaRL shows consistent and monotonic BF1 and R1 improvements on the out-of-domain DEAL dataset.

7 Ablation Studies

The ablations show that PersuaRL’s adaptive expert selection improves persuasive dialogue quality over indiscriminate activation and prompt-based routing. Full expert coordination and reinforcement-learning-based selection produce stronger alignment, relevance, fluency, and persuasion.

  • Expert-module ablation: Removing any expert consistently degrades performance, with Engagement and Intent modules producing the largest drops in semantic alignment and relevance.The full PersuaRL model performs best overall, while Keyterm and Sentiment have comparatively smaller individual effects.
  • All tools versus PersuaRL: PersuaRL outperforms the AllExpert baseline across all metrics for both Phi and Llama-3B variants.The strongest gains occur in semantic relevance and lexical overlap, especially for ROUGE-1 and BERT-F1.
  • Prompting versus PersuaRL: PersuaRL outperforms prompt-based routing across Phi and Llama-3B, improving fluency, relevance, and overall generation quality without reducing output diversity.The comparison attributes these gains to preference-aligned expert coordination rather than generic routing.
  • Human evaluation: Human evaluation finds PersuaRL highest across all five dimensions for Llama 3.2 3B, Qwen 2.5 3B, and Mistral 24B.The evaluation used domain-expert annotators on 30% of the randomly sampled InsureDial test set.
  • Qualitative analysis: Qualitative examples show PersuaRL combining empathetic, persuasive reassurance with coverage details more effectively than single-shot and SFT responses.The comparison highlights stronger user-centric framing and contextual reassurance in PersuaRL outputs.

11 Limitations

The paper identifies limitations in dataset realism, scalability, deployment cost, evaluation setting, and model-size coverage. It also describes safeguards intended to support ethical and factually accurate persuasion.

  • Dataset: InsureDial’s semi-automated GPT-4o generation and human filtering may introduce synthetic artifacts and incompletely represent real user behavior.
  • Scalability: The binary expert-selection mask creates an exponentially growing action space as the number of experts increases, limiting scalability.
  • Deployment: Multiple expert modules increase inference latency, computational cost, and system complexity relative to single-model baselines.
  • Evaluation: Offline evaluation uses gold dialogue histories, leaving interactive rollouts and real-user outcomes unassessed.
  • Model scale: Applying the full framework with large-scale LLMs as both selector and generator was infeasible under available computational resources.
  • Ethics: The work frames persuasion as decision support that preserves user autonomy and emphasizes factual accuracy in high-stakes settings.

5. Why are some closely related persuasion and routing baselines not included?

The paper positions its comparisons around strong, directly comparable baselines while distinguishing domain-specific persuasion systems and reporting computational trade-offs. It interprets transfer beyond insurance as a test of expert coordination under domain shift, not universal generalization.

  • Baseline selection: The primary comparisons target single-shot, supervised fine-tuning, and heuristic-routing baselines under identical backbones.
  • Baseline selection: Some intent-to-strategy and persona-aware models require additional supervision or assumptions that are not directly comparable to expert selection.
  • Training comparison: PersuaRL applies reinforcement learning to expert-selection actions while updating the generator through supervised fine-tuning on ground-truth responses.
  • Training comparison: Frozen reward models and indirect reward influence reduce the opportunity for the generator to exploit reward artifacts or circularity.
  • Robustness: Ablations show graceful degradation after removing individual experts, although noisy expert outputs are not explicitly simulated.
  • Deployment trade-off: PersuaRL takes 1.4 times more inference time than SFT baselines, reflecting a trade-off between computation and persuasion quality.
  • Out-of-domain evaluation: DEAL was selected because its multi-turn negotiation setting includes resistance handling, objective trade-offs, and strategic adaptation relevant to persuasion dynamics.
  • Out-of-domain evaluation: The out-of-domain study tests whether expert coordination remains beneficial under domain shift rather than claiming universal persuasion generalization.

A Appendix

The appendix documents dataset construction, experimental regimes, baselines, tables, and the specialized experts used by PersuaRL. It also details how supervised fine-tuning and expert modules support persuasive response generation.

  • Appendix contents: The appendix covers dataset construction, experimental settings, additional results, and the prompts used in the experiments.
  • Dataset annotation: Five trained annotators supported staged InsureDial annotation, beginning with 75 manually labeled dialogues used as gold-standard references.
  • Dataset annotation: Human verification continued during annotation to support high-quality, contextually accurate labels, with an example presented in Table 19.
  • Experimental regimes: Experiments compare single-shot generation, SFT, and PersuaRL, while larger 32B–70B models provide high-capacity reference points.
  • Experimental regimes: Single-shot generation uses the full dialogue context and a simple prompt without task-specific fine-tuning, intermediate reasoning, experts, or reward-based selection.
  • Experimental regimes: SFT trains next-token prediction on InsureDial and improves fluency and domain-specific strategy alignment, but does not dynamically adapt strategies at inference.
  • Expert modules: PersuaRL uses Engagement, Intent, Keyterm, and Sentiment experts, each contributing distinct dialogue-understanding signals to the final response.
  • Expert modules: The Engagement Expert classifies each dialogue context into six persuasion strategies, including logical, credibility, emotional, personal, persona, and neutral appeals.

2. Intent Expert

The framework combines specialized experts and reward components to align generated insurance responses with user intent, persuasion strategy, context, and conversational quality. Evaluation uses automated and human measures across response-level and dialogue-level dimensions.

  • Intent Expert: The Intent Expert classifies the user’s underlying goal to support contextually relevant and strategy-aligned responses.Its listed intents include requesting quotes, asking coverage details, expressing concerns, requesting information, confirming interest, and asking about price.
  • Keyterm Expert: The Keyterm Expert extracts domain-specific insurance terms so the generator can produce precise, informative responses.Examples include depreciation, roadside assistance, and personal accident coverage.
  • Sentiment Expert: The Sentiment Expert classifies messages as Positive, Neutral, or Negative to support empathy, reassurance, and trust-building.Its output enables sentiment-aware response generation in multi-turn dialogues.
  • Reward Components: BERT-based classifiers provide engagement-strategy and intent-consistency rewards, with reported InsureDial accuracies of 82.1% and 84.9%, respectively.The rewards align generated responses with predicted persuasion strategies and user intents.
  • Reward Components: The reward design also scores contextual appropriateness, non-repetitiveness, and high-level persuasive quality while penalizing inefficient or imbalanced expert selection.Contextual appropriateness weights the current utterance twice and normalizes the weighted similarity score.

D Additional Experiments

Additional experiments examine reward weighting, reward-component ablations, tool-selection strategies, and training dynamics. Across these analyses, combined rewards and reward-driven selection are associated with the strongest reported performance.

  • Reward Weighting: Including all reward components yields the best perplexity score among the tested reward-weight combinations.Weights were selected using a 15% InsureDial holdout set.
  • Reward Ablation: PersuaRL achieves B2 = 0.375, BF1 = 0.760, D2 = 0.991, and R1 = 0.609 in the Qwen reward-component ablation.The combined reward signal is reported as the most effective configuration.
  • Reward Ablation: R1, R2, and R5 are identified as the most impactful reward components, while R3 and R4 provide complementary contextual-alignment and diversity benefits.The analysis links the combined structure to coherent, persuasive, and non-repetitive responses.
  • Tool Selection: PersuaRL reaches BLEU-2 0.398, BERT-F1 0.771, and ROUGE-1 0.631 with Llama-3B, outperforming AllExpert and Prompting Tools.The reported advantage is attributed to activating contextually relevant modules per turn.
  • Base-Model Training: Training the base models with GRPO and the same reward functions consistently outperforms the single-shot setting across evaluated metrics.The result supports reward-driven optimization beyond the standard single-shot baseline.
  • Training Dynamics: Llama achieves the highest final training reward, Qwen converges stably, and Phi converges faster but to a lower reward.The curves reflect differences in model capacity and learning stability.

D.3.4 Ablation study w.r.t Generator

Generator adaptation is a central component of PersuaRL’s ablation results. Untrained generators and indiscriminate expert activation underperform, while fine-tuning improves the use of expert signals and can yield gains even for untrained base models.

  • Generator Ablation: Removing generator fine-tuning causes substantial performance drops across all metrics, approaching the single-shot setting.The authors identify the generator step in alternating optimization as critical to PersuaRL’s gains.
  • All-Expert with Untrained Generator: Providing expert signals to an untrained generator is insufficient for coherent, persuasive, domain-aligned responses.The untrained generator cannot effectively interpret and integrate structured expert outputs.
  • All-Expert with Trained Generator: Fine-tuning an all-expert generator improves performance, but uniformly activating every expert remains lower than learned selective routing.Indiscriminate activation can introduce redundant or conflicting signals.
  • Untrained Small Models: Untrained models inside the full PersuaRL pipeline achieve BERT-F1 scores of 0.621 for Llama 3.2 3B and 0.628 for Llama 3.1 8B.The authors report only marginal gains from scaling in this untrained regime and identify domain adaptation through fine-tuning as the critical bottleneck.
  • Human Evaluation: Human evaluation shows trained all-expert generators outperform untrained variants, including Llama 3.2 3B Resistance Handling rising from 3.06 to 3.21.Strategy Appropriateness also rises from 2.57 to 2.76 in the cited comparison.

Prompt for Keyterm Expert

The Keyterm Expert prompt instructs the model to identify motor-insurance terms in the latest utterance and justify their relevance. It supports both standard coverage terminology and vehicle-specific terms.

  • Prompt Definition: The Keyterm Expert identifies important motor-insurance-related terms mentioned explicitly or implicitly in the user’s latest utterance.The prompt frames keyterm extraction as the expert’s central task.
  • Prompt Definition: The prompt includes coverage, liability, roadside assistance, depreciation, deductibles, renewal, personal accident cover, and IDV as example keyterms.It also allows user- or vehicle-specific terms such as Tesla Model 3, EV, and 2024 vehicle.
  • Output Requirements: For each extracted keyterm, the prompt requires a brief justification explaining its insurance relevance.The required output format is an extracted keyterm followed by its justification.
  • Prompt Example: The prompt’s example extracts “Policy premium” and “2024 Tesla Model 3” from a pricing question.The justification connects the terms to vehicle-specific cost estimation.

Prompt for Sentiment Expert

The prompt defines two expert tasks: generate persuasive motor-insurance dialogues under strict structural and strategy constraints, and classify emotional tone using only sentiment-focused evidence.

  • Sentiment classification: The sentiment expert classifies input into Positive, Negative, or Neutral and provides a concise explanation based only on emotional tone.The output must contain only the sentiment label and supporting explanation.
  • Sentiment classification: The sentiment analysis must focus on emotional tone, word choice, and sentiment-laden phrases without summarizing or inferring intent.The supplied example labels “way too expensive” as Negative because it conveys frustration and disappointment.
  • Dialogue generation: The dialogue generator creates multi-turn conversations between a motor-insurance customer and representative, with the user initiating and turns alternating strictly.Dialogues must contain 6–15 turns, remain coherent, conclude naturally, and use brief, crisp, informative responses.
  • Dialogue generation: Each conversation must use at least three persuasion strategies, while every agent utterance uses exactly one strategy without naming it.Allowed strategies include logical, emotional, credibility, persona-based, personal, and default appeals.
  • Dialogue generation: Generated dialogues must implicitly reflect user attributes and remain within the specified motor-insurance domain, vehicle requirements, premium range, and sentiment scope.The user’s age group, occupation, driving frequency, budget sensitivity, risk attitude, and family orientation should be inferred rather than explicitly stated or queried.
  • Dialogue generation: Motor-insurance knowledge is background context only and must not be reproduced verbatim or used to make unrealistic guarantees.The prompt also identifies the material as a dataset-generation prompt and frames the agent as an expert conversational generator.
Loading 2609.01188v1…