Profile-Driven User Simulation via Reinforcement Learning with Hierarchical Rewards for Task-Oriented Dialogues

• Task-oriented dialogue systems commonly use modular natural-language understanding, state tracking, policy learning, and generation components.Source: §1 · p. 1
• Financial-service dialogue systems must meet compliance, accuracy, and professional-rigor requirements, while high-quality labeled data is scarce because of expert annotation, regulatory review, and privacy concerns.Source: §1 · p. 1
• PROFILESIM combines explicit user and business profiles with real-voice-data training and hierarchical SFT/GRPO to simulate financial outbound calls.The profiles represent user demographics and Big Five traits alongside business processes and compliance constraints.Source: §1 · pp. 1–2
• Existing vertical-domain simulators lack effective business-process and compliance modeling, while written-text training misses the disfluencies and informality of voice interactions.Source: §1 · p. 1
• The authors report stronger colloquial naturalness, profile adherence, and downstream agent stress than zero-shot LLM baselines, with measurable gains from each component.Source: §1 · p. 2

• Classical task-oriented systems use modular NLU/DST/policy/NLG pipelines, while MultiWOZ and refinements remain primary multi-domain benchmarks.Source: §2 · p. 2
• Prior user simulators progressed from rule- and agenda-based formulations to neural and LLM-based methods, including fine-tuning, verification, and profile-conditioned approaches.Source: §2 · p. 2
• Prompt-based simulators offer diverse utterances but can hallucinate, repeat themselves, or incompletely fulfill goals; later methods use fine-tuning or verification to address these issues.Source: §2 · p. 2
• Existing LLM-based methods target mostly general written-text scenarios; PROFILESIM instead uses explicit demographic-personality and business profiles reverse-engineered from in-domain call logs.Source: §2 · p. 2
• Research on user simulation for outbound marketing remains sparse, with prior work focusing on the system side and giving little attention to user behavior under business-process constraints.Source: §2 · p. 2
• Figure 1 depicts a profile-conditioned simulator and SOP/RAG-constrained service agent interacting to generate closed-loop multi-turn dialogue data.Source: Figure 1 · p. 3

3 Methodology

• ProfileSim uses interacting user-simulator and customer-service agents to generate multi-turn dialogue data for training, evaluation, and cold-start deployment.The simulator generates profile-conditioned user responses while the service agent responds under business-process constraints.Source: §3 · p. 3

3.1 User Simulator Construction

• The simulator pipeline cleans real outbound-call data, constructs profile-conditioned training examples, and applies supervised fine-tuning followed by GRPO.The corpus retained 72,296 valid dialogues from approximately 242,438 sessions after desensitization, transcription correction, and quality filtering.Source: §3.1 · p. 3, §3.1.1 · p. 4

3.1.1 User Profile Extraction

• User profiles combine observable demographics and language style with Big Five traits to control response content and linguistic style.The described objective attributes include gender, age range, occupation, and income status; personality dimensions include Extraversion, Agreeableness, Conscientiousness, Emotional Stability, and Openness to Experience.Source: §3.1.1 · p. 4

3.1.2 Business Profile Schema

• Business profiles are natural-language descriptions reverse-engineered from real dialogue intents, with turn-to-turn intent-group transitions estimated from observed conversations.The intent space contains 73 intents organized into 9 groups; transition dynamics are estimated at the group level from consecutive customer-service dialogue turns.Source: §3.1.2 · pp. 4–5

3.1.3 Supervised Fine-Tuning

• Supervised fine-tuning adapts the simulator to colloquial outbound-call language and conditions response generation on product information, dialogue history, and user and business profiles.The training sample is formalized as a quintuple containing those inputs and the target user response.Source: §3.1.3 · p. 5

3.1.4 GRPO-Based Reinforcement Learning

• GRPO optimizes the simulator using dialogue-level profile consistency and utterance-level coherence, non-repetition, naturalness, and business consistency rewards.Full-dialogue rollouts support profile consistency over cumulative responses; turn-level advantages preserve differences between turns while retaining group-relative normalization.Source: §3.1.4 · pp. 5–7
• During multi-turn rollouts, business profiles are dynamically sampled from empirical intent-group transitions, allowing business focus to change across turns.The customer-service agent remains fixed during reinforcement-learning training and supplies responses within the simulated dialogue.Source: §3.1.4 · pp. 5–6

3.2 Customer Service Agent Construction

• The customer-service agent completes the simulation loop by providing a structured interactive environment for evaluating the user simulator.Its responses are governed by procedural constraints and grounded in retrieved domain knowledge.Source: §3.2 · p. 7, §3.2.1 · p. 7, §3.2.2 · p. 7

3.2.1 SOP-Constrained Dialogue

• The SOP is represented as a directed graph of dialogue stages, restricting the agent to remain at its current stage or move to a legal subsequent stage.This constrains the dialogue to the prescribed business workflow despite diverse user input.Source: §3.2.1 · p. 7

3.2.2 Retrieval-Augmented Generation

• At each turn, RAG retrieves the recognized business intent, candidate next stages, and stage-specific templates, then filters candidates using the current SOP state.The retrieval-and-filter mechanism is intended to balance responses to diverse queries with adherence to business processes.Source: §3.2.2 · p. 7

3.2.3 Structured Input and Output

• The agent receives dialogue history, product information, current stage, SOP-filtered candidates, and retrieved knowledge, then returns JSON with the predicted next stage and response.The structured output supports integration with downstream dialogue-state tracking and execution modules.Source: §3.2.3 · p. 7

4.1 Experimental Settings

• Training uses 72,296 real multi-turn dialogues; manual validation judged 95.4% of sampled dialogue–profile pairs valid, and the simulator receives SFT followed by online GRPO optimization.SFT samples 97,126 utterances; GRPO interacts with a separate customer-service model using 4,000 user profiles.Source: §4.1 · p. 8
• The primary baselines are zero-shot LLMs given the same prompts and inputs, while the simulated customer-service agent uses RAG and SOP-constrained dialogue management.The authors exclude direct UserLM comparison because its general-domain, open-domain evaluation setting differs from Chinese outbound business conversations.Source: §4.1 · pp. 8–9

4.2 Distributional Alignment Analysis

• Full SFT reaches 17.52 PPL and the Full Model reaches 17.49 after GRPO; explicit profile conditioning, especially user profiles, accounts for the larger alignment gains.User-profile-only conditioning yields 22.24 PPL versus 43.04 for business-profile-only conditioning and 58.48 for the SFT model without profiles.Source: §4.2 · pp. 9–10, Table 1 · p. 9

4.3 Generated Dialogue Dataset

• The generated corpus contains 3,000 dialogue sessions, about 20,000 user–agent utterance pairs, and averages seven turns, with 58% of user utterances tied to scenario-specific intents.Business profiles evolve across turns using transition probabilities estimated from real outbound-call data.Source: §4.3 · p. 10

4.4.1 Behavioral realism

• Fidelity assessment covers spoken naturalness and utterance length, persona consistency across turns, and adherence to conditioned business guidance.Colloquial naturalness reuses the GRPO naturalness rubric; profile adherence compares an extracted profile with the target, and business compliance reuses the GRPO business rubric.Source: §4.4.1 · p. 10, §4.4.2 · p. 11

4.4.3 Downstream utility

• Long-tail stress testing uses intents in the bottom 20% of training-corpus frequency and measures response quality from a fixed customer-service agent with its model and RAG index held constant.Lower agent scores indicate that generated user behaviors expose more weaknesses in that agent.Source: §4.4.3 · p. 11

4.4.4 Results and analyses

• 52.25% business-profile compliance is comparable to Full SFT and Qwen3-Max but below GPT-5, while shorter, more implicit phrasing can be less likely to meet explicit semantic criteria.The authors caution that business-profile compliance alone does not capture user-simulation quality.Source: §4.4.4 · pp. 11–12
• 4.97 colloquial score, 0.9255 profile score, and 3.49 agent score make the Full Model best on these dimensions among evaluated generators; its mean utterance length is 11.14 characters.The agent score measures performance on long-tail intent turns, where lower scores indicate greater stress on the fixed agent.Source: §4.4.4 · p. 11, §4.4.3 · p. 11, Table 2 · p. 12

5 Conclusion

• The framework combines supervised fine-tuning on real voice dialogue data with GRPO preference optimization using dialogue- and utterance-level hierarchical rewards.It is paired with a customer service agent constrained by SOP and RAG to produce closed-loop multi-turn dialogues.Source: §5 · p. 12
• ProfileSim jointly models user and business profiles to provide fine-grained control over simulated behavior.The generated dialogue data is designed to support intent-recognition training, agent evaluation, and rapid cold-start deployment.Source: §5 · p. 12
• ProfileSim achieves the highest colloquial naturalness and user profile adherence among evaluated generators, with substantially shorter utterances and the lowest agent score on long-tail intents.The agent score indicates greater challenge for the fixed customer service agent; ablations also report gains from profile conditioning and GRPO.Source: §5 · p. 12

Limitations

• The static user profile does not capture conversational shifts in attitude or emotion, and the framework has been validated only for financial marketing outbound calls.The authors identify cross-domain experiments as necessary to further assess generality.Source: Limitations · p. 12

A.1 Objective Feature Extraction

• Objective extraction covers demographics, work and education, relationships, routines, personality, experiences, and goals using information supported by the user's utterances.The listed fields include gender, age, location, occupation, education, family and social relationships, daily habits, personality traits, other experiences, and goals or plans.Source: §A.1 · p. 16
• The extraction workflow asks for specific, directly relevant attributes and distinctive communication style while prohibiting quotation, paraphrase, or close restatement of user utterances.The output is structured as JSON.Source: §A.1 · p. 16

A.2 Subjective Feature Extraction

• The subjective-profile prompt requires evidence-based scoring and rationale while prohibiting reproduced or closely paraphrased user speech.Only dimensions without clear evidence should be marked uncertain; the output is JSON.Source: §A.2 · p. 16
• Subjective extraction assesses five personality dimensions: extraversion, agreeableness, conscientiousness, emotional stability, and openness.The prompt asks for a high, low, or uncertain score for each dimension, with a concise conclusion and supporting rationale.Source: §A.2 · p. 16

A.3 Natural-language Profile Generation

• The generation prompt asks for coherent, vivid descriptions and discourages vague wording when describing subjective traits.It also prohibits quoting, paraphrasing, or closely reproducing user speech.Source: §A.3 · p. 17
• Generated profiles must omit specific product, financial-domain, and institutional references.The prompt also bars reusing the user's specific wording or speech patterns as examples.Source: §A.3 · p. 17
• Natural-language profile generation converts structured objective and subjective traits into two fluent paragraphs written in the second person.The first paragraph states objective facts and the second describes subjective traits, preserving supplied details without unnecessary elaboration.Source: §A.3 · p. 17

B.1 Contextual Coherence Prompt for Contextual Coherence

• Contextual coherence is scored from 1 to 5 based on consistency with dialogue history, logical response, and continuity of user style.The rubric distinguishes justified attitude changes prompted by the agent from factual contradictions, which should be penalized.Source: §B.1 · p. 17
• The coherence prompt returns a score and a brief evidence-based reason in JSON.The reason should be one or two sentences directly tied to the score.Source: §B.1 · p. 17

B.2 Colloquial Naturalness

• Colloquial naturalness evaluates spontaneous phone-like speech, including rhythm and spoken-language features, while penalizing sustained scripted or written style.The rubric says not to judge from an isolated word and distinguishes mildly formal but still plausible speech from replies that no longer sound like normal phone conversation.Source: §B.2 · p. 17
• The naturalness rubric ranges from realistic, spontaneous replies to mechanical, incoherent, or nonsensical output.A short response is not required for the top score; overall real-time conversational feel is the criterion.Source: §B.2 · pp. 17–18

B.3 Business Profile Consistency

• The business-consistency prompt returns a binary score and a brief reason tied to the query and intent definition.The required output is JSON.Source: §B.3 · p. 18
• Business-profile consistency is scored 1 when a query's core request matches the intent definition and 0 when it does not.Keyword overlap alone is insufficient, and conflicting key information weighs against a match.Source: §B.3 · p. 18

C Example of a Generated Dialogue

• The example is a multi-round financial product recommendation dialogue between a customer service agent and a simulated user.Each round contains one agent utterance followed by a user response.Source: C Example of a Generated Dialogue · p. 18
• The conversation covers product introduction, clarification of product details, and questions about returns and account access.Source: C Example of a Generated Dialogue · p. 18
• The user eventually expresses interest in receiving further information.Source: C Example of a Generated Dialogue · p. 18

D.1 Downstream Agent Stress

• The prompt evaluates a customer-service reply against the user’s current utterance, rather than the entire dialogue.Source: §D.1 · p. 18
• Core intent matching has highest priority; politeness, sales enthusiasm, overall service quality, and reply length do not earn extra credit.Source: §D.1 · p. 18
• Basic correctness also considers factual errors, contradictions, fabricated information, and unreasonable promises.Source: §D.1 · p. 18
• A score of 5 means the reply directly addresses the core intent with few obvious problems, while 4 allows minor deviation or redundancy.Source: §D.1 · p. 18
• A score of 3 indicates partial response with omissions or misunderstanding; 2 indicates mostly irrelevant content or clear misunderstanding.Source: §D.1 · p. 18
• A score of 1 indicates no response to the user’s intent or a reply clearly contrary to the request.Source: §D.1 · p. 18
• The required output is JSON containing a 1–5 score and a one- or two-sentence reason with evidence directly tied to the score.Source: §D.1 · p. 18

E Business Profile Compliance Cases

• Table 4 illustrates the distinction between semantic business-profile compliance and colloquial user expression.Source: E Business Profile Compliance Cases · p. 18, Table 4 · p. 19
• With the same business profile, general-purpose LLMs explicitly elaborate context and restate the target business intent, whereas the Full Model produces a shorter, more implicit utterance.Source: E Business Profile Compliance Cases · pp. 18, 20, Table 4 · p. 19
• The Full Model’s less explicit wording better reflects concise, spontaneous expression patterns in real telephone conversations.Source: Table 4 · p. 19, E Business Profile Compliance Cases · p. 20