Source-linked AI summary
Proactive Dialogue Policy Optimization via Cognitive-State Transition
Minghui Ma, Mengqi Chen, Bin Guo, Jingqi Liu
TL;DR
Proactive dialogue research needs policies that adapt to user feedback, but many simulators do not explicitly track changing cognition and strategy-only actions miss utterance-level variation. This paper combines Cog-Sim's constrained cognitive-affective state transitions with CSTPO's hierarchical strategy–utterance optimization. Across three tasks, Cog-Sim is preferred over prompt-based simulators for naturalness, and CSTPO brings Qwen3-14B to performance comparable to GPT-5.5-based planning methods.
Problem
Many dialogue simulators do not explicitly model evolving user cognition, and strategy-only actions cannot distinguish alternative utterance realizations.
Source: §1 · p. 2Method
Cog-Sim models persistent cognitive-affective states with constrained transitions, while CSTPO optimizes hierarchical strategy–utterance actions with separate advantages.
Source: §1 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
The framework jointly models user-state evolution and optimizes both strategic decisions and their linguistic realizations in proactive dialogue.
Source: §1 · p. 2, §6 · p. 9The methods are evaluated only on public datasets and simulated interactions and are not claimed ready for real-world deployment.
Source: ETHICS STATEMENT · p. 10
Abstract
from arXiv · showhide
Proactive dialogue requires agents to continually adapt their policies to user feedback while progressing toward task objectives over multiple turns. To move beyond imitation learning on static datasets, recent approaches use user simulators to collect interactive data for policy optimization. However, many simulators do not explicitly model the evolution of user cognition, limiting the consistency and state dependence of feedback across turns. Moreover, representing each action only by a high-level strategy label overlooks the large utterance space and cannot distinguish alternative realizations of the same strategy. To this end, we jointly design a $\textbf{Cog}$nitive User $\textbf{Sim}$ulator $\textbf{(Cog-Sim)}$ and $\textbf{C}$ognitive-$\textbf{S}$tate $\textbf{T}$ransition--Driven $\textbf{P}$olicy $\textbf{O}$ptimization $\textbf{(CSTPO)}$. Cog-Sim maintains the user's cognitive and affective states and generates responses through constrained state transitions across turns, so feedback depends on both the realized utterance and the user's current state. CSTPO organizes each action as a hierarchical strategy--utterance representation: a high-level strategy label constrains utterance sampling, and utterances are optimized within each label. Sparse complete-branch sampling reuses shared dialogue prefixes and estimates separate strategy-level and utterance-level advantages, enabling fine-grained optimization at both levels. Across three tasks, Cog-Sim exhibits monotonic dose--response relationships and is preferred over prompt-based simulators for naturalness. CSTPO improves Qwen3-14B's performance to a level comparable to that of GPT-5.5-based planning methods.
1 INTRODUCTION
Proactive dialogue agents must adapt over multiple turns, but existing approaches either rely on existing planning abilities or fit demonstrations without repeatedly learning from interaction feedback. The paper addresses state-dependent user feedback and the tension between expressive actions and sampling efficiency with Cog-Sim and CSTPO.
1 INTRODUCTION
- Prompt-based planning uses current model abilities, while supervised planners learn from demonstrations; neither repeatedly adjusts policy from feedback generated by its own actions.Simulator-based reinforcement learning provides interactive trial, but prompt-based simulators often leave information interpretation, stance revision, and intention formation implicit.Source: §1 · pp. 1–2
- Label-only actions obscure differences among utterances using the same strategy, while richer action spaces require more trajectory comparisons under terminal-only rewards.Complete-trajectory sampling creates an efficiency bottleneck, so restricted action spaces trade behavioral granularity for lower sampling costs.Source: §1 · p. 2
- Without persistent state records and transition constraints, simulators may miss state-dependent responses to the same utterance.Conflating argument acceptance with action commitment risks premature goal pursuit or repeated use of mismatched strategies; some simulations also show excessive acceptance or persistent rejection.Source: §1 · p. 2
- Cog-Sim models persistent cognitive-affective states, while CSTPO optimizes both strategy selection and conditioned utterance realization.Across three tasks, Cog-Sim shows consistent dose-response behavior and is preferred over prompt-based simulators for naturalness; CSTPO-trained Qwen3-14B performs comparably to GPT-5.5-based planners.Source: §1 · p. 2
2 RELATED WORK
Prior work advances proactive dialogue through policy planning, user simulation, and multi-turn credit assignment. The paper positions CSTPO and Cog-Sim as addressing remaining gaps in action expressiveness, dynamic cognition modeling, and decomposing strategy and utterance contributions.
Policy planning for proactive dialogue
- CSTPO factorizes each turn into a strategy and its conditioned utterance, optimizing both from interaction outcomes rather than limiting expressiveness to predefined or abstracted strategies.Earlier methods include prompt-based planning, supervised planners, and interaction-trained policies such as PPDPP and DialogXpert.Source: §2 · p. 3
User simulation in dialogue policy learning
- Cog-Sim represents cognition and affect as persistent variables with bidirectional, constrained transitions, enabling state-dependent and counterfactual responses.Profile-conditioned simulators often fix profiles before interaction, while many state-based approaches emphasize task progress rather than controlled bidirectional cognitive change.Source: §2 · p. 3
Multi-turn RL and credit assignment
- CSTPO decomposes terminal-return credit into strategy-selection and utterance-realization advantages using strategy-conditioned values and shared-prefix branching.Prior methods address multi-turn credit assignment or hierarchical strategy-response optimization, but do not explicitly separate the contribution of strategy choice from linguistic realization.Source: §2 · p. 3
3 METHOD
The method models proactive dialogue as a finite-horizon interaction with sparse terminal feedback, then combines persistent cognitive-affective user simulation with hierarchical policy optimization. Shared-prefix branches support separate credit assignment to strategies and their utterances.
3.1 PROBLEM FORMULATION
- The policy selects a strategy and then a conditioned utterance, which Cog-Sim uses to generate the next observation before terminal task utility is assigned.The Actor observes dialogue history and role-visible task information; the objective is based on terminal utility over trajectories induced by the Actor and Cog-Sim.Source: §3.1 · pp. 3–4
3.2 COG-SIM: COGNITIVE-STATE-DRIVEN USER SIMULATION
- Cog-Sim maintains a persistent user profile and dynamic cognitive-affective state, using user-specific processing tendencies and constrained updates to model responses.An utterance-based Action Type Classifier distinguishes Influence, Elicit, and Social: Influence invokes the cognitive updater, while Elicit and Social preserve the BDI state; all modes update affect and generate a response.Source: §3.2 · pp. 4–5
3.3 CSTPO: COGNITIVE-STATE-ASSISTED HIERARCHICAL OPTIMIZATION
- CSTPO estimates state and strategy-conditioned values, then uses their baselines to separate strategy-selection and utterance-realization advantages.At selected checkpoints, it continues multiple strategies and utterance realizations from the same pre-action context; averaging returns within the main strategy reduces dependence on one wording, while alternative tails fit the Critic and strategy-level return estimates.Source: §3.3 · p. 5
4 EXPERIMENTAL SETUP
The experiments evaluate policy optimization on three multi-turn dialogue tasks using task-specific terminal returns and compare against simulator and policy baselines. The setup also specifies model backends, training initialization, and rollout sampling.
4.1 TASKS AND EVALUATION
- Evaluation covers emotional support, donation persuasion, and price negotiation, with 200/50/100 interaction seeds for training, development, and final evaluation.Mean task-specific terminal return is primary; ESConv combines emotion improvement and action-plan feasibility, P4G measures valid voluntary donation commitment, and CraigslistBargain measures valid deals and clipped price utility.Source: §4.1 · p. 6
4.2 BASELINES
- Policy baselines range from direct generation and prompting methods to RL-based planners, supervised initialization, and CSTPO.Simulator comparisons include Persona, Persona-Resist, and BDI-Prompt; policy comparisons include Proactive, ProCoT, Ask-an-Expert, MI-Prompt, PPDPP, and DialogXpert.Source: §4.2 · p. 6
4.3 IMPLEMENTATION DETAILS
- GPT-5.5 evaluates prompt and planning baselines, DeepSeek-V4-Pro-0813 backs simulators and the terminal evaluator, and Qwen3-14B is the trainable backbone.Task-specific SFT mixes human dialogues and teacher trajectories 1:1 for four epochs with LoRA; RL samples 10 seeds per step and expands two checkpoints for P4G and CB and four for ESConv.Source: §4.3 · p. 6
5 EXPERIMENTS
Across six evaluations, Cog-Sim showed dose-responsive feedback and higher naturalness preference, while CSTPO improved dialogue-policy outcomes, survived ablations and an 8B test, and shifted strategy use by task.
5.1 USER SIMULATOR VALIDATION
- Cog-Sim’s mean response scores rose monotonically across input levels on all three tasks, while prompt controls changed less at ESConv and P4G endpoints.On CB, Cog-Sim responded progressively to intermediate offers, whereas controls changed most sharply at the full-price offer.Source: Figure 3 · p. 7, §5.1 · p. 7
- Cog-Sim earned higher naturalness preference in all nine task–comparator pairs and gains across evaluation dimensions on ESConv, P4G, and CB.The pairwise evaluation table reports LLM judgments; positive values favor Cog-Sim.Source: Table 1 · p. 7, §5.1 · p. 7
5.2 DIALOGUE POLICY PERFORMANCE
- Terminal return reached 0.624 on ESConv (vs. GPT-5.5 0.632), 0.460 on P4G (vs. SFT Init. 0.420), and 0.491 on CB (vs. GPT-5.5 0.489).ESConv gains came mainly from action-plan feasibility; CB gains came mainly from price utility, with comparable emotion improvement and deal rate, respectively.Source: §5.2 · p. 7, Table 2 · p. 8
5.3 COMPONENT ABLATIONS
- Without cognitive-state inputs, tail-50-step returns were lower on all tasks; without the hierarchical critic, training failed to improve over SFT initialization.The full model scored 0.620 vs. 0.604 on ESConv, 0.462 vs. 0.448 on P4G, and 0.496 vs. 0.478 on CB against w/o Cog-State.Source: Figure 4 · p. 8, §5.3 · p. 8
5.4 SMALLER-BACKBONE GENERALIZATION
- The 8B Actor improved over task-specific SFT initialization on all three tasks, with the largest absolute gain on CB, but remained below 14B endpoints.The evidence supports transfer across the two tested scales, not backbone-independent performance.Source: Table 3 · p. 8, §5.4 · p. 8
5.5 HUMAN EVALUATION
- CSTPO received significant positive preferences in 13 of 27 human comparisons, with consistent ESConv gains in identification and comforting and P4G gains in persuasiveness.Three comparisons favored GPT-5.5 baselines, including suggestion on ESConv and naturalness on P4G and CB.Source: Table 4 · p. 9, §5.5 · p. 9
5.6 STRATEGY ADAPTATION AFTER RL
- After RL, strategy distributions shifted toward task-relevant categories: Soften on ESConv, Probe on CB, and Advance on P4G, while Other declined across tasks.Soften rose from 43.2% to 54.4% on ESConv; Probe from 14.2% to 19.8% on CB; and Advance from 25.8% to 33.6% on P4G.Source: Figure 5 · p. 9, §5.6 · p. 9
6 CONCLUSION
The work combines Cog-Sim’s explicit cognitive–affective user states with CSTPO’s joint optimization of strategic decisions and utterance realizations. Across three tasks, the authors report more consistent and natural feedback from Cog-Sim and higher terminal returns for CSTPO-trained Qwen3-14B than strong baselines.
6 CONCLUSION
- Across emotional support, donation persuasion, and negotiation, CSTPO-trained Qwen3-14B achieves higher terminal returns than strong baselines including GPT-5.5-based ProCoT and PPDPP.Cog-Sim provides more consistent and natural user feedback than prompt-based simulators.Source: §6 · p. 9
- Cog-Sim grounds user responses in explicit cognitive and affective states, while CSTPO jointly optimizes strategic decisions and linguistic realizations.The paper connects user cognition with proactive dialogue policy learning through these two components.Source: §6 · p. 9
- Future work will calibrate cognitive and affective transitions with human interaction data and investigate broader user profiles and contexts.The authors also call for independent evaluators, cross-simulator validation, and human interaction studies to assess whether simulated improvements translate to effective real-world dialogue.Source: §6 · p. 9
AI USE STATEMENT
The appendix records AI use and evaluation boundaries, then details Cog-Sim’s state representation, utterance-based action classification, constrained transitions, and recoverable checkpoints.
AI USE STATEMENT
- AI tools supported conceptual discussions, experimental design and implementation, hyperparameter selection, analysis, trajectory generation, and strategy annotation; authors revised and reviewed the work.Strategy instances were independently annotated three times, with majority labels retained when available and disagreements manually annotated; AI tools were not used for mathematical derivations or proofs.Source: AI USE STATEMENT · p. 10
- The work uses public datasets and simulated interactions, does not claim real-world deployment readiness, and cautions that its state representations are not clinical assessments or reliable estimates of real users’ internal states.It also notes possible undue influence concerns for deployed persuasive dialogue policies.Source: ETHICS STATEMENT · p. 10
APPENDIX ROADMAP
- Table 5 summarizes the appendix contents and their relation to the main paper.The appendix is organized around mechanism, reproducibility details, evaluation evidence, and prompt specifications.Source: APPENDIX ROADMAP · p. 15, Table 5 · p. 15
A.1 STATE REPRESENTATION
- Cog-Sim represents users with a persistent profile, dynamic beliefs–desires–intentions state, affective state, and stable user-specific processing tendencies.Belief, desire, and intention records include semantic content, strength, centrality, activation status, and, where applicable, direction.Source: §A.1 · p. 15
A.2 ACTION TYPE, PROCESSING ROUTE, AND JUDGMENT
- An action-type classifier uses the realized utterance—not its task-specific strategy label—to assign Influence, Elicit, or Social, which determines whether cognition changes or is preserved.All three modes may update affect and generate a user response; Influence uses route and judgment rules, including acceptance and rejection regions.Source: §A.2 · pp. 15–16
A.3 CONSTRAINED COGNITIVE TRANSITION
- For Influence acts, an LLM proposes a sparse cognitive change and response plan, then a deterministic updater applies transition constraints; the response is generated from the updated state.The updater clips strengths, enforces capacity limits, and rejects unsupported changes; Elicit and Social preserve the cognitive state, and affect is smoothed with mode-specific rates.Source: §A.3 · p. 16, Table 6 · p. 16
A.4 RECOVERABLE CHECKPOINTS
- Each branch continuation restores a complete pre-action checkpoint, so alternatives share the same observed prefix and latent user state.The checkpoint includes dialogue history, profile, cognitive parameters, task events, termination status, remaining horizon, and random stream.Source: §A.4 · p. 16
B CSTPO TRAINING DETAILS
CSTPO samples alternative strategy–utterance continuations from shared dialogue prefixes, trains hierarchical critics, and updates the actor on main-path action tokens. The appendix also defines nested conditional-return quantities used to motivate this training structure.
B.1 CRITIC OBJECTIVE
- The critic objective combines main-path and branch losses, while branch continuations supervise only sampled strategy-conditioned values and unsampled strategies receive no artificial zero target.The main continuation is counted only in the main loss, and the frozen old critic forms advantages before fitting the current batch.Source: §B.1 · p. 18
B.2 SAMPLING AND ACTION MASKS
- At selected checkpoints, CSTPO samples distinct strategies and multiple utterance realizations, reusing the stored main continuation to reduce additional tails.Each RL step uses 10 seeds; two checkpoints are selected per main trajectory for P4G and CB and four for ESConv, yielding seven and thirteen rollouts per seed, respectively.Source: §B.2 · p. 18
- Only main-path strategy and utterance tokens enter the actor loss; additional continuation tokens and non-action inputs have zero actor mask.Strategy probabilities use trie-constrained label support, while utterance probabilities use the regular vocabulary.Source: §B.2 · p. 18
C THEORETICAL PROPERTIES OF CSTPO
- The theoretical setup defines nested return expectations conditioned on observation, training state, strategy, and utterance.These quantities are H(O), V(X), U(X,K), and Q(X,K,U), respectively.Source: C THEORETICAL PROPERTIES OF CSTPO · p. 18
C.1 TRAINING-STATE REFINEMENT
The theoretical section relates richer conditioning to return prediction and decomposes joint-action value variation across strategy and utterance levels. It also qualifies what the population identities do—and do not—establish.
C.1 TRAINING-STATE REFINEMENT
- The training-state refinement gap measures additional return-predictive information in the full training state, but does not attribute that gap exclusively to cognition.The state also contains task and horizon variables; the ablation study is identified as the empirical way to isolate cognitive-affective fields.Source: §C.1 · p. 18
C.2 HIERARCHICAL ORTHOGONAL DECOMPOSITION
- The joint-action advantage separates into strategy-level and within-strategy utterance-level components that are orthogonal in expectation.The proof uses conditional expectation given the training state and strategy.Source: §C.2 · p. 19
C.3 ACTION ALIASING AND THE UNIFIED DECOMPOSITION
- A positive action-aliasing gap means utterances sharing a strategy label can have different long-term values, motivating an explicit utterance component.The four increments are pairwise orthogonal under the fixed-policy trajectory distribution, but the identity does not assert exact finite-sample advantage decomposition, learned critics equal conditional expectations, or PPO convergence to an optimal policy.Source: §C.3 · p. 19
D.1 DATASETS AND ACTION SPACES
The appendix specifies the three task-specific action spaces, how dialogue seeds preserve state and role boundaries, and the training and evaluation protocols used for Cog-Sim and CSTPO.
D.1 DATASETS AND ACTION SPACES
- ESConv, P4G, and CB use distinct strategy spaces; CB also records structured offer, acceptance, and termination events separately from strategy labels.Training, development, and test-time evaluation use 200, 50, and 100 dialogue seeds per task, respectively.Source: §D.1 · p. 19, Table 7 · p. 19
D.2 SEED SCHEMA AND INFORMATION BOUNDARIES
- Seeds preserve role-specific task views, dialogue prefixes, user profiles, initial cognitive-affective states, random streams, and task-specific information.The Actor sees only its role-visible view and dialogue history; Cog-Sim receives the simulated user's private task view and cognitive state, while terminal evaluation excludes latent state.Source: §D.2 · p. 20
D.3 STATE ANNOTATION AND QUALITY CONTROL
- Initial states are annotated from evidence available by the rollout cutoff, excluding reactive changes from later dialogue and future source outcomes.An automatic validation pass and human review check grounding, role visibility, outcome leakage, structural validity, and evidence-supported state changes; failed seeds are corrected or excluded.Source: §D.3 · p. 20
E.1 SFT AND REINFORCEMENT LEARNING
- Each task receives a separate Qwen3-14B Actor, initialized by SFT and then trained with PPO using a separate Critic and no SFT-reference KL penalty.SFT mixes human dialogues and frozen teacher trajectories 1:1; RL runs for 250 steps on ESConv and CB and 300 on P4G, with rollouts capped at 30 Actor–User turns.Source: §E.1 · p. 20
E.2 TERMINATION AND TERMINAL UTILITIES
- Terminal utilities score task outcomes, including ESConv emotion/hope improvement and action-plan feasibility, P4G commitment, and CB deal outcomes.Numeric judgments are averaged across three evaluations, Boolean fields use majority vote, and paired comparisons use 2,000 case-clustered bootstrap samples with 95% confidence intervals.Source: §E.2 · pp. 20–21
E.3 COG-SIM VALIDATION PROTOCOL
- Cog-Sim validation tests four ordered input levels from the same pre-action checkpoint, with two paraphrases per level, and evaluates response change, naturalness, consistency, and dose appropriateness.The 100 frozen seeds per task are scored by a frozen LLM Judge for response change; pairwise preference is reported as (W −L)/N, with ties included in N.Source: §E.3 · p. 21
F.1 COMPLETE QWEN3-14B RESULTS
The appendix reports complete Qwen3-14B results, describes strategy-use shifts, and tests whether gains extend to an 8B Actor. It also characterizes those 8B gains as positive but below the 14B counterparts.
F.2 DIFFERENCES FROM THE QWEN3-14B STANDARD POLICY
- CSTPO improves terminal return over the Qwen3-14B Standard policy on all three tasks, with the largest absolute difference on CB; these mean differences are descriptive, not separate significance tests.Table 8 provides the complete Qwen3-14B evaluation, and Table 9 reports differences computed from its means.Source: §F.2 · p. 21, Table 9 · p. 21, §F.1 · p. 21
F.3 STRATEGY MAPPING AND FINE-GRAINED SHIFTS
- The cross-task strategy analysis maps 27 task-specific labels into four functional groups without changing the policy's action space.Tables 10–13 provide the mapping and individual label definitions.Source: §F.3 · p. 21, Table 10 · p. 22, Table 11 · p. 22
- On ESConv, Affirmation and Reassurance rises 9.5 points while Question and Others fall 6.0 and 7.2 points; on CB, Propose Price and Inform rise 4.5 and 3.1 points while social falls 5.6 points.On P4G, Donate Request has the largest increase at 6.0 points; these reallocations do not establish a label's causal contribution or utterance quality.Source: §F.3 · p. 23, Figure 6 · p. 23
G.1 SMALLER-BACKBONE GENERALIZATION
- With an 8B Actor, training improves over task-specific SFT initialization on all three tasks, with the largest absolute gain on CB, but endpoints remain below 14B counterparts.The result supports transfer across the two tested model scales, not backbone-independent performance.Source: §G.1 · p. 23, Table 14 · p. 23, Figure 7 · p. 24
G.2 ACTOR–CRITIC TRAINING DYNAMICS
The training traces show that both value heads are optimized across tasks, while the policy-gradient loss remains bounded; these diagnostics do not establish optimality.
G.2 ACTOR–CRITIC TRAINING DYNAMICS
- The strategy-conditioned U-head loss falls early and stays low, the state-value V-head loss declines across all three tasks, and the policy-gradient loss stays near zero with bounded oscillation.The U-head supports utterance-level credit assignment, while the V-head supports strategy-level credit assignment; Figure 8 shows the 14B traces and Figure 9 the 8B traces.Source: §G.2 · p. 24, Figure 8 · p. 24, Figure 9 · p. 24
- The loss curves provide diagnostic evidence of value-head optimization, not proof that training converges to an optimal policy.The 14B training-dynamics plot places the two Critic value losses on the left axis and the Actor policy-gradient loss on the right.Source: §G.2 · p. 24, Figure 8 · p. 24
H HUMAN EVALUATION DETAILS
Human comparisons favor CSTPO in 13 of 27 comparisons, with advantages concentrated in task-relevant dimensions. Case studies illustrate strategy changes and expose limits in what terminal outcomes establish.
H HUMAN EVALUATION DETAILS
- CSTPO is significantly preferred in 13 of 27 human comparisons, with advantages concentrated in ESConv identification and comforting and P4G and CB persuasiveness.The three significant negative comparisons concern ESConv suggestion against ProCoT and naturalness against Standard on P4G and CB; cases are illustrative rather than average-performance evidence.Source: H HUMAN EVALUATION DETAILS · pp. 24–25, Table 15 · p. 25, I CASE STUDIES · p. 25
I.1 ESCONV: ADAPTING FROM AFFECT ELICITATION TO ACTION FORMATION
- In the ESConv case, the Actor moves from eliciting anger and schooling constraints to travel information and reassurance as the user forms a research plan.The analysis flags overconfident visa claims requiring external verification and notes that the high terminal score reflects emotional and action progression, not factual reliability.Source: §I.1 · p. 25, Case 1: (continued) · p. 26
I.2 P4G: RESOLVING INFORMATION BARRIERS BEFORE COMMITMENT
- In the P4G case, the Actor shifts from Donation Information to Inquiry Response after the user asks about donation frequency and allocation, preceding a stated one-time commitment.The terminal score captures expressed intent, not a completed donation or verified organizational claims.Source: §I.2 · p. 26
Case 3: CB (continued)
- In the CB case, the Actor raises its offer to $3,000 and shifts to stance and social acts, but the user's agreement remains conditional on inspection despite an evaluator deal=true outcome.The reported return is an evaluator outcome for this trajectory, not evidence of an unconditional deal.Source: Case 3: CB · p. 26, Case 3: CB (continued) · p. 27
J PROMPT SPECIFICATIONS
- The appendix presents proposed-framework prompts as compact operational templates, represents repeated schema boilerplate and runtime state by named fields, and excludes baseline prompts and settings.The Actor protocol strips duplicated role prefixes or embedded label lines before passing utterances to Cog-Sim while retaining sampled tokens and log probabilities for training.Source: J PROMPT SPECIFICATIONS · p. 27, Actor prompts and hierarchical output… · p. 28