Source-linked AI summary
Implicit Beliefs as Scalable Process Rewards for Strategic Persuasion
Minqian Liu, Yuexi Shen, Heajun An, Qi Zhang, Pamela J. Wisniewski, Sang Won Lee, Jin-Hee Cho, Lifu Huang
TL;DR
Multi-turn strategic persuasion needs realistic simulations and dense supervision tied to interaction dynamics. The paper introduces PERSUARENA and PROBE-RPO, which use the simulated persuadee’s belief transitions as process rewards. PROBE-RPO outperforms outcome-only GRPO by 16.7 percentage points in full acceptance when process rewards are weighted moderately.
Problem
Learning policies for complex, multi-turn persuasion requires systematic training, but LLM simulations can produce unrealistically compliant or arbitrarily obstinate persuadees.
Source: §1 · pp. 1–2Method
PERSUARENA simulates psychologically structured persuadees, while PROBE-RPO probes their logits each turn and uses belief-state changes as process rewards alongside outcome rewards.
Source: §1 · p. 2, §3.2 · p. 5Results
16.7 percentage points higher full acceptance than the strongest outcome-only GRPO baseline with moderately weighted process rewards; the learned policy also transfers to unseen persuadee models.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
The results support implicit beliefs as a source of effective process supervision for strategic dialogue, with belief read-outs replacing a per-turn judge model.
Source: §6 · p. 10, §7 · p. 10Transfer to human interlocutors remains untested, and reward faithfulness depends on the simulated persuadee’s calibration and the probe questions’ scope.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Training large language models (LLMs) for multi-turn strategic persuasion faces two coupled obstacles: (i) realistic simulation of psychologically grounded persuadees, and (ii) dense, faithful supervision for credit assignment in strategic dialogues. Outcome-only rewards are sparse and uninformative about which conversational moves drove success, while LLM-judge process rewards are expensive and decoupled from the underlying interaction dynamics. We address both challenges with a unified framework. First, we introduce PersuArena, a persuasion simulation environment that produces realistic, non-monotonic trajectories. Second, building directly on this environment, we propose Process Rewards via Online Belief Elicitation for Relative Policy Optimization (Probe-RPO), an online RL framework that converts the persuadee's internal state into dense process rewards. At each turn, three introspective probes are read from the frozen persuadee's logits, yielding a multi-dimensional reward signal that is faithfully aligned with persuasion dynamics. With extensive experiments on PersuArena spanning healthcare, financial advising, workplace management, education, and personal relationships, we demonstrate that our Probe-RPO achieves the full acceptance rate of 52.5%, compared with the 35.8% for the strongest GRPO ablation and 25.8% for the best off-the-shelf LLM. We also conduct comprehensive analyses on the generalization across persuadee models and the persuasion dynamics across conversations. Our work offers a promising paradigm that can provide effective process supervision for strategic dialogue.
1 Introduction
Multi-turn persuasion requires realistic, psychologically grounded simulation and dense supervision tied to the interaction. The paper introduces PERSUARENA and PROBE-RPO to address these challenges, reporting a 16.7-percentage-point full-acceptance gain over the strongest outcome-only GRPO baseline when process rewards are weighted moderately.
1 Introduction
- Persuasion simulation is difficult because targets differ psychologically and change dynamically, while existing LLM simulations can be unrealistically compliant or arbitrarily obstinate.Outcome-only rewards also fail to identify which conversational moves drove success, and LLM-judge process rewards are costly and decoupled from interaction dynamics.Source: §1 · pp. 1–2
- PROBE-RPO derives dense, multi-dimensional rewards from turn-by-turn changes in the persuadee’s belief state and combines them with outcome rewards for critic-free policy optimization.The framework uses introspective probing of the persuadee’s logits.Source: §1 · p. 2
- 16.7 percentage points higher full acceptance than the strongest outcome-only GRPO baseline, with process rewards weighted moderately.The authors report that the gain indicates high-quality dense supervision.Source: §1 · p. 2
- PERSUARENA operationalizes psychological mechanisms into executable constraints across 120 scenarios in five domains, producing realistic, non-monotonic persuasion trajectories.The scenarios cover healthcare, financial advising, workplace management, education, and personal relationships.Source: §1 · p. 2
2 PERSUARENA
PERSUARENA uses hidden psychological profiles and explicit behavioral rules to simulate varied, dynamic persuadees. Its taxonomy, nuanced terminal states, and 120 scenarios structure the environment around realistic persuasion dynamics.
2 PERSUARENA
- PERSUARENA compiles a hidden persuadee profile into explicit behavioral constraints and a priority-ordered decision procedure, with a deterministic episode state machine enforcing termination.The design targets dynamic phenomena such as graduated reactance escalation and cross-turn inoculation that generic role-play prompts leave to the LLM’s priors.Source: §2 · p. 2, §2.1 · p. 3
2.2 Taxonomy
- The simulation taxonomy maps psychological constructs to parameters, prompt instructions, or decision rules, with resistance type as its main strategic axis.Four hidden resistance types—Entrenched Belief, Identity Threat, Autonomy Resistance, and Ambivalent Avoidance—call for different strategies that the persuader must diagnose.Source: §2.2 · p. 3
- Reactance escalation, inoculation saturation, and asymmetric trust ratcheting produce non-monotonic trajectories; at least 15% of turns carry a negative process reward for every evaluated persuader.The share reaches 36.1% for the untrained policy.Source: §2.2 · p. 4
2.3 Decision States
- The decision procedure distinguishes five terminal states, including conditional acceptance pending one external verification and a stalled conclusion without a clear yes or no.A strict priority order checks for stalled conversations, missing concrete requests, value violations, and then whether concerns are resolved.Source: §2.3 · p. 4
2.4 Scenario Coverage
- The benchmark contains 120 scenarios across healthcare, financial advising, workplace management, education, and personal relationships, with combinatorial coverage of psychological and situational factors.Each scenario specifies the persuader’s goal, the persuadee’s hidden profile, and the initial context.Source: §2.4 · p. 4
3 PROBE-RPO
PROBE-RPO uses the frozen persuadee both to generate dialogue and to provide dense supervision from its changing belief state. It combines per-turn process rewards with terminal outcomes in a group-relative optimization framework.
3.1 Problem Setup
- The persuader observes dialogue history and acts in episodes with a frozen persuadee, which replies under a hidden profile.Episodes end after at most Tmax turns.Source: §3.1 · p. 4
3.2 Eliciting Implicit Beliefs
- Each persuadee turn uses one pass to sample its visible reply and a discarded context fork to measure probes, leaving dialogue generation stochastic and measurement deterministic.The measurement never enters the transcript observed by the persuader.Source: §3.2 · p. 5
- Three fixed, positively valenced probes measure cognitive concern-address, strategic persuasion-effectiveness, and conative acceptance-shift dimensions.Yes is favorable to the persuader on all three probes; the basis is not tuned.Source: §3.2 · p. 5, Table 1 · p. 5
- The belief state reads the persuadee’s Yes/No probability mass for each probe, then averages probe values into an attitude summary.The state is read from the persuadee itself rather than inferred by the agent.Source: §3.2 · p. 5
3.3 Reward Design
- Per-turn reward uses belief change rather than belief level, avoiding payment for receptiveness the persuader did not create.The increments telescope over an episode, but may favor ending at a higher belief; a terminal anchor and weight λ constrain that effect.Source: §3.3 · p. 5
3.4 Optimization
- Optimization combines a GRPO outcome advantage with a group-pooled dense return-to-go, with λ controlling the process contribution and λ = 0 recovering outcome-only GRPO.The clipped surrogate updates persuader tokens only; persuadee replies and probe tokens are masked out.Source: §3.4 · p. 6
4 Experiments
Experiments compare Probe-RPO with prompted persuaders and reward ablations, then examine cross-model transfer, conversational moves, and belief trajectories. The results show strong full-acceptance performance, with outcomes sensitive to process-reward weight and persuadee model.
4.1 Experimental Setup
- The evaluation uses 120 held-out PERSUARENA scenarios, with 1,800 distinct training instances and prompted LLM and reward-ablation comparisons.The persuader backbone is Qwen3-4B-Instruct; the frozen persuadee and judge are separate Qwen3-14B instances.Source: §4.1 · p. 7
4.2 Main Results
- 52.5% full acceptance for PROBE-RPO, versus 35.8% for GRPO (Goal+Strategy) and 25.8% for Claude Haiku 4.5.PROBE-RPO also reaches 73.3% full-or-conditional acceptance; its pending rate is 17.5%, versus 39.2% for GRPO (Goal+Strategy).Source: §4.2 · p. 7
- The best-checkpoint acceptance rate peaks at 52.5% with λ = 0.5, versus 35.8% at λ = 0 and 30.0% at λ = 1.The paper reports that too little process-reward weight can leave GRPO groups with identical outcomes and zero gradient, while too much lets an accept-blind shaping signal dominate.Source: §4.2 · pp. 7–8, Figure 3 · p. 8
4.3 Generalization across Persuadee LLMs
- PROBE-RPO exceeds prompted persuader baselines against all three frozen persuadees, with reported acceptance of 52% on Qwen3-14B, 75% on GPT-5.6 Luna, and 45% on Claude Haiku 4.5.It was trained only against Qwen3-14B; the 45%–75% spread shows transfer is substantial but not invariant across persuadee models.Source: §4.3 · p. 8, Figure 4 · p. 8
4.4 Analysis of Persuasion Dynamics
- PROBE-RPO uses diagnostic questions in roughly 80% of first-turn utterances, then increases plan details and commitment asks in the middle turns.The untrained initialization relies more on generic autonomy cues and produces substantially fewer concrete plans; Claude Haiku 4.5 exhibits a similarly staged pattern.Source: Figure 5 · p. 8, §4.4 · p. 9
4.5 Analysis of Probe Trajectories
- Under PROBE-RPO, all three probes rise from near −1 to roughly +0.7 by mid-conversation, while the untrained policy’s PE and AC remain negative.Both policies raise concern-address (CA), but only PROBE-RPO also raises persuasion effectiveness (PE) and acceptance shift (AC).Source: §4.5 · p. 9, Figure 6 · p. 9
- Acceptance and conditional-acceptance belief trajectories separate from the others within roughly two to three turns; pending trajectories rise slightly before flattening near zero.The paper describes this pending pattern as a distinct signature of stalling rather than opposition.Source: §4.5 · p. 9, Figure 6 · p. 9
5 Related Work
Related work spans computational persuasion, social-agent simulation, and dense reinforcement-learning supervision. The paper positions PERSUARENA as a controlled persuasion simulator and PROBE-RPO as an alternative to external per-turn judging.
Persuasion and Strategic Dialogue
- Computational persuasion has moved from task-specific corpora (Wang et al., 2019) to LLM-based simulation and evaluation, including parameterized social-agent profiles in SOTOPIA (Zhou et al., 2024).Recent studies use simulated or human interlocutors (Salvi et al., 2024; Durmus et al., 2024; Liu et al., 2025), while training methods include retrospective feedback, dialogue-level RL, and offline-RL stage planning (Hong et al., 2025; Guo et al., 2026a; An et al., 2026).Source: §5 · pp. 9–10
- PERSUARENA targets controlled policy learning with measurable persuasion goals and hidden resistance profiles, not human-level simulation fidelity.The paper distinguishes it from general social simulators on that narrower scope.Source: §5 · p. 10
Process Rewards and Dense Supervision for RL
- PROBE-RPO derives dense process rewards from the opponent’s token probabilities, unlike external LLM-judge approaches that can be expensive, biased, and decoupled from latent state.The passage also notes that probes can be raised without closing, so the terminal outcome remains necessary; its audit detects only failures within the judge’s questions’ scope.Source: §5 · p. 10
6 Discussion
The discussion frames belief probes as an inexpensive source of process supervision while emphasizing that they are limited proxies. Their reliability depends on simulated persuadee calibration and probe scope, and transfer to human interlocutors remains open.
What the probes add
- A few single-token read-outs from the persuadee already in memory replace a per-turn judge model, while dense rewards provide gradient where outcome-only GRPO has none.The paper says the gains mainly shift unresolved conversations toward decisions rather than avoiding rejection.Source: §6 · p. 10
What they do not guarantee
- Probe rewards reflect simulated persuadee self-reports, depend on model calibration and question scope, and do not establish transfer to human interlocutors.The passage also says the three probe levels are strongly correlated and λ must remain anchored to the outcome; ethical limits for more capable persuasion agents remain open.Source: §6 · p. 10
7 Conclusion
The conclusion presents PERSUARENA and PROBE-RPO as a unified framework for multi-turn persuasion training. It reports improved persuasion outcomes, transfer to unseen persuadee models, and an emergent diagnose-then-act structure.
7 Conclusion
- The framework combines PERSUARENA’s principled persuadee simulation with PROBE-RPO’s belief-state process rewards, and experiments report stronger results than prompted LLMs and outcome-only RL baselines.The learned policy generalizes to unseen persuadee models and acquires a diagnose-then-act conversational structure through policy optimization.Source: §7 · p. 10
A Limitations •
The authors identify limits in probe validity, scenario realism, human evaluation, reward parameterization, and ethical safeguards. PERSUARENA itself uses static prompts and a deterministic state machine rather than a numerical psychological controller.
A Limitations •
- The probe reward assumes the frozen persuadee’s next-token distribution tracks the named constructs, an assumption that may fail for poorly calibrated or very small models.Scenarios are synthetic and lack human-agreement validation, and the trained agents have not been evaluated with human interlocutors; probability-space rewards also discard magnitude information preserved by log-odds.Source: A Limitations • · p. 14
- Ethical constraints remain future work despite PERSUARENA including metrics for detecting manipulation.The authors note risks from unethical persuasion and dual use, including misinformation and cybergrooming.Source: A Limitations • · p. 14
B PERSUARENA Details
- PERSUARENA has no numerical controller: static prompts and a deterministic parser with an episode state machine govern termination, while dialogue history drives other cross-turn changes.No trust, reactance, or phase variable is tracked.Source: B PERSUARENA Details · p. 14
- A single rejection is recoverable, conditional acceptance gets a non-resetting upgrade window, and exhausted turn budgets preserve a recorded decision or default to pending.The parser checks CONDITIONAL ACCEPT before ACCEPT; final-exchange instructions are ephemeral, and persuader-emitted decision tokens are stripped before the persuadee sees them.Source: B PERSUARENA Details · p. 14
B.1 Scenario Construction
Scenarios are built by expanding sparse seeds into coherent psychological profiles and contexts, then split into training and held-out sets with materially different coverage and populations. Shared prompt templates give persuaders strategy guidance and standardize how persuadees express decisions.
B.1 Scenario Construction
- Each scenario expands a sparse DailyPersuasion seed into a context, hidden persuadee profile, resistance and motivation patterns, relationship and trust, persuader profile and goals, and a prompt-ready narrative.The annotator is instructed to keep resistance and motivation coherent, trust plausible, and the situation nontrivially agreeable.Source: §B.1 · pp. 14–15
B.2 Scenario Statistics
- Held-out scenarios specify every hidden block, whereas training scenarios sometimes omit resistance, motivation, or relationship fields and differ in stakes, professionalism, age, and motivation.Held-out results therefore measure transfer to independently authored scenarios rather than in-distribution performance.Source: §B.2 · p. 15
B.3 Example Scenario
- The example’s strongest argument—administrative necessity—would escalate its persuadee, whose hidden resistance is entrenched belief and whose undisclosed secret complicates the decision.The persuader is not given the persuadee’s resistance, motivation, or secret.Source: §B.3 · pp. 15–16
C Prompt Templates
- The shared persuader template directs agents to diagnose concerns, fit moves to resistance, and pivot after pushback, while requiring substantive, goal-focused turns.Its strategy advice is a shared prior across systems, so held-out success partly measures adherence; the persuadee template defines decision states and a procedure for pending, rejection, and acceptance.Source: C Prompt Templates · pp. 16–17
D Training and Probe Details
The training details specify how signed probe signals and terminal rewards enter optimization, how probes are read and checked, and how the strategy judge and computational setup are used. Probe reuse avoids a separate judge model, though not lower latency than an equal-sized judge call.
D.1 Full Objective and Probe Signs
- Each probe has a fixed valence sign, and the experiments use three positive probes; probability-space signals are bounded, unlike unbounded log-odds that can require clipping or temperature tuning.The persuader is updated with a token-level clipped surrogate over persuader tokens, regularized toward a frozen reference policy.Source: §D.1 · p. 18
D.2 Probe Interface and Fidelity
- Concurrent probes read yes/no token probabilities from context forks, and probe read-outs closely match offline full-vocabulary scores except mainly when the persuadee is undecided.AC measures change and is differenced again, making it a trend signal; 83.3% of read-outs have an absolute state above 0.95.Source: §D.2 · p. 18
D.3 Strategy Judge
- The frozen strategy judge scores diagnostic probing, profile–strategy fit, and adaptation after resistance, separately from the goal reward.Its score is mapped to a −2 to +2 strategy modifier, or zero if it fails after three retries.Source: §D.3 · pp. 18–19
D.4 Configuration
- The persuadee and strategy judge run as separate frozen services, preventing either from receiving gradient during training.The stated configuration uses FSDP for the actor and SGLang for rollouts.Source: §D.4 · p. 19
D.5 Cost
- With prefix caching, probe read-out costs 14.3% of visible-reply latency, but its advantage is reusing the persuadee already in memory rather than reducing latency versus an equal-sized judge.Single-token decoding avoids the judge’s parse-failure and retry path.Source: §D.5 · p. 19, Table 4 · p. 19
E.1 Full Results and Paired Tests
On the 120-scenario held-out set, PROBE-RPO improves outcomes over its initialization without longer conversations, while analyses show that performance varies by resistance type and persuadee. Belief traces distinguish acceptance from stalling and expose turn-level differences in strategic credit.
E.1 Full Results and Paired Tests
- PROBE-RPO wins 58 scenarios and loses 1 against its initialization, improves terminal state on 76 scenarios, and uses fewer turns (5.7 vs. 8.1).The most common change is PENDING → ACCEPT (40 scenarios); paired tests against prompted baselines are significant, with McNemar p < 10^-5 and Wilcoxon p ≤ 1.3 × 10^-3.Source: §E.1 · pp. 19–20, Table 5 · p. 20
E.2 Breakdown by Hidden Profile and Domain
- Ambivalent Avoidance is the hardest type for PROBE-RPO, with 23.5% acceptance matching Claude Haiku 4.5; acceptance varies more across resistance types (23.5–63.9%) than domain families (40.9–68.4%).On Ambivalent Avoidance, 76.5% of PROBE-RPO conversations end pending or conditional, and none ends in rejection. Of the 62 scenarios that none of the five prompted baselines closed, PROBE-RPO closes 41.9%.Source: §E.2 · p. 20
E.5 Case Study
- In held-out scenario 112, a concrete time-bound protocol earns 71.6% of episode process reward at turn 3 as CA and AC flip while PE stays at −0.64; PE follows at turn 4 after decision handback.Claude Haiku 4.5 reaches belief +1.0 by turn 6, then stalls for turns 7–10 and ends pending. Two turns account for 95.9% of the episode’s total absolute dense reward.Source: Table 8 · p. 21, §E.5 · p. 22
E.3 Cross-Persuadee Grid
- On the fixed 60-scenario subset, PROBE-RPO reaches 75.0% full acceptance against a GPT-5.6 Luna persuadee, versus 38.3% for GPT-5.6 Luna persuading itself.Which resistance type is hardest depends on the persuadee; the appendix’s Ambivalent Avoidance drop is specific to Qwen3-14B.Source: §E.3 · p. 21, Table 7 · p. 21
E.4 Belief Statistics
- Belief changes are comparable across systems from an initial mean of −0.96; final belief ranks systems similarly to full acceptance, and dense reward correlates with the goal bucket (Spearman ρ = 0.735).At least 15% of turns receive negative process reward under every persuader, while probe turn-level increments are less correlated than probe levels.Source: §E.4 · p. 21
F The Process-Reward Weight λ
The process-reward weight balances dense belief shaping against terminal outcomes: a moderate weight performs best, while too much can favor receptiveness over commitment. The section also shows that shaping and acceptance reward different conversational moves, and that training acceptance does not reliably select the strongest held-out checkpoint.
F The Process-Reward Weight λ
- The shaping return is bounded by [−λ, +λ], but at λ = 0.5 its mean is 0.479 for accept and 0.461 for conditional accept, a 0.018 gap.The goal-bucket accept-versus-conditional gap is 2.0; across checkpoints, final belief spans +0.46 to +0.67 while closing efficiency spans 22–70%.Source: F The Process-Reward Weight λ · p. 22
- Move rankings for process reward and P(ACCEPT) are uncorrelated (r = 0.001): implementation detail is best for acceptance (+16.4 pp) but worst for dense reward (−21.4 × 10−3).The diagnostic question is best for dense reward (+26.3 × 10−3) and not significant for acceptance, so the proxy and goal reward disagree about what makes a good turn.Source: F The Process-Reward Weight λ · p. 22
- Training acceptance favored λ = 1, but its best held-out checkpoint was worst; across seven checkpoints, training-to-held-out rank correlation is ρ = 0.57 (p = 0.18).From step 60 to 80, λ = 1 training acceptance rises while held-out acceptance falls from 25.0% to 17.5%; held-out evaluation allows 10 turns versus 6 in training.Source: F The Process-Reward Weight λ · pp. 22–23
- 52.5% best-checkpoint full acceptance at λ = 0.5 exceeds 35.8% at λ = 0 and 30.0% at λ = 1; the shared step-60 rates are 21.7/52.5/25.0%.Across the seven held-out checkpoints, all runs use goal + strategy terminal reward; the table caption says checkpoints move persuadees about equally far but differ in conversion.Source: F The Process-Reward Weight λ · p. 22, Table 9 · p. 23
- At λ = 0.5, 10.3% of groups are dead, and the dense term gives 83% of them a nonzero advantage; its share of advantage magnitude is ≈15%.At λ = 0, dead groups rise from 21.0% to about 35% by step 180; Figure 7 shows dead-group shares, dense-term separation, and the two Eq. 3 terms at λ = 0.5.Source: F The Process-Reward Weight λ · p. 22, Figure 7 · p. 23
- At λ = 1, goal and strategy rewards correlate mildly negatively (r = −0.291), alongside held-out degradation; the audit cannot detect failures outside the judge’s criteria.The correlations are positive at λ = 0 (r = 0.517) and λ = 0.5 (r = 0.529).Source: F The Process-Reward Weight λ · p. 23