PaperlayerView exact revision on arXiv

Source-linked AI summary

$Ψ$-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues

Peixuan Han, Hongyi Du, Jiayu Liu, Yihang Sun, Yutong Liu, Jiaxuan You

arXiv:2606.02754v1cs.LG

TL;DR

Existing personalization research largely treats agents as passive responders, while persuasion evaluations often ignore individualized users. Ψ-Bench evaluates persona-sensitive influencing across realistic scenarios and finds that client profiles improve performance by an average of 18.24%.

  • Problem

    Existing personalization and persuasion research largely evaluates passive response or generic influencing, leaving profile-grounded proactive persuasion insufficiently assessed.

  • Method

    Ψ-Bench evaluates persuasion across three realistic scenarios using simulated clients instantiated from dialogue-history-based user profiles and user-specific judging metrics.

  • Results

    Persona-sensitive influencing remains challenging and varies substantially across 10 frontier LLMs, while providing client profiles yields an average performance gain of 18.24%.

  • Takeaways & Limitations

    The benchmark highlights persona-sensitive influencing and user-specific profile modeling as important directions for evaluating proactive personalized LLM agents.

  • Takeaways & Limitations

    Simulated clients cannot fully represent the breadth of real-world users, and scaling all query-persona combinations is limited by computational costs.

Abstract

from arXiv · show

Personalization is a crucial capability of modern language agents. However, current research primarily positions personalized agents as passive responders to user preferences, limiting their ability to interact with users and provide suggestions or guidance proactively. To systematically evaluate such proactive personalization in realistic interactions, we propose $Ψ$-Bench, a benchmark for assessing LLMs' ability to influence realistic users through conversation. We design three real-world interaction scenarios that involve persuasion in $Ψ$-Bench, and endow simulated clients with personal characteristics through explicit user profiles derived from dialogue histories. We evaluate 10 frontier LLMs on $Ψ$-Bench and find that while most models can produce coherent and reasonable arguments, even state-of-the-art models still leave considerable room for improvement in persuasion. We also find that providing access to client profiles yields an average performance gain of 18.24\%, highlighting the importance of user-specific information for effective persuasion. Overall, our work highlights persona-sensitive influencing as a challenging yet practical direction for evaluating and developing more proactive personalized LLM agents. Codes are available at: https://github.com/Hanpx20/Psi-Bench.

1 Introduction

Ψ-Bench addresses gaps in passive personalization and nonpersonalized persuasion evaluation by testing whether LLMs can proactively influence profile-grounded clients in realistic conversations. Results show substantial limitations in personalized persuasion, while access to client profiles improves performance.

  • Existing personalization methods typically treat LLMs as passive responders rather than agents that proactively support users in real-world workflows.
  • Generic persuasion evaluations often omit individualized target-user profiles and may reflect judges’ default preferences instead of the actual user’s preferences.
  • Ψ-Bench evaluates persona-sensitive influencing across three realistic persuasion scenarios using nearly 700 real-world queries and dialogue-history-based profiles for simulated clients.Client profiles remain hidden from tested LLMs during evaluation, and judgments use objective, traceable LLM-as-a-judge metrics.
  • Less than 67% of the full score is achieved by even GPT-5.1, although most evaluated models generate coherent and reasonable arguments.The evaluation covers 10 frontier LLMs, whose personalized-client persuasion performance varies substantially.
  • 18.24% average performance gain results when models receive access to client profiles, showing the importance of user-specific information for effective persuasion.Profile access consistently improves the performance of all evaluated models.
  • An RL-based profile analyzer infers client profiles from conversations and significantly improves persuasion performance when profiles are hidden from the tested model.

2 Benchmark Construction

Ψ-Bench constructs persuasive dialogues across three real-world scenarios, pairing queries with synthesized persona profiles and evaluating models on conversation quality, personalization, and persuasion effect. Its scenarios target viewpoint change, positive mindset development, and helpful everyday actions, using curated or generated data with scenario-specific grounding.

  • Scenarios: Ψ-Bench comprises three scenarios that test whether LLMs can influence clients’ beliefs, mindsets, or actions.The scenarios are Viewpoint Debate, Psychological Consultation, and Everyday Request.
  • Viewpoint Debate: Viewpoint Debate requires persuading a client to change their opinion on a controversial topic.It uses 2,131 Change My View discussion threads, split into 1,631 training and 500 test threads, with Delta labels indicating successful persuasion.
  • Psychological Consultation: Psychological Consultation casts the model as a therapist helping a client develop a more positive mindset through empathetic, sensitive, and professionally competent conversation.The scenario selects 90 psychotherapy queries, each paired with four therapist responses scored for professionalism.
  • Everyday Request: Everyday Request asks the model to persuade a client to take a helpful action, testing social reasoning and pragmatic persuasion.The final dataset contains 100 profile-grounded request instances generated from 20 categories with five requests per category.
  • Persona-Grounded Dialogues: Each query is paired with a synthesized persona containing personality traits, speaking style, and related characteristics, while the client role-plays it invisibly to the tested LLM.Persona profiles are adapted from PersonaMem-v2 and refined or reconstructed with scenario-specific information.
  • Evaluation: Three LLM-based metrics—Conversation Quality, Personalize Response Level, and Persuasion Effect—are scored by DeepSeek-v3.2 on 9-point scales.They assess general conversation quality, tailoring arguments to clients, and influence over clients’ opinions or behaviors.

3 Preliminary Experiments

Preliminary experiments validate Ψ-Bench’s evaluation framework, showing that the LLM judge aligns with human judgments and that persona-augmented simulated clients support realistic persuasion evaluation. Comparisons with human clients further support simulated clients as a scalable proxy for human evaluation.

  • 3 Preliminary Experiments: The preliminary experiments assess the benchmark design and evaluation framework.These experiments are conducted to validate Ψ-Bench’s design and evaluation methodology.
  • LLM-Judge Validation: Effect in Debate and Quality in Consultation achieve high ROC-AUC against corresponding human annotations, indicating reliable judgment of persuasive dialogues.Personalize also aligns strongly with human signals, suggesting personalized responses are more often associated with favorable human judgments.
  • Client Simulation: The persona-augmented client achieves an AUC of 66.9, outperforming the baseline without persona when reconstructing final user turns in Debate.The experiment compares judge-scored reconstructed dialogues with persuasion outcomes from original conversations.
  • Human Study: Simulated-client conversations produce metrics comparable to human-client conversations across all three metrics, with moderate correlations ranging from 0.4 to 0.5.Participants provide self-described profiles and interact with the same persuader in paired human-client and profile-enhanced simulated-client settings.

4 Benchmarking Results

Across Ψ-Bench, LLMs generally produce high-quality dialogues but remain limited at persuading realistic clients. Personalization is closely associated with persuasion effectiveness, and successful interactions depend on identifying client-specific information and adapting strategies accordingly.

  • Overall performance: Most tested LLMs score above 7 on Quality, producing professional and contextually appropriate responses.Examples include fact-oriented debate responses and empathetic psychotherapy responses.
  • Overall performance: Even the best-performing models average below 6 on Effect, while weaker models achieve around 4 points.Strong dialogue quality therefore does not necessarily produce successful persuasion.
  • Metric analysis: Personalize correlates 0.77 with Effect, slightly exceeding Quality’s 0.75 correlation, and has a more discriminative score distribution.Quality scores are more concentrated, making Personalize an informative indicator of persuasion effectiveness.
  • Long-horizon interactions: Stronger models gain consistently from longer conversations, whereas weaker models saturate earlier; Personalize saturates earlier than Effect.Smaller persuader models also repeat similar arguments more often, lowering Quality and Personalize scores.
  • Qualitative persuasion patterns: Successful persuasion identifies explicit or inferred client-specific signals and adapts its strategy, whereas generic high-Quality responses can remain misaligned.Examples include reframing a lab technician’s perspective, shifting toward pragmatic protest arguments, and avoiding treatments mismatched with a results-oriented client.

5 Ψ-Bench with Profile Analyzer

The section introduces profile analyzers that infer client information from dialogue histories to improve persuasion on Ψ-Bench. An RL-trained analyzer substantially improves downstream Effect and generalizes beyond its training scenario, indicating that dialogue-relevant client information matters more than exact profile reconstruction.

  • Oracle setting: The Oracle setting improves all 10 evaluated LLMs, raising Personalize by 41.19% and Effect by 18.24% on average.The Oracle setting gives the LLM complete client information before dialogue generation.
  • Profile analyzer design: Profile analyzers predict client profiles in JSON format from dialogue histories for proactive personalization.The section evaluates both an off-the-shelf LLM analyzer and a smaller RL-trained analyzer.
  • Profile prediction: Qwen3-4B-RL outperforms all baselines on held-out Debate-domain prediction, achieving 55% average Sim from conversation alone.Sim measures average semantic similarity between predicted and ground-truth profiles across fields such as education level, occupation, and speaking style.
  • Persuasion outcomes: Compared with baseline, DeepSeek-v3.2 and Qwen3-4B-RL achieve 6.4% and 9.1% higher Effect, respectively.Qwen3-4B-RL sometimes approaches or surpasses the Oracle setting with complete ground-truth client profiles.
  • Generalization: The RL-trained analyzer transfers effectively from Viewpoint Debate to other scenarios, showing strong Effect gains despite slightly lower Sim than the zero-shot analyzer.The findings suggest identifying dialogue-relevant information is more important than exactly reconstructing the client profile.

6 Related Work

Prior work has advanced user personalization through training-time and test-time adaptation, while benchmarks assess user-trait inference and preference-tailored responses. Separately, research studies LLMs’ persuasive content generation, persuasive potential, and methods for training persuasive models.

  • Personalization: Personalization research has explored reinforcement learning, parameter-efficient fine-tuning, and test-time methods to adapt agents to specific users.The cited training-time approaches include reinforcement learning and parameter-efficient fine-tuning.
  • Personalization: Existing benchmarks evaluate inferring user traits from sparse interactions and generating responses tailored to users, but primarily frame LLMs as assistants catering to preferences.This framing motivates evaluating more proactive interaction with users.
  • Persuasion: Persuasion research has demonstrated LLMs’ persuasive potential and proposed training persuasive models through human imitation and reinforcement learning.The passage identifies both empirical studies of persuasive potential and training-based approaches.

7 Conclusion

Ψ-Bench evaluates persona-sensitive influencing in LLM agents across three realistic interaction scenarios with simulated clients whose profile-grounded information is hidden from evaluated models. Experiments on 10 frontier LLMs show that this capability remains challenging for current models.

  • Ψ-Bench is a benchmark for evaluating persona-sensitive influencing in LLM agents.
  • The benchmark covers viewpoint debate, psychological consultation, and everyday requests.It uses simulated clients with profile-grounded information hidden from the evaluated models.
  • 10 frontier LLMs were evaluated, and persona-sensitive influencing remained challenging for current models.

Limitations

The benchmark’s simulated clients do not capture the full breadth of real-world users, and scaling all query–persona combinations is limited by computational costs. Future versions may scale the data in quadratic order and analyze further.

  • Client representation: Simulated clients cannot fully represent users’ diverse educational, digital-literacy, socioeconomic, and cultural backgrounds.These dimensions illustrate gaps between the benchmark’s personas and the breadth of real-world users.
  • Benchmark scaling: The benchmark does not enumerate all query–persona combinations because doing so incurs computational costs.Although such scaling is possible in principle, the paper opts not to perform it.
  • Future work: Future versions can scale the data in quadratic order and analyze further.The paper identifies this as a direction for addressing the benchmark’s current scaling limitation.

Ethical Considerations

The work evaluates personalized influence by LLM agents while acknowledging manipulation and misuse risks, especially in high-stakes scenarios. Ψ-Bench is therefore framed as an evaluation benchmark rather than a deployment framework and incorporates safeguards.

  • Risks: The study acknowledges that personalized influence by LLM agents could enable manipulation or misuse, particularly in high-stakes scenarios.
  • Safeguards: Ψ-Bench is designed as an evaluation benchmark rather than a deployment framework to mitigate risks associated with agent influence.
  • Safeguards: The authors adopt safeguards including screening all queries with LLaMaGuard.

A Data Construction Details … B.3 Evaluate Semantic Matching

Ψ-Bench constructs scenario-specific persona profiles from behavioral and linguistic information, then analyzes their distributions and evaluates semantic matching against effective human responses. Although stronger models generally achieve higher Match scores, the modest Match–Effect correlation shows that static similarity is insufficient for persuasion evaluation.

  • A Data Construction Details: Persona templates are adopted from PersonaMem-v2 and tailored to each scenario’s context-relevant features.
  • A.1 Persona Templates: Viewpoint Debate profiles include education, occupation, interests, personality, political views, speaking tone, formality, and clarity.
  • A.1 Persona Templates: Psychological Consultation and Everyday Request profiles additionally include age, gender, religion, family status, and cultural identity.
  • A.2 LIWC scores: Viewpoint Debate profiles use LIWC features covering clout, authenticity, analytic style, tone, six-letter-word percentage, and words per sentence.These features characterize confidence, self-revelation, thinking style, emotional valence, language complexity, and sentence length.
  • A.3 Construction Prompts: DeepSeek-v3.2 generates realistic user profiles from behavioral prompts shown in Figures 9–11.
  • B.1 Persona Statistical Distribution: Client ages approximately follow a normal distribution, while occupations, personalities, and hobbies are diverse and education and political views are more centered.
  • B.2 Human Study Process: The human study procedure is documented with Python-style pseudocode and an annotator webpage screenshot containing the interface and instructions.
  • B.3 Evaluate Semantic Matching: 0.41 is the correlation between Match and Effect, indicating that semantic similarity with effective reference responses is not fully reliable for persuasion evaluation.Match uses BERTScore; effective responses are marked “∆” for Viewpoint Debate and scored ≥4/5 for Psychological Consultation, while Everyday Request lacks human dialogues and is excluded.

B.4 Correlation of Different Metrics … D Cases

The appendix reports metric correlations, RL training settings, inference and evaluation prompts, and representative successful and failed conversations from Ψ-Bench. It also documents profile construction, judging criteria, and client-profile examples used across the benchmark’s scenarios.

  • B.5 RL Hyperparameters: Table 9 lists the hyperparameters used to train the profile analyzer in Section 5.
  • B.4 Correlation of Different Metrics: Figure 7 shows correlations between Quality, Personalize, Match, and the persuasion effect across scenario–model combinations.Each dot represents a combination of scenario and model.
  • D Cases: Profile-construction prompts use browsing topics and LIWC scores, survey information and counseling dilemmas, or rich PersonaMem-v2 persona records.The resulting personality profile is required to be a single JSON object matching the specified structure.
  • D Cases: Figure 15 illustrates client-profile information appended to the persuader’s prompt in the Oracle setting or when using the profile analyzer.The documented profiles include varied backgrounds, preferences, traits, and communication styles.
Loading 2606.02754v1…