Source-linked AI summary
PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios
Ali Zamani, Soroush Esfahanian, Erfan Shahabi, Mohammad Taher Pilehvar, Azadeh Shakery, Yadollah Yaghoobzadeh
TL;DR
Prior social-intelligence benchmarks often rely on synthetic or curated data and evaluate isolated aspects, despite the importance of assessing LLMs in social contexts. PERSO evaluates cognitive and behavioral intelligence using real-world ChangeMyView persuasion dialogues and an LLM-in-the-loop task-construction pipeline. Performance generally improves with model scale, but the strongest model slightly exceeds the best observed human on the behavioral task while the best human remains ahead on the cognitive task.
Problem
Prior benchmarks largely rely on synthetic or curated data and tend to assess isolated aspects of social intelligence, although LLMs increasingly interact with people and other agents.
Source: §1 · p. 1Method
PERSO uses real-world ChangeMyView dialogues and an LLM-in-the-loop pipeline to construct tasks evaluating mental-state inference and persuasion-response selection.
Source: §1 · p. 2Results
Performance generally improves with scale but not universally; the strongest model slightly surpasses the best observed human on behavior, while the best human exceeds the top model on all three cognitive dimensions.
Source: §5 · p. 8Takeaways & Limitations
Emotion is the most difficult dimension for both models and human participants, and reasoning selectively improves emotion inference.
Source: §5 · p. 8PERSO covers English-language, single-turn conversations from one subreddit, leaving generalization to other contexts, languages, and cultures open.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-agent environments. Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals. Persuasion provides a natural testbed in which both aspects are simultaneously at play, yet prior benchmarks typically evaluate them in isolation using synthetic scenarios. We introduce PerSo, a benchmark that evaluates both aspects on real-world persuasion dialogues from Reddit's ChangeMyView community, using an LLM-in-the-loop pipeline to construct the evaluation tasks, validated through human annotation. Our evaluation of diverse LLMs shows that performance generally scales with model size but varies across dimensions. The strongest LLM slightly surpasses the best observed human performance on the behavioral task, while the best human participant remains ahead on the cognitive task.
1 Introduction
The paper frames social intelligence as both inferring others’ mental states and acting appropriately toward social goals. It introduces PERSO to evaluate these complementary abilities in real-world persuasion dialogues, addressing limits of prior benchmarks.
1 Introduction
- Social intelligence comprises cognitive intelligence, inferring others’ beliefs, emotions, and intents, and behavioral intelligence, selecting and executing actions toward social goals.The distinction follows Li et al. (2024).Source: §1 · p. 1
- Prior benchmarks largely use synthetic or curated data and tend to evaluate isolated aspects rather than multiple facets of social intelligence.Assessing social intelligence matters as LLMs enter direct human interactions and multi-agent environments.Source: §1 · p. 1
- Persuasion provides a testbed for both dimensions: anticipating a counterpart’s mental state and judging which response is more persuasive.PERSO evaluates social intelligence across cognitive and behavioral aspects in persuasion scenarios.Source: §1 · p. 1
- PERSO uses ChangeMyView dialogues to forecast persuadees’ beliefs, emotions, and intents, and to identify the response associated with persuasion.The behavioral task uses the persuadee’s delta award as the indication of persuasion.Source: §1 · p. 2
- The authors construct PERSO from real-world persuasion dialogues using an LLM-in-the-loop pipeline for mental-state annotation and question generation.Source: §1 · p. 2
- The targeted tasks jointly evaluate cognitive and behavioral social intelligence on one dataset, and the authors assess models across scales and architectures.The evaluation reveals distinct patterns in social-intelligence performance.Source: §1 · p. 2
2 Related Work
Prior work evaluates cognitive or behavioral social intelligence in varied conversational, narrative, interactive, and simulated settings. PERSO differs by using real human persuasion dialogues to evaluate both dimensions on the same data.
2 Related Work
- Cognitive benchmarks assess mental-state inference across dialogues and social contexts, including beliefs and evolving beliefs, emotions, and intentions.Most ask models to recognize states from observed dialogue, whereas PERSO forecasts the persuadee’s state before the next response.Source: §2 · p. 2
- Behavioral benchmarks assess social-action selection and goal achievement in role-play, constructed narratives, and individual, group, or multi-agent settings.Source: §2 · p. 2
- Unlike primarily constructed or simulated settings, PERSO uses real human persuasion interactions to evaluate cognitive and behavioral intelligence on the same data.Source: §2 · p. 3
3 Method
PERSO builds paired persuasion tasks from real ChangeMyView discussions to assess mental-state forecasting and response selection, then checks the cognitive labels and their alignment with a social-intelligence measure.
3.1 Data Collection
- The dataset contains 200 sampled threads from 872 qualifying threads, requiring contrasting delta outcomes, higher community approval for the successful comment, and OP replies of at least 100 tokens.Persuasive replies are on average 56% longer than non-persuasive replies.Source: §3.1 · p. 3
- An LLM-in-the-loop pipeline generates belief, emotion, and intent questions from each dialogue branch, using the OP’s subsequent reply to derive the gold state and plausible distractors.Three computer science annotators reviewed a random 10% subset of generated questions.Source: §3.1 · pp. 3–4
3.2 Task Definition
- Both tasks use the same ChangeMyView dataset, enabling an integrated evaluation of cognitive and behavioral intelligence on identical real-world persuasion data.Source: §3.2 · p. 4
3.2.1 Behavioral Intelligence Task
- The behavioral task asks models to select the more persuasive of two responses to an OP’s post, without showing the OP’s later replies; delta awards determine the reference outcome.Models rate each response on a four-point scale, and the two scores must differ.Source: §3.2.1 · p. 4
3.2.2 Cognitive Intelligence Task
- The cognitive task forecasts the OP’s belief, emotion, and intent after a persuader’s response using separate four-option questions, while withholding the reply that expresses the gold state.Source: §3.2.2 · p. 5
3.3 Data Validation
- Human answer-selection agreement was strongest for belief (κ=0.911), followed by intent (κ=0.709) and emotion (κ=0.623).The annotators were shown the final OP utterance, unlike evaluated models.Source: §3.3 · p. 5, Table 4 · p. 5
- In the small participant group, behavioral rankings matched TSIS Social Skills rankings, while the ordering of participants B and C differed on Social Information Processing and cognitive dimensions.Participant A ranked highest on Social Information Processing and all three cognitive dimensions.Source: §3.3 · p. 6
4 Evaluation
The evaluation compares diverse LLMs and three human participants on behavioral persuasion judgments and cognitive mental-state inference, finding different performance patterns across the tasks. The strongest model slightly exceeds the best observed human on the behavioral task, while the best human remains ahead on each cognitive dimension.
4.1.1 Model Evaluation
- The model evaluation spans open-source Gemma, Qwen, and DeepSeek families and closed-source Claude and Gemini models, including Qwen reasoning variants and Gemini 2.5 Pro for a self-bias check.Experiments used temperature 0 and top-k 1 for deterministic outputs.Source: §4.1.1 · p. 6
4.1.2 Human Evaluation
- Three human participants completed the same tasks; behavioral evaluation sampled 20 instances and cognitive evaluation sampled 40 instances, each 10% of its full dataset.The participants were also the annotators, and this evaluation preceded gold-label annotation.Source: §4.1.2 · p. 6
4.2 Behavioral Intelligence Results
- Behavioral accuracy generally rises with model scale, but DeepSeek V4-Pro scores 62%, below V3.1 at 67.5% and V3.2 at 69%.DeepSeek models were evaluated without reasoning capability for comparison with other models.Source: §4.2 · p. 6
- Qwen-3.5-397B leads the behavioral task at 76%, ahead of the best closed-source model, Sonnet-4.6, at 71.5%; the model ordering largely persists after length outliers are removed.The robustness check used a 167-instance subset and retained Qwen-3.5-397B as best-performing.Source: §4.2 · pp. 6–7
- 76% accuracy from Qwen-3.5-397B without thinking is 1 percentage point above the best human participant’s 75% on the behavioral task.The authors characterize the model and human as performing at a comparable level.Source: §4.2 · p. 7
4.3 Cognitive Intelligence Results
- Emotion was the most difficult cognitive dimension: Gemma-3-4B and Gemma-3-12B scored 20.5% and 23.25%, respectively, on four-option questions, and human annotator agreement was also lowest for emotion.Emotion scores ranged from 20.5% to 59%, while intent scores clustered from 60% to 80.5%.Source: §4.3 · p. 7
- Cognitive performance generally scales with model size, and Sonnet-4.6 scores 59% on emotion, over 10 percentage points above the second-best model on that dimension.Sonnet-4.6 leads all three cognitive dimensions; the best human participant remains ahead by 2 points on belief, 6 on emotion, and 4.5 on intent.Source: §4.3 · pp. 7–8
4.4 Thinking Analysis
- Enabling thinking improves emotion accuracy for 35B (35.75% to 41.25%) and 397B (41.0% to 48.75%), leaves 4B nearly unchanged, but lowers belief accuracy across all three sizes.Belief accuracy falls from 66.0% to 63.5% for 4B, 71.75% to 69.75% for 35B, and 80.5% to 79.0% for 397B. Thinking offers no clear benefit on intent or behavioral dimensions.Source: §4.4 · p. 8
5 Conclusion
PERSO evaluates LLM social intelligence across cognitive and behavioral dimensions using real-world ChangeMyView persuasion conversations. Performance generally improves with scale but varies by dimension, and the strongest model slightly exceeds the best observed human on behavior while humans lead on cognition.
5 Conclusion
- PERSO uses real-world Reddit ChangeMyView conversations to evaluate cognitive and behavioral social intelligence together in one persuasion benchmark.This addresses prior benchmarks’ reliance on synthetic data or evaluation of the dimensions in isolation.Source: §5 · p. 8
- Across diverse LLMs, performance generally improves with scale but not universally, and emotion is harder to infer than belief or intent for both models and humans.Reasoning selectively improves emotion inference but offers little benefit and sometimes slight harm on other dimensions.Source: §5 · p. 8
- The strongest model slightly surpasses the best observed human on behavior, while the best human remains ahead of the top model on all three cognitive dimensions.The paper identifies cognitive social reasoning, particularly emotion, as an ongoing challenge for current LLMs.Source: §5 · p. 8
Limitations
The human comparison is based on only three participants, and the benchmark’s data covers a specific online community, English conversations, and single-turn exchanges.
Limitations
- Human-model comparisons have limited statistical reliability because the human reference results include only three participants.The authors call for a larger and more diverse participant pool for more definitive comparisons.Source: Limitations · p. 9
- PERSO uses only English-language conversations from ChangeMyView, a community with its own norms, demographics, and argumentative conventions, leaving generalization to other contexts open.The paper specifically identifies other persuasion contexts, languages, and cultural settings as open questions.Source: Limitations · p. 9
- The dialogues are limited to single-turn persuader-persuadee exchanges; longer conversations were not feasible given the volume of qualifying data.The authors identify multi-turn persuasion as a direction for future research.Source: Limitations · p. 9
Ethical Considerations
The paper reports consent, anonymization, and limits on how participant assessments and publicly available Reddit data are used.
Ethical Considerations
- The three volunteer participants provided informed consent and were told they could withdraw at any time.Individual results are anonymized as A, B, and C.Source: Ethical Considerations · p. 9
- TSIS scores are included only as context for interpreting human performance and should not be used to judge individual participants.The scores reflect personal psychological assessments.Source: Ethical Considerations · p. 9
- The Reddit data came from publicly available ChangeMyView posts and comments and excludes private communications.Source: Ethical Considerations · p. 9
A.1 Conceptual Grounding of the Evaluation Tasks
The section connects social intelligence’s understanding and action components to complementary persuasion tasks: forecasting a recipient’s mental state and selecting a persuasive response. These tasks are definition-grounded measures of specific facets, not a complete validation of social intelligence.
A.1 Conceptual Grounding of the Evaluation Tasks
- Cognitive intelligence is operationalized as forecasting the recipient’s belief, emotion, or intent from the initial post and persuader’s response.The recipient’s subsequent reply informs the reference answer but is hidden during evaluation; the task predicts a future expressed state, not the model’s endorsement of the argument.Source: §A.1 · p. 11
- Behavioral intelligence is operationalized as selecting the response expected to achieve the local persuasion goal for a particular recipient.The reference selection uses the recipient’s observed delta award, and the task evaluates action selection rather than generating or executing a response in live interaction.Source: §A.1 · p. 11
- Persuasion brings the facets together through a recipient, a communicative action, and a social goal, while the tasks ask about mental-state understanding and response effectiveness.The example distinguishes the recipient’s resulting mental state from which response achieved the acknowledged view change.Source: §A.1 · p. 12
- PERSO links interpersonal understanding with effective action by evaluating complementary cognitive and behavioral facets on the same persuasion interactions.The section grounds this connection in established definitions of social intelligence and persuasion.Source: §A.1 · pp. 11–12
- The tasks measure theoretically grounded facets in persuasion, but do not establish complete psychometric validation, cognitive independence, internal forecast use, or competence across interpersonal settings.Persuasive success alone does not establish social wisdom.Source: §A.1 · p. 12
A.2 Additional Annotation Results
The additional annotation analyses report consistent multi-annotator agreement estimates and assess the quality of gold answers and distractors. Because ratings concentrated near the top of the scale, the quality analysis uses descriptive statistics rather than agreement coefficients.
A.2 Additional Annotation Results
- Fleiss’ κ and Krippendorff’s α closely match average pairwise Cohen’s κ across dimensions, supporting consistency across agreement measures.Table 6 reports the joint agreement of annotators A, B, and C using two complementary reliability metrics.Source: §A.2 · p. 12, Table 6 · p. 12
- Gold answers were rated as accurately capturing final mental states, while distractors were generally plausible but incorrect.Table 7 reports mean ± standard deviation on a 5-point Likert scale.Source: §A.2 · p. 12, Table 7 · p. 12
- Because most answer-quality ratings were near the scale’s upper end, the authors report descriptive statistics rather than agreement coefficients.They state that the resulting ceiling effect makes agreement-based reliability measures uninformative for this assessment.Source: §A.2 · p. 12
A.3 Behavioral vs. Cognitive Intelligence
Cognitive and behavioral performance is positively associated across most model families, but the dimensions scale differently and can diverge within a family. The results caution against treating a high score on one dimension as a reliable proxy for the other.
A.3 Behavioral vs. Cognitive Intelligence
- Gemma’s behavioral spread is 10.5 percentage points from 4B to 27B, versus approximately 8.4 points across cognitive dimensions; Qwen and Claude show the reverse pattern.Qwen’s cognitive margin averages approximately 15 points versus 12 on behavioral performance; Claude’s behavioral gap is 2.0 points versus approximately 15.7 across cognitive dimensions.Source: §A.3 · p. 13
- DeepSeek V4-Pro ranks lowest in its family on behavioral performance at 62.0%, yet highest on belief at 80.0%, emotion at 39.75%, and intent at 75.0%.This reversal illustrates that family variants can have different performance orderings across the two tasks.Source: §A.3 · p. 13
- Cognitive and behavioral scores tend to rise together across most model families, but their scaling differences indicate distinct capabilities rather than interchangeable measures.The authors caution that a high score on one dimension is not a reliable proxy for the other.Source: §A.3 · p. 13
A.4 Behavioral Answer Distribution Analysis
This section documents analyses of answer-choice patterns, topic coverage, non-LLM baselines, and reply-length robustness, alongside the prompts used for cognitive evaluation. It also presents a worked example of converting a paired persuasion branch into cognitive and behavioral tasks.
A.4 Behavioral Answer Distribution Analysis
- 12 of 15 models select the first behavioral option 45%–58% of the time, while Gemma-27B and DeepSeek-V3.1/V3.2 favor the second at 60%–65.5%.Human choice proportions partly reflect answer correctness because the gold answer was usually second, so human distributions are not clean measures of position bias.Source: §A.4 · p. 13, Figure 2 · p. 14
A.5 Cognitive Answer Distribution Analysis
- Most cognitive-task models select each option 17.5%–33% of the time, but Gemma-4B favors A across belief, emotion, and intent and rarely selects D.Choice-distribution correlations range from strongly negative to strongly positive across models; human-model correlations vary by annotator and dimension, while annotator correlations are usually low or negative. These correlations summarize four option proportions, not agreement on individual questions.Source: §A.5 · pp. 13–14
A.6 Topic Distribution
- Ethics & Morality, Politics & Government, and Society & Culture account for most discussion topics across PERSO’s 200 CMV threads.Source: §A.6 · p. 19
A.7 Non-LLM Baseline Results
- Behavioral surface-feature classifiers achieve 62.5–68.0%, near the 65.5% length heuristic, while SBERT scores 21.00%, 23.25%, and 11.75% on belief, emotion, and intent.The classifiers use five-fold cross-validation on 14 linguistic features, with predictive power largely driven by reply length. SBERT is below the 25% chance level on every cognitive dimension; on intent, the gold answer is least similar in 49.2% of instances, consistent with gold answers abstracting away from dialogue wording.Source: §A.7 · p. 19, Table 9 · p. 19
A.8 Robustness to Reply Length
- After removing 33 extreme-length-difference instances, accuracy decreases modestly but model ordering largely holds and Qwen-3.5-397B remains best.The filter removes successful replies more than 4.97 times longer than their counterparts, leaving 167 instances; the authors conclude reply-length differences have limited effect on evaluation conclusions.Source: §A.8 · pp. 19–20, Table 10 · p. 21
A.9 Prompt Templates
- The cognitive pipeline first generates gold belief, emotion, and intent labels from the full dialogue, then creates three plausible, definitively incorrect distractors for each.Evaluation asks models to infer the persuadee’s next-turn mental state from the body utterance; distractors should match the gold answer’s abstraction level and avoid shortcut eliminations. For behavioral evaluation, models rate two persuader comments on a 1–4 scale, and the higher-scoring response is selected.Source: §A.9 · pp. 20, 22
- The templates generate definitely-wrong, stylistically matched distractors for fixed persuadee belief, emotion, and intent questions.Source: §A.9 · p. 22
A.10 Processed Example from PERSO
- Figure 17 illustrates a paired persuasion branch converted into two cognitive tasks and a behavioral comparison of the persuaders’ responses.It includes the source thread, a cognitive task for each persuader, and the corresponding behavioral comparison.Source: §A.10 · p. 24, Figure 17 · p. 25