Source-linked AI summary
Evaluating the Capabilities of LLMs for Persuasive Dialogue
Jordan Robinson, Angus R. Williams, Katie Atkinson, Anthony G. Cohn
TL;DR
The paper examines whether subjective perceptions of LLM persuasiveness correspond to logical argumentative strength. It introduces PERSUASIO, a formal multi-agent dialogue platform, and evaluates human and LLM debates using both adjudication and crowdsourced judgements. LLMs dominate subjective rankings but perform substantially worse under logical evaluation, while the exploratory design and self-adjudication introduce important interpretive limits.
Problem
It remains unclear whether perceived dialogical persuasiveness corresponds to logical persuasiveness, because evaluations rarely measure rhetorical appeal and argument structure jointly.
Method
PERSUASIO implements formal persuasion dialogues and compares human and LLM interlocutors using automated adjudication and crowdsourced pairwise judgements.
Results
All nine LLM configurations ranked highest subjectively but underwent substantial rank inversions under logical evaluation, where human debaters were more competitive.
Takeaways & Limitations
Evaluation protocols relying exclusively on human preference risk conflating persuasive appeal with argumentative validity in deliberative domains.
Takeaways & Limitations
The evaluation is exploratory and lacks preregistered hypotheses, inferential uncertainty estimates, and null-distribution comparisons; self-adjudication may also introduce model-dependent concession biases.
Abstract
from arXiv · showhide
Large language models (LLMs) can generate apparently highly persuasive text, but does sounding persuasive mean arguing well? We introduce \textsc{Persuasio}, a multi-agent dialogue platform grounded in a formal argumentation-based theory of persuasion dialogues that adjudicates logical winners during free-text debates. Using this system, we generated 192 debates on a UK political topic between humans and LLMs, and evaluated 22 interlocutors through both automated adjudication and 9,702 crowdsourced pairwise judgements across 1{,}386 annotation instances. We observed a consistent decoupling between subjective and formal persuasiveness: LLMs dominated the subjective ranking yet performed substantially worse under argumentation-theoretic adjudication, where humans remained competitive. Multi-agent and retrieval-augmented variants further widened this divergence. These findings reveal a systematic gap between rhetorical fluency and formal argumentative strength in LLM-based persuasive dialogues.
1 Introduction
The paper asks whether perceived persuasiveness aligns with logical persuasiveness and introduces PERSUASIO to compare them in controlled human–LLM debates. Its evaluations reveal a systematic divergence between rhetorical appeal and formal argumentative strength.
- Existing evaluations rarely jointly measure rhetorical appeal and argument structure, leaving the relationship between fluency and inferential rigour unclear.
- PERSUASIO implements formal persuasion dialogues with turn-taking, commitment tracking, and termination conditions for automated logical adjudication.
- The study generated 192 UK political debate transcripts involving 22 interlocutors across human–human, human–model, and model–model configurations.
- Persuasiveness was evaluated through formal logical adjudication and 9,702 crowdsourced pairwise judgements across 1,386 samples, with both modeled using Bradley–Terry models.
- All 9 LLM configurations ranked highest subjectively but dropped substantially under logical evaluation, exposing a systematic gap between the two dimensions.
2. We release a corpus of 192 political debate
The paper releases a corpus of political debate transcripts involving human and LLM interlocutors, together with crowdsourced annotations and annotator demographics. It uses these resources to compare logical and subjective persuasiveness over the same interlocutor set.
- The released resources include debate transcripts involving 22 distinct human and LLM-based variants, 9,702 crowdsourced annotations, and annotator demographic data.
- The evaluation compares logical and subjective persuasiveness using Bradley–Terry rankings inferred over the same 22 interlocutors.
2 Related Work and Background
Prior work studies LLM persuasion, persuasive dynamics, and argument quality through separate rhetorical or computational lenses. The paper situates its approach within formal argumentation-based persuasion dialogues.
- LLMs have been studied for generating personalised messages, exploiting cognitive biases, and conducting effective multi-turn interactions.
- Persuasiveness research uses debate platforms, online forums, and computational argumentation features to study persuasive dynamics and rank persuasive texts.
- Formal persuasion dialogues resolve opinion conflicts through regulated exchanges using locutions, turn-taking protocols, and commitment stores.
3 Multi-Agent System and Dialogue Game Design
PERSUASIO structures human and LLM debates as formal, turn-based persuasion dialogues, with base, multi-agent, retrieval-augmented, and human configurations. Shared classification and commitment updates support controlled comparison and logical analysis.
- Shared Architectural Components: PERSUASIO implements turn-based exchanges between two speakers using a base LLM, structured MAS, optional RAG, or human participant, with formal termination conditions.
- Shared Architectural Components: The protocol constrains each turn through permissible replies while allowing multi-move, multi-reply, non-immediate, and non-deterministic argumentative exchange.
- Shared Architectural Components: All agents classify utterances and update commitments, adding, accepting, or removing propositions according to formal move effects.
- Structured Multi-Agent System: The base agent generates protocol-compliant responses from claims, commitments, and dialogue history without multi-agent decomposition or RAG.
- Structured Multi-Agent System: The MAS divides response generation among protocol reasoning, candidate generation, persuasive selection, disambiguation, and commitment updating agents.
- Graph-Based Retrieval: RAG combines SBERT cosine-similarity retrieval with political alignment constraints and graph relations to supply aligned examples for persona construction and runtime responses.
- Human Sub-Graph: The human sub-graph accepts free text and applies the same utterance classification and commitment updates as artificial agents for controlled cross-agent comparison.
4 Dialogue Transcript Generation
The study generated controlled UK political debates among humans and LLMs using fixed viewpoints, a formal dialogue protocol, and multiple model configurations. The design included base, multi-agent, and retrieval-augmented systems, with reproducible settings and human participants.
- Corpus and participants: All dialogues addressed one UK political topic to control for domain variation.The topic concerned trade-offs of government intervention across healthcare, immigration, and welfare.
- Corpus and participants: Thirteen undergraduate Computer Science students debated from assigned left- and right-wing perspectives without knowing whether opponents were human or artificial.Opponent identities were disclosed after the workshop under the stated ethics protocol.
- Dialogue design: Dialogues assigned political personas, began with stance-aligned claims, and constrained responses through a turn-taking protocol.The second speaker argued against the initial claim, ensuring closure under negation.
- Model configurations: Each LLM used near-deterministic settings and was tested with BASE, MAS, and MAS with RAG configurations.The settings were temperature=0, top_p=1, and seed=123; content filters were removed for politically sensitive debates.
- Model configurations: GPT-4o and Grok-3 were selected for the full human evaluation to increase ideological diversity across providers.The selection aimed to reduce the risk that rankings reflected a single provider’s alignment regime.
5 Quantifying Logical and Subjective Persuasiveness
The evaluation modeled logical and subjective persuasiveness from complementary debate outcomes and human transcript judgements. Bradley–Terry rankings and distance measures enabled comparison across the 22 interlocutors.
- Evaluation sources: Logical persuasiveness came from formal debate outcomes, while subjective persuasiveness came from crowdsourced pairwise transcript judgements.The two sources were used to construct complementary win matrices.
- Sampling: 231 unique interlocutor pairings each received six balanced excerpt pairs spanning left- and right-wing perspectives.Excerpts contained four consecutive dialogue turns and matched Speaker A’s ideological position across the pair.
- Evaluation sources: 1,386 transcript-pair samples were annotated independently by seven crowdworkers each.Annotators compared Speaker A across excerpts and also reported confidence on a five-point Likert scale.
- Win matrices: 192 debate outcomes formed the logical win matrix, with ties contributing 0.5 wins to each interlocutor.The matrix recorded pairwise wins among n=22 interlocutors.
- Ranking model: Bradley–Terry models fitted with I-LSR produced latent strengths used to rank interlocutors.Higher θ_i values indicated greater estimated strength, with pairwise preference probabilities derived from the parameters.
- Ranking model: Ranking comparisons used normalized Kendall’s τ distance, Spearman’s footrule distance, and ordinal Krippendorff’s α.The metrics assessed distance and agreement between subjective, logical, and subgroup rankings.
6 Experimental Evaluation
The experiments found a clear separation between subjective and logical persuasiveness. LLMs led subjective rankings, whereas humans remained competitive under formal adjudication, and demographic aggregates were stable subjectively but divergent logically.
- Subjective versus logical rankings: All nine LLM configurations ranked in the top nine subjectively, while the highest-ranked human was tenth.Model-generated excerpts were consistently judged more persuasive than human-produced excerpts.
- Subjective versus logical rankings: A human debater ranked first logically, and humans occupied six of the top ten logical positions.Logical persuasiveness was not dominated by agentic variants.
- Subjective versus logical rankings: GPT-4o_MAS_RAG and Grok-3_MAS_RAG fell to mid-table under logical evaluation despite leading subjectively.Mistral-Medium_BASE instead rose sharply under logical criteria.
- Agentic enhancements: MAS and RAG consistently improved subjective persuasiveness but affected logical persuasiveness inconsistently and sometimes negatively.The enhancements increased fluency, structure, and evidential density without reliably improving inferential coherence.
- Interlocutor variability: Humans showed greater variance across both rankings, with strong humans matching or surpassing LLMs logically.LLM configurations were more tightly clustered, especially subjectively.
- Ranking divergence: Global subjective and logical rankings had ordinal α = 0.410, indicating moderate agreement alongside substantial positional shifts.The rankings captured overlapping but distinct aspects of persuasion.
- Demographic stability: Every demographic subgroup closely reproduced the subjective ranking but diverged substantially from the logical ranking.Right-leaning annotators aligned more closely with the logical ranking, though the passage advises caution because subgroup sizes differed.
- Discussion: The authors characterize LLMs as favoring rhetorical polish over formal argumentative rigour.Formal adjudication rewards different properties from those driving human perceptions of persuasiveness.
7 Conclusions and Future Work
The paper presents PERSUASIO as a formal platform for comparing logical and subjective persuasiveness in human–LLM debates. Its central finding is that rhetorical appeal does not reliably translate into logical soundness, motivating evaluation and training approaches that assess both.
- Conclusions: PERSUASIO implements a formal argumentation-based persuasion dialogue system for controlled exchanges between humans and LLMs.The platform was applied to UK political discourse.
- Conclusions: The study collected 192 debates and evaluated 22 interlocutors using logical adjudication and 9,702 crowdsourced pairwise comparisons.The two axes were logical persuasiveness and subjective persuasiveness.
- Main finding: All nine LLM configurations led subjective rankings but showed substantial logical rank inversions, where humans were more competitive.Fluency, organisation, and assertive presentation did not reliably produce logically consistent or well-supported arguments.
- Implications: Preference-only evaluations risk conflating persuasive appeal with argumentative validity in deliberative domains.The passage specifically identifies legal reasoning and public policy as relevant domains.
- Future work: Future work should test logical quality signals, hybrid human–AI systems, and larger balanced samples for demographic analyses.The proposed signals include prompting or post-training approaches that align rhetorical and logical dimensions.
Limitations
The study identifies limitations in its ranking model, exploratory design, sample composition, and generalisability. These constraints affect statistical interpretation and the breadth of conclusions about persuasiveness.
- Exploratory Design: The evaluation is exploratory and does not establish whether observed ranking agreements or disagreements differ from chance.The study reports rank-distance and agreement metrics without preregistered hypotheses, p-values, confidence intervals, or null-distribution comparisons.
- Small Ranking Space: With n=22 interlocutors, rank-based metrics have limited discriminative power and make fine-grained comparisons difficult.Small changes in win matrices can notably shift rankings, particularly among interlocutors with similar BT strength estimates.
- Demographic Subgroup Analysis: Subgroup rankings may reflect uneven sample sizes and uncorrected multiple comparisons rather than genuine demographic effects.The authors treat demographic patterns as hypotheses requiring larger, balanced confirmatory studies.
- Annotator Population: The Prolific annotator sample may not represent the broader population, while cultural, linguistic, educational, engagement, and debate-literacy factors may add noise.The study notes that some human debaters were unfamiliar with left- and right-wing distinctions.
- Logical Persuasiveness: Logical outcomes rely on participant self-adjudication rather than an independent neutral model, and sparse pairings may bias BT estimates.Different models may vary in willingness to concede, while a separate adjudicator could produce different outcomes.
- BT Model Assumptions and Tie Handling: The BT model assumes one transitive, context-independent strength parameter per interlocutor, although persuasiveness may depend on topic, opponent, or context.Regularisation and half-win tie handling introduce additional assumptions that may obscure genuine differences or treat uncertain ties as uninformative.
- Generalisability: The findings are specific to one UK political topic, selected interlocutor configurations, and the study’s evaluation protocols.Generalisability to other domains, languages, debate formats, agentic sub-graphs, or LLM architectures remains open.
Ethical Considerations
The study describes ethical approval, informed consent, participant protections, and formal dialogue safeguards. Its evaluation uses structured protocols that regulate speech, commitments, and termination.
- Ethical approval: The study received ethical approval from the University of Liverpool, University of Leeds, and Alan Turing Institute.The approved process covered removal of content filters and the deliberate choice of political debate topic.
- Human participants: Thirteen human debaters gave informed consent, were paid £18.60 per hour, and could withdraw from distressing exchanges.Participants were not told whether opponents were human or LLM-based until each dialogue concluded.
- Crowdsourced annotation: Prolific annotators provided informed consent, and dialogue excerpts were screened for offensive content before annotation.Submissions failing mandatory attention checks were returned, while other responses were treated as subjective judgements.
- Dialogue safeguards: The formal persuasion framework specifies legal replies, turn-taking, commitment effects, and outcome rules.Communication locutions include claim, why, concede, retract, since, and question.
- Commitment handling: Commitments are updated after each turn, with concede or retract moves potentially terminating the dialogue and determining the winner.These operations enforce consistency around publicly undertaken propositions and initial claims.
- Automated processing: The system assigns exactly one context-sensitive dialogue-move label per sentence before downstream protocol enforcement and commitment updates.Classification distinguishes claim, since, why, question, concede, and retract using preceding turns.
C.6.2 Label-Set Ablations
Label-set ablations show that GPT-4o’s utterance classification improves when the confusing interrogative categories why and question are removed. The reduced four-label set was most reliable.
- Ablation results: Removing why increased mean F1 to approximately 0.76–0.78 with narrower confidence intervals.The result indicates substantial confusion involving why, likely because it overlaps with question.
- Figure 7: Figure 7 compares GPT-4o’s ablated label-set F1 scores with green full-label-set baseline scores.The figure provides the reference baseline needed to interpret each reduced-label configuration.
- Ablation results: Excluding question improved F1 to 0.72–0.76, suggesting symmetric confusion between the two interrogative categories.The ablation directly tests the contribution of question to classification difficulty.
- Ablation results: Removing both why and question yielded stable performance around 0.74–0.76, making claim, since, concede, and retract the most reliable label set.Across ablations, F1 exceeded the full configuration, supporting confusion between why and question as the main source of degradation.
- Experimental choice: Despite improved ablation reliability, the experiments retained the full label set because why and question occurred in human dialogues.The ablations instead informed interpretation of anticipated misclassification.
D.1 Task Overview
Annotators compared the persuasiveness of Speaker A across two same-topic debate transcripts. They made subjective pairwise choices based only on transcript content and also rated confidence.
- Comparison setup: Each annotation instance presented two debate transcripts on the same topic, with Speaker A representing the same ideological position in both.This design focused comparisons on argumentative effectiveness rather than speaker identity.
- Subjective evaluation: Participants were told that, apart from attention checks, there were no correct or incorrect answers and that honest subjective judgements were required.This instruction framed the evaluation as a subjective comparison rather than an objectively scored test.
- Instructions: Annotators were instructed to base decisions solely on transcript content and compare only the two Speaker A participants.The procedure included reviewing the topic and both debates before making the comparison.
- Judgement task: Annotators selected which Speaker A was more persuasive overall, or whether both were equally persuasive.They then reported confidence in their choice.
- Task structure: The task required two questions: a persuasiveness choice and a confidence rating.Responses also included mandatory attention checks.