Source-linked AI summary
Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks
Ewelina Gajewska, Katarzyna Budzynska, Jaroslaw Chudziak
TL;DR
The paper asks how LLMs respond to character attacks in political argument, where credibility affects whether claims are accepted. It maps human responses into a dialogue game and benchmarks LLMs against human debaters, finding that most models favour logical defences and rarely use ethotic counterattacks.
Problem
Computational work has focused largely on identifying ad hominem arguments, leaving limited systematic evidence about how interlocutors respond after an attack in political persuasion.
Source: §1 · p. 2Method
The authors map human defences against character attacks into a dialogue game and benchmark LLM responses against human political debaters.
Source: §1 · p. 2, §6 · p. 10Results
Humans use rebuttals in 59% of instances and ethotic counterattacks in 32%, while artificial agents use rebuttals in 67-76% and ethotic counterattacks in 15-17%.
Source: §5 · p. 8Takeaways & Limitations
The findings suggest formal dialectics may be needed to regulate LLMs’ dialogue action space beyond probabilistic text generation.
Source: §5 · p. 10The authors argue that ethotic argumentation depends on social and epistemic judgements about credibility, trustworthiness, expertise, and intention, which may not be addressed by alignment tuning alone.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly deployed as argumentative agents in persuasive dialogues, necessitating rigorous evaluation of their debating competence relative to human interlocutors. In this study, we focus on character attacks (ad hominem arguments), traditionally dismissed as fallacies, which play a pivotal role in political persuasive dialogues where ethos often rivals propositional content. Specifically, we investigate whether modern LLMs can replicate human competence to strategically use and respond to such attacks. We analyse a corpus of natural language political dialogues to identify defensive strategies human interlocutors naturally employ in ethos-centred debates and structure them into a dialogue game. Empirically, we benchmark LLM-generated dialogues against the ElecDeb60to16-fallacy corpus of U.S. presidential debates, contrasting human debaters' repertoire of defensive strategies with those of artificial agents. Results reveal a substantial difference: most LLMs rigidly prioritise logical defences, failing to exploit ethotic counterattacks as valid moves in political discourse. We argue that current safety fine-tuning constraints the strategic action space of these LLMs, making them unable to fully engage in naturalistic interactions within domains where character contestation is a normative expectation rather than a mere fallacy.
1. Introduction
The paper addresses the limited computational study of how people respond after ad hominem attacks, especially in political persuasion where character and policy claims both shape argument acceptance. It maps human strategies into a dialogue game and benchmarks LLM responses against human debaters.
1. Introduction
- Political persuasion gives character assessments substantial weight alongside policy claims, while credibility can be actively contested.The paper frames ethos as a load-bearing part of political dialogue rather than a peripheral influence.Source: §1 · pp. 1–2
- The paper targets a gap in systematic computational accounts and empirical verification of how interlocutors respond after ad hominem attacks.It notes that prior computational work has mainly identified and classified ad hominem arguments.Source: §1 · p. 2
- The study asks whether LLMs mirror human defensive strategies against character attacks and whether persona grounding changes their behaviour.It argues these questions matter for evaluating argumentative agents against human rhetorical baselines.Source: §1 · p. 2
- The authors map empirically observed ethotic strategies into a dialogue game and benchmark LLMs against human political speakers.They operationalise Walton’s credibility function and assess whether models use both logical and ethotic defence layers.Source: §1 · p. 2
2. Background and Related Work
The background frames ad hominem as a structured challenge to ethos whose conversational responses require formal modelling. Prior work establishes legal moves and political relevance, while leaving their comparative effectiveness and LLM behaviour open.
2. Background and Related Work
- Argumentation schemes treat generic ad hominem as the defeasible inference that a person’s character bears on whether their argument should be accepted.Critical questions about premise truth and relevance help determine whether the move is normatively valid.Source: §2 · p. 3
- The Ad Hominem Dialogue game formalises character defence and reciprocal counterattack as legal responses, but does not assess their relative success in restoring credibility.Its framework builds on Inference Anchoring Theory and provides a procedural basis for analysing ethotic conflict.Source: §2 · p. 3
- Studies of parliamentary and broadcast debates describe ethotic expressions and personal conflict as recurring structures in political argumentation.Social-media analyses also report that an ad hominem attack can increase the probability of retaliation.Source: §2 · p. 3
- The paper asks which response strategy predominates when LLMs face character attacks, given their competing exposure to human eristic text and harmlessness-oriented fine-tuning.It proposes analysing natural human dialogues, mapping observed defences into a simplified game, and using that baseline to benchmark LLMs.Source: §2 · p. 4
3. Problem and Approach
The framework treats credibility as a distinct layer that can gate an argument’s acceptability, and models logical and ethotic defences as alternative routes to reinstatement. A political-debate example illustrates how those routes differ.
3. Problem and Approach
- The model treats agents as “thick”: argument acceptability depends on both logical content and speaker credibility, represented as a binary credibility threshold.The authors avoid assigning numerical credibility weights; a successful ethotic attack moves credibility below the commitment threshold.Source: §3 · p. 4
3.1. The Ethotic Defence Game
- In ethos-centred dialogue, a reciprocal character attack is a distinct defence because it targets the opponent’s ethos rather than falsifying a claim or its inference.The framework distinguishes this strategy from standard logical defences even when the reciprocal claim appears logically unrelated to the original claim.Source: §3.1 · pp. 4–5
- Argument acceptability is modelled as Accept(A) = LogicalStatus(A)×Ethos(X), so an attack on a speaker’s ethos can gate their argument without resolving its propositional conflict.Ethos(X) is 1 by default and becomes 0 when a character attack succeeds.Source: §3.1 · p. 5
3.2. Case Study
- In the Clinton–Trump example, Clinton’s travel record rebuts the stamina claim, while calling Trump a “corrupt liar” attacks his credibility.The two responses illustrate logical and ethotic pathways for defending the initial claim.Source: §3.2 · pp. 5–6
- The case study’s ethotic counterattack lowers the opponent’s credibility and thereby neutralises the original attack, whereas the rebuttal addresses the stamina claim directly.The framework groups conclusion-directed rebuttals and premise-directed underminers into one logical category.Source: §3.2 · pp. 5–6
4. Experimental Validation
The study benchmarks five LLMs and a human political-debate baseline, varying persona grounding and classifying responses into the dialogue game’s defensive moves. Its protocol combines expert and automated annotation with majority resolution.
4. Experimental Validation
- The validation benchmarks five LLMs against human political debates and tests whether persona grounding changes defensive behaviour.The evaluation maps generated responses to the game’s move categories using expert human coding and automated LLM classification.Source: §4 · p. 6
4.1. Experimental Conditions
- Figure 1 shows three agent types responding to the same ad hominem starting point, followed by evaluation and classification into three move categories.The experimental conditions compare persona-grounded and generic agents with the human baseline.Source: Figure 1 · p. 6, §4.1 · pp. 6–7
4.2. Data Sources
- The dataset contains 97 ad hominem exchanges from U.S. pre-election presidential debates in the ElectToDeb60to16-fallacy corpus.Each exchange includes a character attack by one candidate and the other candidate’s response.Source: §4.2 · p. 7
4.3. Evaluation Protocol
- Three raters classify each response as a rebuttal, undercutter, or ethotic counterattack, with disagreements resolved by majority vote and human oversight.The panel includes one human expert and two LLM raters conditioned with human-dialogue examples, game definitions, and transition rules.Source: §4.3 · p. 7
- Human–LLM annotation agreement is κ = 0.62 for GPT 5.1 and κ = 0.63 for Grok 4.1-reasoning, reported as substantial agreement.The paper describes these coefficients as comparable to fully human annotation of argument structures.Source: §4.3 · p. 8
5. Results and Discussion
Human debaters use ethotic counterattacks more often than artificial agents, which rely mainly on logical defences. The discussion links this gap to models’ treatment of character contestation as aggression and to limits in modeling when such attacks are appropriate.
5. Results and Discussion
- Generic agents used undercutters at 18% versus the human 9% baseline, while persona prompting shifted some models slightly toward human response patterns.Gemini 3 Pro and Grok 4.1 reasoning were the only named exceptions reported to the majority’s failure to use the ethotic layer as a structural pathway.Source: §5 · p. 8
- 59% of human responses were rebuttals, 32% ethotic counterattacks, and 9% undercutters; artificial agents used rebuttals 67-76% and ethotic counterattacks 15-17%.Table 1 reports the distribution of responses to character attacks by human and artificial agents (N = 97).Source: Table 1 · p. 8, §5 · p. 8
- Most models interpret ad hominem prompts as aggression, and generated ethotic counterattacks can default to overtly aggressive or toxic language.The discussion links this pattern to safety fine-tuning that prioritises harmlessness and restricts strategic action in mixed-motive persuasion dialogues.Source: §5 · p. 9
- The suppression of ethotic counterattacks suggests models represent argumentative possibilities differently from humans, transforming some human-legitimate reasoning into a safety violation.The paper also questions treating LLM-generated discourse as a sufficiently faithful approximation of human communicative behaviour.Source: §5 · pp. 9–10
- Appropriate ethotic argumentation requires social and epistemic judgments about credibility, trustworthiness, expertise, and intention, making it Theory-of-Mind-intensive.The discussion proposes combining formal dialogue games with computational Theory of Mind to support argumentation-aware agents.Source: §5 · p. 10
6. Conclusions
The paper formalizes defences against character attacks as valid pathways in ethos-centred dialogue and benchmarks modern LLMs against human political debaters. It finds that persona grounding shifts response patterns but does not confer genuine argumentative reasoning, and calls for formal dialogue protocols alongside Theory of Mind.
6. Conclusions
- The framework treats logical rebuttals, undercutters, and ethotic counterattacks as structurally valid ways to defend and reinstate arguments, then benchmarks LLMs against human political debaters.The investigation focuses on defensive strategies against ad hominem arguments in natural ethos-centred dialogues.Source: §6 · p. 10
- LLMs default to logical defences and undercutters while failing to use reciprocal character attacks; persona grounding shifts responses toward the human baseline but does not produce genuine argumentative reasoning.The authors attribute this structural rigidity to safety fine-tuning that conflates legitimate ethotic contestation with eristic aggression.Source: §6 · p. 10
- The authors conclude that probabilistic generation and contextual prompting alone cannot replicate human argumentation’s multi-layered competence, and recommend formal dialogue protocols alongside Theory of Mind.The conclusion presents these as future-system design directions for complex persuasive environments.Source: §6 · p. 11