Source-linked AI summary
Retrieving Relations, Detecting Fallacies: A RAG Approach to Political Debate Analysis
Deborah Dore, Greta Damo, Elena Cabrio, Serena Villata
TL;DR
Political fallacy detection requires world knowledge and argumentative context beyond surface text. The paper guides retrieval with support and attack relations, then evaluates the approach across models and configurations, finding higher macro-F1 for both detection and classification.
Problem
Fallacy analysis needs world knowledge and argumentative relations, but prior structural information was typically used only as static classifier features.
Method
The paper uses a guided RAG methodology that dynamically retrieves political evidence using argumentative relations and combines it with debate context for prediction.
Results
Retrieval raises the best macro F1 from 0.772 to 0.864 for detection and from 0.653 to 0.725 for classification.
Takeaways & Limitations
Argumentatively guided external knowledge improves fallacy detection and classification across the evaluated political-debate setting.
Takeaways & Limitations
The winning retrieval configuration is optimized for the three strongest baseline models, so it is only reasonable for the remaining models.
Abstract
from arXiv · showhide
Fallacies are arguments that employ invalid reasoning, making their automatic detection critical in sensitive contexts such as high-stakes political debates, where public opinion is shaped. Spotting a fallacious argument requires contextual knowledge beyond its pure surface text. This entails world knowledge pertaining to the subject matter under discussion, as well as knowledge of the relationships that exist between arguments within the argumentative discourse. Prior work on fallacy analysis has shown that argumentative discourse structure can beneficially improve classification performance. However, such structure is typically encoded only as static classifier features, limiting its flexibility. Building on this intuition while addressing this limitation, we introduce a guided retrieval-augmented methodology for fallacy detection and classification that leverages argumentative relations of support and attack to dynamically steer the extraction of relevant documents. We evaluate our approach on the ElecDeb60to20 benchmark across 42 retrieval configurations and 14 models, performing retrieval over a 15GB knowledge base of collected political-related documents. Our approach improves macro-F1 up to 0.864 for fallacy detection and up to 0.725 for classification over non-retrieval baselines. These results show that incorporating external knowledge significantly enhances fallacy detection and classification when retrieval is argumentatively guided.
1 Introduction
Fallacy detection in political debate requires contextual knowledge beyond surface text, including world knowledge and argumentative relations. The paper proposes using support and attack relations to guide external retrieval rather than treating structure only as static features.
- Fallacies can manipulate public opinion despite appearing superficially valid, making their recognition important for critical evaluation of political information.
- Argumentative structure alone is often insufficient because fallacy analysis also requires world knowledge beyond the argument’s surface text.
- The proposed method dynamically uses support and attack relations between argument components to steer targeted external-knowledge retrieval.
- The evaluation covers 42 retrieval configurations and 14 models using a 15 GB political-domain knowledge base.
2 Related Work
Prior fallacy work uses semantic, argumentative, contextual, and model-generated knowledge, while retrieval from external corpora has not been used for these tasks. This paper investigates argumentatively guided retrieval of political evidence.
- Earlier studies combine semantic and argumentative features or contextual information for fallacy and propaganda detection.
- Recent approaches incorporate background knowledge, explainable reasoning, or implicit context through language models.
- Existing added knowledge is hand-crafted or model-generated rather than retrieved from an external corpus.
- RAG has extended from knowledge-intensive generation to discriminative tasks, including classification, and retrieval supports fact checking and argumentation.
- This work adapts RAG to retrieve political knowledge relevant to each instance’s argumentative structure and condition predictions on that evidence.
3 Methodology
The method is a four-stage pipeline that generates structure-informed queries, retrieves external evidence, combines it with debate context, and predicts fallacy outcomes. Query variants progressively add related components and explicit support or attack labels.
- Fallacy detection determines whether a sentence is fallacious, whereas classification assigns a known fallacy to exactly one of six types.
- The pipeline comprises query generation, retrieval, input generation, and model prediction.
- Query generation progresses from the target sentence to variants incorporating argumentative relations and relation labels.
- Relation labels are verbalized as clauses such as related supports or attacks target, directly encoding relation direction.
- Retrieved evidence is combined with local debate context into one model input for prediction, and the pipeline can extend to domains with available argumentative relations.
4 Resources for Fallacy Analysis of Political Debates
The study builds a temporally grounded political knowledge base and evaluates retrieval components for fallacy analysis. Its resources combine legislation, people, geography, history, definitions, and labeled exemplars, indexed for multiple retrieval strategies.
- Dataset: ElecDeb60to20 provides political-debate annotations for argumentative components, support and attack relations, and six fallacy types.
- Knowledge base: The knowledge base contains six targeted collections covering legislation, people, geography, history, fallacy definitions, and labeled exemplars.
- Temporal grounding: History records are annotated by year so temporal context is retained after chunking, while temporal filtering can restrict retrieval to years preceding each debate.
- Knowledge base: Exemplar records are checked for overlap with evaluation instances to prevent label leakage during retrieval.
- Knowledge base statistics: Bills dominate collection volume, while geography and people contribute the next-largest document counts; smaller collections provide targeted temporal and fallacy-specific knowledge.
- Indexing: Documents are normalized with retrieval metadata, chunked to at most 1,000 tokens with 10% overlap, and indexed separately for each embedding model.
- Retrieval evaluation: The study compares dense and lexical retrievers, similarity and hybrid search, three query formulations, and temporal constraints across 42 configurations.
5 Experimental Setting
The experiments evaluate balanced fallacy detection and imbalanced six-type classification across encoder and decoder models, retrieval settings, and five-seed macro-F1 measurements.
- Datasets and tasks: Detection uses 2,612 training, 327 validation, and 327 test sentences after balancing Fallacy and NoFallacy labels.
- Models: The study evaluates ten encoder-only models and four decoder-only LLMs, using sentence and speech-turn context with retrieved context where applicable.
- Experimental design: The study compares 43 configurations per task, comprising 42 retrieval configurations plus a no-retrieval baseline, with staged model and hyperparameter selection.
- Evaluation: The benchmark reports Macro-F1 mean and standard deviation over five seeds, treating macro F1 as primary because it weights every class equally.
6 Results
Retrieval improves encoder performance on both tasks, with the strongest gains from argumentative structure and Sentence-RooseBERT retrieval, while decoder-only models degrade with retrieved context.
- Detection: 0.864 macro F1 is the best detection result, improving 0.100 over ModernBERT’s baseline and 0.092 over the best baseline model.
- Detection: Retrieval improves every tested encoder from +0.016 to +0.100 macro F1, except NeoBERT, which fails to converge reliably.
- Model comparison: Decoder-only models lose 0.126–0.193 macro F1 after retrieved context is added, despite competitive baselines.
- Classification: 0.725 macro F1 is the best classification result, improving 0.148 over ModernBERT’s baseline and 0.072 over the best baseline model.
- Error analysis: Temporal filtering helps detection but hurts classification, indicating different informational needs for fallaciousness detection and fine-grained type classification.
- Error analysis: Retrieval redistributes classification errors: Slogans recall rises from 0.25 to 0.75, while Ad Hominem precision drops from 0.86 to 0.69.
- Retriever comparison: Sentence-RooseBERT consistently outperforms other retrievers, likely because parliamentary pretraining aligns with debate discourse.
7 Ablation Study
Ablations show that performance depends on the combination of argumentative queries, dense retrieval, hybrid search, reranking, and task-specific temporal filtering.
- Detection ablations: 80.2 macro-F1 versus 75.4 without retrieval is achieved by the full detection pipeline, while removing relations from the query causes a 7.2-point decline.
- Detection ablations: Structured queries outperform sentence-only retrieval: BM25 reaches 75.7 with structure but 72.7 with only the target sentence.
- Detection ablations: Dense retrieval is necessary but insufficient for detection, since replacing the dense retriever costs 1.6–2.4 points while removing argumentative structure removes the gain.
- Classification ablations: 71.8 macro-F1 versus 65.3 for the baseline is achieved by the full classification pipeline, but alternative retrievers reduce performance by 11.7–13.8 points.
- Classification ablations: For classification, sentence-only queries reach 66.4, whereas sentence+relations queries reach 61.0, showing that gains depend on a specific retriever-query pairing.
8 Conclusion
The paper presents argumentatively guided RAG for political fallacy analysis, combining external and structural knowledge and achieving improved detection and classification performance.
- Conclusion: The method combines external knowledge with argumentative structural knowledge for fallacy detection and classification.
- Conclusion: Detection macro F1 rises from 0.772 to 0.864, while classification rises from 0.653 to 0.725 under retrieval.
- Conclusion: Retrieved context is presented as a step toward interpretable fallacy explanation alongside prediction.
Limitations
The evaluation is constrained by model-selection and domain scope. Retrieval configurations were selected using only a subset of models, and the knowledge base and experiments focus on English-language U.S. presidential debates.
- 42 retrieval configurations were explored only with the three strongest baseline models per task, so the selected configuration may not be optimal for the remaining models.The winning configuration was applied to all 14 models, but its optimality is established only for the three models used during configuration search.
- All experiments use one English-language corpus of U.S. presidential debates from 1960 to 2020.The associated knowledge base is specifically constructed from U.S. legislative, census, and congressional-biography sources.
A Additional Experiments
Additional experiments test whether relation-aware retrieval generalizes from ElecDeb60to20 to propaganda detection. Using automatically recovered relations and a re-tuned dense retriever, the approach improves substantially over the baseline on PROPAGANDA.
- Dataset and task: The PROPAGANDA corpus contains news sentences annotated with 18 propaganda techniques, and its authors report a sentence-level macro F1 of 60.98.The experiment evaluates binary sentence-level propaganda detection rather than fragment-level technique classification.
- Relation transfer: LIARArg supplies 2,832 claims paired with justifications and annotated support, attack, partial support, and partial attack relations.The relation classifier is trained on this relation-annotated corpus before transfer to PROPAGANDA.
- Relation transfer: Partial LIARArg labels are merged into support and attack, producing a three-way support/attack/no_relation classification task.Models are selected by macro F1 averaged over five seeds, with results reported in Table 4.
- Relation-guided retrieval: Llama is used to label PROPAGANDA sentence pairs, whose predicted support and attack relations form retrieval queries as in the main experiments.Each annotated sentence is paired with every other sentence from its article.
- Relation-guided retrieval: Sentence-BERT is the best dense retriever after testing the ElecDeb60to20-best configuration with all three dense models on PROPAGANDA.The protocol otherwise follows the procedure described for the main evaluation.
- Results: 10.4 points of macro F1 improvement over the baseline are obtained on PROPAGANDA, while longformer with retrieval reaches 85.5 Macro F1 versus 75.1 for the best baseline Llama.The difference is significant at p < 0.001 with d = 5.471; Table 5 reports baseline, RAG, and ∆ values.
B Implementation Details: Prompts and Hyper-parameters
The appendix documents prompts, model settings, and task-specific label definitions. It covers hyperparameters for the evaluated tasks and exact output constraints for fallacy, propaganda, and relation classification.
- B Implementation Details: Prompts and Hyper-parameters: Appendix B reports the prompts and hyperparameters used in the work, with B.1 covering best-model settings and B.2 covering LLM prompts.Table 6 summarizes task-specific hyperparameters across baseline and RAG settings using five seeds.
- B.2 LLMs Prompts: The fallacy-detection prompt asks whether a political-debate sentence contains a fallacy and requires exactly Yes or No.It defines a fallacy as a deceptive argumentative move that appears valid but violates correct-reasoning principles.
- B.2 LLMs Prompts: The fallacy-classification prompt restricts outputs to six named fallacy types and requires exactly one type name.The prompt states that every sentence contains exactly one of the six types, so None is not valid.
- B.2 LLMs Prompts: The classification prompt defines Ad Hominem as attacking an opponent’s character instead of addressing the argument.Examples include name-calling and labelling, tu quoque, and bias.
- B.2 LLMs Prompts: The prompt definitions include appeals to non-experts or popularity, emotional language, false causation, and exaggerated slippery-slope outcomes.These correspond to Appeal to Authority, Appeal to Emotion, False Cause, and Slippery Slope descriptions in the prompt.
- B.2 LLMs Prompts: The propaganda-detection prompt asks for exactly Yes when a sentence contains a persuasive propaganda technique and No otherwise.Examples include loaded language, name calling, exaggeration, appeal to fear, and flag waving.
- B.2 LLMs Prompts: The LIARArg relation prompt classifies Sentence 2 toward Sentence 1 as Support, Attack, or No Relation.Support and Attack correspond to reasoning in favor of or against Sentence 1, while No Relation indicates no argumentative connection.