Counter with Evidence! A Multi-Agent Memory Efficient Reasoning Framework for Hate Category Informed Counterspeech Generation

Sujoy Nath; Aswini Kumar; Tanmoy Chakraborty

Source-linked AI summary

Counter with Evidence! A Multi-Agent Memory Efficient Reasoning Framework for Hate Category Informed Counterspeech Generation

Sujoy Nath, Aswini Kumar, Tanmoy Chakraborty

arXiv:2608.23152v1Aug 24, 2026cs.CL

TL;DR

Prior counterspeech methods often treat hate speech as homogeneous despite its varied semantic forms, limiting category-specific responses. FIRE decomposes hate into categories, retrieves or reasons over supporting evidence, and generates grounded counterspeech with compact agents. The framework substantially outperforms established baselines while matching larger state-of-the-art models, although its scope and tool dependencies impose limitations.

Problem
Prior counterspeech methods often emphasize stylistic control while treating diverse hate speech as homogeneous, despite different abuse types requiring distinct logical refutations.
Method
FIRE decomposes hate speech into five categories, uses specialized agents and retrieved evidence for reasoning, and synthesizes targeted counterspeech with compact models.
Results
FIRE substantially outperforms established baselines and matches larger state-of-the-art LLMs while using compact agents.
Takeaways & Limitations
Decomposing generation into intent analysis and grounded response synthesis provides an efficient approach to safe, context-aware counterspeech.
Takeaways & Limitations
FactualCS is not exhaustive, small models may limit reasoning on ambiguous language, and web-search failures can propagate errors into final responses.

Abstract

from arXiv · show

Counterspeech effectively neutralizes the impact of online hate. Although prior work explores automated counterspeech generation, it largely emphasizes stylistic control while treating hate speech as homogeneous, overlooking that distinct forms of abuse require fundamentally different counterspeech strategies. To address this gap, we introduce FIRE (Factuality Informed Multi-Agent Reasoning Framework) that first decomposes hate speech into one of the five distinct categories (misinformation, stereotype, conspiracy, dehumanizing, non-factual), and then maps it to a targeted counterspeech style. To facilitate FIRE, we curate FactualCS, a novel dataset of $4,784$ instances that provides the annotations regarding hate categories, reasoning traces, and evidence mappings, which are critical elements for grounded generation that are missing in prior work. A comprehensive evaluation across $28$ baseline configurations demonstrates that FIRE significantly surpasses existing methods, despite using compact agents ($<$2B). FIRE achieves a $\sim$ $12 \%$ and $\sim$ $11 \%$ improvements in factual and category-specific accuracy respectively, while simultaneously reducing toxicity by $\sim$ $11 \%$ relative to the strongest baselines. Further human evaluation confirms that responses generated by FIRE are significantly preferred over the strongest baselines, underscoring its effectiveness for real-world deployment. These findings show that decomposing the underlying intent of hate speech is essential for generating safe, effective, and contextually precise counterspeech.

1 Introduction

Existing counterspeech systems often treat diverse hate as homogeneous, while FIRE decomposes abuse into categories and uses specialized, evidence-grounded reasoning to generate targeted responses.

Automated counterspeech is important because manual intervention cannot scale to the volume of online content.
Different hate categories require distinct logical refutations, but stylistically controlled methods typically homogenize abusive content.
Monolithic language models struggle to distinguish abuse types, verify facts, and maintain stylistic coherence simultaneously.
FIRE uses specialized agents to diagnose abuse before generation, supporting planned and factually grounded counter-responses.
FIRE identifies one of five hate categories and uses FactualCS annotations covering categories, reasoning traces, and evidence mappings.

2 Related Works

Counterspeech research has progressed from limited expert-crafted datasets and generic text generation toward systems that control response style and address bias, while reasoning remains a motivation for agentic approaches.

Early counterspeech resources used expert-crafted static corpora that offered high quality but limited coverage.
Later pipelines steered generation toward tones such as politeness or detoxification rather than relying only on text fluency.
Recent strategies increasingly target bias dismantling and use agentic AI in response to monolithic models’ reasoning and hallucination struggles.

3 The FactualCS Dataset

FactualCS is designed around the semantic diversity of hate, pairing categorized abuse with annotations that support category-specific logical refutation and grounded counterspeech generation.

Effective counterspeech must address the core premises of varied abuse, because a stereotype rebuttal may not counter a conspiracy theory.
The dataset maps hate categories to annotation rationales, target groups, retrieval queries, and supporting evidence through a multi-stage annotation protocol.
FactualCS targets hate categories including stereotypes, misinformation, dehumanization, conspiracies, and non-factual claims.
These structured annotations help models learn why a response fits a specific abuse type, not only what response to generate.

4 Proposed Methodology

FIRE decomposes counterspeech generation into analysis and response stages, using memory retrieval, hate-category prediction, conditional evidence search, and category-aware generation. Its modular workflow combines compact agents with a trained encoder while keeping language models frozen.

FIRE separates counterspeech generation into an analysis stage and a response-generation stage aimed at categorical accuracy and quality alignment.The framework is designed to adapt responses to the abuse type while preserving constructive, non-toxic behavior.
The task maps hate speech to counterspeech while also generating the hate category, target group, reasoning trace, search query, and supporting evidence.These auxiliary variables form the conditioning context for grounded generation.
Memory Module: A 22M-parameter encoder retrieves hate-type exemplars by comparing input embeddings with a fixed memory index.The memory contains annotated records, while only the hate speech text is used to learn representations.
Memory Module: The predicted hate type determines which exemplars are retrieved, and the encoder is the only trained module, avoiding language-model fine-tuning.Retrieved entries provide reasoning patterns for both the analyst and generator.
Phase 1: HEAL: When retrieval confidence is ambiguous, the Hate Speech Analyst receives exemplars from the top two hate types and must choose between them with justification.For confident retrieval, it uses exemplars from the single most likely type; the analyst predicts the target group, type, reasoning trace, and query.
Phase 1: HEAL: Web search retrieves factual evidence for misinformation, stereotypes, and conspiracies, while non-factual hate receives no evidence.A manual audit of 100 retrieved instances found 92% of evidence came from institutional or mainstream informational sources.
Phase 2: CARE: The Counterspeech Generator conditions on the original hate speech, structured analysis, retrieved exemplars, and evidence to produce a category-aware response.With frozen model parameters, specialization comes from how this conditioning context changes the generation distribution.

5 Experimental Setup

The evaluation compares FIRE with multiple baseline pipelines and language-model architectures under zero-shot, few-shot, retrieval-based, and supervised settings using lexical, semantic, and counterspeech-specific metrics.

The baselines include GPS, DialoGPT, CoARL, and HiPPrO, alongside six encoder-decoder and decoder-only architectures.
The study evaluates zero-shot prompting, few-shot prompting with five exemplars, retrieval-based generation, and supervised fine-tuning.
Evaluation uses lexical metrics including METEOR and ROUGE, semantic metrics including BERTScore and Cosine Similarity, and counterspeech-specific dimensions.
Table 2 marks higher- and lower-is-better metrics with directional arrows and distinguishes best from second-ranked results.

6 Experimental Results

FIRE outperforms existing techniques across most evaluated categories, combining stronger semantic, factual, strategic, and safety performance with substantially lower memory use. An ablation and human-evaluation design further examine the sources and practical significance of these gains.

The study combines quantitative comparisons, ablation analysis, and human evaluation against strong baselines to assess FIRE’s effectiveness.The human evaluation uses randomly sampled outputs and compares FIRE with LLaMA-3.1-8B-Instruct, HiPPrO, and CoARL.
FIRE achieves the best performance in nine of thirteen evaluated categories.
Semantic and Lexical Alignment: FIRE leads deep semantic metrics with BERTScore 0.886, METEOR 0.247, and CoSIM 0.574 while retaining competitive ROUGE scores.ROUGE scores are R1: 0.294, R2: 0.071, and RL: 0.197.
Response Quality and Safety: Toxicity falls to 0.016, while Repetition Rate reaches 0.118, Novelty 0.730, and Diversity 0.873.These correspond to 11.1% lower toxicity, 7.1% lower repetition, 9.1% higher novelty, and 1.3% higher diversity than the stated baselines.
Strategic and Factual Integrity: Category Accuracy reaches 0.702 and Factual Score reaches 0.969, representing increases of 11.1% and 12.2%, respectively.The results are presented as evidence of better strategy alignment and factual reliability.
Overall Performance: Using sequentially activated 1.7B agents, FIRE reduces peak VRAM by 75% while maintaining comparable end-to-end inference latency.Peak VRAM is approximately 4GB versus approximately 16GB for monolithic 8B models.

7 Error Analysis

The error analysis contrasts FIRE with the strongest baseline on a stereotype involving Muslim compatibility with Western societies.

The baseline’s hedged rebuttal conditionally frames integration and can unintentionally validate the stereotype.
FIRE counters the universal generalization with concrete counter-evidence, including London Mayor Sadiq Khan.
FIRE’s response directly rejects the stereotype rather than relying on the baseline’s passive neutrality.

FIRE (Ours)

FIRE is presented as a constructive, factually grounded alternative to the strongest baseline in a stereotype-related counterspeech example. Its response uses a concrete example and reframes compatibility through mutual respect and dialogue.

FIRE (Ours): FIRE uses London Mayor Sadiq Khan as concrete counter-evidence that compatibility between Islamic values and Western societies is possible.
FIRE (Ours): The response frames mutual respect and dialogue as a constructive alternative to division.
FIRE (Ours): Table 4 reports an error analysis comparing FIRE with the best baseline.

8 Conclusion

FIRE combines hierarchical agentic reasoning with FactualCS to produce category-targeted, factually supported counterspeech using compact models. The paper also identifies boundaries involving dataset coverage, model capacity, external-tool reliability, annotation representation, language, and long-term impact.

Conclusion: FIRE decomposes counterspeech into intent analysis and grounded response synthesis, using compact models under 2B parameters without finetuning.The framework is presented as outperforming established baselines and matching larger state-of-the-art LLMs.
Limitations: FactualCS covers five abuse categories, but its scope may limit generalization to uncovered hate forms and intersectional attacks across 14 target communities.The authors explicitly describe the dataset as non-exhaustive.
Limitations: Small language models may constrain reasoning depth for highly ambiguous linguistic nuances, despite improving efficiency.
Limitations: External web-search failures can propagate errors into final responses and compromise factual grounding.
Limitations: The evaluation focuses on immediate response quality rather than long-term conversational impact or potential conflict escalation.
Ethics Statement: The paper frames automated counterspeech as ethically complex and notes that generated responses may fail to convey intended meanings accurately.

A.1 Hatespeech Selection

FactualCS aggregates hate speech from four established datasets to capture varied forms of online abuse. Multi-stage filtering removes duplicates and context-poor instances before retaining samples with clearly articulated harmful intent or targeted content.

A.1 Hatespeech Selection: FactualCS combines IntentCONANv2, HatEval, ETHOS, and the Gab Hate Corpus to capture conversational abuse, identity-targeted attacks, discrimination, violence, and extreme dehumanization.
A.1 Hatespeech Selection: The initial collection was reduced to 8,000 unique samples after removing redundant hate speech instances across the source datasets.
A.1 Hatespeech Selection: Samples under five words were removed when they lacked sufficient context for meaningful analysis, while self-expressive hate with clear harmful intent or targeting was retained.
A.1 Hatespeech Selection: The final source counts were 2,384 from IntentCONANv2, 1,500 from HatEval, 550 from ETHOS, and 350 from the Gab Hate Corpus.

A.2 Annotator Demographics

FactualCS was annotated by six volunteers with computational and domain expertise in natural language processing and hate speech analysis.

A.2 Annotator Demographics: Six annotators—four male and two female—participated in the annotation process.
A.2 Annotator Demographics: All annotators had computational backgrounds and expertise in natural language processing and hate speech analysis.
A.2 Annotator Demographics: The team included researchers with publications in content moderation, computational social science, and counter-narrative generation.

A.3 Annotation Process

The annotation process operationalizes five mutually exclusive hate categories and links each instance to reasoning, targets, queries, and evidence for category-specific counterspeech. The resulting dataset supports evidence-aware evaluation and shows that retrieval, factuality, and error handling are tied to hate-category distinctions.

Annotation Process: Annotators recorded hate type, rationale, target group, contextual query, and supporting evidence for each hate-speech instance.The process began with a 500-sample pilot to establish shared protocols before broader annotation.
Taxonomy: Five mutually exclusive hate categories serve as functionally distinct logical triggers requiring different counterspeech strategies.The taxonomy distinguishes empirically verifiable claims from abstract or dignity-targeting abuse.
Dataset Statistics: FactualCS contains 4,784 instances split into 3,912 training, 383 validation, and 489 test samples with stratified category proportions.Stereotype Hate is the largest category at 32.8%, while Conspiracy Hate is the smallest at 4.4%.
Evidence Retrieval: 47.0% of instances use web search, with factual categories triggering searches while dehumanizing and non-factual categories rely on logical or moral refutation.This operationalizes evidence retrieval according to the semantic type of abuse.
Evaluation: FIRE significantly improves Novelty, Category Accuracy, and FactualScore over LLaMA-3.1-8B-Instruct under supervised fine-tuning.FIRE also significantly improves Category Accuracy and FactualScore in the few-shot setting, without a statistically significant toxicity difference.
Reliability and Efficiency: A purity threshold of τ = 0.62 flags ambiguous analyst predictions, while 92% of audited retrieved evidence comes from institutional or mainstream informational sources.The threshold separates 88.4% high-confidence cases from 11.6% lower-purity cases, and retrieval filtering limits potentially weak sources to 2%.