A Multi-Aspect Framework for Counter Narrative Evaluation using Large Language Models

• Manual counter-narrative intervention is impractical because online toxicity arrives constantly and spreads rapidly.Source: §1 · p. 1
• BLEU and ROUGE-L can misalign with human judgment, while trained human evaluation is costly and time-consuming.Source: §1 · p. 1
• The proposed framework prompts pretrained LLMs to score and explain counter-narrative quality using specificity, opposition, relatedness, toxicity, and fluency.Source: §1 · p. 2
• The framework is validated by correlating LLM-generated scores with human annotations and qualitatively analyzing evaluator feedback.Source: §1 · p. 2, Figure 2 · p. 2

• Counter-narrative generation has used curated datasets and prompting strategies, including the expert-annotated Multitarget-CONAN dataset.Source: §2 · p. 2
• Earlier evaluation combines overlap and diversity metrics with trained human annotation, but reference comparisons cannot represent key NGO-specified quality aspects.Source: §2 · p. 2
• Expert annotator training can be costly, time-consuming, and difficult to reproduce, motivating alternative evaluation procedures.Source: §2 · p. 2
• The proposed framework extends LLM-based evaluation to counter narratives by prompting models with NGO-derived aspects to produce scores and feedback.Source: §2 · p. 2

Evaluation framework

• Five NGO-derived aspects—specificity, opposition, relatedness, toxicity, and fluency—define the criteria for evaluating counter narrative quality.The aspects target argumentative association, denunciation of hate speech, civility, de-escalation, and response coherence.Source: §3 · p. 3

Evaluation data

• 180 Multitarget-CONAN test examples receive counter narratives from DialoGPT, zero-shot ChatGPT, and Vicuna for evaluation.DialoGPT was trained on 4,003 examples, while ChatGPT and Vicuna represent zero-shot and open/closed-source model settings.Source: §3 · p. 3

Evaluation protocol

• The evaluation compares LLM judgments and reference-based metrics against AMT scores using Pearson, Spearman, and Kendall correlations.Table 2 reports correlations for the entire evaluation set, with the full results in Table G.1.Source: Table 2 · p. 3

Human evaluation

• AMT workers represent human interpretation through qualification, training, monitoring, and manual verification because expert NGO annotation was unavailable.Workers studied task materials and examples, passed qualification tests, and had their scores and feedback checked.Source: §3 · p. 3

Scoring and baselines

• Each candidate receives 1–5 star scores per aspect, an aggregated multi-aspect average, a final 15-star overall score, and explanatory feedback from human and LLM evaluators.Automatic evaluation uses single runs of ChatGPT, Vicuna, GPT-4, and Prometheus, while BLEU, ROUGE-L, METEOR, BERTScore, and BARTScore provide reference-based comparisons.Source: §3 · pp. 3–4

Qualitative evaluation

• Example cases additionally compare evaluator explanations with AMT feedback to assess whether scores are justified for the right reasons.Table 3 presents an example of effective counter narrative evaluation from an LLM evaluator.Source: §3 · p. 4, Table 3 · p. 4

4.1 Evaluation Metric Correlation

• LLM evaluators achieved the highest correlations with AMT-annotated scores, outperforming overlap and BERTScore metrics while offering interpretability and reduced reference reliance.BARTScore Recall also achieved strong correlations, but the authors attribute the LLM advantage to direct evaluation of relevant counter-narrative aspects.Source: §4.1 · p. 4
• Vicuna achieved performance comparable to GPT-4 when evaluated with the multi-aspect framework.Source: §4.1 · p. 4

4.2 Fine-grained Analysis

• LLM evaluators generally outperformed alternative metrics across generation models, but performance declined for Vicuna candidates and for overall scores on ChatGPT-, Vicuna-, and Prometheus-generated candidates.Source: §4.2 · p. 4

4.3 Qualitative Evaluation

• Vicuna corrected initially misaligned Opposition and Toxicity ratings under multi-aspect evaluation while retaining an accurate Specificity rating.Source: Table 4 · p. 5
• Multi-aspect evaluation produced aligned scores where a single overall score diverged and helped weaker open-source models represent intricate NGO criteria.Source: §4.3 · p. 5
• Each evaluator could misunderstand the hate-speech–counter-narrative relationship or conflate aspects, potentially producing unaligned scores and explanations.Source: §4.3 · p. 5
• GPT-4 often underrated Vicuna candidates, which the authors propose may reflect self-enhancement bias against generations unlike its own.Source: §4.3 · p. 5

• The framework uses LLMs to score and provide feedback on counter-narrative candidates across key evaluation aspects derived from NGO guidelines.Source: §5 · p. 5
• Experiments show that LLM evaluators represent socially nuanced NGO guidelines while producing evaluation scores and feedback aligned with human interpretation.Source: §5 · p. 5
• The authors identify future prompting and finetuning as ways to address qualitative errors and use LLM-generated scores for downstream counter-narrative generation.Source: §5 · p. 5

• Human annotation exposed AMT workers to potentially offensive content, so the study applied mitigation procedures, fair compensation, and privacy protections.Source: §6 · p. 6
• The authors caution that automated evaluation should complement rather than completely replace gold-standard human evaluation in human-sensitive tasks.Source: §6 · p. 6
• Future evaluation should combine the framework with diverse human annotations reflecting differing views of hate speech and effective responses.Source: §6 · p. 6

7 Limitations

• Expert NGO annotation could not be reproduced, so the study used crowdsourcing with qualification and active monitoring as an alternative.The authors note that workers were not trained through the multi-week expert procedure used in prior work.Source: §7 · p. 6
• Alternative evaluator designs, including 0-100 scoring, reference-based few-shot prompts, score normalization, and pairwise comparison, remain unexplored.The authors identify these strategies as future work because their effects on evaluator performance are unknown.Source: §7 · p. 6
• Validation covered 180 Multitarget-CONAN pairs generated by DialoGPT, ChatGPT, and Vicuna v1.3 33b, leaving additional targets and approaches for future testing.The authors explicitly call for validation on more examples, target groups, and generation methods.Source: §7 · p. 6

A Counter Narrative Generation

• Zero-shot ChatGPT and Vicuna performed effectively against supervised strategies by AMT judgment, but their counter narratives still struggled with specificity.This result leaves room for improvement in future counter narrative generation models.Source: A Counter Narrative Generation · p. 9

B DialoGPT Implementation

• DialoGPT-medium was finetuned on 4003 Multitarget-CONAN training pairs using Adam and hyperparameters from Tekiro˘glu et al. (2022).Source: B DialoGPT Implementation · p. 9
• Five DialoGPT candidates were generated with top-pk decoding, after which one candidate was randomly selected for each hate speech example.Source: B DialoGPT Implementation · p. 9

C Prompting/API details

• ChatGPT and Vicuna v1.3 33b generated counter narratives using a temperature-1, max_new_tokens-512 zero-shot prompt.Source: C Prompting/API details · p. 9, Figure C.1 · p. 9
• ChatGPT, GPT-4, and Vicuna evaluated candidates with generated 1-5-star aspect rubrics, while Prometheus 13b used an adapted original prompt with repetition penalty 1.03 and maximum output length 256.Source: C Prompting/API details · p. 9

D BARTScore details

• BARTScore was implemented with Precision, Recall, and F1, alongside CNN and CNN-Para finetuned variants.Source: D BARTScore details · p. 10

E AMT Study details

• The AMT study received Institutional Review Board review and used brief work periods, frequent breaks, and communication procedures to mitigate distress from hateful content.Source: E AMT Study details · p. 10
• Workers needed a 95% HIT approval rate, 1000 approved HITs, and a qualification task before annotation, with confidentiality protections and specified compensation.The study paid $2.7 per HIT, estimated 15 minutes per HIT, and reported $1,830 in total annotation costs.Source: E AMT Study details · p. 10

F Interrater Agreement

• Human-annotation reliability was assessed with Krippendorff’s α using FastKrippendorff, with results reported in Table F.1.Source: F Interrater Agreement · p. 10

G Correlations

• Table G.1 contains correlations for all evaluation metrics and candidates, while Tables G.2-G.4 provide model-specific analyses for DialoGPT, ChatGPT, and Vicuna.The correlations use Pearson, Spearman, and Kendall coefficients computed with Scipy.Source: G Correlations · pp. 10–11
• Table F.1 reports interrater agreement for the Amazon Mechanical Turk study using Krippendorff’s α.Source: Table F.1 · p. 11

H Qualitative Examples

• Qualitative examples cover multi-aspect improvements for Vicuna and Prometheus, common errors, and cases where GPT-4 underrated Vicuna.Source: H Qualitative Examples · p. 11

Additional results

• The appendix reports average counter-narrative scores, evaluator mean absolute errors, and AMT annotation variability across generation models.These results are organized in Tables I.1–I.3.Source: I Additional Figures · p. 11

AMT study materials

• The appendix includes AMT qualification materials covering task descriptions, qualification questions, qualification tasks, demographic questions, and the main task.These materials are presented as supplementary figures for the Amazon Mechanical Turk study.Source: Figure E.1 · p. 12, Figure E.2 · p. 13, Figure E.3 · p. 14
• Supplementary tables provide AMT-generated average scores by generation model and demographic information about workers in the study.The model-level AMT scores are reported in Tables A.1 and I.3, while worker demographics appear in Table E.1.Source: Table A.1 · p. 17, Table E.1 · p. 17, Table I.3 · p. 22

Metric correlations

• Correlation tables compare evaluation metrics with AMT scores for the full evaluation set and candidates generated by DialoGPT, ChatGPT, and Vicuna.The tables identify the best and second-best correlations through formatting.Source: Table G.1 · p. 17, Table G.2 · p. 18, Table G.3 · p. 18

Evaluation examples

• Supplementary examples show that the multi-aspect framework can correct Prometheus’s initially misaligned Overall score by revising Opposition and Specificity ratings.The example attributes the correction to using the multi-aspect framework.Source: Table H.1 · p. 19
• Other examples document evaluator errors caused by misunderstanding hate-speech–response relationships, mixing information across aspects, or underrating Vicuna-generated candidates.The examples associate these patterns with specific evaluators and generation models, including DialoGPT and Vicuna.Source: Table H.2 · p. 20, Table H.3 · p. 20, Table H.4 · p. 20