PaperlayerView exact revision on arXiv

Source-linked AI summary

Look What You Made Us Cluster: Hate Narrative Extraction from Reddit Discourse

Annabelle K. L. Chua, Forster J. Khoo, Joel C. R. Tan, Huey Ting Ang, Kheng Hwee Tan, Joel Y. A. Sim, Shirley W. H. Ow, Ria Mundhra, Elsie C. K. Toh, Youfeng Xu, Lynnette H. X. Ng

arXiv:2609.37408v1cs.CLcs.SI

TL;DR

Existing semantic representations can miss the deeper structure of narratives, limiting precision in online hate narrative extraction. The paper uses an LLM pipeline to derive entity evaluations and cluster them into meta-narratives. Its evaluation reports 68.9% end-to-end extraction accuracy and illustrates a coherent cluster whose criticism is not apparent from surface sentiment alone.

  • Problem

    Semantic-representation approaches tend to capture surface-level meaning, limiting precision in extracting online narratives.

    Source: §1 · p. 1

  • Method

    An LLM extracts entity, aspect, judgement type, evaluation, and sentiment features, which are bucketed by entity and polarity, clustered with Leiden, and refined into meta-narratives.

    Source: §3.3 · p. 4, §3.4 · p. 5

  • Results

    68.9% of 90 extracted narratives were fully accurate end to end, and a representative cluster revealed criticism that constituent comments often framed with compliments.

    Source: §4.2 · p. 7, §4.3 · pp. 7–8

  • Takeaways & Limitations

    The pipeline provides an avenue for monitoring online narratives and supporting efforts to detect and understand hate narratives as they emerge.

    Source: §5 · p. 8

    Clustering quality depends on upstream entity and sentiment extraction, whose errors propagate into bucket composition and cannot be recovered downstream.

    Source: §5 · p. 8

Abstract

from arXiv · show

Narrative extraction allows us to identify online hate narratives, supporting the construction of rigorous detection systems. Existing computational approaches, however, are limited in precision as they rely on semantic representations, which tend to capture only surface-level meaning. To detect more precise and interpretable narratives, we present an extraction pipeline that represents narratives as entity-evaluation pairs. Narratives are extracted using a Large Language Model (LLM) reasoning process that extends Aspect-Based Sentiment Analysis, identifying the aspect, classifying its judgement type as the basis for evaluation, and deriving the evaluation accordingly. Extracted narratives are then clustered using Leiden, following which clusters are resolved to an intended level of granularity through an LLM-guided refinement process. We illustrate this narrative pipeline with English Reddit comments from 2024 that criticize Taylor Swift, analyzing a representative cluster that exhibits hate speech patterns to demonstrate its interpretive value.

1 Introduction

Existing semantic-similarity methods can miss narratives expressed indirectly or implicitly, limiting precision in understanding online hate. The paper proposes an LLM-based pipeline for extracting and aggregating narratives, demonstrated on Taylor Swift discourse.

1 Introduction

  • Semantic-similarity methods may capture only surface patterns and struggle with indirect or implicit narratives.The introduction notes that hate narratives can accumulate across seemingly harmless posts.Source: §1 · p. 1
  • The proposed LLM framework extracts and aggregates online narratives to help capture how narratives shape perceptions of social entities.The authors identify potential relevance to platform governance.Source: §1 · p. 2
  • The pipeline combines aspect-based sentiment analysis and judgement-frame analysis, then clusters narratives to surface meta-narratives.It is demonstrated on 2024 English Reddit comments about Taylor Swift and may be adapted to other social categories where online hate detection is a concern.Source: §1 · p. 2

2 Background

Prior narrative approaches can be imprecise or difficult to interpret when they rely on surface-level semantic similarity or infer structure only after clustering. The paper derives discourse structure from theory to extract narratives more precisely and interpretably.

Entity Extraction and Resolution.

3 Methodology

The methodology processes documents through entity resolution, narrative-feature extraction, clustering, and LLM-guided refinement. Its narrative features characterize evaluations of entities, and its clustering groups summaries by entity and sentiment before producing refined meta-narratives.

3.1 Pipeline Overview

  • The pipeline identifies entities, extracts features about their portrayal, groups features into buckets for community detection, and refines clusters into meta-narratives.The application used 13,812 Taylor-related candidate records from English Reddit comments in the Entertainment category.Source: §3.1 · pp. 2–3

3.2 Entity Recognition and Resolution

  • An LLM records each entity’s textual reference, country of residence, and category to resolve mentions to canonical forms and reduce naming fragmentation.Filtering the 13,812 candidate records yielded 6,715.Source: §3.2 · p. 3

3.3 Narrative Extraction

  • Narratives are represented by an entity and an evaluation, with judgement type specifying the basis on which the evaluation is made.The defined judgement types are ethics, capability, and appearance.Source: §3.3 · pp. 3–4
  • An LLM extends aspect-based sentiment analysis by identifying an aspect, classifying its judgement type, and deriving an evaluation and sentiment score.Scores from -2 to 2 are collapsed to positive, neutral, or negative labels for bucketing before clustering.Source: §3.3 · p. 4

3.4 Meta-Narrative Extraction

  • Summaries are bucketed by shared entity and sentiment polarity before Leiden clustering, because narrative clusters need coherence on both dimensions.This contrasts with semantic clustering based on embedding similarity alone.Source: §3.4 · p. 5

Semantic Labelling

  • For each cluster, up to 100 centroid-nearest texts are sampled for an LLM to generate a title and summary.The title is embedded for use in downstream refinement.Source: Semantic Labelling · p. 5

Cluster Refinement

  • LLM-guided refinement uses a user-supplied granularity description to split overly general clusters or merge clusters with overly similar titles.The decisions address clusters that are structurally sound but may be too broad or have near-identical titles.Source: Cluster Refinement · p. 5
  • Clusters with at least six texts are split candidates; merging considers cluster pairs linked at cosine similarity ≥0.75 among each cluster’s three nearest neighbors.Leiden reclusters identified splits at resolution 2.0; connected components of retained merge edges define merge groups.Source: Cluster Refinement · p. 6

4 Results and Discussion

Narrative extraction achieved 68.9% end-to-end accuracy, while entity resolution correctly assigned 89 of 90 Taylor Swift mentions. The recovered meta-narrative brings distinct appearance-related critiques together into a coherent characterization.

4.1 Performance of Entity Extraction and Resolution

  • Taylor Swift mentions were correctly identified and assigned to the canonical bin in 89 of 90 cases, an accuracy of 98.9%.The evaluation manually coded 90 outputs randomly sampled from the Taylor Swift entity bin, with 30 for each judgment type.Source: §4.1 · p. 6

4.2 Performance of Narrative Extraction

  • 68.9% of 90 sampled narratives were fully accurately extracted end to end.Feature-level accuracy used shrinking denominators; judgement type achieved macro-averaged F1 of 1.0, and sentiment ratings had ICC = .93, 95% CI [.89, .96], p < .001. The authors characterize these metrics as a preliminary benchmark.Source: §4.2 · p. 7

4.3 Performance of Meta-Narrative Extraction

  • Distinct comparisons about Taylor Swift’s image, body shape, and other artists converge on a meta-narrative that she lacks sex appeal despite being objectively attractive.The authors describe this as a granular characterization that avoids both narrow feature-specific clusters and undifferentiated appearance or character criticism.Source: §4.3 · p. 7
  • Comments that appear superficially positive or neutral can collectively reveal a broader critical characterization when their underlying evaluations are extracted and aggregated.The paper notes that post-level hate detection based on surface sentiment alone might not flag such comments.Source: §4.3 · p. 8

5 Conclusions and Future Work

The paper presents narrative extraction and clustering as an avenue for more precise, interpretable online narrative monitoring. It identifies upstream errors, LLM variation, static-corpus processing, and the need for cluster-level hate criteria as limitations or future-work directions.

5 Conclusions and Future Work

  • The pipeline is presented as an avenue for online narrative monitoring that supports detecting and understanding hate narratives as they emerge.Source: §5 · p. 8
  • Clustering quality is bounded by upstream entity and sentiment extraction, whose errors propagate into bucket composition and cannot be recovered downstream.Source: §5 · p. 8
  • The pipeline processes a static corpus; incremental clustering, more consistent LLM-guided split and merge decisions, and operational cluster-level hate criteria remain future-work directions.The paper suggests confidence thresholds or ensemble voting for consistency, and possible cluster-level criteria based on average sentiment or concerning label semantics.Source: §5 · pp. 8–9
Loading 2609.37408v1…