PaperlayerView exact revision on arXiv

Source-linked AI summary

From Repetition to Recognition: Inductive Discovery of Disinformation Narratives

Max Upravitelev, Veronika Solopova, Jing Yang, Charlott Jakob, Alexandra Tsiakalou, Neda Foroutan, Vera Schmitt

arXiv:2609.11128v1cs.CL

TL;DR

Existing narrative-mining evaluation is tied to predefined taxonomies, limiting assessment of novel disinformation narratives. The paper introduces a three-tier framework and compares clustering with graph-community pipelines, finding complementary automated behavior, severe topic underrepresentation for clustering in one corpus, and many human-validated singleton candidates from graph outputs.

  • Problem

    Narrative evaluation remains bounded by predefined taxonomies, leaving no reference-based way to assess valid candidates absent from existing label spaces.

  • Method

    The paper evaluates unsupervised narrative-label generation through recovery, mining, and discovery tiers and compares clustering- and graph-based pipelines across seven datasets.

  • Results

    Clustering and graph pipelines are complementary under automated metrics; graph pipelines produce 30–62% singleton labels, and many are validated as disinformation narrative candidates.

  • Takeaways & Limitations

    Narrative discovery can surface candidates for extending existing taxonomies, while recognized narratives need not repeat within the evaluated corpus.

  • Takeaways & Limitations

    Discovery validation is a pilot using two annotators with NLP backgrounds, both co-authors, on two datasets, so it does not establish required agreement or candidate prevalence.

Abstract

from arXiv · show

In disinformation datasets, narratives are often understood as recurring interpretive patterns that group texts under narrative labels. Recent work formalized narrative mining as inductively inferring narrative labels from corpora, but its evaluation stays tied to predefined taxonomies, a closed-world setting that cannot capture narratives absent from the reference labels. We introduce a three-tier evaluation framework for unsupervised narrative label generation: recovery (against a corpus's own taxonomy), mining (against external label sets), and discovery (without predefined labels). Applying it, we compare clustering-based and graph-community-based pipelines across seven disinformation datasets, with human validation of discovery on two. The two families are complementary under automated metrics, but in a corpus with two prominent topics, clustering can reduce one topic to 2% of generated labels while graph-based pipelines stay balanced. Discovery validation also reveals many singletons (narrative labels derived from single claims, 30-62% of graph outputs), which clustering cannot produce. Annotators confirm many as recognizable disinformation narratives, suggesting that in open-world discovery the repetition assumed by narrative mining may be recognized outside the corpus, not within it. We release human-validated narrative candidate labels for the Climate Obstruction and PolyNarrative datasets to support taxonomy development and dataset extension.

1 Introduction

The paper extends disinformation narrative evaluation beyond predefined taxonomies by introducing open-world discovery alongside recovery and mining. Comparing clustering and graph-based pipelines, it finds complementary automated performance, topic imbalance from clustering, and validated singleton candidates that challenge within-corpus repetition assumptions.

  • 1 Introduction: The study introduces recovery, mining, and discovery tiers for evaluating unsupervised narrative labels against own taxonomies, independent taxonomies, or no reference set.Recovery and mining support automated evaluation, while discovery uses human validation.
  • 1 Introduction: Across seven datasets, clustering and graph-based pipelines are complementary under automated metrics but differ structurally in topic coverage and singleton generation.The comparison spans clustering- and graph-based aggregation families.
  • 1 Introduction: At default parameters, clustering can suppress one of two main corpus topics, whereas graph pipelines preserve both topics across the parameter sweep.This imbalance is among the differences not captured by closed-world metrics.
  • 1 Introduction: Discovery identifies singleton labels derived from single claims that clustering cannot produce, while human annotators validate many as recognizable disinformation narratives.The finding suggests that narrative recognition may draw on broader discourse rather than repetition within the corpus.
  • 1 Introduction: The paper releases human-validated narrative candidates for Climate Obstruction and PolyNarrative to support taxonomy maintenance and extension.These candidates are intended to support future taxonomy development.

2 Related Work and Preliminaries

Prior narrative research mainly uses clustering or graph representations and evaluates generated labels against existing taxonomies. This paper positions narrative discovery as an open-world problem because novel candidates lack predefined reference labels.

  • 2 Related Work and Preliminaries: Clustering methods group textual or semantic representations, while graph-based approaches model actors, events, entities, and their relations.Both families recur in computational narrative analysis and narrative mining.
  • 2 Related Work and Preliminaries: Narrative mining summarizes groups into labels but evaluates them against pre-existing reference narratives, keeping assessment within a closed label space.This applies to both claim-cluster pipelines and graph-based community summarization.
  • 2 Related Work and Preliminaries: Narrative discovery differs from generalized category discovery and open-world classification because its novel candidates lack ground-truth labels for evaluation.Known-class labels and reference annotations are unavailable for the candidates of interest.

3 Methodology

The methodology evaluates inductive narrative-label pipelines across closed- and open-world tiers, using similarity-based metrics for reference taxonomies and human validation for discovery. It compares clustering and graph-community configurations, while documenting important metric, language, and annotation constraints.

  • 3.1 Evaluation Framework: The framework has independent recovery, mining, and discovery tiers that test source-taxonomy reproduction, cross-taxonomy matching, and open-world candidate validity.Success in one tier is not assumed to predict success in another.
  • 3.1 Closed-world metrics: Each generated-label set is compared with reference labels using cosine similarity, Weighted Chamfer Distance, Hungarian similarity, and collapse.The metrics capture coverage, alignment, precision, and top-1 collisions from different perspectives.
  • 3.1.2 Open-world validation: Discovery uses annotator judgments of narrative validity and taxonomy matching, with Other marking in-domain labels absent from retrieved taxonomy candidates and None marking out-of-domain labels.Annotators receive retrieved top-2 taxonomy candidates as an aid.
  • 3.1.2 Open-world validation: Discovery validation is a pilot on two datasets with two annotators, produces no gold labels, and withholds corpus-frequency information from judgments.The study reports strict and union agreement rather than claiming population-level prevalence.
  • 3.3 Example Pipelines: PSVO is a no-LLM SVO-triplet baseline whose English dependency parser does not handle the multilingual corpora natively.This language limitation applies specifically to PSVO's extraction step.
  • 3.3 Example Pipelines: The study compares claim-clustering and graph-community pipelines, including Pclr and Pcom, which differ in extracted units and how those units are grouped before LLM summarization.Pclr clusters extracted claims, whereas Pcom constructs a heterogeneous graph and detects Leiden communities.

4 Results

Across seven-dataset recovery and mining benchmarks, clustering and graph pipelines show complementary metric strengths, while discovery validation finds substantial human-validated candidates and supports human involvement.

  • Narrative Recovery and Mining: Clustering wins WCD in 13/14 recovery comparisons and graph aggregation wins Hungarian in 12/14, while graph matches or wins recovery Collapse in all 14.On mining, clustering wins WCD and graph wins Hungarian in 10/14 comparisons; graph has the clearest advantage on mining-Collapse, winning in 14/14 comparisons.
  • Narrative Discovery: The two Pper variants differ only in aggregation and have close automated results, with ΔHun ≤0.03 and ΔWCD ≤0.02, motivating human validation.Using shared components isolates the clustering-versus-graph contribution while keeping the number of generated labels manageable.
  • Narrative Discovery: Human validation reports 47–62% yes, 10–23% other, and 0–9% none across discovery runs, with merged agreement κ of 0.56–0.65.The comparison to PolyNarrative agreement is approximate because the annotation setups differ.
  • Narrative Discovery: Annotators flag largely disjoint sets as out-of-domain despite similar aggregate none rates, with Jaccard overlap of 9/31, 14/53, and 1/8 across reported runs.Thus aggregate rates do not reveal which individual candidates annotators classify as none.
  • Narrative Discovery: LLM judges agree more with one another than with humans and therefore cannot replace human validation in the discovery tier as deployed.Their unanimous votes might still serve as a filter.

5 Analysis

Analysis shows that graph pipelines expose singleton narratives and preserve topical coverage more reliably than clustering under some parameter settings, while validated candidates can extend existing taxonomies.

  • 5.1 Singletons and the repeatability assumption: Graph outputs contain many singleton labels, and even strict agreement validates 14% of CO and 27% of PN graph labels as singletons that clustering cannot reproduce.Across all annotations, singletons comprise 30% of CO and 38% of PN graph outputs.
  • 5.1 Singletons and the repeatability assumption: Lowering HDBSCAN’s min_cluster_size to 2 yields 4069 PN labels, exceeding practical human-validation capacity, while still not reproducing graph singletons.A cluster of size two requires two similar claims, whereas a graph singleton can arise from one isolated claim.
  • 5.1 Singletons and the repeatability assumption: Annotators accepted singleton candidates without within-corpus frequency evidence, suggesting recognition may draw on broader discourse rather than repetition in the corpus.The paper presents this as an interpretation and notes alternatives, including plausibility judgments and example-based guidelines.
  • 5.2 Parameter sensitivity and topical coverage: At HDBSCAN’s default, climate labels collapse to 2.1% on Pclr and 1.7% on Pper-clr because minority-topic groups are treated as noise and dropped.Lowering min_cluster_size restores balance but increases label counts, reaching 4069 labels at mcs=2 on Pclr.
  • 5.2 Parameter sensitivity and topical coverage: Graph pipelines preserve both topics across the full γ-sweep, with neither topic falling below 19.7%, unlike clustering’s default-parameter collapse.At γ=1, Pcom produces a 38%/29% Ukraine/climate split and Pper-com produces 37%/38%.
  • 5.3 Candidate narratives: Human validation identifies a repeated climate-obstruction pattern favoring market forces over state regulation, motivating a proposed “State Regulation & Free Market” category.The largest candidate contains 99 source ads and matched neither retrieved CO nor CARDS2 labels.
  • 5.3 Candidate narratives: On PN, validated candidates work better as a flat list than as a coherent umbrella category, and their in- versus out-of-domain scope is less clear than on CO.A self-reliance candidate with 59 source documents was marked None by both annotators.

6 Conclusions

The evaluation reveals structural differences between clustering- and graph-based pipelines that closed-world metrics miss, while discovery surfaces validated novel candidates for taxonomy extension.

  • Graph pipelines preserved both PolyNarrative topics across parameter settings, whereas clustering underrepresented one topic at default parameters.
  • Many graph-pipeline singletons were validated as disinformation narrative candidates despite lacking within-corpus frequency evidence.
  • Narrative-mining pipelines can surface novel candidates for extending existing narrative taxonomies, and candidate sets were released for Climate Obstruction and PolyNarrative.

Limitations

The study’s discovery validation is a pilot with limited annotator and dataset coverage, while metric, temporal, and prompt-related constraints bound interpretation of the results.

  • Discovery validation used two annotators, including co-authors, on only the Climate Obstruction and PolyNarrative datasets.
  • The annotators’ shared NLP background and project familiarity may bias candidate recognition and acceptance, motivating interdisciplinary validation.
  • The clustering coverage finding comes from one two-topic dataset and requires testing on additional multi-topic datasets.
  • Embedding-based closed-world metrics capture surface meaning more reliably than implicit semantics and cannot be fully separated from embedding-model behavior.
  • LLM knowledge leakage limits absolute novelty claims, so novelty is defined relative to reference taxonomies rather than as historically new.
  • The study harmonized prompts but did not systematically explore prompt optimization, which affects label quality and singleton counts.

Ethics Statement

The study uses public disinformation datasets under their original licenses, releases identifiers and validated candidate labels, and documents its use of language models in manuscript and pipeline work.

  • Experiments use publicly released disinformation datasets under their original licenses, without redistributing raw documents.
  • Released materials include dataset identifiers and human-validated narrative-label candidates for Climate Obstruction and PolyNarrative, not raw documents or gold labels.
  • The candidate labels are intended for taxonomy maintenance and extension, not as training data for generating or amplifying disinformation framings.
  • Claude models supported manuscript editing, reformulation, coding assistance, pipeline development, and analysis, with authors retaining responsibility for final content.

C Metric Correlations

Metric correlations show overlapping and size-sensitive behavior, so the study interprets them directionally and relies on Hungarian for comparisons across pipelines with different label counts.

  • Collapse and C/R are tightly coupled, with correlations of +0.82 for recovery and +0.78 for mining because C/R = Coll/m.
  • Hungarian and Collapse/C/R are negatively correlated on recovery and mining, while Hungarian and WCD agree on mining but not recovery.
  • Recovery has a median |L|/m of 7.5, causing WCD and Hungarian to favor pipelines with very different label counts.
  • On mining, |L|/m is close to 1 with a median of 1.4, so the recovery-side size effect largely disappears.
  • The strong mining correlation between |L| and C/R is partly mechanical because additional labels reduce Collapse’s floor when |L| < m.
  • Hungarian is the most |L|-robust metric for cross-dataset comparisons, whereas WCD remains sensitive to label count on recovery.

D Automated Metrics Correlations: Bootstrap Confidence Intervals

Table 10 reports bootstrap 95% confidence intervals for the Spearman correlations summarized in Table 9, using percentile intervals from 1,000 resamples over each side’s cells.

  • Table 10 reports 95% bootstrap confidence intervals for the Spearman correlations summarized in Table 9.The intervals use the percentile method with 1,000 resamples with replacement over the cells of each side.

E Robustness of Singleton Findings

Multiple robustness checks support the singleton finding, showing stability across Leiden seeds, persona configurations, pipeline variants, and replicate runs, while topical annotation uses a controlled four-way procedure.

  • Seed stability (Leiden): Singleton counts remain nearly deterministic across 10 Leiden seeds, with standard deviations at most 3.2% of the mean.Pper-com produces 70.1 ± 0.3 singletons on CO and 148.6 ± 4.2 on PN; Pcom produces 476.2 ± 15.3 on PN.
  • Pipeline transferability: The singleton phenomenon transfers beyond the persona pipeline, with Pcom producing 507 of 824 singleton narrative labels on PolyNarrative.
  • Persona ablation: Across 15 persona-subset configurations, mean singleton counts remain 150–154 and singleton community fractions 58–71%.These ranges persist across 10 Leiden seeds per configuration.
  • Upstream LLM variance: Across three replicate runs, singleton counts vary from 140 to 159, while singleton community fractions remain 59.8–62.8%.Varying Leiden seeds produces 133–174 singletons, with LLM variation contributing more apparent variance than Leiden variation.
  • Topical distribution: Topical annotation classifies narrative labels into War in Ukraine, Climate, Other, and Noise using zero-shot self-consistency across prompt variants.Each label is classified five times per variant, with majority voting within and across three variants.

J.2 LLM-as-a-judge Evaluation

The LLM panel agrees internally more than with human annotators and tends to overrecognize disinformation framings, especially on PolyNarrative and discovery-relevant labels.

  • Agreement patterns: LLM ensembles agree with each other at κ = 0.76–0.83 for is_narr and 0.73–0.77 on the merged label space, exceeding their agreement with humans.The aggregate LLM–human κ of 0.53–0.64 is therefore misleading as evidence of human-equivalent judgment.
  • Systematic differences: LLM judges overrecognize disinformation framings relative to humans, most clearly on PolyNarrative.Gemma’s yes-rates exceed both annotators’ by 4–28 percentage points; Qwen exceeds them on PolyNarrative by 8–24 points.
  • Discovery-relevant labels: For discovery-relevant labels, LLM recovery of human-flagged clusters is limited, reaching at most 26% for None and 57% for Other at the least strict threshold.With unanimous consensus, only 12 of 76 human-flagged Other candidates on PolyNarrative are recovered, although reverse-direction precision is 55–100%.
  • Interpretation: The authors attribute human–LLM differences partly to annotators’ exposure to narratives outside the corpus, which LLM judges lack.
Loading 2609.11128v1…