PaperlayerView exact revision on arXiv

Source-linked AI summary

SWARM: A Multilingual Human-Annotated Dataset for Russian Propaganda Detection in Search Engine Results

Manuel Tonneau, Abhinav Dubey, Farhan Shaikh, Ilaria Vitulano, Martha Stolze, Hale Dedeoglu, Clara Riechert, Ella Kuka, Maryna Sydorova, Mykola Makhortykh, Elizaveta Kuznetsova

arXiv:2609.12653v1cs.CL

TL;DR

Prior work on propaganda detection offers limited evidence across languages and web domains and often judges sources rather than document content. SWARM provides a multilingual, human-annotated search-results dataset and benchmarks source-based and content-level approaches. Source-based labels miss much narrative support, while zero-shot LLMs outperform supervised baselines on the rare support class, with smaller models over-predicting support.

  • Problem

    Propaganda-detection research has limited domain and language coverage and often uses source-based labels rather than judging whether content supports a narrative.

    Source: §1 · p. 1

  • Method

    SWARM contains 2,183 search-result documents across nine languages, human-labeled for support of recurring Russian propaganda narratives and used to benchmark source-based and content-based models.

    Source: §1 · p. 2, §3 · p. 3, §3.5 · p. 4

  • Results

    Zero-shot LLMs outperform supervised baselines on the minority support class, while the source-based baseline misses most supporting documents and smaller LLMs over-predict support.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings identify content-level, per-language detection as a central goal for future work on search-borne propaganda.

    Source: §7 · p. 9

    Support is rare and operationalized as a fixed binary judgment, which cannot capture partial endorsement, irony, or framing effects.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Russian state propaganda spreads across many languages and online spaces. Yet, most computational work examines only one such space, usually social media, in one or two languages, and analyses sources rather than content. We introduce SWARM (Search-Web documents Annotated for Russian propaganda, Multilingual), a dataset of 2,183 search engine results across nine languages and diverse web domains (e.g., news, blogs, government sites), each annotated by trained coders for whether it supports a recurring Russian propaganda narrative. We benchmark a source-based blocklist, supervised classifiers, and zero-shot LLMs against these labels. The blocklist misses most propaganda-supporting documents, because such content is not confined to flagged "propaganda" outlets but also appears on mainstream ones. Content-level analysis helps, though how much depends on the model: the strongest LLM reaches a positive-class F1 of 0.73, whereas the supervised classifiers reach only about 0.5, with the smaller LLMs over-predicting support, mistaking topical relevance for endorsement. Detecting search-borne propaganda thus requires per-language, content-level evaluation, which we hope SWARM and our evaluation code enable.

1 Introduction

Prior propaganda-detection work is limited in domain and language coverage and often labels sources or narrative presence rather than content-level support. SWARM addresses these gaps with a multilingual, human-annotated search-results dataset and benchmarks source- and content-based detection approaches.

1 Introduction

  • Prior work concentrates on news and social media, covers few languages, often relies on outlet-based labels, and does not directly label narrative endorsement.Existing schemes may identify a narrative or persuasion technique without judging whether a document supports that narrative.Source: §1 · p. 1
  • SWARM contains 2,183 search-engine documents across nine languages, annotated by trained coders for support of a specific pro-Kremlin narrative.The dataset covers twenty narratives and yields narrative-specific labels rather than broad propaganda-presence judgments.Source: §1 · p. 2
  • The source-based baseline misses most narrative-supporting documents, while zero-shot LLMs outperform supervised baselines on the minority support class.Smaller LLMs over-predict support by conflating topical relevance with endorsement.Source: §1 · p. 2
  • The study benchmarks a source-based outlet-reputation baseline against supervised classifiers and zero-shot LLMs using the human labels.The narrative framework covers twenty recurring pro-Kremlin narratives organized around five central actor groups.Source: §1 · p. 2

2 Related Work

Prior datasets largely label persuasion techniques, outlet reputation, or narrative type rather than document-level narrative support, and mostly draw on social media and a small number of languages. Search engines are a distinct, manipulable channel that prior audits characterize without judging narrative support in results.

Veracity, framing and narrative

  • Veracity and stance tasks do not capture whether a document endorses a recurring narrative, which cannot be reduced to topic or sentiment.Propaganda can be true, unfalsifiable, or value-laden, while stance targets a single bounded proposition.Source: §2 · p. 2

Propaganda datasets

  • Existing datasets rarely label narrative support, instead tagging persuasion techniques, outlet reputation, or narrative type, and mostly cover social media and English or Russian.Even EUvsDisinfo and RU22Fact label disinformation or veracity rather than narrative support.Source: §2 · p. 2

Search engines

  • Search engines are an actively manipulated information channel, but prior audits describe surfaced content without detecting narrative support in results.Pro-Kremlin operations have exploited data voids, and audits report substantial misinformation on Russia–Ukraine topics.Source: §2 · p. 3

3 The SWARM Dataset

SWARM labels search-retrieved documents for support of recurring Russian propaganda narratives, using a multilingual audit and human annotation. Its retained corpus spans nine languages and varied web sources, while annotation disagreements and scraping losses qualify the resulting labels.

3 The SWARM Dataset

  • SWARM labels each search-retrieved document for whether it supports the recurring Russian propaganda narrative it was retrieved for.Each narrative is operationalised as a yes/no question.Source: §3 · p. 3

3.1 Query corpus

  • Twenty narratives across five actor groups were paired with framed and neutral queries, translated into nine languages for 360 total queries.The narratives cover politics, economics, war, pro-Russian/anti-Western values, and anti-gender/LGBTQ values.Source: §3.1 · p. 3

3.2 Algorithmic audit

  • The audit issued queries to Google, Bing, Yandex, and DuckDuckGo across six regions on 29–30 January 2024, yielding over 1.3 million result listings.The listings contained 30,265 unique URLs from 7,191 domains.Source: §3.2 · p. 3

3.3 Sampling

  • A stratified sample targeted 300 documents per language, drawing from both query framings and audit regions and engines.The final sample comprised 53% propaganda-framed-query results and 47% neutral-query results.Source: §3.3 · p. 4

3.4 Scraping

  • 2,183 of 2,700 sampled rows were retained after pages were crawled and their main text extracted.The 517 exclusions included 293 extraction failures, 204 texts annotators could not assess, and 6 label ties; recovery reduced source skew, but the retained set may slightly under-count positives.Source: §3.4 · p. 4

3.5 Annotation

  • Two annotators labelled each document independently, with a third adjudicating disagreements; the released set contains 2,183 majority-vote labels.Annotators judged English machine translations, and documents entered the set only when at least one coder could assess them.Source: §3.5 · p. 4
  • Pooled inter-annotator agreement was moderate (Krippendorff’s α = 0.54), and disagreements clustered on evaluative rather than concrete, checkable narratives.Adjudication was needed on 16% of double-coded items; in disagreements, adjudicators chose “not support” in 62% of cases, paralleling the models’ endorsement-boundary error.Source: §3.5 · p. 5

3.6 Dataset description

  • 388 of 2,183 documents (18%) support a narrative, while only 2% of results link to social media and support rates vary by source type.Support was highest on social media (33%) and blogs (25%), around 20% on news and government sites, and lowest on international organisation and Wikipedia pages (3%); blog classification is less reliable.Source: Table 2 · p. 4, §3.6 · p. 5

4 Experiments

The benchmark asks models to classify whether a document supports its associated narrative, using the same labelled items across methods. It compares a source-based baseline, supervised classifiers, and zero-shot LLMs, with F1 reported overall and by language.

4 Experiments

  • Each model predicts narrative support from a document and a yes/no question, and all methods are evaluated on the same 2,183 labelled items.Supervised models are scored out of fold rather than on a held-out split.Source: §4 · p. 5

4.1 Models

  • The benchmark compares a domain blocklist, three supervised baselines, and four zero-shot LLMs.The supervised models include SetFit, a few-shot method chosen to suit the 388 positive labels spread across nine languages.Source: §4.1 · pp. 5–6

4.2 Evaluation

  • Positive-class F1 is reported overall and per language, with bootstrap 95% confidence intervals and both English-translated and native-text inputs.Per-language reporting addresses the risk that aggregate scores mask language-dependent failures.Source: §4.2 · pp. 6–7

5 Results

Content-level zero-shot LLMs outperform supervised baselines on narrative support, but performance varies by model and language. The source blocklist misses many supporting documents, while systems often confuse topical discussion with endorsement.

Results

  • 0.39 positive-class F1 is the blocklist’s overall score; it recovers only 35% of supporting documents, and 69% of its overlooked positives come from mainstream or institutional domains.Blocklist recall is 41% on news pages, 22% on blogs, and 3% on research institute pages; a missed example is a single German taz commentary, not the outlet’s editorial line.Source: §5 · p. 7
  • 0.62–0.71 overall F1 for zero-shot LLMs exceeds the supervised methods’ 0.47–0.51, and the best zero-shot model outscores the best supervised method in every language.The per-language margins range from 0.05 in Portuguese to 0.31 in Ukrainian.Source: §5 · p. 7
  • 0.71 overall F1 makes GPT-5.4 the top model, but the margin over GPT-5-nano (0.70) is small and language winners vary.An oracle choosing the best model and input per language averages 0.74 F1, only 0.01 above GPT-5.4 on native input throughout (0.73); some language comparisons rest on few positives.Source: §5 · p. 7
  • Native versus English-translated inputs shift each model’s positive-class F1 by at most 0.02 in either direction.Native text slightly lowers scores for open LLMs, while GPT-5.4 and most supervised baselines score slightly higher on native text.Source: §5 · p. 8
  • Smaller models over-predict support on on-topic documents that do not endorse a narrative, while SetFit under-predicts support.GPT-5-nano has recall 0.76 and precision 0.65; GPT-5.4 is more precise (0.73) with lower recall (0.69), leading it on F1.Source: §5 · p. 8

6 Discussion

The discussion argues that source is not a reliable proxy for document-level narrative support, and attributes the supervised models’ weaker scores partly to scarce positive labels. It highlights the distinction between discussing a narrative and endorsing it, alongside language variation and remaining limits on detector reliability.

6 Discussion

  • Narrative support appears on ordinary media as well as flagged outlets, so source is not a usable proxy for content.The discussion connects this result to prior critiques of source-based labels and amplification through ordinary media.Source: §6 · p. 8

Zero-shot LLMs beat supervised baselines

  • With only 388 positive examples, supervised encoders may lack enough labels for a robust boundary; they also read only the first 512 tokens.Capping LLMs at 512 tokens costs Qwen2.5-7B 0.03 F1 and Qwen2.5-72B 0.01, yet both remain ahead of the best supervised baseline, so the authors identify label scarcity as the main factor.Source: Zero-shot LLMs beat supervised baselines · p. 8
  • Systems struggle to distinguish raising a narrative from endorsing it, and the authors identify making that boundary explicit as a direct route to reducing over-prediction.They suggest asking for an endorsing passage; narrative-typing datasets record which narrative appears, not whether it is endorsed.Source: Zero-shot LLMs beat supervised baselines · p. 8
  • No system is reliably best across languages, and the authors recommend per-language, narrative-level reporting rather than a single aggregate score.A frontier model reaches 0.73 positive-class F1 on average, which the authors describe as short of reliable; downstream prevalence estimates should correct for classifier error.Source: Zero-shot LLMs beat supervised baselines · pp. 8–9

7 Conclusion

SWARM is a multilingual, human-annotated dataset for detecting support for Russian propaganda narratives in search results, benchmarked against source-based, supervised, and zero-shot methods. Source-level analysis misses most support, while zero-shot LLMs outperform supervised baselines on the rare support class but remain unreliable.

7 Conclusion

  • SWARM is a multilingual, human-annotated dataset for detecting support for Russian propaganda narratives in search engine results.The paper benchmarks source-based, supervised, and zero-shot approaches against its labels.Source: §7 · p. 9
  • Source-level analysis misses most narrative support, much of it from mainstream outlets; zero-shot LLMs outperform supervised baselines on the rare support class but remain unreliable.Smaller LLMs over-predict support by mistaking topical relevance for endorsement.Source: §7 · p. 9
  • The authors identify per-language, content-level detection as a central future goal, including task-tailored prompting, in-domain fine-tuning, and precision-oriented decision rules.They hope SWARM, its guidelines, and evaluation code drive progress toward auditing search results and filtering what language models ingest.Source: §7 · p. 9

Limitations

The benchmark has boundaries in retrieval, labels, time, and evaluation scope. In particular, rare positive labels and the snapshot design constrain interpretation, while supervised and LLM comparisons have additional limitations.

Retrieval selection bias

  • Documents enter the benchmark only when retrievable as clean text; the remaining non-retained results are mildly skewed toward blocklisted sources (18% versus 15%).The authors characterize any remaining under-representation of overtly propaganda-supporting sources as small.Source: Limitations · p. 9

Label scarcity and construct

  • Positive support labels are rare (18% overall; as low as 8% in some languages), widening positive-class metric intervals; binary labels omit partial endorsement, irony, and framing effects.Chance-corrected agreement is moderate and uneven across languages.Source: Limitations · p. 9

Data age and duplication

  • The corpus is a snapshot from 29–30 January 2024, and 40 documents (1.8%) fall into near-duplicate groups, including two groups with conflicting labels.Some cross-validation leakage is possible for supervised baselines, though the paper says it cannot account for their gap to zero-shot models.Source: Limitations · p. 9

Scope of the benchmark

  • Supervised baselines cover three methods and are constrained by scarce positives and 512-token truncation; only six regions and a two-day search window are represented.Other frontier models and fine-tuned LLMs were not evaluated, and LLM detectors have language-dependent biases.Source: Limitations · p. 9

Ethics Statement

The annotation process used fluent or native speakers and allowed them to skip or stop when exposed to harmful content. The dataset is gated for research use because of rights constraints and dual-use concerns.

Ethics Statement

  • Research-team members and collaborating volunteers who were native or fluent speakers annotated the documents; participation was voluntary and informed.Annotators could pace their work, skip items, or stop, given exposure to propaganda and sometimes dehumanising content.Source: Ethics Statement · p. 10
  • A positive label records document support for a narrative, not an author's or publisher's personal view or an outlet's editorial position.Source: Ethics Statement · p. 10
  • The dataset is gated to research-use-only access because of possible terms-of-service and copyright constraints and the dual-use potential of its labels.The release includes a request process and takedown procedure; quoted examples are minimized and stripped of identifying information.Source: Ethics Statement · p. 10

Data and Code Availability

The appendix describes gated data and code access, the annotation procedure, translation checks, narrative targets, and zero-shot setup. Annotations judge support for a narrative in relevant document text, while translation checks found no materially inadequate translations in the rated sample.

Data and Code Availability

  • The 2,183 annotated documents, labels, narrative questions, and source and English text are available through gated research-use access; evaluation code covers the benchmark experiments and analysis.Source: Data and Code Availability · p. 10

A Annotation Codebook

  • Annotators read each question and document, judged only the passage relevant to the query, and could consult the source page when extracted text was unclear.Items also provide document and query information; unassessable items are left blank and excluded.Source: A Annotation Codebook · p. 13
  • A positive label requires clear support presented as fact or supported by evidence; a negative label includes debunking, mixed presentation without a conclusion, or unrelated text.A document need not come from a Russian or otherwise disreputable source to receive a positive label.Source: A Annotation Codebook · p. 13

B Translation Quality

  • 86% of rated translations were adequate and 14% had minor loss; none were rated inadequate in the stratified spot check.Source-language evaluation changed positive-class F1 by at most 0.02; the authors conclude translation preserved content needed for narrative-support detection.Source: B Translation Quality · p. 14, Table 6 · p. 14

C Propaganda Narratives and Questions

  • The twenty narrative questions are issued as both propaganda-framed and neutral queries and span actor groups including the United States, the West, the EU, Russia, and Ukraine.Examples include claims about US economic decline, Western media censorship, EU sanctions, Russian economic stability, and NATO expansion and Ukraine.Source: C Propaganda Narratives and Questions · p. 14, §1 · p. 14, The West · p. 14

D Zero-shot LLM Configuration; E Full Prompt

F Full Results

The full-results section reports overall and per-language evaluation metrics, while domain validation finds high document-weighted category accuracy alongside weaker accuracy for some categories and fallback assignments.

F Full Results

  • Table 8 reports overall precision, recall, positive-class F1, balanced accuracy and confidence intervals; Table 4 gives per-language F1, while Table 11 details blocklist flags and misses.Source: F Full Results · pp. 15–16

G Domain Categorisation and Validation

  • Document-weighted accuracy is 92%; category accuracy is 50% for blogs, 42% for research institutes, 92% for news and government, and 100% for international organisations, Wikipedia, and other.The gpt-5-nano fallback is least reliable, with 72% of its assignments correct.Source: G Domain Categorisation and Validation · p. 16

H Retrieval Selection Bias

The section examines retrieval selection, two prompt and context comparisons, and the dataset’s scope and release conditions. The matched-context comparison finds that truncation reduces open-model F1, but both models remain above the supervised baseline.

H Retrieval Selection Bias

  • The retained set requires successful text extraction and assessment; remaining failures may slightly under-represent overtly propaganda-supporting sources, making observed support rates a lower bound.Recovery removed most of the selection effect, but 293 URLs were lost to extraction failure in the final release.Source: H Retrieval Selection Bias · p. 16

I Prompt Ablation

  • 38 of 2,183 Qwen2.5-7B predictions and 33 of 2,183 Qwen2.5-72B predictions change when the prompt instruction is removed, with opposite F1 directions.Positive-class F1 moves from 0.624 to 0.633 for the 7B model and from 0.655 to 0.653 for the 72B; there is no consistent accuracy effect.Source: I Prompt Ablation · pp. 16–17

J Matched-Context Comparison

  • 0.028 positive-class F1 is lost by Qwen2.5-7B and 0.013 by Qwen2.5-72B after truncation, yet both remain above the strongest supervised baseline at 0.514.Input budget explains part, but not most, of the performance gap.Source: J Matched-Context Comparison · p. 17

K Data Statement

  • The corpus covers search results for twenty narratives in propaganda and neutral framings across nine languages; records include document text, query, operationalising question, and support label.Access is gated and limited to research use, with a request process and takedown procedure.Source: K Data Statement · p. 18
Loading 2609.12653v1…