Agentic Detection of Online Conspiracies

1 Introduction

• The central challenge is inferring whether a speaker endorses conspiracy-related content, since the same surface text may express legitimate concerns, criticism, satire, or mockery.Temporal, conversational, and social contexts can help distinguish commitment from mere presence of conspiracy-related content.Source: §1 · pp. 1–2
• The proposed agent can sample a user’s posts, inspect metadata and profiles, query the user’s social circle, and use network structure as contextual evidence.These signals are contextual hints rather than direct proof of intent.Source: §1 · pp. 1–2
• For H1, contextual workflows recover a sarcastic non-conspiracy reading by using storm-related context, the author’s profile, prior weather posts, and ego-network evidence.Text-only classification mistakes the tweet’s strong conspiracy markers for endorsement.Source: §1 · p. 2, Table 2 · p. 3
• Appending every available context to each tweet may be inefficient, introduce noise, and harm performance, motivating per-case tool selection.The agent can decide which tool to use next or whether available data is sufficient.Source: §1 · p. 2
• F1 improved from 0.535 to 0.730 over the text-only LLM, and the agentic approach achieved the best results against both text-only and corresponding preloaded-context conditions.The evaluation uses a manually annotated dataset of Hebrew tweets.Source: §1 · p. 2
• The Hebrew tweet archive is estimated to cover 80–90% of public Hebrew tweets from December 2018 to June 2023, spanning pandemic waves and five general-election cycles.The paper describes these linguistic and social conditions as making Hebrew conspiracy-tweet classification particularly challenging.Source: §1 · pp. 2–3
• The paper’s contributions include evaluating context-aware agentic retrieval and a two-agent debate framework, analyzing efficient tool-use patterns, and examining token-economy gains.The framework is evaluated on Hebrew tweets.Source: §1 · pp. 2–3

2 Related Work

• Conspiracy-detection research is described as sparse, with prior work modeling narrative structure, using event-relation graphs, or analyzing multimodal discourse.Source: §2 · p. 3
• Existing datasets and classifiers cover multiple topics, while cross-topic generalization remains variable and can decline substantially (Langguth et al. 2023; Liaw et al. 2023; Phillips, Ng, and Carley 2022; Fort et al. 2023).George et al. (2024) report improved classification over text alone after adding psycho-linguistic features.Source: §2 · p. 4
• LLM studies address conspiracy detection and related tasks, and human-labeled Reddit data has been used to distinguish endorsement from criticism, skepticism, and debunking (Pustet, Steffen, and Mihaljević 2024; Liu et al. 2024; Diab, Nefriana, and Lin 2024).Source: §2 · p. 4
• Misinformation research uses content, user, temporal, and network signals, while agentic verification systems retrieve external evidence or assign analytical subtasks (Castillo, Mendoza, and Poblete 2011; Ma et al. 2016; Bian et al. 2020; Li, Zhang, and Malthouse 2024; Li et al. 2025).Source: §2 · p. 4
• Unlike factual verification, this task concerns pragmatic function; the agent selectively retrieves metadata, profiles, prior discourse, and ego-network context, without task-specific fine-tuning.The authors describe this as the first agentic conspiracy-detection framework to selectively retrieve user information, prior discourse, and network context.Source: §2 · p. 4

3.1 Data Curation

• The archive contains 270 million Hebrew tweets from 3.3 million users and is estimated to cover 80–90% of public Hebrew Twitter during the study period.Tweets were collected from December 2018 to March 2023; coverage was extrapolated from tracked terms against a random 10% Decahose sample.Source: §3.1 · pp. 4–5
• High-recall keyword filters retain conspiracy narratives and related discussion, including questions, legitimate vaccine concerns, and anti-vaccination content.The retrieval rules combine topic-specific keywords with general conspiracy cues; tweets must match more than one filter.Source: §3.1 · p. 5
• The retrieval process produced 33,121 unique candidate tweets, while Gemini 3 Flash and GPT-5 disagreed on 3,915 candidates (11.8%).The crop-duster example shows that retrieval detects topic and conspiracy signals but does not determine the tweet’s label.Source: §3.1 · p. 5
• The 504-tweet evaluation sample oversampled cases where Gemini 3 Flash and GPT-5 disagreed.Table 3 reports the composition of both the evaluation set and full candidate pool by the models’ text-only predictions, with disagreement groups balanced through oversampling.Source: §3.2 · p. 5, Table 3 · p. 5

3.2 Data Annotation

• Annotation is demanding because labels depend on communicative function and often require examining context, rather than relying on surface wording alone.The task draws on domain expertise and familiarity with niche memes, in-group language, and internet culture.Source: §3.2 · p. 5
• Two domain experts labeled all 504 tweets through independent review and consolidation, producing 331 non-conspiracy labels (65.7%) and 173 conspiracy labels (34.3%).The first expert labeled from scratch with model predictions hidden; the second reviewed labels and consulted contexts when further evidence seemed necessary.Source: §3.2 · p. 5

4.1 The Agentic Pipeline

• The agent follows an iterative reasoning-and-action workflow, using available tools to gather external context while classifying a target post.The workflow follows the interactive approach of ReAct rather than Toolformer’s more static API invocation.Source: §4.1 · pp. 5–6, Figure 1 · p. 6

4.2 Contexts and Tools

• Four tools retrieve target-tweet metadata, author profile information, recent user history, and a retweet-based ego-network when needed during reasoning.Only data timestamped before the target tweet is served, and returned tweet counts, time windows, and neighbors are capped to curb runtime.Source: §4.2 · p. 6, Tweet Information (metadata) · p. 6, Author Information (metadata) · p. 6

4.3 Baseline Models

• Baselines include a tweet-only LLM, a tool-free model given preloaded context, and a two-agent debate in which a second analyst critiques the first analyst’s reasoning.In the debate, the first analyst reweighs the decision after the second analyst focuses on possible non-literal interpretations; both see the target tweet and the other analyst’s reasoning, not social-context tools.Source: Tweet-only LLM · p. 6, Degenerate Agent · p. 6, Agents-debate Baseline · pp. 6–7

4.4 Experimental Setting

• The experiments separate context availability, evidence type, and access mechanism, contrasting preloaded contexts with agents that choose whether and when to query them.Each setting tests individual contexts or tools and a full-context version; classification is binary at tweet level, with accuracy, precision, recall, and F-Score reported.Source: §4.4 · p. 7
• Gemini-3-Flash serves as the text-only and single-agent model, GPT-5 serves as the debate’s second agent, and all settings use temperature 0.2.The workflows were implemented with LangChain and LangGraph.Source: §4.4 · p. 7

5.1 Overall Performance

• Among single-tool conditions, User History performs best (Accuracy: 0.748; F: 0.698), while Two-agent Debate beats the single-LLM baseline but trails all other settings.The authors interpret the User History result as evidence for the agentic framework over supplying elaborated contexts as input.Source: §5.1 · p. 7
• The full-context agent reduces false positives from 142 to 76 and false negatives from 58 to 30, improving classification across both classes.Table 6 presents the confusion matrices for the text-only LLM and Agentic: All Tools.Source: §5.1 · p. 7, Table 6 · p. 8

5.2 Error Analysis

• Among tweets the baseline classified correctly but the full agent misclassified, 78% were false positives; incomplete context can distort interpretation.The authors note that some words should be taken at face value and that incomplete contexts may depend on hyperparameter settings.Source: §5.2 · p. 7
• Shared errors include confusing conspiracy claims with strong political criticism or corruption accusations, and misclassifying mockery or discussion of conspiracy theories.The authors note that the agent correctly classifies many such cases.Source: §5.2 · p. 7

5.1 Overall Performance

• 0.790 accuracy and 0.73 F-score for Agentic: All Tools exceed text-only (0.6 accuracy, 0.535 F1-score) and degenerate full-context (0.707 accuracy, 0.67 F-score).The comparison is reported in Table 5.Source: §5.1 · p. 7, Table 5 · p. 8

5.3 Tool-Use Patterns

• The agent uses no tools in 10.9% of cases; on those cases, it and the text-only LLM make identical predictions with 94.5% accuracy.When at least one context is requested, text-only accuracy is 56.1%, compared with 77.1% for the agent.Source: §5.3 · p. 8
• User Info and User History are both requested in the first step in 36.3% of cases; the most common three-tool sequence is User Info + User History → Network at 12.3%.User Info is used in 89.1% of cases and User History in 81%.Source: §5.3 · p. 8, Table 7 · p. 8, Table 8 · p. 8

5.4 Analysis of ‘Token Economy’

• The full agent averages 11,434 input characters and 1,048 output characters per instance, versus 8,251 and 614 for the degenerate agent, alongside accuracy gains from 0.707 to 0.79 and F-score gains from 0.67 to 0.73.The authors describe this as a tradeoff and say a more thorough analysis is beyond the work's scope.Source: §5.4 · p. 9

Contexts

• Context helped agents correctly interpret a verbatim conspiratorial quote as mockery and a seeming mockery of conspiracy theories as promotion.The authors describe context as calibrating textual interpretation, with user history, author information, and ego-network cues helping situate the post.Source: §6 · p. 9

Multi-Agent Debate

• In the debate setting, analysts initially disagreed on 17.9% of tweets; Analyst 1 reversed decisions after critique in 60% of cases, with 66.7% of those changes correcting an initial error.A correct decision was reversed in 33.3% of the changes.Source: §6 · p. 9
• The reported debate changes are qualified by incomplete visibility into reasoning and token counts that depend on model BPE and other optimization practices.Source: §6 · p. 9

Limitations

• The analysis is limited to this dataset and the specific models used; other or future models may perform differently in reasoning or token efficiency.The authors did not optimize hyperparameters or explore other model settings.Source: §6 · p. 9

An Ethical Note

• Using user histories and social-network context raises privacy considerations, and incorrect user labels may incur social costs.The authors say the system is better suited to research than automated moderation without human review.Source: §6 · p. 9

7 Conclusion

• The agentic approach significantly outperformed other approaches, even when the exact same contextual information was available.The paper frames Hebrew conspiracy-tweet classification as a context-dependent interpretation task, evaluated using a large local tweet archive and manually labeled tweets.Source: §7 · p. 9

A Prompt Templates and Agent Instructions

• The appendix documents the LLM prompt templates and agent instructions used in the study, including experimental prompts reproduced from implementation.The preliminary sampling prompt is supplied in English translation; its predictions were used only for disagreement-based evaluation-set sampling, not as gold labels or reported results.Source: A Prompt Templates and Agent… · p. 11, §A.1 · p. 11

A.4 On-Demand Retrieval

• On-demand retrieval instructs the agent to use tools only for ambiguous tweets, select tools for missing information, and stop when classification is confident.The listed tools provide tweet metadata, author profiles, prior tweets, and interacting users’ context.Source: §A.4 · p. 12

A.5 Preloaded Context; A.6 Two-Agent Debate

• The preloaded conditions provide contextual information with the tweet, while the debate condition uses two agents and no external contextual tools.Available context was matched across on-demand and preloaded conditions; the debate workflow has an analyst challenge an initial classification when evidence supports an alternative interpretation.Source: §A.5 · pp. 12–13, §A.6 · p. 13

A.6 Two-Agent Debate

• The moderator considers Analyst 2’s argument and revises the original opinion only if Analyst 2 gives a convincing reason.Source: §A.6 · p. 13
• The moderator must return only a JSON object with a conspiracy label, confidence score, and rationale reflecting the final stance after considering Analyst 2.Labels are 1 for conspiracy and 0 for non-conspiracy; the confidence score ranges from 0 (not conspiracy) to 1 (conspiracy).Source: §A.6 · pp. 13–14

• The appendix contains original Hebrew texts for tweet examples.Source: B Original Hebrew Examples and… · p. 14
• English translations accompany the Hebrew examples.Source: B Original Hebrew Examples and… · p. 14
• The examples’ posting dates are also provided.Source: B Original Hebrew Examples and… · p. 14

• Example H1 is identified as the Pfizer storm tweet.Source: §B.1 · p. 14
• Example H1 is dated December 20, 2021.Source: §B.1 · p. 14
• Its English translation links the storm to Pfizer’s alleged secret plan for world takeover and a New World Order.Source: §B.1 · p. 14

B.2 Discussion Examples

• The lemon example recounts a forwarded cancer-cure claim and alleged pharmaceutical concealment, ending with the speaker calling themself the family’s “black sheep.”Source: §B.2 · p. 14
• The Zombie Guide says the label “conspiracy theory” can be used to avoid investigating a supposedly frightening truth.Source: §B.2 · p. 14

B.3 Examples from Table 1

• The crop-duster tweet links photographed planes to deadly air pollution and says responsibility is being blamed on the public.Source: §B.3 · p. 14
• The Rabin-assassination report recounts a teacher’s uncertainty about the killer and the parents’ subsequent complaint to education authorities.Source: §B.3 · p. 14
• The COVID-vaccine example invokes tracking chips and a secret database in a share-this appeal framed as a conspiracy claim.Source: §B.3 · pp. 14–15
• The hospital-strike example combines a patient’s postponed surgery with criticism of politicians and the government.Source: §B.3 · p. 15
• The climate-policy example argues that Israelis could sue their government over its response to global warming.Source: §B.3 · p. 15

C Error Analysis Examples

• One example blames justice-system leaders for eroding public trust, alleging dirty investigations, selective enforcement, and susceptibility to blackmail.Source: C Error Analysis Examples · p. 15
• A satirical example imagines a classified new virus and a 75-year disappearance for those who contract it.Source: C Error Analysis Examples · p. 15

D Conspiracy Topics and Retrieval Lexicon

• The appendix describes lexical resources for candidate retrieval using general conspiracy cues and topic-specific rules; a match is not automatically labeled conspiratorial.Source: D Conspiracy Topics and Retrieval… · p. 15

D.1 Conspiracy Topic Guide

• Its listed topics also include global elites, QAnon, chemtrails, 9/11, climate denial, and a moon-landing hoax.Source: §D.1 · p. 16
• The topic guide includes narratives about UFOs, a hidden deep state, flat-earth claims, and pharmaceutical suppression of cures.Source: §D.1 · pp. 15–16

D.2 Retrieval Rules

• General conspiracy cues are expressions that may indicate conspiratorial framing across multiple topics.Source: §D.2 · p. 17
• A tweet is retrieved when it meets either the topic-specific phrase threshold or keyword threshold, and also meets the general-cue threshold.Source: §D.2 · p. 17
• The listed tables provide topic-specific retrieval rules, including for Flat Earth, Yemenite Children, Rabin assassination, and Health and COVID conspiracy.Source: Table 11 · p. 17, Table 12 · p. 17, Table 13 · p. 18