MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Dataset

1 INTRODUCTION

• Existing datasets need greater scale and breadth, and claim-text-only models may not learn event-independent predictors, motivating social-network and article context.The authors argue that claim texts are inherently event-dependent and that surrounding context should be used.Source: §1 · p. 2
• Because Twitter posts are short, MuMiN links claims to posts plus media, shared articles, user networks, and interactions, and targets multiple languages.The paper also frames misinformation as multimodal, multilingual, and multitopical.Source: §1 · pp. 1–2
• MuMiN-trawl is released as open collection and linking infrastructure so future research can continue building and extending the dataset.Source: §1 · p. 2
• The paper proposes two social-network tasks: determining claim veracity from network context and whether a post discusses a misleading claim.Baseline results include text-only, image-only, and heterogeneous graph neural network models.Source: §1 · p. 2
• MuMiN releases 21 million tweets in 26,000 threads linked to 13,000 fact-checked claims across 41 languages, topics, events, and domains.The dataset spans more than a decade and includes replies, users, images, articles, and hashtags.Source: §1 · p. 2

2 RELATED WORK

• Some datasets focus on a single topic, while PHEME sought multiple events but has fewer events, claims, and tweets than MuMiN.Source: §2 · p. 2
• Fact-checking resources underpin datasets such as Liar and FakeNewsNet, whose labelled news-article counts are smaller than MuMiN’s.The passage reports 240 labelled articles in [37] and 1,056 in.Source: §2 · p. 2
• MM-COVID covers six languages but one topic; MuMiN is larger across claims, tweets, languages, and topics and links content using embedding cosine similarity.MM-COVID links articles to posts by searching for article URLs, titles, and first sentences.Source: §2 · p. 2
• Evidence-based fact-checking approaches use sources such as news, Wikipedia, and social networks to classify claims, while Popat et al. retrieve search results from claim text.Popat et al. also use article linguistic features and source reliability, and simplify some fact-checker labels.Source: §2 · pp. 2–3
• MultiFC includes 26 fact-checking sources but lacks images and social-network data; X-FACT covers 25 languages but contains only text.The cited Hanselowski dataset includes documents from multiple domains but remains strongly text-focused.Source: §2 · p. 3
• FEVER and HoVer use manually annotated Wikipedia-derived claims, but remain monolingual and text-focused compared with MuMiN.HoVer allows claims to require evidence from up to four Wikipedia articles.Source: §2 · p. 3

3.1 Claims

• The Google Fact Check Tools API supplied claims from fact-checking organisations, and MuMiN aimed to expand their diversity in content and language.The API was also used in an earlier misinformation dataset.Source: §3.1.1 · p. 3

3.1.1 Data Collection.

• The collection queried 132 translated coronavirus and covid searches, then scraped 128,070 claims from 115 fact-checking organisations.The initial searches covered 70 languages, with up to 1,000 claims requested per query.Source: §3.1.1 · p. 3
• Claim metadata included source, reviewer, language, verdict, and date; when utterance and review dates were unavailable, a date could be extracted from the article URL.The passage also specifies that only selected claim fields are released.Source: §3.1.2 · p. 3

3.1.2 Data Processing.

• The verdict classifier maps free-text verdicts into misinformation, factual, or other; claims labelled other were excluded from the final dataset.Half-true and half-false verdicts were labelled misinformation, while mostly true verdicts were labelled factual.Source: §3.1.2 · p. 4
• 0.99 macro-F1 for misinformation and factual, and 0.92 with other, for the English-only classifier; the multilingual classifier scored 0.98 and 0.85, respectively.Both scores are reported on the same test set of 1,000 multilingual verdicts.Source: §3.1.2 · p. 4
• The English-only verdict model performed marginally better, but both English-only and multilingual verdict classifiers were released.The English-only approach translates verdicts into English; the multilingual approach uses a multilingual model.Source: §3.1.2 · pp. 4–5

3.2.1 Data Collection.

• The search retrieved about 30 million tweets by querying the first 100 results for each of five keyphrases, with media or link and date constraints.Tweets had to be posted within the three days before a claim or on its date, and replies were excluded.Source: §3.2.1 · p. 5

3.2.2 Data Processing.

• At least five retweets reduced the candidate set by 90%, leaving about 2.5 million tweets; this filter targeted posts with reasonable interest.The authors framed the system's goal as acting on social-media stories before they go viral.Source: §3.2.2 · p. 5
• URLs, hashtags, user descriptions, and shared-image links populated a graph database of about 17 million nodes and 50 million relations.For article URLs, the collection downloaded the title, body, and top image.Source: §3.2.2 · p. 5

3.3 Data Linking

• Translation to English outperformed multilingual embeddings for linking tweets to claims, though the authors note translation can lose context.They used Google Translation API translations to identify tweets discussing a claim.Source: §3.3 · p. 5
• Articles were summarized with BART-large-CNN before embedding, then articles, claims, and tweets were embedded with paraphrase-mpnet-base-v2.Summarization allowed more article text to fit the Sentence Transformers' 512-token limit.Source: §3.3 · p. 5
• Claims were processed in batches of 100, comparing against tweets and articles dated from three days before a claim through its date.This batching avoided computing every tweet-claim and article-claim cosine similarity.Source: §3.3 · p. 5
• The released similarity thresholds of 0.7, 0.75, and 0.8 trade dataset size against label quality: lower thresholds yield more data but more label noise.The thresholds were selected after qualitative evaluation of linked claims.Source: §3.3 · p. 5

3.4 Data Enrichment

• Figure 1 presents the MuMiN dataset's graph schema.The release description names POSTED, MENTIONS, FOLLOWS, DISCUSSES, and IS_REPLY_TO relations.Source: Figure 1 · p. 6
• Table 4 presents examples of claim-article linking at different similarity thresholds.The threshold versions trade dataset size against label noise and quality.Source: Table 4 · p. 6
• For each linked post, the collection sampled users who retweeted it, followed or were followed by its author, replied to it, or were mentioned in it.It then queried each selected user's recent 100 tweets.Source: §3.4 · pp. 5–6

4 DATASET DESCRIPTION

• The authors leave a thorough analysis of the dataset’s scale and diversity to future work, offering only a preliminary analysis here.Source: §4 · p. 6
• The three cosine-similarity-threshold datasets retain heavy class imbalance; trusted-source tweets were excluded because they might come from a different distribution.The thresholds are 0.7, 0.75, and 0.8; the authors note that fact-checking organizations tend to favor novel claims, which tend to be misinformation.Source: §4 · pp. 6–7
• Twitter and copyright restrictions mean the release provides tweet IDs, user IDs, claim keywords, and graph relations rather than tweets, user data, or claim text.The mumin package can use these released components to compile the remaining dataset.Source: §4 · p. 7

4.1 Claim Topic Clusters

• UMAP and HDBSCAN clustering of claim embeddings identified 26 clusters, with a silhouette coefficient of 0.28.Source: §4.1 · p. 7
• The clusters span diverse topics, from natural disasters to political and social events, while approximately half of claims fall in the COVID-19 cluster.Source: §4.1 · p. 7

5 THE MUMIN PACKAGE

• The mumin package enables compilation of the dataset because Twitter terms permit release of tweet and user IDs, but not the underlying data.Source: §5 · p. 7
• MuminDataset’s size argument selects a small, medium, or large dataset, and its constructor can control which data, such as tweet images, are included.Source: §5 · p. 7
• The package exposes graph nodes and relations and can export a heterogeneous graph for DGL; the authors also provide a usage tutorial and package source code.Source: §5 · p. 7

6.1 Dataset Splits

• Validation and test splits comprise distinct claim-cluster events, each covering at least 10% of claims, while the remaining clusters form training data.The authors report that label balance is roughly maintained across splits.Source: §6.1 · p. 7

6.2 Baseline Models

• The two tasks predict claim verdict from a claim and its social subgraph, or predict whether a source tweet discusses a misinformation or factual claim.Source: §6.2 · pp. 7–8
• Baselines include LaBSE text models, a ViT image model, and two-layer HeteroGraphSAGE; the graph model omits profile and article images and user timelines.The authors also release baseline code and an online leaderboard.Source: §6.2 · p. 8
• HeteroGraphSAGE achieves the best overall performance on the two tasks, while text-only LaBSE is not far behind; adding graph hops worsened performance through oversmoothing.The reported HeteroGraphSAGE baseline uses two hops, so it cannot capture all information in the graph.Source: §6.2 · p. 8

7.1 Representative Data Splits

• Because the intended setting is handling new events, the authors compare random splits with their representative splits using LaBSE and HeteroGraphSAGE.They argue that splits should reflect this real-world generalization goal.Source: §7.1 · p. 8
• On claim classification, random splits yielded only marginal improvements over the baselines; LaBSE-random scored 58.85%, 62.80%, and 61.50% on MuMiN-small, medium, and large.HeteroGraphSAGE-random scored 48.50%, 61.40%, and 62.55% on those respective datasets.Source: Table 6 · p. 8, §7.1 · pp. 8–9
• Tweet-task random splits scored substantially higher than the baselines, including 89.50% macro-average F1 for HeteroGraphSAGE-random on MuMiN-medium.LaBSE-random scored 71.10% on MuMiN-small and 73.6% on MuMiN-medium; HeteroGraphSAGE-random scored 74.90% on MuMiN-small and 79.80% on MuMiN-large.Source: Table 7 · p. 8, §7.1 · p. 8

7.2 Limitations

• Automated linking can create erroneous labels; verdicts may also be contentious or misclassified, while deletions can make social-data research difficult to reproduce.The authors note that higher similarity thresholds can partly address linking errors, and provide the fact-checking organization for each claim and verdict.Source: §7.2 · p. 9

7.3 Ethical Considerations

• The dataset uses public Twitter data accessed through the Academic API, but releases IDs and hydration code rather than raw posts, so deleted content becomes unavailable.The authors describe data disappearance as an expected trade-off and report ethics approval from both named review bodies.Source: §7.3 · p. 9

• The two node-classification tasks are challenging, with highest macro-average F1-scores of 62.55% for claim veracity and 61.45% for identifying tweets discussing misleading claims.Source: §9 · p. 9
• MuMiN links 21 million tweets in 26 thousand threads to 13 thousand fact-checked claims across 41 languages, dozens of topics, events, and domains, spanning more than a decade.Source: §9 · p. 9
• The paper releases MuMiN-trawl, a verdict-classification transformer, and the mumin Python package, which compiles the graph and exports it to graph machine-learning libraries.Baselines include text-only, image-only, and heterogeneous graph neural network approaches; data, tutorials, and a leaderboard are available through the project site.Source: §9 · p. 9

• The collection queried 70 languages, of which 41 appear in the final dataset.Source: Table 8 · p. 11, A SUPPLEMENTARY TABLES AND FIGURES · p. 11
• Tables 9–11 report language distributions for the large, medium, and small dataset versions, respectively.Source: Table 9 · p. 11, Table 10 · p. 11, Table 11 · p. 11
• Figure 3 plots the distribution of cosine similarities among tweet-claim pairs.Source: Figure 3 · p. 11, A SUPPLEMENTARY TABLES AND FIGURES · p. 11
• Table 12 lists 115 fact-checking organizations whose claims are included in the dataset.Parenthetical counts indicate how many claims were processed from each website.Source: A SUPPLEMENTARY TABLES AND FIGURES · p. 11, Table 12 · p. 12