PaperlayerView exact revision on arXiv

Source-linked AI summary

Community-based fact-checking reduces the spread of misleading posts on social media

Yuwei Chuai, Moritz Pilarski, Thomas Renault, David Restrepo-Amariles, Aurore Troussel-Clément, Gabriele Lenzini, Nicolas Pröllochs

arXiv:2409.08781v1cs.SI

TL;DR

Real-world evidence on whether community fact-checks reduce misleading-post spread was missing. The study uses repost time series and a Difference-in-Differences design to assess Community Notes on X, finding lower spread and higher deletion odds, while noting that notes may arrive too late for the most viral stage.

  • Problem

    Real-world intervention effects of fact-checking on misleading-post spread had not been quantified, largely because repost time series and intervention data were lacking.

    Source: Introduction · p. 5

  • Method

    The study applies a Difference-in-Differences design and negative binomial regression to 36-hour repost time series from 237,677 community fact-checked X cascades.

    Source: Introduction · p. 5

  • Results

    Community Notes reduced misleading-post spread by an average of 62.0%, and deletion odds were 103.4% higher for posts with displayed notes than for posts without them.

    Source: Discussion · p. 20

  • Takeaways & Limitations

    Community Notes may arrive too late to reduce spread during the most viral stages of diffusion.

    Source: Effect on the Cumulative Repost Count · p. 16

    Only 13.5% of helpful notes were displayed before the 5.75-hour repost half-life, limiting the opportunity to affect the most viral diffusion stage.

    Source: Effect on the Cumulative Repost Count · p. 16

Abstract

from arXiv · show

Community-based fact-checking is a promising approach to verify social media content and correct misleading posts at scale. Yet, causal evidence regarding its effectiveness in reducing the spread of misinformation on social media is missing. Here, we performed a large-scale empirical study to analyze whether community notes reduce the spread of misleading posts on X. Using a Difference-in-Differences design and repost time series data for N=237,677 (community fact-checked) cascades that had been reposted more than 431 million times, we found that exposing users to community notes reduced the spread of misleading posts by, on average, 62.0%. Furthermore, community notes increased the odds that users delete their misleading posts by 103.4%. However, our findings also suggest that community notes might be too slow to intervene in the early (and most viral) stage of the diffusion. Our work offers important implications to enhance the effectiveness of community-based fact-checking approaches on social media.

Introduction

Professional fact-checking faces constraints in volume, reach, and trust, while community-based fact-checking may address some of them. This study examines whether Community Notes reduce misleading-post spread on X, where real-world intervention effects had not been quantified.

Introduction

  • Professional fact-checking can be slow and limited in volume and reach, and many people distrust fact-checkers.Investigating claims can take hours or days, and fact-checking organizations often disseminate findings through their own websites.Source: Introduction · p. 3
  • Community-based fact-checking draws on diverse users’ collective judgments, with studies finding small crowds’ assessments comparable to experts’ and broader post coverage.The approach may also mitigate trust issues associated with simple misinformation flags.Source: Introduction · pp. 3–4
  • Community Notes lets enrolled users flag posts and add explanations that other users rate; helpful notes appear on the posts, but their effect on spread lacked empirical evidence.The paper describes it as the first large-scale attempt to implement community-based fact-checking on a major social media platform.Source: Introduction · p. 4
  • Field studies had not quantified fact-checking’s real-world intervention effect, largely because repost time series and intervention data were missing.Prior field studies mainly compared the virality of misleading and non-misleading posts.Source: Introduction · pp. 4–5
  • The study analyzes 237,677 X cascades, using 36-hour repost time series and a Difference-in-Differences design with negative binomial regression.The cascades were reposted more than 431 million times and spanned over 20 months after Community Notes launched.Source: Introduction · p. 5
  • A key concern is that notes may arrive too late to affect the early, most viral diffusion stage.The authors examine whether display timing diminishes notes’ effect on cumulative repost counts.Source: Introduction · p. 5

Results

Displayed community notes were associated with fewer subsequent reposts and higher deletion odds, but their arrival often lagged the fastest diffusion. Earlier display was associated with larger cumulative reductions.

Data Overview

  • 10.1% of notes were displayed, while mean response time was 61.4 hours versus a 6.25-hour repost half-life for posts with displayed notes.The median response time was 18.1 hours.Source: Data Overview · p. 7, Figure 1 · p. 8

Difference-in-Differences Estimation

Sensitivity Analyses

  • Effects were weaker for posts by verified, older, and higher-follower accounts and for economy, health, and political posts; sentiment and science-topic differences were not significant.Effects were also larger for shorter posts and posts with media elements.Source: Sensitivity Analyses · pp. 13–14

Effect on the Cumulative Repost Count

Effect on Post Deletion

  • 103.4%: displayed notes increased the odds of post deletion; deletion ratios were higher for notes with higher helpfulness scores.The estimate was based on a discontinuity at the 0.4 helpfulness cutoff.Source: Effect on Post Deletion · p. 18, Figure 5 · p. 19

Discussion

Community Notes reduced misleading-post spread on X, but their delayed display limited their impact during the early, often most viral, diffusion stage. Effects also varied by users and content, and the study’s scope was limited to X and primarily English-language posts in the American and Global North context.

Discussion

  • 62.0% lower spread of misleading posts on X with community notes; effects were discounted for influential accounts and varied by content and topic.Attached media amplified the intervention effect, while efficacy was lower for economy-, health-, and politics-related misinformation.Source: Discussion · p. 20
  • Only 13.5% of helpful notes appeared before the 5.75-hour repost half-life, limiting intervention during early, often most viral, diffusion.In its current implementation, community notes reduced the overall number of reposts by 15.3%.Source: Discussion · pp. 20–21
  • The evidence concerns community notes on X and primarily English posts in the American and Global North context.The feature may evolve, so findings may need revalidation; effects across cultural contexts and demographic groups remain future research questions.Source: Discussion · p. 21
  • Community-based fact-checking’s effectiveness hinges on responding swiftly to misinformation’s dynamic dissemination.The discussion frames faster, more effective approaches as important amid the scalability of AI-generated misinformation.Source: Discussion · pp. 21–22

A.1 Data Overview

The dataset contains 237,677 fact-checked English source posts from approximately 20 months, authored by 60,928 users and reposted more than 431 million times. The study also collected user, post, sentiment, and topic characteristics for analysis.

A.1 Data Overview

  • 237,677 English source posts were fact-checked from October 6, 2022, to June 11, 2024, and were reposted more than 431 million times by data collection.The posts came from 60,928 unique users; Table S1 describes the full dataset and displayed-note subsets.Source: §A.1 · p. 36, Table S1 · p. 38
  • User and post controls included author verification, account age, follower and followee counts, program rollout timing, media presence, and word count.Word counts followed text preprocessing that removed leading user mentions and URLs, converted HTML entities, normalized characters, and counted alphanumeric tokens.Source: §A.1 · p. 36
  • Posts were also assigned positive- and negative-emotion probabilities using Twitter-RoBERTa-base and topic labels for politics, health, economy, science, or other.The topic labels were used to study heterogeneity across topics.Source: §A.1 · p. 37

A.2 Fact-Checking Activity Over Time

The supplement describes how often Community Notes were displayed, how note statuses changed, and how posts were classified by topic. About 12% of daily notes were displayed, while most notes remained in “Needs More Ratings”; the topic classifier achieved 0.904 average accuracy.

A.2 Fact-Checking Activity Over Time

  • 12.0% of community notes created each day were displayed on average, with the ratio stable around 0.1 after April 2023 despite declining after January.Figure S1 shows the two-week rolling ratio and the distribution of daily ratios.Source: §A.2 · p. 39, Figure S1 · p. 39

A.3 Stability of Note Status

  • 83.3% of notes remained in “Needs More Ratings,” while 70.0% of notes initially rated helpful retained helpful status in the current and most recent status.No note first rated helpful later became not helpful in this dataset; notes marked “Currently Rated Helpful” are displayed, unlike notes in the other statuses.Source: §A.3 · p. 40

B Identification of Topics

  • A TwHIN-BERT-based classifier was fine-tuned on manually labeled posts and used to assign politics, health, economy, science, or other labels.Two research assistants labeled 1,500 distinct posts, with 175 additional posts labeled by both; agreement was 90.8% and Cohen’s κ was 0.711.Source: B Identification of Topics · p. 41
  • The topic classifier achieved average accuracy of 0.904 and macro-averaged F1 of 0.755 in 10-fold cross-validation.Table S2 reports out-of-sample performance on manually labeled posts.Source: B Identification of Topics · p. 41, Table S2 · p. 42

C Propensity Score Matching

The authors used propensity-score matching to create control and placebo groups with user and post characteristics similar to the treatment group. In two-period ATT models, the Display × After estimate was qualitatively identical with and without additional covariates.

C Propensity Score Matching

  • One-to-one propensity-score matching created control and placebo groups from posts without displayed notes, matching each treated post to its closest match without replacement.The placebo group used a second matching after excluding the first control matches, allowing comparison without actual note display.Source: C Propensity Score Matching · p. 43
  • After matching, user- and post-level variables were balanced across treatment, control, and placebo groups, unlike before matching.Figure S2 shows propensity-score distributions before and after matching; Table S3 reports balance regressions.Source: C Propensity Score Matching · pp. 43–44, Figure S2 · p. 44, Table S3 · p. 45

D.1 Two-Period ATTs

  • The Display × After coefficient was qualitatively identical in the main and extended two-period ATT models.The before-display window was 4 to 2 hours before note display, and the after-display window was 1 to 12 hours after; the hour immediately before display was omitted.Source: §D.1 · p. 46, Table S4 · p. 47

D.2 Parallel Test and Multi-Period ATTs

The multi-period analysis found no statistically significant differential changes in reposts during either of the two pre-display test periods, supporting parallel trends. It then used hourly post-baseline estimates to assess multi-period ATTs.

D.2 Parallel Test and Multi-Period ATTs

  • Before-display estimates were not statistically significant for Display × Before:4 (coef. = 0.015, p = 0.142) or Display × Before:3 (coef. = 0.015, p = 0.148), supporting parallel trends.The second hour before note display was the baseline; the first hour before display was excluded to avoid potential treatment contamination.Source: §D.2 · p. 48
  • Hourly difference-in-differences estimates after the baseline period were used to estimate multi-period ATTs.Table S5 reports the parallel test and multi-period ATT regression results.Source: §D.2 · p. 48, Table S5 · p. 49

E Placebo Analyses

Placebo analyses found no significant extra reduction in reposts without note display, supporting parallel spread patterns in placebo and control groups. Further analyses examined sensitivity across response time, rollout month, rating thresholds, and user and post characteristics.

E Placebo Analyses

  • The placebo group showed no significant extra repost reduction after the display period, and its spread pattern paralleled the control group regardless of note display.The placebo and control groups' repost changes overlapped; hourly and two-period ATT estimates were not statistically significant.Source: E Placebo Analyses · p. 50, Figure S3 · p. 51, Table S7 · p. 53

F.1 Sensitivity Across Response Time

  • Response-time analyses estimated two-period ATTs in four-hour post-age groups, spanning note-display ages from 4 to 24 hours.Posts receiving displayed notes at 4 and 8 hours were grouped together.Source: §F.1 · p. 54, Table S8 · p. 54

F.2 Sensitivity Across Months From Roll-Out

F.3 Sensitivity Across Rating Thresholds

  • Rating-threshold sensitivity analyses used thresholds from 10 to 80 ratings, excluding notes ever rated not helpful.The reported regressions compare two-period ATT estimates across thresholds.Source: §F.3 · p. 57, Table S10 · p. 57

F.4 Sensitivity Across User and Post Characteristics

  • Sensitivity analyses examined efficacy across four user characteristics and eight post characteristics.User characteristics were verification, account age, followers, and followees; post characteristics included words, media, sentiment, and topic.Source: §F.4 · p. 58, Table S11 · p. 59, Table S12 · p. 60

G Analysis of Deleted Posts

A regression discontinuity analysis examined whether displaying a helpfulness-qualified community note changed misleading-post deletion near the 0.4 score cutoff. Displayed notes were associated with a higher estimated deletion effect, which increased with note helpfulness scores.

G Analysis of Deleted Posts

  • The RDD compared deletion probabilities around the 0.4 Core-model helpfulness cutoff, using scores from 0.3 to 0.5 and excluding [0.39, 0.40).Notes scoring at least 0.40 were displayed; NoteScore was the running variable.Source: G Analysis of Deleted Posts · p. 62
  • 1.034 was the estimated treatment effect of note display on deletion, corresponding to posts with displayed notes being 103.4% more likely to be deleted.The estimate's 99% CI was [0.800, 1.297] (p < 0.001).Source: G Analysis of Deleted Posts · pp. 62–63, Table S14 · p. 63
  • The Display × NoteScore coefficient was 2.875 (99% CI: [0.750, 4.999]; p < 0.001), indicating deletion efficacy increased with note helpfulness scores.The NoteScore coefficient was not statistically significant, suggesting deletion probability remained stable across scores without display.Source: G Analysis of Deleted Posts · p. 63

Related papers

Loading 2409.08781v1…