PaperlayerView exact revision on arXiv

Source-linked AI summary

Community notes reduce engagement with and diffusion of false information online

Isaac Slaughter, Axel Peytavin, Johan Ugander, Martin Saveski

arXiv:2502.13322v2cs.SIcs.CYphysics.soc-ph

TL;DR

Research has paid less attention to how platform interventions on false content alter its engagement and diffusion than to the spread of true versus false content. The study uses time-series data for 40,078 posts with proposed notes and synthetic controls to estimate effects of attaching Community Notes. After attachment, notes reduced views 13.5%, replies 21.9%, likes 44.1%, and reposts 46.1%; cascades became less deep and structurally viral than expected from their size reduction.

  • Problem

    Research has paid less attention to how platform interventions on false content alter its engagement and diffusion than to the spread of true versus false content.

    Source: Introduction · pp. 1–2

  • Method

    The study uses time-series data for 40,078 posts with proposed notes and synthetic controls to estimate effects of attaching Community Notes.

    Source: Introduction · p. 2, Estimating Individual Treatment Effects · p. 10

  • Results

    After attachment, notes reduced views 13.5%, replies 21.9%, likes 44.1%, and reposts 46.1%; cascades became less deep and structurally viral than expected from their size reduction.

    Source: Introduction · p. 3, Discussion · p. 9

  • Takeaways & Limitations

    Note attachment changes repost-cascade structure, reducing depth and structural virality more than expected from the overall reduction in cascade size.

    Source: Introduction · p. 3

    The analysis estimates effects only for posts that received notes and cannot establish program coverage or effectiveness across the broader population of misleading posts.

    Source: Discussion · p. 10

Abstract

from arXiv · show

Social networks scaffold the diffusion of information on social media. Much attention has been given to the spread of true vs. false content on online social platforms, including the structural differences between their diffusion patterns. However, much less is known about how platform interventions on false content alter the engagement with and diffusion of such content. In this work, we estimate the causal effects of Community Notes, a novel fact-checking feature adopted by X (formerly Twitter) to solicit and vet crowd-sourced fact-checking notes for false content. We gather detailed time series data for 40,078 posts for which notes have been proposed and use synthetic control methods to estimate a range of counterfactual outcomes. We find that attaching fact-checking notes significantly reduces the engagement with and diffusion of false content. We estimate that, on average, the notes resulted in reductions of 46.1% in reposts, 44.1% in likes, 21.9% in replies, and 13.5% in views after being attached. Over the posts' entire lifespans, these reductions amount to 11.6% fewer reposts, 13.3% fewer likes, 6.9% fewer replies, and 5.5% fewer views on average. In reducing reposts, we observe that diffusion cascades for fact-checked content are less deep and less "viral," but not less broad, than synthetic control estimates for non-fact-checked content with similar reach. This structural difference contrasts notably with differences between false vs.\ true content diffusion itself, where false information diffuses farther, but with structural patterns that are otherwise indistinguishable from those of true information, conditional on reach.

Introduction

Although false information’s spread and platform responses have been studied, less is known about how interventions change false content’s engagement and diffusion. This study estimates Community Notes’ effects, finding reduced engagement and shallower, less viral repost cascades.

Introduction

  • Professional fact-checking commonly uses warning labels, while crowd-sourced fact-checking has been explored as an alternative, including in small-group experiments.The introduction also describes public skepticism toward professional fact-checking.Source: Introduction · pp. 1–2
  • Community Notes lets users propose and rate notes; only notes rated helpful by users with diverse views above a threshold are displayed.The system uses a bridging-based matrix-factorization algorithm to score helpfulness [35].Source: Introduction · p. 2
  • The study tracks 40,078 posts with proposed notes, their engagement over time, and repost histories and user follow graphs to reconstruct diffusion cascades.These records support estimates of note effects and analyses of conditions associated with greater or lesser reductions.Source: Introduction · p. 2
  • Notes significantly reduce views, reposts, likes, and replies, measured both by growth after attachment and total reductions across the post’s lifespan.The two perspectives account differently for engagement before a note was attached.Source: Introduction · p. 3
  • Note attachment reduces cascade depth and structural virality more than expected from the overall reduction in cascade size.The authors contrast this with false-versus-true news, whose cascade structures are similar conditional on reach.Source: Introduction · p. 3
  • The analysis extends prior Community Notes work beyond reposts and deletions and uses synthetic controls rather than difference-in-differences’ parallel-trends assumption [42, 43].The authors describe their estimates, where comparable, as corroborating prior effects.Source: Introduction · p. 3
  • Larger absolute engagement reductions are associated with earlier notes, highly engaging or media posts, altered-media concerns, and moderately long, simpler notes.These are associations across post and note characteristics.Source: Introduction · p. 3

Data Collection

The study collected engagement time series for posts with proposed Community Notes and assembled public repost and reply histories to analyze diffusion. It compared posts receiving helpful notes with synthetic controls built from posts without attached notes.

Data Collection

  • From March 16 to June 23, 2023, the researchers tracked 40,078 posts with proposed notes, recording reposts, likes, replies, and views every five minutes for three weeks after note creation.Engagement histories were aligned to post creation at 15-minute intervals for constructing synthetic-control weights.Source: Data Collection · p. 3
  • Fig. 2 compares average observed engagement with synthetic-control estimates and plots their average treatment-effect differences for views, replies, likes, and reposts.The error bands represent 95% confidence intervals.Source: Figure 2 · p. 4
  • Of the 40,078 posts, 6,757 (16.9%) received helpful notes and formed the treatment group; the other 33,321 proposed-note posts supplied the donor pool.The analysis focused on the 48 hours after attachment, which the authors say closely approximates lifetime effects for posts remaining on the platform.Source: Data Collection · p. 4
  • Full public repost histories were collected for 36,408 posts (90.8%) and full public reply histories for 30,858 (77.0%); deleted or private posts lacked these histories.For those posts, the researchers relied on repost and reply counts returned by the X API.Source: Data Collection · p. 4

Results

Community Notes reduce engagement with noted posts and alter repost-cascade structure. Exploratory analyses associate larger effects with earlier attachment, post characteristics, and note characteristics, but these associations are not causal estimates.

Decline in Average Engagement

  • After note attachment, repost growth fell 46.1%, likes 44.1%, replies 21.9%, and views 13.5% relative to synthetic controls.These are reductions in additional growth after attachment, not total-lifespan reductions.Source: Decline in Average Engagement · p. 5
  • Across posts’ entire lifespans, reposts fell 11.6%, likes 13.3%, replies 6.9%, and views 5.5%.These overall reductions include engagement before note attachment.Source: Decline in Average Engagement · p. 5

Altered Dynamics of Information Diffusion

  • After 48 hours, notes reduced maximum cascade breadth by 46.0%, maximum depth by 39.9%, and structural virality by 48.5% in growth after attachment.Total reductions were 11.7% in breadth, 4.5% in depth, and 3.9% in structural virality.Source: Figure 3 · p. 6, Altered Dynamics of Information… · p. 6
  • Among cascades with similar size growth, noted posts had smaller maximum depth and structural virality than controls, but no difference in maximum breadth.This indicates less viral diffusion beyond what the size change alone explains.Source: Figure 3 · p. 6, Altered Dynamics of Information… · p. 7

Factors Associated with Large Effects

  • Repost reductions were largest for notes attached within 1–12 hours (−49.6% growth) and smallest after 47+ hours (−6.2% growth).The corresponding total repost reductions were 24.9% and 0.1%.Source: Factors Associated with Large Effects · p. 7
  • Posts with more prior reposts had larger absolute drops, while low-repost posts often had small positive changes in visibility.For popular posts, percentage declines were relatively similar, suggesting larger absolute drops mostly reflected larger baseline audiences.Source: Factors Associated with Large Effects · p. 8
  • Effects also varied with post and note characteristics: media posts had larger percentage repost-growth reductions than text-only posts, and altered-media or misleading-satire concerns were associated with large effects.Right-leaning posts had larger estimated repost reductions than left-leaning posts; these factor-effect relationships are exploratory associations, not causal estimates.Source: Factors Associated with Large Effects · pp. 7–8
  • Simpler notes were associated with larger repost reductions: grade level ≤5 corresponded to −47.4% growth, versus −38.8% above grade 10; two- or three-sentence notes were slightly more effective.These are associations, not causal effects of note length or readability.Source: Factors Associated with Large Effects · pp. 8–9

Discussion

Community Notes reduce engagement after attachment, with especially large declines in reposts and likes, while whole-lifespan reductions are smaller. The estimates apply to posts that received notes during March–June 2023, not to X posts generally.

Discussion

  • After attachment, reposts fell 46.1%, likes 44.1%, replies 21.9%, and views 13.5% on average.These are reductions in engagement after notes were attached.Source: Discussion · p. 9
  • Reposts and likes declined more than views or replies, but fewer public signals of support do not necessarily mean lower underlying support.The authors note that users may be less willing to signal support publicly; each prevented view is meaningful because exposure can increase later agreement.Source: Discussion · p. 9
  • Diffusion changes are consistent with stronger behavioral shifts among users who do not follow the original author, possibly through homophily or dyadic social pressures.The authors present these as alternative explanations, not established mechanisms.Source: Discussion · p. 9
  • Across posts’ entire lifespans, views fell 5.5%, replies 6.9%, likes 13.3%, and reposts 11.6%.These overall reductions also account for engagement before note attachment.Source: Discussion · p. 9
  • The estimates cover posts with attached notes during March–June 2023, not program-wide coverage or effects on unnoted posts.The study also could not test whether unnoted posts were more likely to be perceived as accurate or non-misleading, and its results reflect an evolving system.Source: Discussion · p. 10

Materials and Methods

The methods estimate post-level effects by comparing treated posts with weighted synthetic controls, then summarize effects across treated posts and time.

Estimating Individual Treatment Effects

  • Synthetic controls estimate what each treated post’s engagement would have been without a note by matching it to a weighted average of untreated donor posts.Donors had proposed notes that were never found helpful, and were selected using historical data around the treated post’s attachment time.Source: Estimating Individual Treatment Effects · p. 10
  • Weights minimize pre-treatment differences across available engagement and cascade metrics, using a restricted pool of the 1,000 closest control posts.Metrics are standardized before fitting so larger-scale measures do not dominate the match.Source: Estimating Individual Treatment Effects · pp. 10–11
  • Bias correction addresses imperfect pre-treatment matches by using regressions on untreated posts to predict post-treatment outcomes from pre-treatment histories.Ordinary least squares models are fit separately for each metric and post-treatment time point.Source: Estimating Individual Treatment Effects · p. 11

Aggregating Individual Treatment Effects

  • The average estimated treatment effect averages individual effects across treated posts and can be interpreted as the absolute change in the average outcome due to note attachment.The average is defined for each metric and time point.Source: Aggregating Individual Treatment Effects · p. 11
  • Uncertainty is summarized with 95% Gaussian confidence intervals, alongside a permutation test of statistical significance.Subpopulation analyses use confidence intervals restricted to the relevant groups.Source: Aggregating Individual Treatment Effects · p. 11
  • Percentage change normalizes the difference by the estimated outcome without notes, while post-treatment growth measures additional change after attachment.The overall percentage measure also accounts for engagement before a note was attached.Source: Aggregating Individual Treatment Effects · p. 11

A Data Collection Pipeline

The study collected post engagement histories and repost data, defined treatment by helpful-note status, and filtered anomalous or insufficiently observed posts.

A Data Collection Pipeline

  • Posts were monitored from March 16 to June 23, 2023, with engagement counts queried every five minutes for three weeks after note proposal.Collection used the X API and also scanned the public Community Notes data to identify proposed notes.Source: A Data Collection Pipeline · p. 16
  • The final analysis included 40,078 posts after excluding anomalous records, posts lacking one hour of pre-attachment data or 48 hours afterward, and three failed synthetic-control fits.The data-processing filters removed 874 anomalous posts before the duration criteria were applied.Source: A Data Collection Pipeline · p. 16

B Defining Treatment Status

  • A post entered the treatment group if it had a helpful note at any point, even if that status later changed; 93.4% of treated-post views occurred while notes were helpful.For posts with multiple helpful notes, treatment time was set by the first helpful status, and post-level note attributes were time-weighted because the displayed note was unknown.Source: B Defining Treatment Status · pp. 16–17

C Anomalous Post Removal

D Missing Data

The analysis used available metrics rather than excluding posts with incomplete data, reconstructed repost cascades where data permitted, and classified partisanship for English-language posts.

D Missing Data

  • When a metric was consistently unavailable, it was omitted from synthetic-control construction and its treatment effect was not estimated for that post.Table S1 summarizes posts with any observation versus none by metric and treatment status; Table S2 breaks down treatment-post data availability by time window.Source: D Missing Data · pp. 17–18, Table S1 · p. 18, Table S2 · p. 18

E Structural Characteristics of Repost Cascades

  • Repost cascades are directed trees that assign each repost to the most recent followed user who shared the content, or to the original post if none did.Cascade measures include maximum depth, maximum breadth, and structural virality; deleted reposts are excluded, and only posts with sufficient cascade data are analyzed.Source: E Structural Characteristics of Repost… · p. 18
  • Fig. S1 displays 48-hour individual treatment effects by engagement metric, with color showing effect sign and log-scale x-position showing magnitude.The figure distinguishes positive effects, meaning increases, from negative effects.Source: Figure S1 · p. 19

F Post Partisanship Classification

  • Partisanship was labeled for English-language posts using a language model, with categories for left, right, center, unknown, and none.Non-English posts and posts without available text were not labeled.Source: F Post Partisanship Classification · p. 19

2 Distribution of Treatment Effects

Notes reduce engagement on average, although many posts have positive estimated effects; increases are generally less frequent and smaller than decreases. The balance also differs across engagement metrics.

2 Distribution of Treatment Effects

  • 43.0% of posts see increased views after 48 hours, compared with 39.2% for replies, 33.9% for likes, and 33.0% for reposts.Estimated increases are typically smaller in magnitude than decreases, yielding negative average effects.Source: §2 · p. 19
  • -13.5% for views and -21.9% for replies are the average declines after note attachment.The passage contrasts these figures with larger declines for likes and reposts but does not provide those values here.Source: §2 · p. 19
  • Median negative effects are 3.18 and 2.49 times the positive effects for views and replies, versus 4.46 and 4.61 times for likes and reposts.Across percentiles from the median to the 99th, negative-to-positive effect ratios reach 3.26 and 2.63 for views and replies, and 6.27 and 6.14 for likes and reposts.Source: §2 · p. 20
  • 20.2% of posts noted within 12 hours have positive repost effects, compared with 43.0% of posts noted after 47 hours.Post age has one of the stronger pairwise associations with whether note attachment leads to increased reposts; other factors show similar, weaker relationships between positivity and effect magnitude.Source: §2 · p. 20, Figure S2 · p. 20

3 Heterogeneity in Effects on Reposts Based on Popularity Before Note Attachment

Absolute repost effects are larger for more popular posts, while percentage declines are similar across the three larger popularity groups; the least-popular group is more variable.

3 Heterogeneity in Effects on Reposts Based on Popularity Before Note Attachment

  • -67, -231, and -753 reposts are the mean absolute effects for the second, third, and fourth pre-note popularity quartiles, respectively.These correspond to percentage changes of -15.0%, -15.0%, and -10.9%; the first quartile, with 134 or fewer reposts, had a change of 1 repost (95% CI: [-10, 13]), or 1.4%.Source: §3 · p. 21
  • 44.4% of posts in the bottom popularity quartile have positive repost effects, versus 35.8%, 28.9%, and 22.2% in the larger quartiles.The authors suggest notes may sometimes draw additional attention to low-visibility posts; effects in this quartile are also less consistent, with a coefficient of variation of 190 versus -3.92, -2.90, and -2.57 for larger quartiles.Source: §3 · p. 21

4 Factors Associated with Note Effectiveness on the Number of Views, Replies, and Likes

Effects on views, replies, and likes vary with post and note characteristics. The reported patterns differ by partisan lean and by note readability for reposts versus views.

4 Factors Associated with Note Effectiveness on the Number of Views, Replies, and Likes

  • Treatment effects on views, replies, and likes are larger in magnitude for right-leaning than left-leaning posts.Average effects on views and replies for left-leaning posts are positive but not statistically significant.Source: §4 · p. 21
  • More readable notes are associated with larger repost effects but smaller view effects.The authors suggest unclear notes may act as a warning signal, while more comprehensible notes may engage users and make them less likely to later express agreement with the post.Source: §4 · p. 21
  • Figures S3, S4, and S5 stratify views, replies, and likes by note speed and by post or note characteristics, with 95% confidence intervals.The characteristics include partisan lean, embedded media, accuracy concerns, readability, and note length.Source: Figure S3 · p. 22, Figure S4 · p. 22, Figure S5 · p. 22

5 In-Time Placebo Test

The in-time placebo tests assess whether synthetic-control estimates appear before the actual treatment time. The one-hour backdate test reports confidence intervals that include zero for every metric.

5 In-Time Placebo Test

  • An in-time placebo backdates treatment and constructs synthetic controls using observations only up to that earlier time; non-null effects before true treatment would undermine estimate credibility.The paper also uses the typical roughly one-hour delay between a note being recorded helpful and appearing on X as a natural placebo period.Source: §5 · pp. 23–24
  • The permutation test compares actual treatment effects, shown by dashed blue lines, with placebo effects from shuffled treatment assignments, shown by gray lines.The figure displays a random sample of 200 permutations.Source: Figure S8 · p. 24
  • Replies, likes, and reposts do not begin declining meaningfully until about an hour after the first helpful note, when notes typically began appearing to users.The slight initial view increase and subsequent decrease is attributed as likely due to posts appearing in the Community Notes “Rated Helpful” tab.Source: §5 · p. 24
  • Backdating treatment by one hour yields confidence intervals containing zero for all metrics; the estimates were 1,028 views, 1.28 replies, -0.154 likes, and 0.287 reposts.The respective 95% confidence intervals were [-341, 2,397], [-2.36, 4.92], [-9.57, 9.26], and [-0.839, 1.41].Source: §5 · p. 24

6 Permutation Test

The permutation analysis uses a computationally feasible placebo procedure but omits control-set variability; adding post-text embeddings leaves the estimated treatment effects within the original confidence intervals.

7 Synthetic Controls with Additional Post Content Embeddings

  • Embedding-based estimates across views, replies, likes, and reposts remained within the original estimates’ 95% confidence intervals.For views, the 20- and 58-dimensional embeddings yielded estimates of -145,097 and -136,535, versus -159,592 without embeddings.Source: Figure S9 · p. 25, §7 · p. 26

6 Permutation Test

  • Because rerunning the full estimation was prohibitively costly, the permutation procedure sampled treated units using precomputed effects and omitted control-set variability.A typical permutation test would reconstruct synthetic controls in each iteration.Source: §6 · p. 25
  • After 100,000 permutations, observed effects were strictly larger in magnitude than every placebo effect for each metric (p-values 0.00001).The observed effects used both treatment and control units as donors and were smaller than the main-text estimates.Source: §6 · p. 25

7 Synthetic Controls with Additional Post Content Embeddings

  • The authors reduced 768-dimensional Gemini text embeddings to 58 or 20 PCA dimensions and standardized them to match other covariates’ per-dimension weight.Spot checks found that the highest-weight donor posts usually concerned the same claim or type of misinformation.Source: §7 · p. 26

8 Synthetic Controls With Donor Pools Restricted to Control Posts with High-Scoring Notes

The analysis restricts synthetic-control donor pools to control posts with higher-scoring notes, but the smaller pools produce weaker matches that may bias estimates, so the authors focus on the full pool.

8 Synthetic Controls With Donor Pools Restricted to Control Posts with High-Scoring Notes

  • Restricted pools yielded increasingly negative 48-hour view estimates: -200,327, -224,108, and -300,596 at helpfulness thresholds >0.25, >0.3, and >0.35, versus -159,592 unrestricted.Repost estimates were -258, -254, and -262 versus -259 unrestricted; reply estimates were -176, -173, and -204 versus -158; like estimates were -2,275, -2,325, and -2,541 versus -2,134.Source: Figure S10 · p. 27, §8 · p. 27
  • Even with broad covariate matching, observational comparisons may reflect systematic differences between treated and control posts in unmeasured ways.The content-embedding analysis tested one source of mismatch but cannot eliminate unobserved confounding.Source: §8 · p. 27
  • Because notes are shown only after diverse users rate them helpful and their score exceeds 0.4, never-helpful control posts may differ in content or other ways.The authors therefore question whether those controls form a suitable donor pool.Source: §8 · p. 27
  • The thresholds reduced the donor pool from 33,321 posts to 8,760, 5,084, and 2,394 posts, respectively.The score for each note was taken as the maximum it reached during the study period.Source: §8 · p. 27
  • Restricted pools produced larger pre-treatment mismatches, potentially biasing estimates, so the authors prioritize results from the full donor pool.The analysis using restricted pools is presented as a robustness check.Source: §8 · p. 27

Related papers

Loading 2502.13322v2…