Needs Your Help: Understanding Platform-Directed Rating Participation in Community Notes on X

Motivation

• The introduction identifies slow accumulation of evaluations as a bottleneck that can delay public corrections until posts have gained substantial engagement.Community-based fact-checking depends on both relevant notes being available and contributors choosing to rate them.Source: §1 · p. 2

Intervention

• “Needs Your Help” targets selected unresolved notes by prompting a randomized subset of contributors, while leaving them free to choose whether to respond.Candidate notes include recent, high-visibility posts near consensus, especially notes marked “Needs More Ratings” with positive signals or lopsided ratings.Source: §1 · pp. 2–3

Research questions

• The study asks which notes receive population-sampled ratings, how those ratings relate to resolution, and how raters’ activity changes after their first such rating.The research questions cover note-level targeting and resolution status, and rater-level evaluation activity.Source: §1 · p. 3

Data and methods

• The analysis uses X’s public dataset of 220 million ratings on 2.8 million notes, including 1.9 million population-sampled ratings.Source: §1 · p. 3
• The note-level analyses compare notes receiving population-sampled ratings with controls, using matching and difference-in-differences to examine resolution speed and transitions from NMR.A separate staggered difference-in-differences design compares contributors’ behavior before and after their first population-sampled rating.Source: §1 · p. 3

Contributions

• The paper presents its contribution as evidence that platform-directed ratings steer attention toward disputed content, accelerate consensus without volunteer fatigue, and motivate dedicated task feeds.Source: §1 · p. 3

2.1 Crowdsourced Rating

• Crowd ratings can be noisy and vary across task designs and participants, while politically divergent judgments complicate actionable consensus.Prior work describes bridging algorithms that use historical rating diversity to elevate notes with cross-perspective agreement [74].Source: §2.1 · pp. 3–4
• Organic notes can be slow to reach consensus, and prior simulations and predictive models do not examine how real-world platform routing directs human attention.Source: §2.1 · p. 4

2.2 Algorithmic Nudging

• Push notifications and task recommendations can steer contributions, but excessive or poorly timed prompts risk volunteer fatigue and attrition.Source: §2.2 · p. 4
• The literature frames nudging as a design tension between directing volunteer attention and sustaining voluntary labor.Intervention choices across participation, curation, and presentation also have normative implications.Source: §2.2 · p. 4
• Prior fact-checking interventions have emphasized note generation, credibility assessment, or annotation scaffolds, whereas this study focuses on platform-directed task selection.Source: §2.2 · p. 4

downstream evaluation, specifically evaluating how algorithmic prompts nudge raters toward contested notes

• The study’s downstream evaluation examines how algorithmic prompts direct raters toward contested notes and whether prompting relates to spillover or volunteer fatigue.Source: downstream evaluation, specifically… · p. 4

2.3 Targeted Feeds Curation

• Feed-c​uration research considers how interfaces surface content and shape users’ attention.Source: §2.3 · p. 5
• Users may form “folk theories” to explain how curation algorithms filter and prioritize content.Source: §2.3 · p. 5
• Research on Mastodon reports that visible algorithmic seams can support trust and user negotiation of machine-learning and heuristic filtering trade-offs.Source: §2.3 · p. 5
• Users may misunderstand how feeds are constructed, including personalization and ad-targeting controls.Source: §2.3 · p. 5
• Feed-elicitation interviews help users translate nuanced preferences into customized feeds.Source: §2.3 · p. 5
• BONSAI lets users specify natural-language intents and exercise transparent control over their feeds.Source: §2.3 · p. 5
• Prior work reports negative correlations between algorithmic feed amplification and users’ explicit personal values, including selective boosting of moral expressions.Source: §2.3 · p. 5

2.4 Community Notes

• Community Notes uses ratings from contributors to assess whether notes are helpful, somewhat helpful, or not helpful [74].Source: §2.4 · p. 5
• Its matrix-factorization bridging algorithm scores notes using positive ratings from contributors with historically divergent perspectives.Source: §2.4 · p. 5
• Notes reaching a sufficiently high consensus score receive “Currently Rated Helpful” status and appear publicly on the associated post.Source: §2.4 · p. 5
• Publicly displayed notes have been reported to reduce engagement with and retweets of misleading posts.Source: §2.4 · p. 5
• Many notes reach helpful status only after the underlying post has gained substantial engagement, limiting the intervention’s efficacy.Source: §2.4 · p. 5
• Public notes can occasionally polarize discussion, provoking heightened moral outrage and negative sentiment in replies.Source: §2.4 · p. 5
• X introduced “Needs Your Help” notifications to sample contributors and prompt evaluations of pending notes.Ratings received through this channel are logged as “population-sampled ratings.”Source: §2.4 · p. 5
• The background identifies open questions about which note attributes trigger sampling, whether sampled ratings accelerate resolution, and whether prompted tasks alter later contributor evaluation.Source: §2.4 · p. 5

• The empirical analysis covers the population-sampled rating mechanism at the note level.Source: algorithmic performance, our work… · p. 5
• The empirical analysis covers the population-sampled rating mechanism at the rating level.Source: algorithmic performance, our work… · p. 5
• The empirical analysis covers the population-sampled rating mechanism at the rater level.Source: algorithmic performance, our work… · p. 5

3.1 Dataset

• The archive contained 220,472,232 ratings for 2,859,500 notes; the post–February 1, 2025 analysis sample included 93,878,870 ratings, of which 1,929,941 were population-sampled.In that sample, 159,377 notes received at least one population-sampled rating, and 95.6% of notes were still in NMR at their first such rating.Source: §3.1 · p. 6

3.2.1 Estimation Model and Factors.

• The POP-entry analysis examined post-, note-, and pre-event rating characteristics as predictors of observed population-sampled rating.Predictors included post classifications and timing, note content and lifecycle, author history, prior rating activity, Helpful support, and rating entropy; coefficients represented adjusted associations with log odds.Source: §3.2.1 · pp. 6–7

3.2.2 Sampling.

• Treated notes were matched to controls from the same creation month and assigned pseudo-events at the same note age, with covariates measured before the event.The entropy analysis required at least five prior DEFAULT ratings.Source: §3.2.2 · p. 8

3.3 Effects on Rating (RQ1)

• The effects analysis examined whether notes reached CRH or left NMR after the aligned event, as well as resolution status and speed.Non-NMR comprised either CRH or CRNH.Source: §3.3 · p. 8, §3.3.1 · p. 8

3.3.1 Metrics.

• Outcome measures included cumulative resolution attainment, time to CRH or non-NMR, time to 20 ratings, seven-day rating rate, and helpfulness difference and lift.Helpfulness difference compared mean population-sampled and DEFAULT scores; lift measured the change in mean helpfulness when population-sampled ratings were included.Source: §3.3.1 · p. 8

3.3.2 Estimation Model.

• The estimation model included note-author fixed effects and selected predictors across post-, note-, and rating-level characteristics.The supplied passage begins describing post-level characteristics but does not provide the full specification.Source: §3.3.2 · p. 8
• The note-level model accounts for note status, format, age at first population-sampled rating, rating volume and timing, and baseline helpfulness, allowing a quadratic helpfulness relationship.It reports heteroskedasticity-robust pointwise 95% confidence intervals and uses Spearman’s rank correlation for the overall rank association.Source: §3.3.2 · p. 9

3.3.3 Sampling.

• For RQ1, treated notes were still NMR at their first population-sampled rating, and propensity-score matching paired them with unrated controls at similar note ages and times.Matching used pre-event DEFAULT rating count, mean helpfulness, and helpfulness-score standard deviation.Source: §3.3.3 · p. 9

3.4 Effects on Raters (RQ2)

• The RQ2 analysis examines how raters’ behavior changes after their first population-sampled rating and how those changes vary by rater characteristics.Source: §3.4 · p. 9

3.4.1 Metrics.

• Eight rater metrics cover participation intensity, rating selection, evaluative tendency, and diagnostic-tag use.They include activity and rating volume, rating timing and note age, misleading-note selection, Not Helpful share, mean rating score, and daily diagnostic-tag-use intensity.Source: §3.4.1 · pp. 9–10

3.4.2 Estimation Model.

• Rater outcomes are estimated with difference-in-differences comparing seven pre-event and post-event days for POP-exposed raters and matched controls, excluding the event day.The methods also describe outcome-specific regressions using contributor standing, pre-event behavior, and first-rating characteristics as predictors.Source: §3.4.2 · p. 10

3.4.3 Sampling.

• The RQ2 sampling subsection identifies population-sampled groups and matched control groups.Source: §3.4.3 · p. 11

3.4.3 Sampling

• The treatment event was a rater’s first population-sampled rating; controls never submitted one and rated the same note by DEFAULT within two hours, with that timestamp as pseudo-treatment.Source: §3.4.3 · p. 11
• Treated and control raters were matched on note content, status, and rating lifecycle, plus pre-treatment participation, helpfulness, tag-use, and experience measures.Source: §3.4.3 · p. 11
• Across matched samples, pre-event comparisons between treatment and control groups were nonsignificant.The analysis tested pre-event group differences to assess whether the groups had similar pre-event metrics.Source: §3.4.3 · p. 11
• Sensitivity analyses re-estimated DID results after excluding index notes, then all population-sampled notes, and using the full archived matched-rater pool.The analyses assessed whether estimates depended on ratings directly associated with the population-sampling process or on sample construction.Source: §3.4.3 · p. 11

4.1 Correlations of Contextual Factors on Population-Sampled Rating

• Prior notes on the same post reduced sampling likelihood, while notes classified as misleading and several author-selected reasons were positively associated with entry.The association for an existing note on the post was β= −0.602 (95% CI [−0.692, −0.513]); misleading classification was associated with β= 5.018 (95% CI [4.651, 5.385]). These classifications reflect authors’ assessments, not independently verified accuracy.Source: §4.1 · p. 13
• Longer notes, more supporting URLs, reliable-source flags, media, and newer note age were associated with greater population-sampled entry.The reported associations were β= 0.043 for note length, β= 0.095 for supporting URLs, β= 0.457 for reliable-source flags, β= 1.254 for media notes, and β= −0.074 for note age.Source: §4.1 · p. 13
• Authors’ prior CRH share and prior Helpful mean and rating entropy were positively associated with entry, while historical note volume was negatively associated.Recent rating bursts within six hours and ratings received more than 24 hours earlier were also positively associated with entry; activity in the intermediate 6–24-hour window was negative.Source: §4.1 · pp. 13–14

4.2 Effects of Disagreement and Strong Helpful Support

• A 0.1-unit increase in prior rating entropy was associated with a 1.57-percentage-point higher entry probability at mean Helpful 0.50 and 2.05 points at 0.80.The association was negative or indistinguishable from zero at low Helpful levels, indicating that the positive relationship with disagreement was stronger when prior Helpful support was high.Source: §4.2 · p. 14, Figure 3 · p. 14

4.3.1 Effects on Note Resolution Speed.

• Notes receiving population-sampled ratings reached CRH and left NMR faster than matched controls, and acquired their first 20 ratings sooner.Median post-exposure time to CRH was 3.51 vs. 6.35 hours; time to first non-NMR status was 3.47 vs. 6.08 hours. First 20 ratings arrived in 1.74 vs. 2.63 hours.Source: §4.3.1 · p. 15
• Population-sampled ratings were associated with faster acquisition of the first 20 ratings, but not with a significant difference in seven-day rating speed.Median seven-day speed was 5.57 vs. 5.43 ratings per day (paired Wilcoxon p= .926), while the first 20 ratings arrived in 1.74 vs. 2.63 hours.Source: §4.3.1 · p. 15

4.3.2 Effects on Note Resolution Status.

• By 168 hours, 6.0% of notes without an early population-sampled rating reached CRH, versus 10.4–12.4% among notes receiving one to four.CRH accumulation was fastest in the first 24 hours; notes with four or more early ratings reached somewhat higher cumulative CRH attainment than the other rating groups.Source: Figure 3 · p. 14, §4.3.2 · p. 15
• Compared with matched controls, notes exposed to population-sampled ratings were 13.17 percentage points more likely to reach CRH within 12 hours, 15.64 within 24 hours, 17.26 within seven days, and 17.65 within 30 days.All differences were statistically significant (p < .001).Source: §4.3.2 · p. 15
• By 12 hours, 76.3% of the seven-day CRH difference had emerged, rising to 90.6% by 24 hours; median CRH lag was 3.51 hours for treated notes versus 6.35 for controls.The separation in CRH attainment emerged primarily during the first day and persisted through the seven-day follow-up.Source: §4.3.2 · p. 15
• At seven days, matched differences were +13.90 percentage points for leaving NMR, +17.26 for CRH, and −3.37 for CRNH, suggesting movement toward CRH rather than CRNH.The first non-NMR status arrived a median 3.47 hours after exposure in treated notes versus 6.08 hours in controls (Mann–Whitney p < .001).Source: §4.3.2 · p. 16

Discrepancies between population-sampled and default ratings are strongly correlated with note content,

• Notes addressing misleading posts were associated with a larger population-sampled versus default rating difference (β= 0.052, p= .05).Other or Manipulated media were associated with smaller differences, while other categories were non-significant.Source: Discrepancies between… · p. 16
• Population-sampled ratings had higher helpfulness scores than DEFAULT ratings for CRNH notes but lower scores for CRH notes.The authors describe this pattern as a corrective tendency that differs from existing rating trajectories.Source: Discrepancies between… · p. 16
• CRNH notes correlated with a larger rating difference (β= 0.154), while CRH notes correlated with a smaller difference (β= −0.092; both p< .001).The passage says this pattern suggested population-sampled ratings may change note status, but does not establish that outcome here.Source: Discrepancies between… · p. 16
• Media notes were positively associated with a larger difference between population-sampled and default ratings (β= 0.015, p< .001).The authors suggest, rather than establish, that emotionally driven organic reactions to multimedia may explain the difference.Source: Discrepancies between… · p. 16
• Mean lift declined as DEFAULT helpfulness rose, with more positive lift for lower-helpfulness notes and small or negative lift near the upper end.DEFAULT helpfulness was negatively correlated with lift (Spearman’s ρ= −0.436, p< .001).Source: Discrepancies between… · p. 18

5.1 Effects of Population-Sampled Rating

• Four of eight behavioral measures changed significantly: ratings per active day, daily active probability, inter-rating interval, and diagnostic-tag-use intensity.The other four measures showed no significant change within the seven-day post-event period.Source: §5.1 · p. 19
• Ratings per active day increased by 0.062 relative to matched controls, and daily active probability also increased significantly.The provided passage gives the active-probability comparison as 0.831 percentage points for controls (95% CI [0.018, 0.106], p= .005, q= .015); it does not provide the corresponding exposed-rater estimate in this excerpt.Source: §5.1 · p. 19
• The inter-rating interval decreased by 0.881 hours relative to matched controls (95% CI [−1.587, −0.175], p= .015, q= .029).Neither rated-note age nor misleading-note share changed significantly.Source: §5.1 · p. 19
• Neither Not Helpful share nor mean rating score changed significantly after the first observed population-sampled rating.The estimates were −0.071 percentage points for Not Helpful share and −0.008 for mean rating score.Source: §5.1 · p. 19
• Daily diagnostic-tag-use intensity increased by 0.060 tags per rating relative to matched controls (95% CI [0.047, 0.072], p< .001, q< .001).The passage suggests this may reflect greater familiarity with the platform’s diagnostic criteria.Source: §5.1 · p. 19

Appendix B.

• Earned-in status was positively associated with subsequent ratings per active day, daily active probability, and diagnostic-tag-use intensity.All three associations remained significant after Benjamini–Hochberg correction.Source: §5.2 · p. 19, Figure 8 · p. 20

5.2 Correlation Analysis

• Behavioral changes varied across rater characteristics, including first-POP-note age, first-POP rating score, baseline helpfulness, and baseline activity.For example, older first-POP notes were associated with a larger subsequent increase in misleading-note selection, while higher first-POP scores were associated with a smaller increase in Not Helpful ratings.Source: §5.2 · p. 19

Appendix B.

• Figure 8 reports standardized coefficients for eight behavioral outcomes, with black-outlined estimates remaining significant after Benjamini-Hochberg correction.Baseline helpfulness and baseline ratings per active day were also associated with differing behavioral changes.Source: §5.2 · p. 19, Figure 8 · p. 20

6.1 Targeted Sampling and Crowdsourced Evaluation

• Population-sampled entry was more common for notes with high rating entropy and prior Helpful support, extending task routing beyond latency and expertise affinity.The authors connect this pattern to contributors participating in tasks highlighted by the system rather than relying only on opportunistic browsing.Source: §6.1 · p. 20
• Population-sampled evaluations showed a corrective tendency: they were more helpful for CRNH notes and less helpful for CRH notes than DEFAULT ratings.The authors say these differing evaluations may help acquire diverse ratings.Source: §6.1 · p. 21
• After a first population-sampled rating, contributors showed increases in daily active probability, ratings per active day, and diagnostic-tag use, with no significant change in mean rating scores.The authors interpret increased tag use as possible implicit scaffolding that encourages diagnostic rationales alongside ratings.Source: §6.1 · p. 21
• Population-sampled ratings had significantly higher helpfulness scores than DEFAULT ratings for multimedia notes.The authors suggest the dedicated evaluation interface may reduce contextual emotional heuristics, allowing contributors to deliberate on multimodal evidence.Source: §6.1 · p. 21

6.2 Implications

• The authors conclude that targeted ratings for unresolved notes accelerate consensus and propose routing volunteer attention toward high-uncertainty items.They caution that routing should account for notification fatigue.Source: §6.2 · p. 21
• For notes receiving partisan-skewed rejections, the discussion proposes inviting raters with opposing or neutral perspectives to re-evaluate them.The stated aim is to avoid ideological reactance and preserve perceived neutrality of decentralized governance.Source: §6.2 · p. 21
• The discussion proposes targeted requests as implicit on-the-job training, noting that assigned tasks did not degrade throughput or scoring accuracy.It suggests platforms could incorporate targeted tasks into day-to-day participation without increasing onboarding overhead.Source: §6.2 · p. 22
• Because behavioral responses varied across contributor backgrounds, the authors propose context-aware routing that matches note characteristics with contributor standing.They frame this as a way to optimize task throughput and data quality.Source: §6.2 · p. 22

7 Limitations

• Because the analysis uses observational platform data, it cannot establish definitive causal effects of population-sampled nudges on note-resolution speed.The sampling algorithm may target highly active raters or notes already nearing evaluative consensus, confounding observed speedups.Source: §7 · p. 22
• Findings come exclusively from X’s Community Notes, whose platform-specific dynamics may limit generalization to other social platforms or task-based crowdsourcing.The system is characterized by an open-source algorithm, transparent data releases, and a mature community culture.Source: §7 · p. 22
• The analysis includes completed population-sampled ratings, not invitations that failed to produce a submitted rating.Source: §7 · p. 22

8 Ethical Considerations

• Ethical analysis uses the public archive and aggregate behavior measures, without deanonymizing raters or tracing ratings to personal accounts.The study focuses on structural outcomes of prompts rather than the moral or political validity of individual notes.Source: §8 · p. 22

9 Conclusion

• Population-sampled ratings target recently active notes retaining disagreement, accelerate resolution from NMR toward definitive statuses, and coincide with modest increases in rating speed, ratings per active day, and diagnostic tag use.The conclusion reports no significant changes across other behaviors and no reduction in evaluation speed.Source: §9 · p. 23
• Targeted population-sampled nudging is presented as promising for expediting consensus without inducing volunteer fatigue.Source: §9 · p. 23

A Complete Rater-Level DID Results

• Tables 1 and 2 report the complete results for all eight behavioral outcomes.The analyses compare seven pre-event days with seven post-event days.Source: A Complete Rater-Level DID Results · p. 26
• The comparison excludes the treatment-event day.Source: A Complete Rater-Level DID Results · p. 26

B.1 Outcome-Specific Matched Samples

• For four outcomes, maximum-cardinality selection used only pre-event information, including matching covariates and outcome trajectories.No post-event outcome was used to construct samples, and pairs lacking observations for both periods were excluded from the corresponding estimate.Source: §B.1 · p. 26

B.2 Complete DID Estimates and Statistical Tests

• Table 3 reports complete DID estimates and statistical tests for the outcome-specific samples.The table notes distinguish selected matched pairs from pairs with sufficient pre- and post-event observations for an estimate.Source: §B.2 · p. 26, §B.1 · p. 26

B.3 Pre-Event Diagnostic Tests

• Approximate MDEs use observed standard errors, 80% power, and an unadjusted two-sided α=.05, and do not incorporate Benjamini–Hochberg adjustment.Standardized estimates and corresponding bounds and MDEs use the standard deviation of matched-pair DID changes rather than the baseline outcome standard deviation.Source: §B.3 · p. 27
• Table 4 reports observed pre-event balance and trend diagnostics for outcome-specific samples.The checks include baseline balance, differential slopes, placebo DID, and a joint Wald test of treated-control gaps.Source: §B.3 · p. 26, Table 4 · p. 28
• All reported absolute standardized mean differences were below 0.10, but passing observed diagnostics does not prove parallel trends for unobserved potential outcomes.For rebalanced samples, diagnostics describe in-sample design balance, not holdout validation.Source: §B.3 · p. 28