Source-linked AI summary
Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking
Soham De, Michiel A. Bakker, Jay Baxter, Martin Saveski
TL;DR
Many proposed Community Notes never gain enough support from users with diverse views to appear, despite some containing valuable context. The paper proposes Supernotes: LLM-generated syntheses of existing notes selected with a helpfulness model trained on historical ratings. Participants preferred Supernotes to the best existing notes and rated them more helpful than LLM-generated summaries.
Problem
For 91% of posts with proposed notes, no note receives enough helpful ratings from users with diverse views to be displayed, although some proposed notes contain valuable context.
Source: §1 · p. 2Method
The framework uses an LLM to generate Supernote candidates from existing notes and a helpfulness model trained on millions of Community Notes ratings to select candidates likely to be rated helpful by diverse users.
Source: §1 · p. 2Results
Participants chose Supernotes over the best existing notes 75.2% of the time and rated them significantly more favorably across five dimensions.
Source: §1 · p. 2Takeaways & Limitations
In a follow-up comparison, participants rated Supernotes as more helpful than LLM-generated summaries, supporting the effectiveness of candidate scoring for selecting notes that drive consensus among diverse users.
Source: §1 · p. 2, §6 · p. 8The human-subject experiments used surveys outside the platform, where participants rated assigned notes and were compensated, unlike the platform setting.
Source: §6 · p. 8
Abstract
from arXiv · showhide
X's Community Notes, a crowd-sourced fact-checking system, allows users to annotate potentially misleading posts. Notes rated as helpful by a diverse set of users are prominently displayed below the original post. While demonstrably effective at reducing misinformation's impact when notes are displayed, there is an opportunity for notes to appear on many more posts: for 91% of posts where at least one note is proposed, no notes ultimately achieve sufficient support from diverse users to be shown on the platform. This motivates the development of Supernotes: AI-generated notes that synthesize information from several existing community notes and are written to foster consensus among a diverse set of users. Our framework uses an LLM to generate many diverse Supernote candidates from existing proposed notes. These candidates are then evaluated by a novel scoring model, trained on millions of historical Community Notes ratings, selecting candidates that are most likely to be rated helpful by a diverse set of users. To test our framework, we ran a human subjects experiment in which we asked participants to compare the Supernotes generated by our framework to the best existing community notes for 100 sample posts. We found that participants rated the Supernotes as significantly more helpful, and when asked to choose between the two, preferred the Supernotes 75.2% of the time. Participants also rated the Supernotes more favorably than the best existing notes on quality, clarity, coverage, context, and argumentativeness. Finally, in a follow-up experiment, we asked participants to compare the Supernotes against LLM-generated summaries and found that the participants rated the Supernotes significantly more helpful, demonstrating that both the LLM-based candidate generation and the consensus-driven scoring play crucial roles in creating notes that effectively build consensus among diverse users.
1 INTRODUCTION
Community Notes can affect how users engage with misinformation, but most posts with proposed notes never receive enough helpful ratings from users with diverse views for a note to appear. The paper introduces Supernotes, which synthesize existing notes and are selected to foster consensus, and reports that participants preferred them to the best existing notes.
1 INTRODUCTION
- For 91% of posts with proposed notes, no note gains enough helpful ratings from users with diverse views to be displayed.Only 12.5% of posts with a proposed note had a note reach helpful status and appear.Source: §1 · pp. 1–2
- The authors suggest that valuable notes can fail to gain broad support when they are biased or one-sided, or when essential context is fragmented across multiple notes.The proposed explanation is that these notes may not achieve the neutral tone or holistic understanding needed for wider acceptance.Source: §1 · p. 2
- The authors propose Supernotes that synthesize several existing notes and are written to foster consensus among diverse users.An LLM generates candidates, and a helpfulness model trained on millions of public Community Notes ratings selects candidates likely to be rated helpful by diverse users.Source: §1 · p. 2
- Participants preferred Supernotes to the best existing notes 75.2% of the time, and rated them more favorably across five dimensions.The five dimensions were quality, clarity, coverage, context, and argumentativeness; Supernotes also outperformed LLM-generated summaries in a follow-up comparison.Source: §1 · p. 2
2 SUPERNOTE GENERATION: FRAMEWORK
The framework generates diverse Supernote candidates from posts and existing notes, then scores and filters candidates before selecting the highest-scoring one. Scoring combines simulated ratings from a jury of users with checks for effective fact-checking principles.
2.1 Candidate Generation
- Candidate generation prompts an LLM to synthesize the post and subsets of existing notes into diverse candidate Supernotes.Prompts include note-writing guidelines, existing-note ratings and tags, while sampling settings are chosen to increase candidate diversity.Source: §2.1 · p. 3
2.2 Candidate Scoring
- A personalized helpfulness model predicts each sampled rater’s response, and probabilistic sampling preserves uncertainty across helpfulness categories.The model predicts helpful, somewhat helpful, or not helpful; the predicted ratings are aggregated into a score for each candidate.Source: §2.2 · p. 3
- The simulated jury’s predictions are aggregated using the Community Notes matrix-factorization algorithm, which gives higher scores to notes rated helpful by raters who typically disagree.Predicted ratings are projected onto the algorithm’s latent space using least squares.Source: §2.2 · p. 3
- Candidates are filtered for neutrality, lack of speculation or opinion, valid links already present in existing notes, and a maximum length of 280 characters.Candidates failing any criterion are excluded from the pool scored by the simulated jury.Source: §2.2 · p. 3
3 SUPERNOTE GENERATION: IMPLEMENTATION
The implementation generates candidates with GPT-4o-mini and evaluates scoring components using historical Community Notes data. The neural network achieves an AUC of 0.85, and simulated-jury predictions identify the highest-scoring note with Precision@1 of 0.7 across posts.
3.1 Candidate generation
- The implementation generates 100 GPT-4o-mini candidates per post, varying subsets and permutations of eligible notes to promote diversity.The selected notes argue that the post is misleading and have a “Needs More Ratings” status.Source: §3.1 · pp. 3–4
3.2 Candidate Scoring
- The neural network outperforms the other tested models across precision, recall, F1, and AUC, achieving an AUC of 0.85.The models predict whether a rater would find a note helpful, somewhat helpful, or not helpful.Source: §3.2 · p. 4, Figure 3 · p. 4
- Aggregating simulated jurors’ predicted helpfulness ratings identified the highest-scoring note with Precision@1 of 0.7 across all posts, 1.94 times higher than random guessing, against observed Community Notes scores.Precision@1 was 0.76 with two proposed notes and 0.73 with three.Source: §3.2 · p. 4
- Invalid links are the most common principle-alignment rejection at 8%, while argumentativeness and lack of clarity each have rejection rates below 2%.GPT-4o-mini has fewer invalid-link rejections than GPT-3.5 Turbo, which has a 22% rejection rate.Source: §3.2 · pp. 4–5
4 HUMAN EVALUATION
The human evaluation compared Supernotes with the best existing notes on X, and participants rated Supernotes more favorably. In a follow-up comparison, Supernotes also outperformed randomly selected candidate summaries.
4.1 Experimental Setup
- The main experiment recruited 42 US English-speaking adults, split evenly between left- and right-leaning participants, and collected 1,008 ratings.Participants evaluated Supernotes and best existing notes on sampled X posts, with note order randomized and note identity concealed.Source: §4.1 · p. 5
4.2 Results
- 0.76 vs 0.58 mean helpfulness: participants rated Supernotes significantly higher than the best existing notes.The comparison uses the Community Notes numerical scale; the difference remained significant after accounting for participant and post effects.Source: Figure 4 · p. 6, §4.2 · p. 6
- 75.2% of the time, participants chose the Supernote as more helpful than the best existing note.When both notes were rated “helpful,” participants chose the Supernote 68.9% of the time.Source: Figure 5 · p. 6, §4.2 · p. 6
- Supernotes received significantly higher Community Notes helpfulness scores than the best existing notes.The scores were 0.32 for Supernotes and 0.24 for best existing notes; the scoring procedure used experiment ratings alongside public ratings.Source: Figure 6 · p. 6, §4.2 · p. 6
- Participants rated Supernotes more favorably across source quality, clarity, coverage, context, and argumentativeness.The largest differences concerned important context and high-quality sources.Source: §4.2 · p. 6, Figure 7 · p. 7
4.3 Ablation Study: Impact of the Candidate Scoring Component
- 61.5% of the time, participants found Supernotes more helpful than randomly selected candidate summaries.The follow-up experiment compared blinded notes, and the mixed-effects model also favored Supernotes.Source: Figure 8 · p. 7, §4.3 · p. 7
- Supernotes were preferred whether compared summaries used the same existing notes (59.1%) or different notes (62.4%).The authors interpret these comparisons as evidence that candidate scoring helps identify helpful syntheses and select appropriate source notes.Source: §4.3 · p. 7
5 FURTHER RELATED WORK
This section situates Community Notes among scalable fact-checking approaches and compares Supernotes with prior LLM-based methods for aggregating perspectives.
5.1 Scaling Up Fact-Checking
- Crowd-sourced and automated fact-checking offer scalable alternatives to expert fact-checking, and Community Notes is a fully crowd-sourced system deployed on a large social network.Studies find crowd-sourced outcomes often align with expert judgments, resist motivated reasoning, and reach polarized communities [26].Source: §5.1 · p. 7
- Unlike LLM fact-checkers that query external sources, this approach uses proposed notes and early ratings, and predicts notes likely to build consensus.The authors say this is slower but less prone to introducing errors that could erode trust [8].Source: §5.1 · p. 8
5.2 Aggregating Diverse Perspectives with LLMs
- Bakker et al. [3] map opinions to consensus statements with personalized preferences, while this work predicts personalized note helpfulness and uses Community Notes aggregation.Fish et al. [12] generate multiple summaries using prompted models for both generation and reward; this work trains its helpfulness model on historical ratings.Source: §5.2 · p. 8
6 DISCUSSION AND CONCLUSION
The paper presents Supernotes as constrained summaries designed to foster consensus and follow fact-checking principles. In experiments, they outperformed existing notes and LLM-generated summaries on helpfulness, with stated limitations in evaluation setting and input coverage.
6 DISCUSSION AND CONCLUSION
- Supernotes were rated more helpful than the best existing notes, scored higher by the Community Notes algorithm, and significantly more helpful than LLM-generated summaries.Participants also perceived them as having higher-quality sources, broader coverage, and less argumentativeness than the best existing notes.Source: §6 · p. 8
- The human-subject experiments used off-platform surveys, required ratings of specified notes, and did not consider images or videos in misleading posts.The authors note that participants might have behaved differently on-platform; Supernotes also depend on existing notes and ratings and therefore appear later than initial notes.Source: §6 · p. 8
- Future work could incorporate external sources, but the authors identify avoiding inaccuracies and fairly allocating writing impact as deployment concerns.The proposed credit system would recognize users whose notes contributed to a Supernote.Source: §6 · p. 8
7 ETHICAL USE OF DATA
The study used human-subject experiments and publicly available Community Notes data to train its personalized helpfulness models.
7 ETHICAL USE OF DATA
- The human-subjects experiment protocol received an exempt determination from the University of Washington’s IRB.Source: §7 · p. 8
- Personalized helpfulness models were trained using publicly available Community Notes data.Source: §7 · p. 8
- The Community Notes data are anonymized by design and cannot be linked to the X users who contributed notes and ratings.Source: §7 · p. 8
A.1 Sampling technique in the PHM
The PHM uses probabilistic sampling at its final layer to avoid the score inflation seen with greedy sampling and better predict observed note intercepts.
A.1 Sampling technique in the PHM
- Greedy sampling over-represents helpful ratings and removes uncertainty in the model's raw output probabilities.Source: §A.1 · p. 10
- The PHM is a 10-layer neural network trained on publicly available Community Notes data, with probabilistic sampling at its last layer.Source: §A.1 · p. 10
- MAE fell from 0.16 with greedy sampling to 0.09 with probabilistic sampling when predicting observed note intercepts.Greedy sampling also creates an artificial score ceiling at y=0.6; the figure's red dashed line marks perfect predictions.Source: Figure 9 · p. 10, §A.1 · p. 10
A.2 Output probabilities of the PHM
The PHM’s predicted probabilities are well calibrated across its three classes, with no skew toward under- or over-confidence.
A.2 Output probabilities of the PHM
- The PHM’s reliability curve is close to the diagonal, indicating no systematic under- or over-confidence across its three classes.Probabilistic sampling relies more strongly on the model’s raw logits than greedy sampling.Source: §A.2 · p. 10
B.1 Matrix Factorization
The framework aggregates PHM-predicted jury ratings with the Community Notes matrix-factorization approach, while the appendix describes its rating questionnaire and LLM prompts.
B.1 Matrix Factorization
- The Community Notes aggregation scores candidates using fitted note factors and intercepts, selects the highest-intercept candidate, and retains the existing 0.4 helpfulness threshold.This aggregation typically scores lower than a mean, reducing false positives; the note intercept represents residual helpfulness after accounting for user tendencies and the global baseline.Source: §B.1 · p. 10
B.2 Tag Questionnaire
- The survey’s five rating categories assess source quality, clarity, relevance, veracity, and bias in language through statements about sources, comprehension, claim coverage, context, and non-argumentative wording.The corresponding statements and tags distinguish, for example, high-quality from unsupported sources and neutral language from argumentative or speculative language.Source: §B.2 · pp. 10–11
B.3 LLM Prompts
- The LLM prompts summarize existing notes into a consensus-oriented Supernote and separately check adherence to individual fact-checking principles.The synthesis prompt requests unbiased language, high-quality sources where applicable, no invented facts, and a 280-character limit; prompts used gpt-4o-mini-2024-07-18 with default parameters unless specified otherwise.Source: §B.3 · p. 11