PaperlayerView exact revision on arXiv

Source-linked AI summary

GitSearch: Enhancing Community Notes Generation with Gap-Informed Targeted Search

Sahajpreet Singh, Kokil Jaidka, Min-Yen Kan, Preslav Nakov

arXiv:2602.08945v2cs.CLcs.CY

TL;DR

Automated Community Note generation must address limited coverage and uneven quality, while existing approaches may depend on prior notes or retrieve evidence misaligned with raters’ needs. GITSEARCH identifies information gaps in a post and optional incomplete notes, then targets search and grounds note synthesis in retrieved evidence. On PolBench, it achieves 99% coverage versus 54% for Supernotes-Lite and wins 59% of human comparisons against general web-search agents.

  • Problem

    Community Notes have limited coverage and uneven quality, while prior work has rarely used incomplete or conflicting notes to operationalize the information gaps relevant to raters.

    Source: §1 · pp. 1–2

  • Method

    GITSEARCH detects gaps from a post and optional incomplete notes, uses them to guide targeted evidence retrieval, and generates a grounded Community Note.

    Source: §1 · p. 2, Phase I: Gap Detection & Prioritization · p. 3

  • Results

    GITSEARCH achieves 99% coverage versus 54% for Supernotes-Lite and wins 59% of human comparisons against general web-search agents.

    Source: §8 · p. 8

  • Takeaways & Limitations

    GITSEARCH maintains comparable note quality while extending coverage, and incomplete Community Notes can help guide retrieval.

    Source: §8 · pp. 8–9

    Evaluation covers only English-language U.S. political tweets, so generalizability to other languages, domains, or political contexts remains untested.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Community Notes enable scalable, community-based fact-checking, but automated generation faces a coverage--quality trade-off. Note-dependent methods (e.g., summarization-based Supernotes) cannot operate without prior crowd input, while web-search LLMs achieve broad coverage but often retrieve evidence misaligned with what raters need. We introduce GitSearch (Gap-Informed Targeted Search), which treats missing, ambiguous, or disputed information as first-class retrieval targets. It infers information gaps from a post and optional incomplete notes, searches specifically to resolve them, and synthesizes an evidence-grounded Community Note. To evaluate this, we introduce PolBench, a new dataset of 78,698 U.S. political tweets with Community Notes. Keeping single-call search inference budget fixed, GitSearch improves human-rated helpfulness by 0.43 over unconditioned web search and wins 59% of pairwise comparisons. It generates notes for 99% of test cases, including the 46% where summarization methods fail. Human evaluators prefer GitSearch to human-written helpful notes in 69% of comparisons, primarily due to better context coverage. These findings show that how search is organized matters more than information access alone.

1 Introduction

Community Notes face limited coverage and uneven quality, while existing methods do not use incomplete notes to identify what raters still need. GITSEARCH targets those information gaps, and its evaluations report broad coverage and stronger human preference than general web search.

1 Introduction

  • Community Notes have limited coverage and uneven quality: about 92% need more ratings, and unhelpful notes outnumber helpful notes (4.19% vs. 3.57%).Notes also vary in completeness, sourcing quality, neutrality, and adherence to platform recommendations.Source: §1 · p. 1
  • GITSEARCH treats perceived quality gaps as retrieval targets and works both without existing notes and with incomplete notes labeled NEEDS_MORE_RATINGS.Its three-stage pipeline generates gap-targeted notes while maintaining neutrality and sourcing standards.Source: §1 · p. 2
  • GITSEARCH produces notes for 99% of test cases, including the 46% where summarization baselines cannot operate, and wins 59% of comparisons against generic web-search agents.The comparison controls for model and single-call search budget.Source: §1 · p. 2
  • GITSEARCH is presented as identifying missing information, searching for relevant evidence, and generating grounded notes, including in cold-start cases.The paper’s contribution summary says gap-specific search improves notes over general web search under the same search budget.Source: §1 · p. 2
  • The paper introduces POLBENCH, a dataset of 78,698 U.S. political tweets paired with Community Notes.The stated goal is to support research on automated note generation.Source: §1 · p. 2

2 Related Work

Prior AI-assisted Community Notes systems rely on existing human contributions or additional retrieval steps. GITSEARCH identifies information gaps from the post and available notes, then uses them to guide a single web search.

Misinformation and Community Moderation

  • Community Notes can take a median of 17 hours to reach consensus, and most submitted notes never receive sufficient ratings.Contributors also vary in fact-checking expertise, and the system often relies on professional fact-checking sources.Source: §2 · p. 2

AI-assisted Community Notes Generation

  • Prior note-generation approaches either depend on existing human contributions or require additional retrieval steps, whereas GITSEARCH guides a single web search using gaps identified from the post and available notes.Supernotes summarizes crowd-written notes; CrowdNotes+ uses contributor-identified URLs; Commenotes uses post comments; and Li et al. (2024) requires iterative retrieval-augmented reasoning.Source: §2 · pp. 2–3

3 The GITSEARCH Framework

GITSEARCH decomposes note generation into gap identification, targeted retrieval, and constrained synthesis. Its retrieval targets are inferred from the post and any incomplete or conflicting notes, and synthesis is restricted to retrieved evidence.

3 The GITSEARCH Framework

  • The framework investigates whether structured gap identification, targeted retrieval, and constrained synthesis can yield more helpful notes than end-to-end generation.It frames the objective as addressing tweet claims with relevant evidence.Source: §3 · p. 3

Problem Formulation

  • The task is to generate a note addressing a post’s claims using relevant evidence, evaluated for factual correctness, contextual completeness, and utility.GITSEARCH maps gap identification to retrieval and constrained synthesis to address coverage and hallucination limitations of standard end-to-end generation.Source: Problem Formulation · p. 3

Phase I: Gap Detection & Prioritization

  • GITSEARCH infers gaps from tweet claims and incomplete or conflicting notes, using six types: unsubstantiated claim, contradiction, vague reference, missing context, source verification, and missing coverage.The taxonomy captures the information deficits needed for targeted search in 82% of sampled cases; a manual audit found accurate, well-grounded gap identification in 85.4% of sampled cases.Source: Phase I: Gap Detection & Prioritization · p. 3

Phase II: Targeted Search

  • Phase II uses prioritized gaps and the post and notes to steer a web-search LLM in one call toward credible, neutral evidence and a coherent fact-checking article.Web-Agents receive the raw tweet under the same single-call budget, isolating retrieval structure while controlling for model capability and compute.Source: Phase II: Targeted Search · pp. 3–4

Phase III: Constrained Note Synthesis

  • Phase III turns the post and retrieved evidence into a platform-compliant note, barring unsupported claims and requiring cited URLs to appear in the retrieved article.Web search is disabled during synthesis, and the model is instructed to omit claims unsupported by that article.Source: Phase III: Constrained Note Synthesis · p. 4

4 Experimental Setup

The experiments compare note summarization, unconditioned web-search agents, and the full GITSEARCH pipeline. Evaluation combines automated measures, diagnostic LLM judging, and human assessment as the primary measure.

4.1 Baselines

  • The baselines isolate note summarization, raw-tweet web search, and GITSEARCH’s full pipeline; Supernotes-Lite is limited to the 54% of test cases with existing notes.Web-Agents use the same single-call budget as Phase II, enabling comparison of gap-driven retrieval.Source: §4.1 · p. 4

4.2 Evaluation

  • Human evaluation is the primary measure, while automated similarity and source metrics and reference-conditioned LLM judging provide additional assessments.Human annotators assess factuality, completeness, and holistic helpfulness; LLM judging uses five 1–5 dimensions and does not independently verify claims.Source: §4.2 · pp. 4–5

5 The POLBENCH Dataset

POLBENCH contains 78,698 English-language U.S. political tweets linked to observed Community Notes. Its filtering and evaluation design focuses on textually comparable posts and recent tweets with helpful notes.

5 The POLBENCH Dataset

  • A two-stage collection first identifies Community Notes, then retrieves their referenced tweets.The data combine X Community Notes through September 7, 2025, with tweet texts obtained through the X API.Source: §5 · p. 5
  • Political filtering combines MNLI zero-shot classification with curated keyword tagging, retaining the union to improve recall.Tweets were further excluded if essential context depended on images or videos, if they were non-English, or if posted before 2021.Source: §5 · p. 5
  • 78,698 unique English-language U.S. political tweets with associated Community Notes form the dataset.The collection links notes to the tweets they reference, grounding the dataset in observed user-generated fact-checking activity.Source: §5 · p. 5
  • About 92% of notes remain NMR; 3.57% are rated helpful and 4.19% unhelpful.Notes average 28 words, 22.93% lack citations, and the median tweet-to-note creation and resolution lag is five hours.Source: §5 · p. 5
  • Evaluation uses 488 recent tweets with at least one CRH note, posted May 1–September 6, 2025.Only 265 of these tweets have an NMR note, limiting the cases where Supernotes can be evaluated.Source: §5 · p. 5

6 Results

Across evaluations, GITSEARCH combines near-complete coverage with note quality competitive with note-dependent generation, and gap-targeted retrieval outperforms unconditioned web search. Human evaluations favor GITSEARCH over both web agents and human-written helpful notes; incomplete notes can further improve its quality.

6.1 Gap-based Retrieval Improves Quality

  • With Gemini-2.5-Flash, GITSEARCH’s Claim Alignment is 3.744 at 99% coverage, versus Supernotes-Lite’s 3.898 at 54% and Web-Agent’s 3.315 at near-complete coverage.GITSEARCH generates notes for 99–100% of the test set while recovering much of the quality associated with note-dependent generation.Source: §6.1 · p. 6, Table 1 · p. 6

6.2 Human Evaluation Favors GITSEARCH

  • GITSEARCH is preferred to human-written helpful notes in 69% of comparisons and scores 3.87 vs. 3.36 on helpfulness.Its context-coverage score is 0.87 vs. 0.69, while factual accuracy is comparable (F1: 0.99 vs. 0.94).Source: §6.2 · pp. 6–7, Table 2 · p. 7
  • GITSEARCH beats the matched Web-Agent by 0.43 helpfulness points (3.87 vs. 3.44) and wins 59% of pairwise comparisons.The systems share model capability, web-search access, and single-call inference budget; retrieval direction is the stated difference.Source: §6.2 · p. 7, Figure 3 · p. 7

6.3 Ablation Study

  • Adding incomplete notes significantly improves Gemini-2.5-Flash on Claim Alignment (+0.322), Fact Alignment (+0.275), Completeness (+0.255), and Helpfulness (+0.225).GITSEARCH still generates in cold-start settings where Supernotes-Lite cannot operate.Source: §6.3 · p. 7, Table 3 · p. 8
  • Neither tested model shows significant Web-Agent score changes across inference timestamps, and the gap from GITSEARCH persists.The Web-Agent baselines were rerun about three months later, so the reported advantage is not explained by transient search recency.Source: §6.3 · pp. 7–8

7 Diagnostic Analysis

GITSEARCH performance varies with the information gap and the maturity of available evidence. Its difficult cases include ambiguous references and posts requiring implicit context or dispersed evidence.

7.1 Performance Varies by Gap Type

  • Contradictions and unsubstantiated claims score best, while vague references score lowest across evaluation criteria.Missing context and missing coverage gaps have lower Fact Alignment and Completeness scores (about 2.2–2.3); vague references score about 3.27 on Claim Alignment, 1.91 on Fact Alignment, and 2.27 on Completeness.Source: §7.1 · p. 8

7.2 Sensitivity to Evidence Maturity

  • Helpfulness is highest for May tweets (3.51) and lower for June–August tweets (2.98–3.16).The trend is not monotonic and may reflect interactions with topic and gap-type composition; the authors note evidence may consolidate over time.Source: §7.2 · p. 8

7.3 Case Study

  • GITSEARCH sometimes gives less exhaustive counterevidence and struggles with vague references needing conversational or temporal reconstruction.Across systems, recurring errors include contextually misaligned corrections, unresolved underspecified claims, and weak corrections to exhaustive claims when tools are treated as definitive.Source: §7.3 · p. 8

8 Conclusion

The paper presents GITSEARCH, which guides web search with information gaps, and POLBENCH, a dataset of political tweets with Community Notes. GITSEARCH achieves broad coverage, wins human comparisons against general web-search agents and human-written helpful notes, and still struggles with vague references.

8 Conclusion

  • GITSEARCH achieves 99% coverage versus Supernotes-Lite’s 54% while maintaining comparable note quality.POLBENCH contains 78,698 U.S. political tweets and associated Community Notes.Source: §8 · p. 8
  • Human evaluators preferred GITSEARCH over general web-search agents in 59% of comparisons and over human-written helpful notes in 69%.The conclusion identifies vague references as a remaining challenge and proposes evaluating different post-lifecycle stages and adaptive gap detection.Source: §8 · pp. 8–9

Limitations

The authors identify limits in GITSEARCH’s evaluation scope, retrieval and grounding, and note quality on recent events. These constraints motivate human oversight and leave generalizability beyond U.S. English-language political tweets untested.

Limitations

  • Evaluation covers only English-language U.S. political tweets; performance in other languages, domains, and political contexts remains untested.The authors also note that the empirically validated gap taxonomy may need re-derivation elsewhere.Source: Limitations · p. 9
  • Single-call retrieval can under-resolve multi-hop claims, favors crawlable outlets, and struggles with vague-reference and missing-context gaps.The authors report lower performance on the most recent events, suggesting that the maturity of online evidence affects note quality and breaking claims may be less reliable.Source: Limitations · p. 9
  • Grounding relies on prompting rather than architecture, so compliance may degrade on adversarial or out-of-distribution inputs.Residual factual-error and framing-bias risks remain for ambiguous gaps, and inference adds latency over single-call baselines; the authors recommend human oversight rather than fully autonomous moderation.Source: Limitations · p. 9

Ethical Statement

The appendix reports ethical safeguards and validation details for POLBENCH, its gap taxonomy, and automated note evaluation. Its results include taxonomy coverage and identification accuracy, along with source-quality and judge-reliability checks.

Ethical Statement

  • The ethical statement notes that real-world political data may include toxic or offensive content and recommends human-in-the-loop use because generated fact-checks can carry bias or inaccuracy risks.The study uses publicly available data and reports an ethics-review exemption; tweet and note IDs with hydration scripts are redistributed rather than tweet text.Source: Ethical Statement · p. 9

A.1 Definition of Gap Types

  • The six gap types capture distinct verification failures: unsubstantiated claims, contradictions, vague references, missing context, source verification, and missing coverage.In cold-start cases, factual assertions default to unsubstantiated claims; missing coverage includes claims or broader narratives not addressed by existing notes.Source: §A.1 · pp. 11–12
  • Missing coverage captures tweet claims that existing notes leave unaddressed, including broader narratives without contextualization in cold-start cases.Source: §A.1 · p. 12

A.2 Completeness of Gap Taxonomy; A.3 Accuracy of Gap Identification

  • The gap taxonomy covers targeted-search information deficits in 82% of sampled cases, while identified gaps are accurate and well-grounded in 85.4%.For the gap-accuracy annotations, Krippendorff’s Alpha is 0.79 across two annotators.Source: §A.2 · p. 12, §A.3 · p. 12

B.1 Dataset & Code Availability

  • The release provides note and tweet IDs with hydration scripts rather than tweet text; 12.1% of tweets had already been deleted at collection time.Source: §B.1 · p. 12

B.2 POLBENCH’s Filteration Quality Assessment

  • The hybrid political-note filter reaches macro F1 0.892 and recall 0.710, versus 0.428 for MNLI and 0.524 for keywords.The authors say higher recall reduces missed political notes and produces a cleaner dataset; Table 4 presents the quality check.Source: §B.2 · p. 12, Table 4 · p. 12

B.10 LLM-as-a-Judge Reliability

  • Across 300 jointly evaluated notes, GPT-5.2 Judge helpfulness has a moderate positive association with human ratings (Spearman ρ = 0.449; Kendall τ = 0.376).The authors use the judge diagnostically and rely on human evaluation for primary conclusions.Source: §B.10 · p. 14

C.1 Automated Evaluation

  • Domain credibility is 0.92–0.96 across systems, with human notes highest at 0.96 and GITSEARCH at 0.931–0.935; its small bias shifts versus Web-Agent were not significant.The authors attribute occasional credibility drops relative to Supernotes-Lite to fresh retrieval drawing on a broader, less curated source pool.Source: §C.1 · p. 14

C.2 Length-Controlled Robustness Check

Length-controlled analyses find that GITSEARCH’s helpfulness advantage over human-written notes is not explained by note length. The appendix also points to representative misleading post–note examples and intermediate outputs.

C.2 Length-Controlled Robustness Check

  • GITSEARCH’s helpfulness coefficient remains positive and significant after controlling for length (β = 0.52, p < 0.01), while length is not significant (p = 0.17).In pairwise preference, length difference does not predict the winner (p = 0.30), while system identity does (p < 0.001); Tables 8 and 9 report these models.Source: §C.2 · p. 14, Table 8 · p. 15, Table 9 · p. 15
  • GITSEARCH notes average 224.7 characters versus 187.9 for human notes, but the reported models do not attribute its advantage to length.Source: §C.2 · p. 14

D.1 Case Study/Error Analysis Examples

  • Table 10 presents representative misleading post–note pairs across methods, with GITSEARCH intermediate outputs and observations provided in Appendix D.2.Source: §D.1 · p. 15

D.2 Qualitative Examples of GITSEARCH Intermediate Outputs

This section presents four test-set examples tracing GitSearch’s intermediate outputs from tweets and existing notes through gap detection and targeted evidence synthesis to final notes. The examples show how prioritized gaps shape retrieval and evidence distillation, including a case where key pronoun references remain unverified.

Example 4

  • Example 4 shows a failure: Phase II assumes an unnamed document refers to moharebeh, producing a structurally valid but claim-misaligned note scored FE=4, CA=1, FA=1, C=1, and H=1.Although Phase I assigned maximum severity to the vague-reference gap, Phase II did not resolve the referent.Source: §D.2 · p. 15

D.2 Qualitative Examples of GITSEARCH Intermediate Outputs

  • Across Examples 1–3, Phase I ranks the most consequential verification barrier first, while Phase III produces concise notes grounded in retrieved evidence rather than merely truncating longer articles.The examples’ final notes include inline citations.Source: §D.2 · pp. 15–16

Example 1

  • Example 1’s highest-priority vague-reference gap identifies the Epstein Files Transparency Act, and retrieval finds that the Act bars releasing child sexual abuse material and victims’ identifying information.The final note condenses this evidence into a claim that Democrats did not vote to publish such material.Source: Example 1 · pp. 25–26

Example 2

  • Example 2 prioritizes whether “FIRED.” asserts Powell’s dismissal or expresses a preference; retrieved evidence reports that Trump said firing Powell was highly unlikely, supporting the notes that Powell had not been fired.The evidence article also notes that the tweet would foreseeably be read as a factual assertion amid public claims of imminent dismissal.Source: Example 2 · pp. 26–27

Example 3

  • Example 3 resolves two high-priority contradictions in parallel, finding public speech and books documenting Trump’s use of “enigma” and a Grok share that identifies those uses.Its final note cites the 2015 rally and Trump’s books as evidence against the tweet’s claim that he never used the word.Source: Example 3 · pp. 28–29

Example 4

  • The detected gaps included Note 1’s unsupported Trump claim and vague “document” reference, plus missing context and unanswered tweet claims about Carlson and Iran.The system assigned maximum severity to the vague-reference gap and generated a query asking which specific document supported the Trump claim.Source: Example 4 · p. 29
  • The retrieval found no credible evidence Tucker Carlson denied moharebeh’s existence or said he would nuke Iran if it existed; the pronoun mapping remained unverified.It did not recover the specific letter or document referenced by the tweet, and it concluded Note 1’s claim about Trump was unsubstantiated.Source: Example 4 · p. 30
  • The final note says Carlson has not claimed moharebeh does not exist, has warned against war with Iran, and that moharebeh can carry execution as a penalty.It describes moharebeh as “waging war under Iranian Islamic law.”Source: Example 4 · p. 30
Loading 2602.08945v2…