Generating Fact Checking Briefs

• Fact checking is difficult because it requires topic knowledge and understanding misinformation, while a single claim can take professionals 15 minutes to one day to check.Crowdsourced fact-checking accuracy has been reported at around 58%, with volunteer efforts succeeding mainly at the simpler task of claim detection.Source: §1 · p. 1
• Existing automated approaches often frame fact checking as classification, but practitioners prefer tools that improve efficiency in reaching a verdict rather than complete automation.Source: §1 · p. 1
• The paper proposes briefs to provide evidence useful to human fact checkers and increase fact-checking accuracy and efficiency.The investigated forms include retrieved Wikipedia passages, information about mentioned entities, and question-answer pairs.Source: §1 · pp. 1–2, Figure 2 · p. 2
• QABRIEFDATASET contains about 10,000 crowdsourced QABriefs with roughly three question-answer pairs each.QABRIEFER generates claim-conditioned questions, retrieves evidence with a search engine, and produces answers with a finetuned pretrained question-answering model.Source: §1 · p. 2
• 10% higher accuracy and reduced checking time were observed for crowdworkers using QABriefs rather than only a search bar.For volunteer fact checkers, accuracy improved by 4% and the process was 20% faster.Source: §1 · p. 2

2 Briefs for Fact Checking

• Fact-checking briefs provide relevant information before checking a claim through Passage, Entity, and Question Answering Briefs.The proposed forms use modern NLP approaches to gather information about terms, people, bills, and other claim concepts.Source: §2 · p. 2

2.1 Passage Briefs

• Passage Briefs retrieve one relevant Wikipedia passage for a claim using DPR and limit it to 500 tokens for readability.The method keeps Passage Briefs focused on Wikipedia because web-based retrieval often failed to find a single passage addressing an entire claim.Source: §2.1 · p. 2

2.2 Entity Briefs

• Entity Briefs address claims needing multiple information pieces by linking each mentioned entity to Wikipedia and displaying information from multiple entity pages.For each entity, the brief provides the first paragraph of its Wikipedia page.Source: §2.2 · p. 3

2.3 Question Answering Briefs

• QABriefs decompose complex claims into questions and answers, making their information needs more compatible with retrieval systems designed for simpler factoids.The paper gives questions about what social security is and who invented it as an example decomposition.Source: §2.3 · p. 3

3.1 Question Generation

• Questions are collected by having workers read fact-checking articles and reconstruct the investigative process in two to five standalone questions per claim.Source: §3 · p. 3, §3.1 · p. 4
• Question validation rejects redundancy, rephrasing, nonsensical wording, and underspecification, while later review can add questions or clarify details such as years.Source: §3.1 · p. 4

3.2 Question Answering

• Answers are gathered through web search and labeled extractive, abstractive, or no-answer, with separate validation for incorrect answers and unverifiable no-answer cases.Source: §3.2 · p. 4

3.3 QABrief Dataset Statistics

• QABRIEFDATASET contains 6,897 claims and 21,168 question-answer pairs, with 500 claims each reserved for validation and testing.Source: §3.3 · p. 4
• Each claim requires around 3 diverse questions averaging 10.5 words, while answers average 43.5 words and are mainly extractive.Source: §3.3 · p. 5
• Across repeated annotations of 50 claims, about half of workers’ questions were paraphrases, while answer selections often differed in length but remained paraphrastic.Source: §3.3 · p. 5

4 QABrief Model

• QABrief generation is challenging because real claims are diverse, answers are long-form, and the process requires question planning followed by web search and answer generation.Source: §4 · p. 5
• QABRIEFER generates multiple claim-conditioned questions, retrieves a top search result for each, and supplies the evidence document and question to an answering model.Source: §4 · p. 5

4.1 Question Generation

• Question generation uses BART to produce either all questions jointly or sequentially, optionally conditioning on the claim source and previous questions.Source: §4.1 · pp. 5–6

4.2 Question Answering

• The answering component fine-tunes BART first on Natural Questions and then on QABRIEFDATASET, using an abstractive model for extractive, abstractive, and explanatory no-answer outputs.Source: §4.2 · p. 6

5 Experimental Setup

• The main experiment evaluates human accuracy and time on fact-checking claims with and without briefs, comparing crowdworkers and volunteer evaluators.Source: §5 · p. 6, §5.1 · pp. 6–7

5.1 Human Evaluation

• Fact-checking accuracy is agreement between an evaluator’s verdict and professional labels from DATACOMMONS, while time spans task loading through submitted verdict and explanation.Source: §5.1 · p. 6
• Crowdworkers evaluate 500 claims using a three-way true, false, or middle scale, whereas volunteers evaluate 100 claims with the same task and compare search-only, generated-QABrief, and gold-QABrief conditions.Source: §5.1 · pp. 6–7

5.2 Automatic Evaluation of Model Quality

• Question generation is evaluated with BLEU and question answering with F1, while QABRIEFER uses pretrained BART trained for 10 epochs with beam size 5.Source: §5.2 · p. 7, §5.3 · p. 7

6.1 Briefs Increase Fact Checking Quality

• Crowdworkers’ accuracy increases with briefs, and their average time falls from 8.8 minutes to around 7 minutes; Entity Briefs and QABriefs outperform search alone, while Passage Briefs do not.Source: §6.1 · p. 7, §6 · p. 7
• For volunteer fact checkers, accuracy is higher across methods and Gold QABriefs remain best, while QABriefs slightly reduce time taken.Source: §6.1 · p. 7

6.2 QABriefs are Preferred

• Crowdworkers rate QABriefs helpful twice as often as Passage Briefs, and their explanations frequently describe using QABriefs to search for evidence.Source: §6.2 · p. 7

6.3 Generating QABriefs with QABRIEFER

• Iterative question generation performs best, source information improves generation, and pretraining on Natural Questions before QABRIEFDATASET gives the best answering results.Source: §6.3 · p. 8

• Question generation and answering had been studied for FEVER, but primarily to elicit answers rather than improve human fact-checking accuracy or efficiency.Source: §7 · p. 8
• FEVER uses modified Wikipedia sentences, whereas later work on real-world claims often relied on existing reports or retrieved material without ensuring direct supporting or refuting evidence.Source: §7 · p. 8
• Studies improved fact-checking accuracy through feedback, additional time, tooling, or training, while this work adds briefs as another form of human-in-the-loop support.Source: §7 · p. 8
• Existing automated fact-checking systems face challenges, motivating tools that improve efficiency in reaching a verdict rather than automating the entire process.Source: §7 · p. 8

• Briefs are not claimed to improve professional fact checkers’ work because professionals investigate alternative interpretations, explain their process, and handle varied claims with substantial experience.Source: §8 · p. 8
• Around 45% of crowdworkers given a Gold QABrief did not use the search bar, even though accuracy was reduced without it.Source: §8 · pp. 8–9
• Briefs aid accuracy and efficiency but are not sufficient by themselves to produce a verdict.Source: §8 · p. 9
• Professional fact checkers do not agree perfectly, and they may differ in the importance assigned to claim components, checking directions, and uncalibrated scales.Source: §8 · p. 9

• The paper proposes briefs to be read before fact checking and develops Passage, Entity, and Question Answering Briefs as forms of pre-check information.Source: §9 · p. 9
• QABRIEFER and QABRIEFDATASET support creating QABriefs, and empirical studies report improved fact-checking accuracy and efficiency for crowdworkers and volunteers.Source: §9 · p. 9

10.1 Dataset Analysis

• QABRIEFDATASET questions decompose comparisons, historical events, and statements by individuals into component checks involving entities, context, time, location, and attribution.Source: §10.1 · p. 12
• QABRIEFDATASET requires answering complex questions on the open web, where context-sensitive facts such as year-dependent statistics make answer quality difficult to verify.Source: §10.1 · p. 12
• About 50% of annotators entered questions directly into the search bar, the remainder mainly used keywords, and around 83% searched only once.Source: §10.1 · p. 12
• Most annotators submitted answers from the first three search results, often using Wikipedia when it appeared at the top.Source: §10.1 · p. 12

10.2 Additional Human Evaluation Results

• Non-professionals often relied on general understanding rather than validating every detail, unlike professionals who cross-reference statistics by year, measurement, and country.Source: §10.2 · pp. 12–13
• Self-reported difficulty did not correspond well to accuracy: easy claims were only 4% more accurate, while medium and hard claims had the same accuracy.Source: §10.2 · p. 13
• The volunteer fact-checking time distribution was bimodal, and QABriefs shifted it toward shorter durations.Source: §10.2 · p. 13, Figure 9 · p. 13

10.3 Model Training Details

• QABRIEFER training used fine-tuned BART-large models for question generation and question answering, with validation-based checkpoint and generation-parameter selection.Source: §10.3 · p. 13

Important!

• Annotators must avoid yes-or-no questions and direct truth questions, instead asking open questions such as how sharks move.Source: Important! · p. 14
• Automatic checks require at least three questions of five tokens each, limit overlap, forbid exact claim matches, and block submission during the first minute.Invalid submissions trigger explanatory error messages.Source: Important! · p. 14
• Question validation asks annotators to compare questions with the claim and fact-checking article, then add questions when needed or select that none are needed.Additional question-writing follows the original guidelines, and writing more questions earns a bonus.Source: Important! · p. 14
• Questions should be specific enough for web search, adding missing entities or time periods rather than merely rephrasing the claim.Examples add “in 2000” to a population question and identify Senator Smith and March 2000 in a voting question.Source: Important! · p. 14

10.4.3 Question Answering Data Collection

• For each question, annotators use the provided search bar to find an answer, explain when none is available, and avoid answering the claim or predicting its verdict.They are instructed not to use other search bars or submit answers from named fact-checking sites.Source: §10.4.3 · p. 14

• Annotators must use the provided search bar, avoid fact-checking domains, and submit answers to questions rather than verdicts on claims.The workflow prevents submission of URLs from known fact-checking domains and instructs annotators not to predict a verdict.Source: Important! · p. 15
• After searching, annotators choose whether they found copyable answer text, must transcribe an answer from a graph, table, or picture, or found no answer.These branches correspond to extractive, abstractive, and no-answer responses.Source: Important! · p. 15
• Extractive responses require a targeted answer passage and its source URL, with copied text limited to 250 words.Annotators are told not to copy an entire website.Source: Important! · p. 15
• Abstractive responses require at least 20 words describing the answer and a source URL, while no-answer responses require at least 20 words explaining why no answer was found.Source: Important! · p. 15
• Automatic validation requires all branch-specific fields, checks answer-length constraints, and verifies that submitted evidence URLs match returned search-result URLs.The URL check also verifies use of the restricted search bar and prevents fact-checking-domain evidence.Source: Important! · p. 15