PaperlayerView exact revision on arXiv

Source-linked AI summary

FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios

I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu

arXiv:2307.13528v2cs.CLcs.AI

TL;DR

Long-form generated text can lack explicit claims and evidence, making common factuality-detection formulations ill-suited to checking it. FacTool combines tool-based evidence gathering with LLM reasoning to assess factuality. Across four tasks, GPT-4-powered FacTool outperforms baselines, with 89.09 claim-level F1 / 71.79 response-level F1 on KB-based QA.

  • Problem

    Common claim- or evidence-based factuality checks are not well suited to validating long-form generated text that lacks explicit claims and evidence.

    Source: §1 · p. 2

  • Method

    FACTOOL uses tools to gather evidence and LLM reasoning to assess factuality through claim extraction, query generation, tool querying, evidence collection, and verification.

    Source: §1 · p. 2, §7 · p. 12

  • Results

    FACTOOL powered by GPT-4 outperforms all other baselines across all scenarios, with 89.09 claim-level F1 / 71.79 response-level F1 on KB-based QA.

    Source: §6.2.1 · pp. 8–9

  • Takeaways & Limitations

    Experiments across knowledge-based QA, code generation, math problem solving, and scientific literature review demonstrate the potential of tool use for factual-error detection.

    Source: §7 · p. 12

    For math problems, FACTOOL does not verify the reasoning process when its extracted claims involve only mathematical calculations.

    Source: §6.2.3 · p. 11

Abstract

from arXiv · show

The emergence of generative pre-trained models has facilitated the synthesis of high-quality text, but it has also posed challenges in identifying factual errors in the generated text. In particular: (1) A wider range of tasks now face an increasing risk of containing factual errors when handled by generative models. (2) Generated texts tend to be lengthy and lack a clearly defined granularity for individual facts. (3) There is a scarcity of explicit evidence available during the process of fact checking. With the above challenges in mind, in this paper, we propose FacTool, a task and domain agnostic framework for detecting factual errors of texts generated by large language models (e.g., ChatGPT). Experiments on four different tasks (knowledge-based QA, code generation, mathematical reasoning, and scientific literature review) show the efficacy of the proposed method. We release the code of FacTool associated with ChatGPT plugin interface at https://github.com/GAIR-NLP/factool .

1 Introduction

Generative AI brings many tasks into a natural-language interface, but its outputs can contain factual errors and long-form generations challenge claim-by-claim checks. The introduction presents FACTOOL as a tool-augmented framework for detecting such errors across tasks and domains.

1 Introduction

  • LLM-generated content can sound credible while being factually incorrect or imprecise, limiting its use in high-stakes fields such as healthcare, finance, and law.The introduction argues that systematic error identification is important for improving generated content's usefulness and reliability.Source: §1 · p. 1
  • Existing approaches are often task-specific, while claim- or evidence-level definitions do not fit long-form generations that lack explicit claims and evidence.The paper argues for a more comprehensive framework that is versatile across tasks and domains.Source: §1 · pp. 1–2
  • FACTOOL gathers evidence with tools such as search, code interpreters, Python, and LLMs, then uses LLM reasoning to assess factuality across four task types.The benchmark covers knowledge-based QA, code generation, math problem solving, and scientific literature review writing.Source: Figure 1 · p. 1, §1 · p. 2
  • The paper extends factuality detection to better audit current generative AI models and connects tool use with a versatile framework spanning tasks and domains.The authors also use FACTOOL to evaluate modern chatbots.Source: §1 · p. 2
  • GPT-4 has the best factuality across almost all scenarios, while Vicuna-13B performs reasonably in KB-based QA but poorly on more challenging tasks.The challenging scenarios include code generation, math problem solving, and scientific literature review writing.Source: §1 · p. 2

2 Related Work

Prior factuality-detection work generally assumes a supplied claim or evidence and targets specific use cases, while FacTool addresses detection without explicit claims or evidence across varied scenarios. The related work also situates this approach among language-model tool-use frameworks.

Factuality Detection in Natural Language Processing

  • Prior work varies in its response, claim, and evidence formulations, including FEVER-based claim verification, summary consistency, Wikipedia-fact support, and RARR’s prompted retrieval approach.Examples include FEVER (Thorne et al., 2018a), models by Zhong et al. (2020) and Krishna et al. (2022), FactCC (Kryscinski et al., 2020), QAGS (Wang et al., 2020), WICE (Kamoi et al., 2023), and RARR (Gao et al., 2022a).Source: §2 · pp. 2–3
  • FacTool tackles factuality detection without explicit claims or evidence, a more challenging and practical setting than prior use-case-specific approaches.The proposed framework is intended to address this setting across a variety of scenarios.Source: §2 · p. 3

Tool use in Large Pretrained Language Models

  • Prior language-model tool use supports tasks such as question answering and dialogue, while FacTool explores tool use specifically for factuality detection.Examples include internet information gathering, interaction with five tools, connections to AI models, and links to existing APIs (Press et al., 2022; Komeili et al., 2022; Schick et al., 2023; Shen et al., 2023; Liang et al., 2023).Source: §2 · p. 3

3 Revisiting Factuality in Generative AI

FacTool broadens factuality detection from text to generated signals such as code and mathematics, judging claims against evidence rules tailored to each task. It also defines fine-grained assessment and instantiates it across four application scenarios.

3.1 Definition

  • FacTool defines factuality across generated text, code, and mathematical signals as whether claims are supported by task-specific evidence rules, unifying detection beyond text.Evidence may come from a knowledge base, a programming-language verification rule, or a mathematical rule.Source: §3.1 · p. 3
  • Fine-grained detection can pinpoint inaccuracies and support reward models, while claim-definition-based few-shot prompting addresses granularity and claim-extraction challenges.The task is organized around a prompt, response, claim, and evidence.Source: §3.1 · p. 3

3.2 Instantiations in Different Scenarios

  • In knowledge-based QA, factuality measures whether each generated answer claim is supported by world knowledge, with FacTool targeting challenging long-form open-domain answers.The task uses a given knowledge base or open-domain source such as Wikipedia.Source: §3.2 · p. 3
  • For code generation, factuality requires the generated code to execute correctly in the target language and meet the query’s requirements, assessed against test inputs and expected outputs.Source: §3.2 · p. 4
  • For math problem solving, factuality concerns both whether individual claims follow calculation rules and whether the overall solution addresses the problem.Source: §3.2 · p. 4
  • For scientific literature reviews, factuality means correctly citing existing research, including accurate author names and publication years.Source: §3.2 · p. 4

4 Approach

FacTool uses domain tools and LLMs in a unified factuality-detection framework that extracts claims, generates queries, gathers evidence, and verifies agreement. Its procedures adapt claim definitions and verification methods to different task domains.

4 Approach

  • The framework combines domain expertise for evidence gathering with LLM use of multiple tools across five stages: claim extraction, query generation, tool querying, evidence collection, and agreement verification.It is designed to apply a unified approach across tasks.Source: §4 · p. 4

4.1 Claim Extraction

  • Claim extraction uses LLM prompts guided by task-specific claim definitions, reducing annotation and training needs; the authors report strong ChatGPT performance on extracting atomic claims.The paper states that claim definitions can be supplied textually for systems or datasets, and reports detailed extraction results in Section 6.1.Source: §4.1 · p. 4
  • Claims are defined as atomic facts in QA, code snippets in code generation, arithmetic operations in math solutions, and (paper title, year, authors) tuples in literature reviews.For QA, claims are limited to 15 words and resolved for coreference or ambiguity; math claims include an operation and its calculated answer.Source: §4.1 · p. 5

4.2 Query Generation

  • Queries are tailored by task: two search queries for QA, test inputs and candidate solutions for code, executable checks for math, and paper-title searches for literature reviews.Code experiments use three simulated inputs and three potential solutions; literature searches assume an existing paper appears first in Google Scholar results.Source: §4.2 · pp. 5–6

4.3 Tool Querying & Evidence Collection

  • Evidence comes from Google Search snippets for QA, code execution across test-input and candidate-solution pairs, executable math checks, and Google Scholar metadata for literature claims.Search snippets can include answer boxes, knowledge graphs, and organic results.Source: §4.3 · p. 6

4.4 Agreement Verification

  • Verification labels claims true or false using task-specific rules: evidence-based LLM judgment for QA, majority-vote pseudo-golden outputs for code, execution outcomes for math, and metadata matching for literature.Literature titles and years require exact case-insensitive matches, while the extracted author list must be a subset of the retrieved list; every field must match.Source: §4.4 · pp. 6–7

5 Dataset Construction

Section 5 describes a four-task dataset suite and task-dependent procedures for forming claims and labeling factuality.

5.1 Prompt and Response Collection

  • The evaluation collected task-specific prompts and responses for knowledge-based QA, code generation, math problems, and scientific literature review.QA uses 100 RoSE reference summaries and 50 FactPrompts responses (Liu et al., 2022; Lin et al., 2022); code responses use processed HumanEval prompts (Chen et al., 2021; Chen et al., 2022), and ChatGPT answers 100 sampled GSM-Hard prompts with positive target values (Gao et al., 2022b). Literature prompts follow self-instruct (Wang et al., 2023), cover five fields, and request technical or research responses with a literature citation; positive math targets avoid illogical negative-value scenarios.Source: §5.1 · p. 7

5.2 Claim Collection

  • FactPrompts and GSM-Hard claims are extracted using ChatGPT, while each HumanEval code response itself serves as its claim.For FactPrompts and GSM-Hard, the method reuses claim prompts from §4.1; ChatGPT is chosen for cost efficiency and effectiveness in extracting fine-grained claims.Source: §5.2 · p. 7

KB-based QA & Scientific Literature Review

  • Authors label extracted claims factual or non-factual, and a response is non-factual if any claim is non-factual.For code, claim and response labels coincide, with execution against HumanEval tests determining response factuality; for math, response labels use GSM-Hard target values.Source: KB-based QA & Scientific Literature… · p. 7

6 Experiments

The experiments assess claim extraction and FACTOOL against self-check baselines across four tasks, then use FACTOOL to evaluate chatbot factual accuracy.

6.1 Exp-I: Claim Extraction Evaluation

  • Claims extracted by GPT-4, ChatGPT, and Flan-T5 closely match human-annotated ACUs under ROUGE and BERTScore.ChatGPT is selected for Exp-II because Flan-T5's 512-token context is too short for lengthy responses, while ChatGPT is more cost-efficient than GPT-4 at similar effectiveness.Source: Table 4 · p. 8, §6.1 · p. 8

6.2.1 Result

  • GPT-4-powered FACTOOL achieves the highest claim-level and response-level F1 in each of four evaluated tasks.Its claim-level / response-level F1 is 89.09 / 71.79 on KB-based QA, 92.11 / 92.11 on code generation, 98.97 / 80.36 on math problems, and 95.24 / 94.74 on scientific literature review.Source: §6.2.1 · pp. 8–9, Table 5 · p. 9
  • GPT-4 FACTOOL exceeds GPT-4 Self-Check (3) in response-level F1 on QA (71.79 vs 57.14), code (92.11 vs 85.26), math (80.36 vs 70.59), and science (94.74 vs 21.54).Self-check models also have lower response-level precision, suggesting they label claims True more often and are less sensitive to errors; ChatGPT Self-Check (0) nevertheless outperforms ChatGPT FACTOOL on KB-QA, where agreement verification can reason inconsistently despite sufficient evidence.Source: Table 5 · p. 9, §6.2.1 · pp. 9–10

6.2.2 Performance Analysis

  • Scenario analyses link performance to evidence retrieval and reasoning, test generation, Python verification, and literature author-list matching.GPT-4 FACTOOL handles recent-event and high-precision-statistic claims, while its strong code-test and solution generation helps identify errors; both GPT-4 and ChatGPT FACTOOL excel at math checks and author-list verification.Source: §6.2.2 · pp. 10–11

6.2.3 Failure Analysis

  • KB-QA failures stem from reasoning errors despite sufficient evidence, conflicting evidence, and ambiguous claims; code failures arise from limited test-case variety or bugs in generated solutions.Majority voting mitigates but does not eliminate bugs in generated solutions.Source: §6.2.3 · p. 11
  • Math failures include Python round-off errors and missed reasoning errors because FACTOOL checks calculations rather than the solution's reasoning; literature-review failures include title- and author-matching errors.The math limitation can leave an incorrectly substituted cost expression undetected, while literature title abbreviations can also cause mismatches.Source: §6.2.3 · p. 11

6.3 Exp-III: Using FACTOOL to Evaluate the Factuality of Modern Chatbots

  • GPT-4 has the best weighted claim-level and response-level factual accuracy among GPT-4, ChatGPT, Bard, Claude-v1, and Vicuna under FACTOOL evaluation.GPT-4 leads on both accuracy measures in most scenarios; Vicuna-13B does reasonably well on common KB-QA prompts but less well on challenging math, code, and scientific prompts. The evaluation uses 30 KB-QA prompts and 10 each for code, math, and scientific prompts, with weighted claim-level accuracy reflecting scenario prompt ratios.Source: Table 6 · p. 12, §6.3 · p. 12

7 Conclusion

FACTOOL is a task- and domain-agnostic, five-step tool-augmented framework for detecting factual errors in generative AI. Experiments across four task types demonstrate the potential of diverse tools for factual error detection.

7 Conclusion

  • FACTOOL addresses lengthy generated text, unclear fact granularity, and scarce checking evidence through claim extraction, query generation, tool querying, evidence collection, and verification.The framework expands factuality to focus on auditing generative AI capabilities.Source: §7 · p. 12
  • Experiments demonstrate the potential of Google Search, Google Scholar, code interpreters, Python, and LLMs for factual error detection across knowledge-based QA, code generation, math problem solving, and scientific literature review writing.The authors believe the framework can be extended to more scenarios.Source: §7 · p. 12

A Prompts

The appendix lists the claim-extraction, query-generation, and agreement-verification prompts used in the paper; all are user prompts and share the system prompt “You are a brilliant assistant.”

A Prompts

B Example cases of FACTOOL

The appendix presents FACTOOL example cases across knowledge-based QA, code generation, mathematics, and scientific claims, including verified claims and error cases. The examples report generated queries, collected evidence, reasoning, and final decisions.

B Example cases of FACTOOL

  • KB-QA examples show verified claims and reasoning or other error cases for FACTOOL powered by ChatGPT or GPT-4.The figures report queries, evidence, reasoning, and final decisions for the GPT-4 examples.Source: Figure 10 · p. 20, Figure 9 · p. 20, Figure 15 · p. 23
  • Code examples show claims verified by GPT-4 and by both ChatGPT and GPT-4, alongside a code error case.The figures report generated queries, collected evidence, reasoning, and final decisions.Source: Figure 11 · p. 21, Figure 12 · p. 22, Figure 16 · p. 24
  • Math examples show claims verified by ChatGPT and GPT-4, with a separate GPT-4 error case.The example figure reports queries, evidence, reasoning, and the final decision.Source: Figure 13 · p. 23, Figure 17 · p. 25
  • Scientific examples show claims verified by ChatGPT and GPT-4, with a separate GPT-4 error case.The figures report the verification responses, including queries, evidence, reasoning, and final decisions.Source: Figure 14 · p. 23, Figure 18 · p. 25
  • The appendix flags limited variety in synthetic test cases and potential errors in code generation.Source: B Example cases of FACTOOL · p. 24

Related papers

Loading 2307.13528v2…