PaperlayerView exact revision on arXiv

Source-linked AI summary

FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios

I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu

arXiv:2307.13528v2cs.CLcs.AI

TL;DR

Generative AI increases the need to detect factual errors in lengthy, claim-granularity-unclear text without explicit evidence. FacTool addresses this with a tool-augmented, task- and domain-agnostic framework, and GPT-4-powered FacTool outperforms baselines across four evaluated scenarios.

  • Problem

    LLM-generated content can appear credible while being factually incorrect, while existing factuality detection is poorly suited to long-form generations without explicit claims and evidence.

  • Method

    FacTool uses claim extraction, query generation, tool querying, evidence collection, and verification to assess factuality across diverse tasks and domains.

  • Results

    FACTOOL powered by GPT-4 outperforms all baselines across knowledge-based QA, code generation, math problems, and scientific literature review, achieving 89.09 claim-level F1 on KB-based QA.

  • Takeaways & Limitations

    The framework demonstrates that tools including search, scholarly search, code interpreters, Python, and LLMs can support factual error detection across varied scenarios.

  • Takeaways & Limitations

    For math problems, FacTool does not verify the reasoning process of mathematical solutions and can misclassify calculations because of Python round-off errors.

Abstract

from arXiv · show

The emergence of generative pre-trained models has facilitated the synthesis of high-quality text, but it has also posed challenges in identifying factual errors in the generated text. In particular: (1) A wider range of tasks now face an increasing risk of containing factual errors when handled by generative models. (2) Generated texts tend to be lengthy and lack a clearly defined granularity for individual facts. (3) There is a scarcity of explicit evidence available during the process of fact checking. With the above challenges in mind, in this paper, we propose FacTool, a task and domain agnostic framework for detecting factual errors of texts generated by large language models (e.g., ChatGPT). Experiments on four different tasks (knowledge-based QA, code generation, mathematical reasoning, and scientific literature review) show the efficacy of the proposed method. We release the code of FacTool associated with ChatGPT plugin interface at https://github.com/GAIR-NLP/factool .

1 Introduction

Generative AI broadens language-model applications while increasing the need to detect factual errors in long-form outputs. FacTool addresses this gap with a tool-augmented, task- and domain-agnostic framework evaluated across four scenarios.

  • Generative AI supports question answering, code generation, mathematical problem solving, and scientific literature generation through a unified natural-language interface.
  • Generated content can appear credible while being factually incorrect or imprecise, limiting its use in high-stakes areas such as healthcare, finance, and law.
  • Existing factuality methods usually target individual tasks, motivating a framework versatile across the diverse domains handled by large language models.
  • Conventional claim- or evidence-centered definitions do not fit long-form generations that lack explicit claims and evidence.
  • FacTool combines tool use with language-model reasoning to gather evidence and assess factuality across knowledge-based QA, code generation, math problem solving, and scientific literature review writing.Its tools include Google Search, Google Scholar, code interpreters, Python, and language models themselves.
  • The paper revisits factuality detection to enable broader auditing of current generative AI models.

2 Related Work

Prior factuality-detection research generally assumes a specific task and often provides a claim or evidence for verification. FacTool instead targets factuality detection without explicit claims or evidence across varied scenarios.

  • Existing approaches commonly focus on a single use case, including retrieval-based QA verification, summarization hallucination detection, and execution-based code evaluation.
  • RARR prompts language models to generate queries, retrieve evidence, and determine factuality.
  • FacTool introduces factuality detection without explicit claims or evidence, extending the setting across multiple scenarios.

3 Revisiting Factuality in Generative AI

The paper broadens factuality beyond text claims supported by a knowledge base to generated signals evaluated under task-specific rules. It also emphasizes fine-grained claims and defines factuality across four application scenarios.

  • 3.1 Definition: Factuality is extended to whether claims in generated text, code, or mathematical expressions are supported under knowledge-base, programming, or mathematical rules.
  • 3.1 Definition: Fine-grained factuality assessment can locate inaccuracies and serve as a reward model for refining generative systems.
  • 3.1 Definition: Claim-definition-based few-shot prompting addresses the challenges of specifying granularity and extracting claims at the intended level.
  • 3.1 Definition: The task formalizes prompts, responses, claims, and evidence for fine-grained factuality detection.
  • 3.2 Instantiations in Different Scenarios: For knowledge-based QA, factuality measures whether claims in long-form open-domain answers are supported by world knowledge.
  • 3.2 Instantiations in Different Scenarios: For code, math, and scientific reviews, factuality concerns executable correctness, adherence to calculation rules, and accurate citation of authors and publication years, respectively.

4 Approach

The framework applies tool-augmented factuality detection across tasks by extracting task-specific claims, querying external tools for evidence, and verifying agreement. It operationalizes these steps differently for natural-language, code, mathematical, and literature-review outputs.

  • The framework unifies claim extraction, query generation, tool querying, evidence collection, and agreement verification across tasks.
  • Claim Extraction: LLM-guided claim extraction uses task-specific textual definitions to identify verifiable claims in generated responses.The paper argues this reduces annotation and training costs and provides reusable claim definitions.
  • Claim Extraction: For math problems, claims are arithmetic operations converted into executable Python checks that return whether calculated answers match.
  • Query Generation: For code generation, generated test inputs and potential solutions are executed, producing input-output evidence for verification.The framework later compares verified solutions against pseudo-golden outputs established by majority vote across solutions.
  • Tool Querying & Evidence Collection: For scientific literature reviews, paper titles are queried through Google Scholar, which supplies titles, authors, and publication years for comparison.The paper assumes an existing paper should appear as the first Google Scholar result when its title is queried.
  • Agreement Verification: Each extracted claim receives a TRUE or FALSE label according to support from collected evidence.The labeling process is applied to every individual claim.

5 Dataset Construction

The evaluation assembles task-specific datasets and generated responses for knowledge-based QA, code generation, math problems, and scientific literature review. Claims and responses receive factuality annotations using task-appropriate procedures.

  • KB-based QA: Knowledge-based QA uses 100 RoSE reference summaries and 50 FactPrompts responses generated by ChatGPT.RoSE provides fine-grained atomic content units, while FactPrompts contains real-world prompts and ChatGPT responses.
  • Code Generation: HumanEval supplies code-generation problems with unit tests, and ChatGPT generates responses from prompts containing instructions without demonstrations.
  • Math Problems: Math evaluation samples 100 GSM-Hard prompts with positive target solution values and generates responses using ChatGPT.Positive targets help avoid illogical situations and additional assumptions involving negative quantities.
  • Scientific Literature Review: Scientific-literature evaluation creates 100 diverse self-instruct prompts across computer science, business, law, medicine, and physics.Each prompt requests a technical or research-oriented response containing at least one relevant literature citation.
  • Claim Processing: ChatGPT extracts claims from FactPrompts and GSM-Hard responses, while HumanEval code snippets are treated as single claims.
  • Annotation: Annotations classify extracted claims as factual or non-factual, with any non-factual claim making the entire response non-factual.
  • Annotation: HumanEval responses are annotated by executing generated code against test cases, distinguishing responses that pass all tests from those that do not.
  • Annotation: Math responses are annotated using the target value provided in GSM-Hard.

6 Experiments

Across four tasks, FACTOOL powered by GPT-4 outperformed the evaluated baselines, while performance varied by backbone and task. The experiments also identify task-specific strengths and failure modes in evidence use, reasoning, claim extraction, and synthetic test generation.

  • Claim extraction by GPT-4, ChatGPT, and Flan-T5 closely matched human-annotated atomic content units under ROUGE and BERTScore.ChatGPT was selected for Exp-II because Flan-T5’s 512-token context was too short for lengthy responses and ChatGPT was more cost-efficient than GPT-4 with similar effectiveness.
  • FACTOOL powered by GPT-4 achieved the highest reported performance across KB-based QA, code generation, math problems, and scientific literature review.It reached 89.09/71.79 claim-level/response-level F1 on KB-QA, 92.11/92.11 on code, 98.97/80.36 on math, and 95.24/94.74 on scientific review.
  • FACTOOL powered by GPT-4 outperformed the GPT-4 Self-Check (3) baseline in response-level F1 across all four scenarios.The comparisons were 71.79 vs. 57.14 on KB-QA, 92.11 vs. 85.26 on code, 80.36 vs. 70.59 on math, and 94.74 vs. 21.54 on scientific literature review.
  • In scientific literature review, FACTOOL powered by GPT-4 substantially exceeded Self-Check (3) in both claim-level and response-level F1.The respective comparisons were 95.24 vs. 36.71 and 94.74 vs. 21.54; Google Scholar was highly robust for citation finding relative to the LLM itself.
  • Self-check models classified claims as True more frequently and had lower response-level precision than FACTOOL powered by GPT-4.For GPT-4 Self-Check (3) versus FACTOOL, precision was 63.16 vs. 87.50 on KB-QA, 76.43 vs. 89.74 on code, 58.33 vs. 69.23 on math, and 12.73 vs. 100.00 on scientific review.
  • ChatGPT-powered self-check models outperformed ChatGPT-powered FACTOOL on KB-QA, where reasoning errors confused agreement verification despite sufficient evidence.The paper gives a self-contradictory fortune-cookie example as an instance of this failure.
  • Task-specific analysis links code-generation performance to synthetic test cases and candidate solutions, and math performance to accurate Python verification snippets.Both GPT-4 and ChatGPT correctly detected that 23 × 4319216 does not equal 99305768.

7 Conclusion

FACTOOL is presented as a task- and domain-agnostic, five-step framework for detecting factual errors in generated text using external tools and verification. The paper demonstrates it across knowledge-based QA, code generation, mathematics, and scientific literature review.

  • FACTOOL is designed as a task- and domain-agnostic framework for factual error detection in generative AI.
  • The five-step pipeline performs claim extraction, query generation, tool querying, evidence collection, and verification.
  • The framework uses tools including Google Search, Google Scholar, code interpreters, Python, and LLMs to support factuality detection.
  • FACTOOL is evaluated on knowledge-based QA, code generation, math problem solving, and scientific literature review writing.

A Prompts

The appendix lists prompts for FACTOOL’s claim extraction, query generation, agreement verification, code solution generation, and code test-case generation. These prompts impose structured output formats for the corresponding model responses.

  • The prompt appendix covers claim extraction, query generation, agreement verification, and code-generation-related prompts.
  • The KB-QA query-generation prompt asks for two skeptical search-engine queries for verifying each claim.
  • The code-solution prompt requires Python and a dictionary containing reasoning and python_solution fields.
  • The experiments also use prompts for generating three distinct function calls as code test cases.

B Example cases of FACTOOL

The appendix presents worked FACTOOL examples and error cases across KB-QA, code, mathematics, and scientific claims. Examples expose generated queries, collected evidence, reasoning, and final decisions, while error cases illustrate failures and limited testcase variety.

  • Examples: An example of recent-event fact checking evaluates the claim that Argentina has not won the World Cup since 1986 against evidence about a 2022 victory.
  • Examples: A high-precision statistics example evaluates an obesity-rate claim for Ireland against evidence reporting different prevalence figures.
  • Examples: Worked examples report FACTOOL’s generated queries, collected evidence, reasoning process, and final decisions for KB-QA, code, math, and scientific claims.
  • Error cases: The appendix includes separate error-case figures for KB-QA, code, math, and scientific factuality checking.
  • Error cases: The appendix identifies limited variety in synthetic test cases as an error-related issue in code-generation evaluation.
Loading 2307.13528v2…