PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem

Yuhong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, Hui Zhao

arXiv:2403.03558v1cs.CL

TL;DR

Hallucination in ambiguous contexts can lead LLMs to produce unreliable conjectures. This paper evaluates it in question answering with unanswerable math word problems, a dataset, and a method combining text similarity with mathematical-expression detection. Experiments find that hallucination varies with model size, input form, and RLHF use.

  • Problem

    LLMs can produce unreliable conjectures in ambiguous contexts, motivating evaluation of hallucination in question answering.

    Source: §1 · p. 1

  • Method

    The paper introduces UMWP, a dataset of answerable and unanswerable math word problems, and evaluates responses using text similarity and mathematical-expression detection.

    Source: §1 · p. 2

  • Results

    Experiments find that hallucination varies with model size, input form, and RLHF use.

    Source: §1 · p. 2

  • Takeaways & Limitations

    UMWP provides a feasible way to assess hallucination in LLMs.

    Source: §6 · p. 5

    The study evaluates hallucination in English question answering, not other tasks such as summarization or code generation.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Large language models (LLMs) are highly effective in various natural language processing (NLP) tasks. However, they are susceptible to producing unreliable conjectures in ambiguous contexts called hallucination. This paper presents a new method for evaluating LLM hallucination in Question Answering (QA) based on the unanswerable math word problem (MWP). To support this approach, we innovatively develop a dataset called Unanswerable Math Word Problem (UMWP) which comprises 5200 questions across five categories. We developed an evaluation methodology combining text similarity and mathematical expression detection to determine whether LLM considers the question unanswerable. The results of extensive experiments conducted on 31 LLMs, including GPT-3, InstructGPT, LLaMA, and Claude, demonstrate that in-context learning and reinforcement learning with human feedback (RLHF) training significantly enhance the model's ability to avoid hallucination. We show that utilizing MWP is a reliable and effective approach to assess hallucination. Our code and data are available at https://github.com/Yuki-Asuuna/UMWP.

1. Introduction

LLMs can produce unreliable conjectures in ambiguous contexts, which can mislead people. The paper uses unanswerable math word problems to evaluate hallucination and introduces a dataset and evaluation method for this purpose.

1. Introduction

  • Unanswerable math word problems offer a QA setting for assessing hallucination, where answers often take the form of a numerical value or variable expression.The approach evaluates the correctness of that output rather than relying on additional text retrieval.Source: §1 · p. 1
  • The paper introduces UMWP, a dataset of 5,200 questions split evenly between answerable and unanswerable questions across five unanswerability categories.The dataset addresses the scarcity of unanswerable questions in existing MWP datasets.Source: §1 · p. 2, Table 1 · p. 2
  • The proposed evaluation method combines text similarity and mathematical-expression detection to judge whether LLM responses reflect unanswerability.Source: §1 · p. 2
  • Experiments find variation in hallucination by model size, input form, and RLHF use.Source: §1 · p. 2

2. Related Work

Prior work covers answerable math word problems, LLM mathematical ability, and hallucination benchmarks for general questions; the related-work section situates this study across these areas.

2.1. Math Word Problem Benchmark

  • Existing MWP benchmarks vary in construction, difficulty, and scope, including datasets for elementary problems and challenging competition mathematics.GSM8K and MATH are described as the two most difficult MWP datasets.Source: §2.1 · p. 2

2.2. Mathematical Ability of LLM

  • Research on LLM mathematical ability includes evaluations of ChatGPT and GPT-4, chain-of-thought prompting, and fine-tuned models such as MetaMath.MetaMath-7B exceeds same-size state-of-the-art models by 11.5% and 8.7% on GSM8K and 19.4% on MATH; MetaMath-70B reaches 82.3% accuracy on GSM8K.Source: §2.2 · p. 2

2.3. Hallucination Benchmark

  • Hallucination benchmarks address general questions through truthfulness, self-knowledge, hallucination recognition, or biography-based factuality evaluation.Source: §2.3 · p. 3

3. Dataset Construction

UMWP is built from existing answerable MWP datasets by modifying questions to make them unanswerable, then validating the annotations. It contains both answerable and unanswerable items, with five categories of unanswerability.

3. Dataset Construction

  • UMWP combines 2,600 answerable and 2,600 unanswerable questions derived from SVAMP, MultiArith, GSM8K, and ASDiv.Two annotators modify questions, and three volunteers validate them; a question is accepted when all three mark it unanswerable.Source: §3 · p. 3

3.1. Unanswerable Question

  • Unanswerable questions are classified into five categories by reason, and the intended response is uncertainty rather than a precise answer.The categories are missing key information, ambiguous key information, unrealistic conditions, an unrelated object, and a missing question.Source: §3.1 · p. 3

3.2. Answerable Question

  • Answerable questions have definite answers, and GSM8K descriptions are longer by token count than those in the other three datasets.Source: §3.2 · p. 3

4. Evaluation Method

The evaluation method identifies responses that signal unanswerability through similarity to uncertainty templates or through detected variable expressions. It then evaluates detection using F1, treating unanswerable questions as positive cases.

4. Evaluation Method

  • For instruction and in-context-learning inputs, the method compares an output sentence with unanswerable templates using a similarity function; direct inputs may use uncertainty words such as “unknown” or “unsure”.An output is regarded as unanswerable when its maximum template similarity meets the predetermined threshold.Source: §4 · p. 3
  • When a response appears as a variable expression, the method assumes the LLM may have identified potential variables and checks the output for valid variable expressions using regex.The process tokenizes the output with Spacy and removes common vocabulary and spaces before detection.Source: §4 · pp. 3–4
  • The method evaluates hallucination with F1, labeling unanswerable questions positive and answerable questions negative.Source: §4 · p. 4

5. Experiment

Experiments evaluate LLM hallucination on UMWP across model series and input forms. Results compare model size, prompting approaches, RLHF, evaluation methods, and performance against a human benchmark.

5. Experiment

5.1. Setting

  • The evaluation uses SimCSE with a similarity threshold of 0.75, generation temperature 0.7, and a six-token sliding window.The window parses output sentences into semantic chunks to address similarity errors from target-reference length differences.Source: §5.1 · p. 4

5.4. Experiment Results Analysis

  • Across the three LLaMA input forms, F1 score improves continuously as model size increases; in the InstructGPT series, this trend is generally observed, except for text-babbage-001.Source: §5.4 · p. 4
  • Instruction and ICL improve hallucination recognition over Direct input, while their F1 gap shrinks as parameter size increases.Figure 3 presents InstructGPT, Claude, and LLaMA results across Direct, Instruction, and ICL inputs.Source: §5.4 · p. 4, Figure 3 · p. 5
  • RLHF substantially improves F1 across three input forms in matched LLaMA-v2 chat and non-chat models.LLaMA-v2-13b-chat can compete with LLaMA-65b despite having significantly fewer parameters.Source: §5.4 · p. 4
  • Template-based evaluation combined with mathematical-expression detection improves agreement with human judgment, with Cohen’s kappa above 0.75.Set U aggregates “unanswerable” answers from 31 LLMs and is manually filtered to remove incorrect strings.Source: §5.3 · p. 4, §5.4 · p. 5, Table 3 · p. 5
  • GPT-4 achieves an F1 score of 85.24%, below the human benchmark of 93.16%.The human benchmark averages five volunteers’ F1 scores on 200 randomly selected samples.Source: §5.2 · p. 4, §5.4 · p. 5, Figure 4 · p. 5

5.5. Noise Analysis

  • Unrelated generated content is rare among five manually examined LLMs and has limited impact on benchmark results.The models were selected because they had relatively lower capabilities within their respective series.Source: §5.5 · p. 5

6. Conclusion

The conclusion presents UMWP as a way to assess LLM hallucination and reports findings about model size, ICL, and RLHF.

6. Conclusion

  • The authors create UMWP from existing MWP datasets and introduce an evaluation method combining text similarity with mathematical-expression detection.They apply the approach to GPT-3, InstructGPT, Claude, and LLaMA series.Source: §6 · p. 5
  • Experiments highlight model size, In-Context Learning, and RLHF as factors associated with hallucination mitigation.Source: §6 · p. 5
  • The authors conclude that UMWP offers a feasible way to assess hallucination in LLMs.Source: §6 · p. 5

Ethics Statement

The ethics statement describes the dataset’s research-use restrictions, sourcing and annotation practices, and privacy and legal safeguards.

Ethics Statement

  • UMWP is curated for academic and research purposes under CC-BY-SA-4.0; commercial and unlawful, harmful, or unethical use is prohibited.Source: Ethics Statement · p. 6
  • Answerable questions come from GSM8K, MultiArith, ASDiv, and SVAMP, while three annotators manually modify unanswerable questions.Five volunteers complete random samples and are compensated at the local average hourly wage.Source: Ethics Statement · p. 6
  • The authors report obtaining required permissions and following relevant laws, regulations, and data-collection principles.Source: Ethics Statement · p. 6
  • The authors report anonymizing personal identity information and sensitive data to safeguard privacy.Source: Ethics Statement · p. 6

Limitations

The study’s stated scope is English question-answering hallucination benchmarking, and its experiments address mitigation through prompt engineering rather than fundamental causes or solutions.

Limitations

  • The benchmark focuses on English QA and does not evaluate other tasks such as summarization or code generation.The authors state that UMWP could cover additional languages.Source: Limitations · p. 6
  • The experiments explore prompt engineering for mitigation but do not investigate hallucination’s fundamental causes or solutions in UMWP.Source: Limitations · p. 6

A. Appendices

The appendices report noise checks, document how questions and prompts were prepared, and add answerable-question accuracy results, non-answer strings, and categorized hallucination examples.

A.2. Modification Strategy

  • Eight strategies make answerable math word problems unanswerable by hiding, obscuring, or modifying key conditions.These changes make a single numerical answer impossible to determine accurately.Source: §A.2 · p. 8

A.3. Input Form Template

A.5. The Detail of Set U

  • Set U contains 24 possible non-answer strings, including “The answer is unknown” and “It is impossible to know.”Source: §A.5 · p. 8

A.1. Noise Analysis Result

  • Experimental results show that frequency of generating irrelevant content is below 0.15% for recently matured commercial Claude-1 and gpt-3.5-turbo-0301, and within 1.5% for open-source Llama-7b and Llama2-7b.Source: §A.1 · p. 8

A.4. Accuracy of Answerable Question

  • InstructGPT accuracy on 2,600 answerable questions gradually increases with model parameter size.The evaluation uses the instruction input form.Source: Figure 5 · p. 8, §A.4 · p. 8

A.6. Case Study

  • Case-study examples include misidentified hidden information, answerable-question misclassification, deleted key information, arbitrary assumptions, and ignored scope conditions.Source: §A.6 · pp. 9–10, Table 6 · p. 10

Related papers

Loading 2403.03558v1…