PaperlayerView exact revision on arXiv

Source-linked AI summary

TruthfulQA: Measuring How Models Mimic Human Falsehoods

Stephanie Lin, Jacob Hilton, Owain Evans

arXiv:2109.07958v2cs.CLcs.AIcs.CYcs.LG

TL;DR

Language models often generate false statements, raising questions about why fluent systems reproduce human misconceptions and whether scaling improves truthfulness. The paper introduces TruthfulQA to measure imitative falsehoods and finds that large models are much less truthful than humans in zero-shot evaluation.

  • Problem

    Language models generate false statements, motivating research into why they reproduce imitative falsehoods and whether scaling alone addresses the problem.

  • Method

    TruthfulQA uses questions designed to elicit imitative falsehoods and evaluates generated answers for truthfulness and informativeness using human judgments.

  • Results

    Large models were much less truthful than humans on TruthfulQA in the zero-shot setting.

  • Takeaways & Limitations

    TruthfulQA provides a benchmark for measuring imitative falsehoods, which the paper concludes are unlikely to be solved by scaling up models alone.

  • Takeaways & Limitations

    TruthfulQA covers general-knowledge questions rather than long-form generation or interactive settings, so performance may not transfer to other tasks or deployed systems.

Abstract

from arXiv · show

We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. The best model was truthful on 58% of questions, while human performance was 94%. Models generated many false answers that mimic popular misconceptions and have the potential to deceive humans. The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size. However, this result is expected if false answers are learned from the training distribution. We suggest that scaling up models alone is less promising for improving truthfulness than fine-tuning using training objectives other than imitation of text from the web.

1 Introduction

TruthfulQA measures whether language models avoid false statements, especially imitative falsehoods that reflect human misconceptions. The introduction argues that truthfulness requires dedicated measurement because scaling models alone may worsen this problem.

  • Motivation: Fluent language models can generate false statements, creating risks of accidental misuse, blocked medical or legal applications, and malicious deception.These risks include user distrust, reluctance to deploy models where factual accuracy is essential, and disinformation or fraud.
  • Motivation: Measuring truthfulness across contexts can clarify how often models make false statements and support the development of more truthful systems.The introduction presents better measurement as a way to understand the risks posed by deceptive models.
  • Imitative falsehoods: GPT-3 answers “14812” to “What is 1241 × 123?”, illustrating failure to reliably generalize multiplication knowledge from its training data.The passage distinguishes this failure from false answers incentivized by the training objective.
  • Imitative falsehoods: Imitative falsehoods are false answers with high likelihood on a model’s training distribution, and TruthfulQA is designed to elicit them.These questions target misconceptions that existing question-answering benchmarks are less likely to cover.
  • Results: Larger models were generally less truthful across model families, an inverse-scaling trend that contrasts with most NLP tasks.One explanation is that larger models learn the training distribution better and therefore produce more imitative falsehoods.
  • Benchmark: TruthfulQA tests zero-shot generation on 817 questions spanning 38 categories.The benchmark and code are publicly available.
  • Results: 58% of questions were answered truthfully by the best tested model, GPT-3-175B with a “helpful” prompt, versus 94% for humans.The same model produced answers that were both false and informative 42% of the time, compared with 6% for humans.
  • Results: 90-96% accuracy was achieved on held-out models by an automated metric predicting whether answers were true or false from human evaluations.Together with the multiple-choice task, this metric provides a quick and reproducible assessment method.

2 The TruthfulQA Benchmark

TruthfulQA is a zero-shot benchmark that evaluates whether models avoid false factual claims under a strict literal-truth standard. Its adversarial questions target imitative falsehoods, while evaluation distinguishes truthfulness from informativeness and includes validation of the references.

  • Truth standard: TruthfulQA counts a claim as true only when it describes literal real-world truth, typically supported by reliable public evidence.Belief-system or traditional claims are counted as false.
  • Evaluation: An answer is truthful iff it avoids asserting a false statement, so refusals, uncertainty, and true but irrelevant answers count as truthful.The benchmark separately evaluates informativeness because perfectly truthful answers could refuse every question.
  • Benchmark design: 817 questions span 38 categories and are intended for zero-shot evaluation, with true and false reference answers plus supporting sources.Most questions are one sentence long, with a median length of 9 words.
  • Adversarial construction: 437 filtered questions were retained after GPT-3-175B repeatedly answered them falsely, and 380 additional unfiltered questions were written to elicit expected false answers.The adversarial procedure targets false statements with high likelihood on the training distribution.
  • Limitations and validation: The adversarial procedure may also expose non-imitative weaknesses, and external validation found disagreement on 6-7% of reference-answer judgments.The authors suspected some validation mistakes and judged that this range would not affect the main results.

3 Experiments

TruthfulQA experiments compare multiple model families and sizes under a true zero-shot setup, using generation as the main task and a multiple-choice variant. Generation outputs are scored by human evaluation, with GPT-judge used as an automated truthfulness and informativeness evaluator.

  • Models and prompts: Different model sizes are evaluated for each family, while GPT-3-175B is additionally tested with different prompts.The experiments also include a default QA prompt for all families and sizes except UnifiedQA, which uses no prompt because it is already fine-tuned for question answering.
  • Prompts: TruthfulQA is evaluated in the true zero-shot setting, with no gradient updates, no TruthfulQA examples in prompts, and no tuning on TruthfulQA examples.Prompts may contain natural-language instructions, and the authors recommend their prompts and hyperparameters for comparison with the baselines.
  • Tasks: The main generation task produces full-sentence answers using greedy decoding, with model and sampling parameters otherwise left at API defaults.Additional experiments at higher temperatures are reported in Appendix B.8.
  • Tasks: The multiple-choice task reuses the generation questions and scores the normalized likelihood of true reference answers against true and false reference answers.Each reference answer is evaluated independently conditional on the default prompt and question.
  • Evaluation: Human evaluation scores generation responses by the percentage judged true or informative, while GPT-judge is fine-tuned to classify truthfulness and informativeness.The evaluation procedure was designed to make judgments replicable and consistent across evaluators.

4 Results

TruthfulQA reveals poor model truthfulness: the best model achieved 58% true answers versus 94% for humans, and larger models were generally less truthful despite being more informative. Results across controls, paraphrases, and model families suggest that imitative falsehoods substantially contribute to this inverse-scaling pattern.

  • Generation results: 58% of answers were true for the best model, GPT-3-175B with a helpful prompt, versus 94% for the human participant.The best model produced 21% true and informative answers, while humans produced 87%.
  • Generation results: 42% of the best model’s answers were false and informative, compared with 6% for the human participant.Different prompts significantly affected GPT-3-175B truthfulness but not its percentage of true and informative answers.
  • Category results: The best model was less truthful than humans on almost all question categories, and performance remained poor for categories with non-trivial deception risk.Law and health were identified as potentially more deceptive categories than proverbs or myths and fairytales.
  • Falsehood analysis: Truthfulness improved with model size on matched controls, while paraphrases preserved the inverse-scaling trend, suggesting that poor performance is not mainly caused by question syntax or form.The largest GPT-3 and GPT-Neo/J models still performed worse than smaller family members on paraphrased questions.
  • Evaluation: GPT-judge predicted human truthfulness evaluations with 90-96% validation accuracy and achieved 90% on UnifiedQA despite training only on GPT-family answers.This provides a cheap alternative to human evaluation and generalizes across answer formats.

5 Discussion

TruthfulQA targets false claims that standard language-model training does not discourage, because models learn to predict human text rather than directly learn truthfulness. The benchmark is intended to evaluate truthfulness in foundation models used for downstream contexts where it is necessary.

  • 5 Discussion: TruthfulQA questions are designed so standard language-model objectives do not incentivize correct answers.Baseline models predict human text and do not directly learn to be truthful.
  • 5 Discussion: Models are likely to repeat false claims that humans commonly state, which TruthfulQA is designed to test.The benchmark targets claims that are often repeated in human text.
  • 5 Discussion: Truthfulness is necessary in many contexts, including downstream tasks using foundation models such as GPT-3.TruthfulQA provides a way to test models expected to be truthful even when the foundation model is misaligned.

6 Related Work

Related work situates truthfulness among factual-question benchmarks and neighboring generation concepts, while linking imitative falsehoods to broader misalignment between training objectives and human values.

  • Factuality benchmarks: Factual-question benchmarks typically test correctness, whereas truthfulness also permits non-committal responses.Most benchmarks use multiple choice; some require short, single-phrase generated answers.
  • Related concepts: Truthfulness is related to factuality, veracity, and hallucination avoidance, with prior work distinguishing truthfulness from honesty.Truthfulness is relevant to applications including news generation and summarization.
  • Alignment and imitative falsehoods: Imitative falsehoods resemble models learning offensive or prejudiced language because high-probability training examples can conflict with human goals.The cited examples frame avoiding falsehoods and offensive language as alignment goals beyond web-text imitation.
  • Alignment and imitative falsehoods: GPT-3 models trained on GitHub provide another example of training-objective misalignment by learning to produce buggy code.Increasing the safety and alignment of pretrained models remains challenging.

7 Conclusion

TruthfulQA addresses the challenge of measuring and improving truthfulness by targeting imitative falsehoods that scaling alone is unlikely to solve. Large models are less truthful than humans zero-shot, while benchmark performance is informative about robustness but not sufficient for specialized-domain truthfulness.

  • Conclusion: TruthfulQA measures imitative falsehoods, failures of truthfulness that are unlikely to be solved by scaling up models alone.The benchmark is intended to help develop tools for measuring truthfulness.
  • Conclusion: Large models are much less truthful than humans in the zero-shot setting.
  • Conclusion: Strong TruthfulQA performance does not guarantee truthfulness in specialized domains, whereas poor performance indicates a lack of robustness.The benchmark may still be useful for general-purpose and specialized models because its questions require no specialized knowledge and are source-supported.

8 Ethics and Impact … B.2 Truthfulness and informativeness across all models

The paper documents TruthfulQA’s scope and limitations, illustrates models’ imitative falsehoods, and evaluates automated truthfulness and informativeness metrics across model families. GPT-judge is designed for benchmark-specific evaluation, while the reported metrics distinguish truth, informativeness, and their combination.

  • 8 Ethics and Impact: TruthfulQA does not establish truthfulness on long-form generation, interactive adversarial settings, or deployed real-world questions.Its questions resemble real-world questions but were not collected from a deployed system.
  • 8 Ethics and Impact: A deceptive model would need to produce false answers infrequently, whereas a low TruthfulQA score requires answering almost all questions falsely.The benchmark is therefore described as unlikely to help construct useful deceptive models for malicious purposes.
  • 8 Ethics and Impact: TruthfulQA has shallow coverage of general-knowledge topics rather than extreme topic specificity.This limitation is stated in the discussion of the benchmark’s usefulness for malicious deception.
  • A Additional examples from TruthfulQA: GPT-J answers illustrate falsehoods that mimic human misconceptions, while GPT-3 examples show truthfulness often declining with model size.The figures use human evaluation labels displayed as binary true/false values, although the underlying truth scores are scalar.
  • B.1 Automated metrics vs human evaluation: GPT-judge is a GPT-3-6.7B classifier fine-tuned on question-answer-label triples to classify TruthfulQA answers as true or false.The final model is trained on examples from all models and is intended to evaluate truth only on TruthfulQA questions.
  • B.1 Automated metrics vs human evaluation: GPT-judge is evaluated on unseen model families, alongside ROUGE1, BLEURT, and GPT-3-Sim reference-answer similarity metrics.For each held-out family, GPT-judge is trained on the other model families; GPT-info separately evaluates informativeness, while All-true labels every answer informative.
  • B.1 Automated metrics vs human evaluation: GPT-judge struggles with longer, multi-sentence answers, especially qualified, mixed-truth, overly detailed, or indirect responses.It also has a strong bias toward labeling longer answers as informative.
  • B.2 Truthfulness and informativeness across all models: The all-model results report truth score, Truth*Info score, % True, % Info, % True+Info, % True (GPT-judge), and Truth score unf.These metrics respectively summarize scalar truth, truth–informativeness products, thresholded truth or informativeness, their conjunction, automated truth, and truth on unfiltered questions.

B.3 Results on newer language models … B.9 Paraphrased questions

Newer models improve TruthfulQA generation performance and restore positive scaling at large sizes, but remain well below human truthfulness. Additional analyses show robustness across question sets, prompts, temperatures, model distributions, and paraphrases.

  • B.3 Results on newer language models: Newer models improve TruthfulQA generation performance and restore positive scaling for the largest model sizes, although WebGPT remains far below humans when truthfulness and informativeness both matter.Information retrieval, prompt engineering, and finetuning appear more efficient than scaling alone.
  • B.3 Results on newer language models: 48% is the extrapolated score for an InstructGPT model with 10^20 parameters, compared with a 95% human baseline, because positive scaling remains very slow.Anthropic3, Gopher, and InstructGPT show positive scaling at large sizes, while Anthropic’s model without context distillation retains inverse scaling.
  • B.4 Adversarially filtered vs unfiltered sets of questions: Truthfulness is evaluated separately on adversarially filtered and unfiltered TruthfulQA questions, rather than only on their combined set.The benchmark contains 437 adversarially filtered questions and 380 unfiltered questions.
  • B.5 Performance broken down by category of question: GPT-3-175B performance is broken down by question category and compared with the average human score, including practical and non-practical categories.Non-practical categories include Fiction, Proverbs, Myths and Fairytales, and Subjective questions, which are unlikely to deceive humans.
  • B.6 Performance of GPT-3-175B under different prompts: GPT-3-175B is evaluated under multiple prompts, including QA, help, harm, null, chat, and long-form, with prompts producing different answer behaviors.The helpful prompt instructs GPT-3 to be truthful, while the harmful prompt primes conspiracy-theorist-style answers.
  • B.7 Distribution of truthful models: Over 80% of benchmark questions receive false and informative answers from at least half of the models.The distribution counts truthful and false/informative models across 19 total models per question.
  • B.8 Higher sampling temperatures: Higher sampling temperatures do not reverse the trend of worse truthfulness at larger model sizes.The analysis uses GPT-judge across model sizes and prompts, comparing sampling and “Best of 20” generation.
  • B.9 Paraphrased questions: Paraphrased questions closely preserve the original results in both model ranking and absolute truthfulness scores, while GPT-3 produces similar imitative falsehoods on both versions.Paraphrases were generated with PEGASUS and filtered or edited to preserve meaning.

C Dataset construction · C.1 Reference answers · C.2 Control questions

TruthfulQA constructs reference answers from sourced true answers and expanded sets of imitative false answers, while control questions test whether errors reflect non-imitative weaknesses. The construction acknowledges incomplete coverage, especially for smaller models.

  • C Dataset construction: TruthfulQA’s dataset construction combines sourced reference answers, misconception-focused false-answer searches, and edited control questions.The control questions preserve question form while converting items into straightforward trivia or common-sense questions.
  • C.1 Reference answers: Reference answers begin with true answers taken directly from Wikipedia or the listed source.The process then seeks coverage of common variations on each answer.
  • C.1 Reference answers: For “Where is Walt Disney’s body?”, true references include cremation and interment at Forest Lawn Memorial Park, Glendale, California, and the U.S.These examples illustrate coverage across multiple correct formulations of the same answer.
  • C.1 Reference answers: False-answer generation uses a similar process but searches for common misconceptions, superstitions, and conspiracies to capture additional imitative answers.Such searches identified claims that Disney’s body was frozen, in suspended animation, buried under Disneyland, or buried under a Pirates of the Caribbean attraction.
  • C.1 Reference answers: The reference sets cannot cover every true or false statement models generate, particularly irrelevant answers from smaller models.The authors expect this limitation to diminish as models grow, because larger models tend to answer more informatively and directly.
  • C.2 Control questions: Control questions edit 1–3 words of TruthfulQA questions while preserving their form and converting them into straightforward trivia or common-sense questions.They test whether the original questions exploit non-imitative weaknesses.
  • C.2 Control questions: GPT-3 performs well on control questions without making the error types it exhibits on the original questions.This pattern argues that the original-question errors are not explained by comparable non-imitative weaknesses.

D Human evaluations · E Prompts

The paper uses a blinded, label-based human-evaluation procedure to make truthfulness and informativeness judgments objective and reproducible. It also presents QA, harmful, helpful, chat, and long-form prompts, including instructions to research carefully and avoid common myths.

  • D Human evaluations: Evaluators were blinded to both the model name and the prompt that generated each answer.
  • D Human evaluations: Evaluators assigned one of 13 qualitative truthfulness labels, each mapped to a fixed truth score before evaluation.For example, “mostly true” maps to 0.9, improving interpretability and consistency.
  • D Human evaluations: Truth scores were thresholded at 0.5, with scores ≥0.5 classified as truthful for the final binary evaluation.This sacrifices granularity but reduces the impact of disagreements among nearby truthfulness categories.
  • D Human evaluations: Around 80% of evaluated answers closely matched sourced reference answers, while around 19% were tautologies, contradictions, or nonsense.For remaining answers, evaluators searched sources including Our World in Data and Wikipedia.
  • D Human evaluations: Informativeness was evaluated using a similar label-and-example procedure, with the assigned labels and scores provided in Table 9.
  • E Prompts: QA prompts asked factual questions about topics including United States life expectancy and the 1955 presidency.
  • E Prompts: Harmful prompts elicited conspiracy-oriented claims about humanity’s enemies and “The Storm,” including allegations involving political figures and elites.
  • E Prompts: The helpful prompt instructed Professor Smith to interpret questions literally, research answers carefully, avoid common myths, and say “I have no comment” when uncertain.The paper also includes chat and long-form prompt formats.

F Checking for data quality and disagreement

TruthfulQA evaluations may contain labeling errors or ambiguities, with people disagreeing on roughly 2–6% of questions despite revisions to reduce ambiguity. This uncertainty could explain very small model-score differences, but not the much larger differences among baseline models.

  • Validator agreement: 7% of questions received a different label from an independent validator, with the authors suspecting 3–4% reflected genuine disagreement or ambiguity.The validator independently judged randomly sampled reference-answer pairs without feedback.
  • Human baseline check: 6% of a participant’s answers were marked false, of which the authors suspect 2% represented disagreement with the evaluation and the remainder were participant mistakes.The participant answered 250 randomly sampled questions without seeing reference answers, and spent less than two minutes per question.
  • Ambiguity reduction: 43 questions, or 5.3% of the total, were modified to make them less ambiguous.The revisions followed the validator and participant data on evaluation disagreement.
  • Estimated disagreement: 2–6% of questions is the authors’ rough estimate for disagreement among people who read the evaluation instructions.Because the benchmark includes informal and somewhat ambiguous everyday questions, disagreement below 0.5% may not be feasible.
  • Implications for score comparisons: 2–6% evaluation disagreement could explain very small differences between model scores, but current baseline-model differences are too large for this concern to apply.The benchmark’s reference answers may include a small number of labeling errors, and people may disagree because they interpret questions differently.
Loading 2109.07958v2…