PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-tuning Language Models for Factuality

Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, Chelsea Finn

arXiv:2311.08401v1cs.CLcs.AIcs.LG

TL;DR

LLMs can confidently generate incorrect claims, while human factuality labeling is costly. This paper fine-tunes models using automatically generated factuality preference rankings and finds that both reference-based and confidence-based tuning reduce factual errors across biography and medical question-answering tasks.

  • Problem

    LLMs can confidently generate incorrect claims, and maximum-likelihood training does not always encourage factual predictions.

  • Method

    The paper fine-tunes language models for factuality without human annotation using preference-based learning from reference-based and model-confidence factuality scores.

  • Results

    FactTune consistently improves factual accuracy over RLHF and decoding-based baselines, with gains of at least 23% on biographies and 12% on medical question-answering.

  • Takeaways & Limitations

    Automatically constructed factuality preference pairs reliably reduce hallucinated facts, and factuality tuning can further improve RLHF chat models without significantly decreasing the total number of facts.

  • Takeaways & Limitations

    The proposed benchmarks cover representative but not the full range of scenarios where factuality improvements are desired.

Abstract

from arXiv · show

The fluency and creativity of large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as 'hallucinations.' These errors can inadvertently spread misinformation or harmfully perpetuate misconceptions. Further, manual fact-checking of model responses is a time-consuming process, making human factuality labels expensive to acquire. In this work, we fine-tune language models to be more factual, without human labeling and targeting more open-ended generation settings than past work. We leverage two key recent innovations in NLP to do so. First, several recent works have proposed methods for judging the factuality of open-ended text by measuring consistency with an external knowledge base or simply a large model's confidence scores. Second, the direct preference optimization algorithm enables straightforward fine-tuning of language models on objectives other than supervised imitation, using a preference ranking over possible model responses. We show that learning from automatically generated factuality preference rankings, generated either through existing retrieval systems or our novel retrieval-free approach, significantly improves the factuality (percent of generated claims that are correct) of Llama-2 on held-out topics compared with RLHF or decoding strategies targeted at factuality. At 7B scale, compared to Llama-2-chat, we observe 58% and 40% reduction in factual error rate when generating biographies and answering medical questions, respectively.

1 Introduction

LLMs can generate confident factual errors, while factuality is difficult to optimize and expensive to label manually. The paper addresses this by constructing factuality preferences automatically and fine-tuning models for long-form generation.

  • LLMs can confidently generate incorrect claims despite extensive pretraining and instruction fine-tuning.
  • Maximum-likelihood pretraining can reward probability spread across many responses rather than factual predictions.
  • Human factuality annotation is costly: professional fact-checkers took approximately 9 minutes per biography, and 505 biographies cost about $2,000.
  • The approach estimates truthfulness using either external-knowledge fact checking or model confidence, then ranks sampled completions automatically.
  • Fine-tuning on automatically generated factuality preferences improves factuality over RLHF and complements decoding strategies on biographies and medical questions.

2 Preliminaries

The paper frames factuality tuning as reinforcement learning from preferences and uses direct preference optimization to learn from ranked candidate responses. DPO avoids explicit reward modeling and policy sampling during training, leaving preference-pair construction as the central challenge.

  • Reinforcement learning fine-tunes a language-model policy by maximizing rewards assigned to responses for input queries.
  • Unconstrained reward maximization can overoptimize idiosyncrasies of the reward function, so practical objectives constrain deviation from a reference model.
  • PPO is commonly used for this constrained objective, but its implementation and tuning are complex.
  • DPO learns from preference pairs in which a preferred response is ranked above a less-preferred response under a Bradley-Terry model.
  • DPO trains on fixed preference data without fitting an explicit reward function or sampling from the policy during training.

3 Constructing Preferences Encouraging Factuality in Long-Form Text

The paper constructs factuality preference data without human labels by scoring sampled responses with reference-based or reference-free truthfulness estimators, then applying DPO. Reference-based scoring checks claims against retrieved knowledge, while reference-free scoring uses calibrated model confidence after claim-focused question reformulation.

  • 3 Constructing Preferences Encouraging Factuality in Long-Form Text: Preference learning requires response rankings, so the paper generates factuality preferences automatically rather than using human annotations.The proposed framework uses external-reference consistency or calibrated model probabilities to score candidate responses.
  • 3.1 Reference-Based Truthfulness Estimation: FactScore extracts atomic claims, checks each against Wikipedia with a fine-tuned fact-checking model, and scores text by the fraction estimated as supported.The procedure uses GPT-3.5 for claim extraction and natural language inference for support judgments.
  • 3.1 Reference-Based Truthfulness Estimation: Reference-based scoring depends on relevant, high-quality reference texts and a reliable support-checking model, limiting use where retrieval or ground-truth documents are difficult.The paper motivates a reference-free alternative to avoid retrieving external knowledge and checking consistency.
  • 3.2 Reference-Free Confidence-Based Truthfulness Estimation: Reference-free scoring extracts claims, converts them into minimally ambiguous questions, and averages model confidence over the resulting factual claims.This separates factual uncertainty from stylistic or multi-fact effects in the original passage.
  • 3.2 Reference-Free Confidence-Based Truthfulness Estimation: Ambiguous question reformulation can lower measured confidence for reasons unrelated to factual uncertainty, so the method specifies the answer type explicitly.For example, “What instrument does Yo-Yo Ma play?” is preferred to the broader “What does Yo-Yo Ma play?”
  • 3.2 Reference-Free Confidence-Based Truthfulness Estimation: For each claim-question pair, the method samples 20 answers, groups equivalent answers, and uses the largest-bin fraction as the fact’s truthfulness score.Equivalence is assessed through heuristic string matching or GPT-3.5 semantic judgments.
  • 3.3 Factuality Tuning: Putting it all Together: Factuality tuning samples multiple temperature-1.0 responses per unlabeled prompt, scores them with either estimator, and prefers the higher-scoring response in each pair.Equal-scoring pairs are excluded from the resulting preference set.
  • 3.3 Factuality Tuning: Putting it all Together: The resulting preference pairs train the model with DPO, while all sampled responses serve as targets for the supervised fine-tuning stage.This combines automatically scored preferences with the DPO training pipeline.

4 Experiments

Experiments evaluate automated factuality preference tuning across biographies and medical question answering, including reference-based and reference-free scoring, chat models, decoding combinations, and scoring-design choices. FactTune improves factuality across domains and complements existing interventions, while qualitative and metric-validation analyses identify trade-offs and robustness checks.

  • Fine-Tuning for Factuality Across Domains: FactTune-FS improves factual accuracy over RLHF and decoding-based baselines by at least 23% on biographies and 12% on medical question answering.It reduces factual errors while maintaining no more than a slight decrease, or sometimes an increase, in correct information generated.
  • Fine-Tuning for Factuality Across Domains: FactTune-MC reduces error rates and improves RLHF-model factuality on both datasets without external reference information.The model uses confidence-based preference scores as a reference-free alternative.
  • Fine-tuning Chat Models for Factuality: Factuality tuning further improves Llama-2-7b-Chat factuality without significantly decreasing the total number of facts.The method combines factuality and semantic-entropy-based rewards with an RLHF dialogue model.
  • Complementary Benefits of Factuality Tuning and Decoding-Time Factuality Interventions: DOLA further increases factuality for most factuality-tuned model–dataset combinations, except Llama-2 on biographies.The result suggests decoding and fine-tuning may operate through complementary mechanisms.
  • Impact of Design Decisions of Open-Ended Model Confidence Scoring: Atomic question extraction generally outperforms named entity extraction, while maximum confidence improves biographies under the atomic-question setting.GPT-3.5 equivalence checking produces worse preference pairs than the simple string-matching heuristic, likely because it introduces noisier semantic-entropy estimates.
  • Validating Metrics for Factuality: Human ratings and GPT-4 evaluations both support improved factuality, with GPT-4 ratings highly correlated with FactScore and lower average errors for FactTune-FS than SFT.These evaluations address whether factuality improvements result from exploiting the evaluation protocol.

5 Related Work

Prior work detects or prevents factual errors using perturbation sensitivity, response diversity, external knowledge, prompting, or internal representations. This paper differs by directly optimizing factual statement generation, including reference-free truthfulness estimation for long-form text.

  • Factual-error detection has used prompt perturbations, response diversity, external knowledge, and internal activation properties.
  • Some methods prevent errors with prompting or internal-representation perturbations rather than post-generation correction.
  • This work directly optimizes generation of factual statements instead of relying on a fixed internal factuality heuristic.
  • The paper studies reference-free confidence-based truthfulness estimation for long-form, unstructured text, beyond prior short-form settings.

6 Conclusion

The paper fine-tunes language models for factual long-form generation using two truthfulness estimators and preference-based learning without human annotation. Experiments report fewer incorrect facts, while future work remains on broader tasks, reward combinations, and larger models.

  • The paper develops reference-based and reference-free truthfulness estimators and optimizes them through preference-based learning.
  • Fine-tuning with either truthfulness criterion reliably reduces incorrect facts in long-form generations.
  • Reference-free truthfulness estimation provides scalable self-supervision without a corpus of gold texts.
  • The benchmarks cover representative but not all factuality scenarios, and the experiments use only 7B models.
  • Future work includes combining factuality rankings with RLHF rewards, integrating other factuality methods, and scaling to larger models and preference datasets.

A.1 Prompts

The appendix lists GPT-3.5 prompts for converting atomic statements into questions used in model-confidence truthfulness estimation.

  • GPT-3.5 is used to convert statements into questions.
  • The prompts support model-confidence-based truthfulness estimation.
  • The appendix documents these prompts in Table 7.

A.2 Sample Model Generations

The appendix provides sample generations for several models and biography prompts. It notes that factuality tuning produces somewhat terser responses.

  • After factuality tuning, model responses become somewhat terser.
  • Table 7 contains GPT-3.5 prompts for converting atomic facts into questions.
  • Table 8 compares SFT, FactTune-FS, and FactTune-MC Llama-1-7b samples for a Greta Gerwig biography prompt.
  • Table 9 compares SFT, FactTune-FS, and FactTune-MC Llama-1-7b samples for a Mukesh Ambani biography prompt.
Loading 2311.08401v1…