Source-linked AI summary
Language Models (Mostly) Know What They Know
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, Jared Kaplan
TL;DR
The paper asks whether language models can assess the correctness of their own claims and predict which questions they can answer. It uses calibration, P(True) self-evaluation, and P(IK) prediction across tasks and contexts, finding strong in-distribution performance, partial cross-task generalization, and improved predictions with relevant sources or hints.
Problem
The paper asks whether language models can recognize what they know and accurately evaluate the confidence and correctness of their own outputs.
Method
The authors study calibration, ask models to score the correctness of sampled answers with P(True), and train value heads to predict P(IK) without a proposed answer.
Results
Models are well-calibrated and perform well at self-evaluation, while P(IK) partially generalizes across tasks and increases appropriately with relevant source materials and mathematical hints.
Takeaways & Limitations
These findings provide a basis for studying and potentially bootstrapping more extensive forms of self- or intermodel supervision related to honesty.
Takeaways & Limitations
The analysis focuses on pretrained language models and does not distinguish truth from what humans say; methods may work less well for RLHF policies or knowledge beyond human pretraining.
Abstract
from arXiv · showhide
We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.
1 Introduction
The paper studies whether language models can recognize what they know, evaluate the correctness of their own outputs, and generalize this self-knowledge across tasks and contexts. It finds strong calibration and self-evaluation in suitable settings, while P(IK) generalization is only partial and calibration suffers out of distribution.
- Calibration: Large language models are well-calibrated on diverse multiple-choice questions when formatted appropriately, with calibration improving through model size and few-shot prompting.The paper also reports good calibration for True/False distinctions and significant degradation when an option is replaced by “none of the above.”
- Self-evaluation: Models can evaluate whether their own sampled answers are correct using P(True), and self-evaluation improves as model size and capabilities increase.Self-evaluation remains more challenging because models tend to find their own samples plausible.
- Self-evaluation: Providing many of a model’s own samples before evaluating one possibility significantly improves self-evaluation performance.The authors describe this as brainstorming other possibilities before judging a specific answer.
- Self-knowledge: The paper trains models to predict P(IK), the probability that they will answer a question correctly, without relying on a proposed answer.P(IK) is usually computed with a binary classification head on top of a language model.
- Self-knowledge: P(IK) generalizes partially from TriviaQA to math, code, and story-completion questions, but calibration is poor on these out-of-distribution tasks.Comparisons between models with different pretraining distributions suggest better generalization when training reflects what the evaluated model knows.
- Self-knowledge: P(IK) increases appropriately when relevant source materials or mathematical hints are added, including responses to correct, incorrect, and distracting hints.The models were trained on standalone questions but generalized their predictions to these in-context information changes.
- Evaluation tasks: The experiments evaluate generative tasks including TriviaQA, Lambada, GSM8k, Codex HumanEval, arithmetic, and natural function synthesis.The paper uses these tasks to study self-evaluation and prediction of questions models can answer correctly.
2 Larger Models are Calibrated on Diverse Multiple Choice Questions
Large language models produce well-calibrated probabilities on diverse multiple-choice questions when answer choices are explicitly shown and appropriately formatted. Calibration generally improves with model size and few-shot prompting, while formatting choices substantially affect performance.
- Large models produce well-calibrated probability distributions across diverse multiple-choice tasks when answer choices are explicitly presented by label.This includes BIG Bench, TruthfulQA, QuALITY, and LogiQA.
- Figure 5 compares expected calibration error with normalized accuracy across BIG Bench tasks and finds no noticeable within-task correlation between them.Marker size represents the number of problems in each task.
- Visible answer choices are crucial because they reduce ambiguity among possible paraphrases and specializations of an answer.
- Calibration improves with model size and when evaluation changes from zero-shot to few-shot prompting.
- Calibration is evaluated with expected calibration error after binning predictions into 10 equally represented bins.The metric compares probabilistic predictions with observed frequencies.
3 From Calibration to Knowing What You Know
The paper extends calibrated multiple-choice prediction toward self-evaluation by asking models to judge proposed answers as True or False. This approach avoids difficulties caused by “none of the above,” while large models can remain well-calibrated in the reformulated setting.
- 3.1 Replacing an Option with ‘None of the Above’ Harms Performance and Calibration: Replacing an answer choice with “none of the above” significantly degrades both accuracy and calibration.Models appear biased against selecting this option, and additional few-shot examples do not reliably fix the problem.
- 3.2 Models are Well-Calibrated on True/False Tasks: The True/False format asks models to judge whether a proposed answer is correct, with P(True) representing the probability that the answer is valid.
- 3.2 Models are Well-Calibrated on True/False Tasks: On BIG Bench reformulated as True/False questions, the 52B model is quite well-calibrated except near the probability tails, where it is overconfident.
- 3.2 Models are Well-Calibrated on True/False Tasks: The reformulated evaluation pairs each correct answer with a randomly chosen incorrect answer and asks the model to determine whether each proposed answer is correct.
- RLHF policies appear poorly calibrated, but temperature adjustment with T = 2.5 largely fixes calibration issues across three evaluations.
4 Ask the AI: Is your proposed answer True or False?
The paper asks models to judge whether their own sampled answers are correct by assigning P(True), finding that this self-evaluation improves with model size and with additional comparison samples. Few-shot prompting yields reasonably calibrated confidence, though judging self-generated answers is harder than judging externally supplied possibilities.
- Basic Self-Evaluation: Self-evaluation performance improves with model size, and 52B-model answers labeled P(True) > 50% are substantially more likely to be correct than generic responses.The separation between overall and P(True)-conditioned accuracy grows with model size.
- Basic Self-Evaluation: The procedure samples an answer and then asks the model to evaluate the probability that its proposed answer is correct.It is applied to TriviaQA, Lambada, Codex HumanEval, GSM8k, arithmetic, and other generation tasks.
- Basic Self-Evaluation: Self-evaluation is challenging because models must assess their own samples, which may be near their validity decision boundary or seem overly plausible.Zero-shot P(True) is poorly calibrated and often near 50% for typical samples.
- Showing Many T = 1 Samples Improves Self-Evaluation: Showing five T = 1 samples for the same question significantly improves evaluation of any one sample, especially on short-form tasks.The method provides other samples as comparison possibilities before judging the selected answer.
- Showing Many T = 1 Samples Improves Self-Evaluation: Few-shot self-evaluations are reasonably calibrated, while larger k primarily improves calibration rather than AUROC for separating correct and incorrect responses.The paper reports less benefit from comparison samples on long-form Codex and GSM8k tasks.
- Conclusion: Overall, models given a few examples from a distribution can generate samples and productively differentiate correct from incorrect answers with reasonably calibrated confidence.The conclusion covers the self-evaluation setting despite the greater difficulty of judging the model’s own samples.
5 Training Models to Predict Whether They Can Answer Questions Correctly
The paper trains models to predict P(IK), the probability that they know an answer without evaluating a proposed response. P(IK) generalizes partially across tasks and responds appropriately to context and hints, but calibration degrades on new distributions.
- P(IK) Prediction: P(IK) denotes the probability that a model can correctly answer a free-form question, making the prediction a claim about the model itself.The paper studies whether this signal reflects model-specific knowledge rather than only general question difficulty.
- Training Approaches: The paper considers a value head and a natural-language approach for predicting P(IK), then uses the value-head approach after early natural-language experiments showed no major out-of-distribution gains.The value head is added independently of the language-modeling logits.
- In-Distribution Performance: Training on TriviaQA lets P(IK) separate correct and incorrect questions well and calibrates it very well on held-out TriviaQA.Training uses sampled answers to construct soft ground-truth P(IK) labels.
- Out of Distribution Generalization: TriviaQA-trained P(IK) generalizes to Lambada, arithmetic, GSM8k, and code, with AUROC and calibration generally improving as model size increases.Calibration on Lambada is poor because the model produces uniformly low P(IK) scores.
- Out of Distribution Generalization: Training on task-specific P(IK) distributions improves performance, indicating a significant generalization gap.The comparison concerns in-distribution versus cross-task performance.
- Context and Hints: P(IK) rises from 18% to 78% for a 52B model when a relevant Wikipedia article is added to the context.The model increases P(IK) without further training when source material contains the answer.
- Context and Hints: For GSM8k, more hint information generally raises P(IK), and correct hints produce higher scores than incorrect hints.A model trained across four tasks performs better and more consistently, especially with partial hints.
- Model-Specific Knowledge: Comparisons between models with distinct pretraining distributions provide mixed but encouraging evidence that P(IK) reflects model-specific knowledge.Each model tends to assign higher P(IK) to questions it answers correctly than to questions answered correctly by the other model.
6 Discussion
The discussion frames calibration, self-knowledge, and truthfulness as parts of a broader honesty agenda. Results are encouraging but limited by pretrained-model scope, human-imitation training, possible deception, and narrow evaluation coverage.
- The paper treats honesty as an umbrella covering truthfulness, calibration, self-knowledge, explainability, and non-deceptiveness.
- Large language models are well-calibrated on diverse multiple-choice questions, perform self-evaluation across subjects, and predict what they do and do not know with some generalization.
- Calibration and self-evaluation improve with model size or capability and additional few-shot examples.
- The authors interpret improving self-evaluation with scale as evidence that verification can have an advantage over generation.
- Honesty generalizes across domains to some degree, but calibration suffers out of distribution.
- Limitations: The study focuses on pretrained language models and does not establish whether its methods work as well for purpose-finetuned models such as RLHF policies.
- Limitations: Because pretraining imitates human writing, the analysis does not distinguish truth from what humans say.
- Limitations: The study does not address intentional deception, and its generalization findings are limited in power and scope because only five sampling-based datasets were studied.
7 Contribution Statement
The contribution statement credits distributed training, sampling, evaluation infrastructure, research design, and writing across Anthropic contributors, while the supplied figure passage identifies RMS calibration trends on BIG Bench.
- Model Training: Model pretraining was led by Sam McCandlish, Nicholas Joseph, Tom Brown, and Jared Kaplan, with broad technical-staff contributions.
- Sampling and Evaluation: Efficient sampling was led by Tom Brown, while Tom Conerly carried out major system design, implementation, and support.
- Cluster: Nova DasSarma and Eli Tran-Johnson managed and maintained the research cluster supporting the work.
- Research: Saurav Kadavath and Jared Kaplan designed and conducted most experiments, including P(IK) training and large-scale evaluations.
- Writing: The paper was drafted by Jared Kaplan and Saurav Kadavath, with additional suggestions from other Anthropic members.
- Other contributions: The paper's ideas developed through conversations with many Anthropic staff members.
- Figure 21 shows RMS calibration-error trends on BIG Bench for multiple-choice and separate True/False formats.
A Metrics, Formatting Details, and P(IK) Training
The appendix defines calibration metrics, question formatting, self-evaluation prompts, and P(IK) training procedures across several task families.
- A.1 Calibration Charts: Calibration charts bin predictions into ten equal-count groups, plotting mean probability against the fraction of correct options.
- A.3 Brier Score: The Brier score uses pi = P(True) for a sample and ci = 0 or 1 according to whether that sample is incorrect or correct.
- A.4 BIG Bench Default Format: BIG Bench's default format asks models to rewrite complete responses, whereas the lettered-choice format requires outputting only the answer letter.
- A.4 BIG Bench Default Format: Calibration is poorer when models rewrite complete responses than when they select lettered choices.
- A.5 Prompt for Self-Evaluation: The self-evaluation prompt uses examples from arithmetic, trivia, and fill-in-the-blank tasks, with an optional set of brainstormed examples.
- A.6 Training for P(IK): P(IK) training pairs each question with labels derived from 30 candidate answers sampled at T = 1 and scored for correctness.
- A.6 Training for P(IK): Evaluation defines ground-truth P(IK) as the fraction of T = 1 samples the model answers correctly.
- A.7 Formatting of Questions for P(IK): The P(IK) tasks use varied formats: few-shot trivia, arithmetic, story completion, Python synthesis, GSM8k reasoning, anatomy questions, and Lambada completion.
A.2 Expected Calibration Error
The appendix describes expected calibration error and RMS calibration error as metrics computed from calibration-chart coordinates, while comparing methods that discriminate correct from incorrect samples without calibration.
- A.2 Expected Calibration Error: Expected calibration error is computed from the (xi, yi) coordinates of equal-count calibration bins.
- A.2 Expected Calibration Error: The ECE calculation uses only top predictions for each multiple-choice question, producing a larger value than using all predictions.
- A.2 Expected Calibration Error: RMS Calibration Error is also evaluated and is described as less common in the literature but theoretically sounder.
- The section compares methods that distinguish correct and incorrect samples without achieving calibration.
B.1 Loss of the Sample
The model’s loss on a generated answer provides some signal about whether that answer is correct, while answer entropy distinguishes questions the model answers correctly from incorrectly. However, entropy’s usefulness varies with task and model size.
- Loss of the Sample: A generated answer’s mean token loss can discriminate between correct and incorrect samples.The method averages loss across tokens; summing would confound results with candidate-answer length.
- Entropy of the Answer Distribution: Correct TriviaQA questions have lower average answer entropy than incorrect questions.The entropy distributions differ between questions the model gets correct and incorrect.
- Entropy of the Answer Distribution: 200 sampled answers estimate H(A|Q), the entropy of the answer distribution for each question.The entropy is computed as the expected negative log probability of sampled answers.
- Entropy of the Answer Distribution: Entropy has predictive power, but its AUROC decreases with model size on HumanEval.Larger models solve harder HumanEval problems while often producing diverse correct answers, conflicting with the hypothesis that knowing implies low entropy.
B.3 Loss of Sampled Answers Stuffed into a Single Context
Stuffing multiple generated answers into one context and measuring token loss provides a signal of whether the model knows the answer. Mean loss controls for answer-length differences, but task-specific performance varies.
- Loss of Sampled Answers Stuffed into a Single Context: Stuffing 30 model-generated samples into context and measuring mean token loss provides discriminative power for model knowledge.The method uses samples generated at T = 1 and evaluates the mean loss across the sequence.
- Loss of Sampled Answers Stuffed into a Single Context: Mean token loss is used instead of total loss to avoid confounding results with sampled-answer length.For HumanEval, correct-answer length tends to correlate with question difficulty.
- Loss of Sampled Answers Stuffed into a Single Context: Entropy-based AUROC distinguishes whether the model knows an answer rather than whether an individual sample is valid.For HumanEval, the figure thresholds correct answers using Pass@10 > 0.5 instead of the main-text Pass@1 criterion.
- Loss of Sampled Answers Stuffed into a Single Context: On TriviaQA, token loss clearly differs between sets of samples associated with correct versus incorrect underlying answers.The corresponding correlation is much weaker on Codex HumanEval.
C More P(True) Evaluation Results and Details
Zero-shot True/False self-evaluation separates correct from incorrect samples, but its probability estimates are poorly calibrated; few-shot evaluation is needed for good calibration.
- More P(True) Evaluation Results and Details: Zero-shot P(True) judgments separate correct and incorrect samples but are poorly calibrated.Good calibration is obtained when evaluation is performed few-shot, as in the comparison-sample setting.
D Mixed-Arithmetic and Function Synthesis Dataset Descriptions
The paper describes a programmatically generated arithmetic dataset and a Python function-synthesis dataset built from tested GitHub code, alongside additional evaluation figures for self-evaluation and P(IK).
- Mixed-Arithmetic Dataset: Mixed-Arithmetic is a programmatically generated dataset of arithmetic problems.Table 4 provides an overview and examples of its question types.
- Function Synthesis Dataset: The Python function-synthesis dataset was scraped from GitHub repositories containing at least one file named ‘test’.Pytest coverage identified non-test functions with at least half their lines covered by passing tests, which supplied evaluation tests.
- Function Synthesis Dataset: 8,000 Python functions and corresponding tests comprise the final function-synthesis dataset, split into 6,000 training and 2,000 test examples.The tests were used to evaluate model responses.
- P(True) Evaluation: P(True) histograms compare correct and incorrect samples when models inspect five T = 1 samples for the same question.The comparison-sample setting is used to improve judgment.
- P(True) Evaluation: P(True) evaluation is assessed with base accuracy, filtered accuracy, Brier scores, AUROC, and calibration curves.The figures compare prompting and few-shot methods, including comparison examples.
- P(True) Evaluation: Smaller models can appear calibrated when they answer nearly every question incorrectly, because their predictions become trivial.This caveat applies especially to GSM8k, Codex, and Arithmetic.
- Cross-Evaluation: Larger evaluators better assess correctness, while samples from smaller models are easier to evaluate.Correctness evaluation improves with evaluator size even when the evaluator judges its own samples.
- Calibration: Zero-shot P(True) is poorly calibrated, although calibration generally improves as models become larger and more capable.Additional figures examine calibration under task reformulations and larger question subsets.