Alignment for Honesty

1 Introduction

• Honesty research has received less attention than helpfulness and harmlessness despite its importance for reliable and safe AI.The authors identify honesty as one of the HHH alignment criteria and highlight the limited prior focus on it.Source: §1 · p. 1
• Honesty requires models to answer questions they know and admit when they do not, while avoiding excessive conservatism.The paper grounds this definition in the spirit of Confucius and illustrates it as answering known questions correctly or refusing unknown ones.Source: §1 · p. 1, Figure 1 · p. 2
• Models’ knowledge boundaries are difficult to inspect, so the paper evaluates whether they can correctly answer particular questions rather than directly measuring stored knowledge.This shifts the target from internal knowledge to externally observable question-answering capability.Source: §1 · p. 2
• Honesty alignment can reduce hallucinations and improve trustworthiness by encouraging models to acknowledge their limitations.The paper also argues that more honest models can remain usable and potentially reduce reliance on external resources.Source: §1 · p. 2
• The framework defines idk responses, introduces prudence and over-conservativeness metrics, and evaluates changes before and after alignment.The metrics target appropriate refusal of unknown questions without excessive refusal of answerable ones.Source: §1 · p. 2
• The work contributes external approximations of knowledge boundaries, automated honesty-data synthesis, and evaluation across in-domain, generalization, and alignment-tax settings.It also presents several honesty-oriented fine-tuning methods and reports that they improve honesty without significantly reducing helpfulness.Source: §1 · p. 2

2.1 LLM Alignment

• Alignment is modeled as an iterative process that transforms a pretrained model through successive alignment cycles using a value function and strategy.The response is generated by M_t, values are assigned through v(·), and f(·) represents an alignment strategy such as supervised fine-tuning.Source: §2.1 · p. 3

2.2 Alignment for Honesty

• The framework approximates inaccessible model knowledge by judging external response accuracy rather than internal representations.The categorization function c(·) supports approximations such as k(x) = I(c(x, y) = 1), while alternative definitions of k(·) are explored later.Source: §2.2 · pp. 3–4
• Honesty alignment requires correct responses for known questions and explicit idk responses for unknown questions.Responses are categorized as idk, correct, or wrong according to refusal signs and whether the correct answer appears in the response.Source: §2.2 · p. 4

2.3 Evaluation Methodology

• The evaluation framework compares paired response types from M_t and M_t+1 to define evolutionary metrics over alignment changes.A second-order c(x, y_t, y_t+1) records the before-and-after response categories across an evaluation dataset of N samples.Source: §2.3 · p. 4
• Prudence score measures whether models decline questions they do not know or would answer incorrectly, while over-conservativeness score measures refusals on questions they previously answered correctly.The two metrics jointly assess refusal behavior and excessive caution after alignment.Source: §2.3 · pp. 4–5
• The framework currently compares two alignment iterations, while higher-order comparisons across more iterations are left for future work.Table 1 illustrates response-type transitions, including a formerly correct response becoming a refusal as a sign of excessive caution.Source: §2.3 · p. 4, Table 1 · p. 4
• The framework distinguishes honesty changes from general capability changes by using accuracy to detect disruption of original abilities.Cases where alignment converts correct answers into wrong answers are not treated as dishonesty because they may reflect performance loss or catastrophic forgetting.Source: §2.3 · p. 5
• Responses containing the correct answer are counted as loosely correct even when they also include idk signs, and accuracy uses this expanded category.This handles cases where uncertainty language co-occurs with a correct answer.Source: §2.3 · p. 5

3 Training Methodology

• The methodology transforms a model M_t into a more honesty-aligned model M_t+1 through training procedures defined by the honesty objective.It includes both training-free prompting and parameter-updating supervised fine-tuning.Source: §3 · p. 5, §3.1 · p. 6, §3.2 · p. 6

3.1 Training-free Method

• Prompting permits explicit refusal without parameter updates, but its effectiveness depends on instruction following and in-context learning and is sensitive to prompt choice.The approach is convenient, but its results are described as insufficiently robust.Source: §3.1 · p. 6

3.2 Supervised Fine-tuning

• Supervised fine-tuning estimates whether a question is known by defining k(·) from the model’s expected accuracy across multiple sampled responses.Expected accuracy is the ratio of correct responses among m candidate responses, and the resulting strategies vary in k(·) and training-sample annotation.Source: §3.2 · p. 6, Figure 3 · p. 6

3.2.1 ABSOLUTE

• ABSOLUTE treats questions with expected accuracy at least τ as known, trains on sampled correct answers for them, and uses predefined idk responses for unknown questions.This directly maps estimated knowledge status to target outputs.Source: §3.2.1 · p. 6

3.2.2 CONFIDENCE

• CONFIDENCE augments correct answers with numerical or verbal expressions of the model’s estimated confidence.For example, an expected accuracy of 0.9 can yield a correct answer paired with approximately 90% or absolute-confidence language.Source: §3.2.2 · pp. 6–7

3.2.3 MULTISAMPLE

• MULTISAMPLE uses the set of sampled responses to expose varying confidence levels during training and expands the dataset by a factor of m.With m = 10, one incorrect response can become an idk target while nine correct responses become correct targets for the same question.Source: §3.2.3 · p. 7

4.1 Training Settings

• The experiments use 8,000 TriviaQA samples with LLaMA2-Chat models, evaluate temperature-0 responses using accuracy and evolutionary metrics, and compare aligned models with unaligned and fine-tuned baselines.Unknown and correct responses are identified through heuristic rules, answer matching, and ChatGPT analysis.Source: §4.1 · p. 7, §4.2 · p. 7, §4.3 · p. 7

4.4.1 Overall Results

• Honesty-oriented fine-tuning consistently improves honesty while preserving high accuracy, with larger prudence gains and no greater false abstention than prompt-based methods.These methods more often abstain on unknown questions without substantially avoiding known ones.Source: §4.4.1 · p. 8
• Expected-accuracy signals improve honesty: CONFIDENCE provides explicit confidence supervision, while MULTISAMPLE learns from sampled answer and abstention proportions.Both methods become slightly more over-conservative but obtain markedly improved honesty scores.Source: §4.4.1 · p. 8
• MULTISAMPLE achieves the highest honesty score, while CONFIDENCE-VERB achieves the best accuracy, second only to the unaligned baseline.MULTISAMPLE sacrifices answers to a small portion of known questions, whereas CONFIDENCE-VERB’s accuracy may reflect inference computation or explicit confidence prefixes.Source: §4.4.1 · p. 8, Table 3 · p. 8

4.4.2 Scalability and Adaptability

• The methods scale across model sizes and adapt to multiple open-source LLMs, including InternLM, Qwen, and Baichuan2.Additional results are reported for smaller and larger models.Source: §4.4.2 · pp. 8–9

4.5 Exp II: Out-of-distribution Evaluation

• CONFIDENCE-VERB transfers across three out-of-distribution datasets, improving refusal of unknown questions while minimizing loss of original performance.The authors interpret performance across different data distributions as evidence against a shortcut based only on TriviaQA.Source: §4.5 · p. 9, Table 4 · p. 9
• On prior-known questions, the fine-tuned baseline reduces accuracy by 10 points, which the authors associate with training models to answer beyond their knowledge boundaries.The baseline training data included about 25% questions the model could hardly be expected to know.Source: §4.5 · p. 9

4.6 Exp III: Alignment Tax

• CONFIDENCE-VERB and MULTISAMPLE achieve helpfulness similar to the unaligned baseline, indicating little measured helpfulness cost from honesty alignment.Helpfulness is evaluated on diverse non-QA requests by AUTO-J and GPT-4.Source: §4.6 · pp. 9–10

5.1 Pitfalls in Defining Honesty

• The study assumes that incorrect answers to general knowledge questions usually reflect fabrication rather than learned false beliefs, while excluding deliberate lying by default.It distinguishes honesty—stating what the model believes—from truthfulness—stating what is objectively true.Source: §5.1 · p. 10
• The paper notes that its external-behavior approximation does not fully capture latent knowledge and calls for methods that can elicit hidden knowledge and separate dishonesty from catastrophic forgetting.These issues are identified as directions for further research rather than resolved within the study.Source: §5.1 · p. 10, §5.2 · p. 10

5.2 Future Work

• Confidence-oriented methods express varying uncertainty, but calibrating free-form confidence and defining feasible calibrated expressions remain outside the paper’s scope.The experiments focus on idk signs or correct answers rather than calibrated confidence itself.Source: §5.2 · p. 10
• Future work could explore representation-level alignment to adjust behavior for both known and unknown questions with minimally invasive interventions.The paper contrasts this goal with prior representation-engineering work focused on eliciting truthful answers to known questions.Source: §5.2 · p. 10

6 Conclusion

• The framework requires LLMs to decline questions when appropriate, introduces idk responses and honesty metrics, and evaluates honesty-oriented fine-tuning methods.The authors report extensive experiments validating the feasibility of alignment for honesty.Source: §6 · p. 11

A Glossary of Important Concepts in LLM

• The paper’s long-term goal is to foster inherent honesty without relying heavily on complex prompt engineering or external resource retrieval.The glossary clarifies distinctions among closely related concepts in LLM knowledge manipulation.Source: A Glossary of Important Concepts in LLM · p. 17

B Related Work

• Existing alignment research emphasizes helpfulness and safety, whereas this work addresses honesty using labeling that relies predominantly on the model rather than external feedback.The paper describes its task as a more nuanced form of alignment for honesty.Source: B Related Work · p. 17
• Prior hallucination and confidence-calibration methods assist truthful answering or trust assessment but do not explicitly give models the ability to refuse unknown questions.This paper investigates autonomous abstention without making models overly cautious.Source: B Related Work · p. 17

C.1 TriviaQA and Non-AmbigQA

• TriviaQA contains over 70,000 training pairs and 9,960 evaluation samples, while Non-AmbigQA provides 5,325 clear questions with non-ambiguous answers.Ambiguous questions are removed because unclear speaker intent can yield multiple valid answers.Source: §C.1 · p. 17
• TriviaQA and Non-AmbigQA responses are scored by extracting candidate short answers with ChatGPT and comparing them against dataset gold answers.This addresses the unreliability of exact match or Rouge-L when zero-shot responses are longer than short-phrase references.Source: §C.1 · pp. 17, 19

C.2 PUQA

• PUQA contains 1,000 challenging questions about 2023 scientific literature that are intended to fall outside the model’s knowledge scope.Questions use confusing titles without explicit time cues, and responses lacking idk signs are treated as hallucinating.Source: §C.2 · pp. 19–20

C.3 PKQA

• PKQA contains 1,000 model-generated questions across 22 topics, filtered for diversity, short answers, and correctness to represent questions the model is likely to know.The unaligned model must answer each retained question correctly, and aligned models are expected to preserve high accuracy.Source: §C.3 · p. 20

C.4 MMLU

• MMLU supplies around 14,000 four-choice questions across humanities, social sciences, hard sciences, and other subjects for evaluating generalization.Questions are reformatted as free-form prompts and scored with ChatGPT-based correctness checks.Source: §C.4 · pp. 20–21

C.5 Helpfulness-related Tasks

• Eval-P− contains 1,320 samples from 55 scenarios spanning summarization, code, writing, rewriting, communication, and NLP tasks.These scenarios are selected to differ substantially from knowledge-intensive question answering.Source: §C.5 · p. 21
• Responses are generated from checkpoints before and after alignment and rated from 1 to 10 by AUTO-J and GPT-4.The evaluation omits the knowledge-QA instruction because helpfulness tasks have distinct requirements.Source: §C.5 · pp. 21–22

D.1 Heuristic Rules for Idk Response

• The evaluation detects idk responses with string matching against phrases such as “i apologize,” “not aware of,” and “not familiar with.”These heuristic expressions operationalize abstention detection in the response text.Source: §D.1 · p. 22

D.2 Output formats for CONFIDENCE

• CONFIDENCE-NUM expresses uncertainty with percentage prefixes, whereas CONFIDENCE-VERB uses verbal expressions such as “absolutely certain.”The two variants use distinct response-prefix formats.Source: §D.2 · p. 22

• 8,000 uniformly sampled questions are drawn from a TriviaQA subset balanced between known and unknown questions according to each base model’s responses.The balance uses temperature = 0 responses, and datasets differ across base models because their correctly answered questions differ.Source: §D.3 · p. 23

D.4 Training Details

• Training used full-parameter fine-tuning with AdamW, a 1e-6 learning rate, 0.1 weight decay, and one or two training epochs depending on the method.Experiments used batch size 8, a 0.05 warm-up ratio, and A100 GPUs.Source: §D.4 · p. 23

D.5.1 The Effect of Refusal Threshold

• Raising the refusal threshold makes the model more reliable but more conservative, providing a direct way to prioritize trustworthiness.ABSOLUTE uses τ = 0.1, allowing an answer when the model answers correctly in at least 1 of 10 attempts.Source: §D.5.1 · p. 23, Figure 4 · p. 23

D.5.2 Scalability

• CONFIDENCE-VERB improves honesty across 7B, 13B, and 70B models while better preserving original accuracy, with larger models showing stronger gains in prudence.The larger-model trend is accompanied by only a marginally higher over-consv. score.Source: §D.5.2 · p. 23, Table 18 · p. 24

D.5.3 Adaptability

• CONFIDENCE-VERB consistently improves honesty across backbones with comparable accuracy, whereas PROMPT-BASED can fail when instruction-following ability is weak.Qwen-Chat-7B is given as an example that cannot return valid PROMPT-BASED replies.Source: §D.5.3 · p. 24, Table 19 · p. 24

D.6 Generalization to Multiple-Choice QA

• On MMLU, models rarely refuse despite available idk responses, and CONFIDENCE-VERB and MULTISAMPLE improve accuracy over the unaligned baseline but retain relatively low honesty.Four-option questions permit a 25% chance of guessing correctly, complicating honesty evaluation.Source: §D.6 · pp. 24–25, Table 20 · p. 24
• Adding 284 MMLU examples improves the honesty of CONFIDENCE-VERB and MULTISAMPLE, while reducing FINE-TUNED BASELINE accuracy when unknown questions include gold answers.The added examples supplement the original 8,000 TriviaQA training samples.Source: Table 20 · p. 24, §D.6 · p. 25

• Honesty-oriented fine-tuning maintains helpfulness performance while also producing strong honesty performance across the evaluated scenarios.The evaluation uses results reported in Tables 21 and 22.Source: §D.7 · p. 25, Table 21 · p. 25, Table 22 · p. 25

D.8 Harmlessness Evaluation

• Honesty-oriented supervised fine-tuning has almost no impact on the models’ inherent harmlessness in evaluation on potentially harmful prompts.The evaluation uses 700 BeaverTails-Evaluation prompts assessed by GPT-4o as safe, unsafe, or controversial.Source: §D.8 · p. 25, Table 23 · p. 25

D.9 Case Study

• After alignment, models refuse unknown questions and can decline controversial questions with explanations rather than supplying unsupported answers.The case studies contrast hallucinated, divergent answers before alignment with refusal or qualified non-answer behavior afterward.Source: Table 24 · p. 26, §D.9 · p. 26, Table 25 · p. 27