Source-linked AI summary
Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity
Irina Proskurina, Mayank Kumar, Oyindolapo O. Komolafe
TL;DR
The paper asks whether instruction tuning changes rationale lexical diversity alongside model confidence in question answering. It evaluates matched base and instruction-tuned models with confidence, calibration, and diversity measures, finding increased confidence and reduced cross-rationale diversity despite limited accuracy changes, while surface lexical diversity varies and the patterns persist under controls.
Problem
The study addresses the limited evidence on whether instruction-tuning-induced confidence changes accompany corresponding changes in the lexical diversity of generated answer rationales.
Method
The paper conducts paired evaluations of three base/instruction-tuned model families across three reasoning benchmarks, measuring confidence, calibration, and rationale diversity while controlling answer selection and rationale length.
Results
Instruction tuning consistently increases confidence and reduces cross-rationale diversity without corresponding accuracy improvements, while surface-level lexical diversity varies across models and benchmarks.
Takeaways & Limitations
Confidence and rationale diversity reflect distinct effects of instruction tuning, so uncertainty assessment should consider both predictive confidence and variation in generated rationales.
Takeaways & Limitations
The study is limited to three English multiple-choice benchmarks and lexical diversity measures.
Abstract
from arXiv · showhide
Instruction-tuned language models achieve strong performance across a range of generation tasks, but have also recently been shown to exhibit verbalized overconfidence. In question answering, verbalized model overconfidence may be associated with the consistency of the generated supporting rationales. In this paper, we study whether corresponding changes in the lexical diversity of generated answer rationales accompany changes in model confidence induced by instruction tuning. We evaluate three matched base and instruction-tuned models across question-answering benchmarks and find that instruction tuning consistently alters answer confidence, despite limited changes in predictive accuracy and decreases in likelihood-based calibration. Secondly, we observe a non-uniform effect of instruction tuning on rationale diversity: cross-rationale diversity consistently decreases, whereas surface-level lexical diversity varies in both direction and magnitude across models and benchmarks. Finally, we find that these differences persist after controlling for answer selection and rationale length, confirming that confidence and rationale diversity capture distinct effects of instruction tuning.
1 Introduction
Instruction tuning changes model confidence and raises questions about whether rationale lexical diversity shifts alongside it. This study pairs base and instruction-tuned models to examine confidence, calibration, and rationale diversity across question-answering settings.
- Reliable confidence estimates matter for language models used in medicine, finance, and law, while instruction following can improve performance.
- Prior work shows that post-training alters model confidence distributions, but confidence can be estimated through multiple proxies.
- The paper investigates whether instruction-tuning differences in likelihood-based uncertainty and verbalized confidence accompany shifts in rationale lexical diversity.
- The evaluation pairs base and instruction-tuned models across three model families and three reasoning benchmarks, with controls for answer selection and rationale length.
- Instruction tuning increases confidence without corresponding accuracy improvements, while rationale-diversity changes are heterogeneous and not uniformly associated with calibration.
2 Related Work
Related work establishes that confidence and diversity are each active research areas, but their relationship in generated rationales remains insufficiently studied. The paper addresses this gap by examining lexical diversity alongside verbalized confidence and calibration.
- Calibration measures how closely predicted confidence matches the empirical probability of correctness, yet confidence can still be misaligned with accuracy.
- Research has evaluated output diversity, including lexical and semantic diversity, as an indicator of creativity by comparing model outputs with human text or judgments.
- The relationship between lexical or token-level diversity and verbalized confidence or calibration had not yet been investigated.
3 Methodology
The study evaluates confidence and rationale diversity using paired multiple-choice models, complementary uncertainty measures, repeated rationale sampling, and benchmark comparisons. Its metrics distinguish answer-selection uncertainty, verbalized confidence, lexical richness, and cross-generation similarity.
- The task is multiple-choice question answering, with each input question x associated with candidate answers Y = y1, . . . , yM.
- Model prediction is the candidate answer with the highest conditional likelihood.
- Choice uncertainty is measured with entropy over normalized candidate-answer likelihoods, where higher Hchoice(x) indicates greater uncertainty.
- Verbalized confidence is elicited in a second forward pass after likelihood-based answer prediction, using a numerical probability that the selected answer is correct.
- Five chain-of-thought rationales are sampled per question, and diversity is assessed with Unique-2 lexical richness and Self-BLEU similarity across rationales.
- The experiments compare Qwen2.5-7B, Llama-3.1-8B, and Mistral-7B-v0.3 base/instruction-tuned pairs on ARC-Easy, MMLU, and CommonsenseQA.
4 Results
Instruction tuning consistently increases confidence while affecting rationale diversity unevenly: cross-rationale variability falls, but surface-level lexical diversity changes by model and benchmark. These patterns are not explained by answer switching or rationale length, and diversity changes do not consistently track calibration.
- Confidence and accuracy: Instruction tuning consistently increases model confidence, reflected by lower answer entropy and higher verbalized confidence, despite less consistent accuracy changes.For Llama on ARC-Easy, accuracy remains 82.2% while verbalized confidence increases and choice entropy decreases.
- Rationale diversity: Cross-rationale diversity decreases across all models and benchmarks, whereas Unique-2 changes vary in direction and magnitude.Mistral’s 1-SelfBLEU decreases from 0.813 to 0.626 on ARC-Easy, while CSQA Unique-2 increases from 0.719 to 0.750.
- Uncertainty-diversity relationship: Decreases in answer uncertainty do not consistently coincide with reduced lexical diversity, with directions depending on the model and diversity measure.For Qwen, lower uncertainty coincides with lower diversity in 61.8% of Unique-2 and 69.3% of 1-SelfBLEU cases; Mistral shows divergent patterns across measures.
- Controlled analyses: Diversity shifts persist after controlling for answer selection and rationale length, while lexical-diversity changes are not consistently associated with calibration changes.For Qwen on ARC-Easy, verbalized confidence, Unique-2, and 1-SelfBLEU all decrease; for Llama on MMLU, likelihood-based and verbalized ECE increase from 0.5 to 5.9 and 16.6 to 23.7.
5 Conclusion
The paper finds that instruction tuning affects confidence and rationale diversity differently in question answering. It argues that uncertainty estimation should jointly consider predictive confidence and variation in generated rationales.
- Instruction tuning reduces cross-rationale variability, while surface-level lexical diversity follows benchmark-dependent patterns.
- The differing confidence and diversity patterns persist after controlling for answer selection and rationale length.
- The findings motivate uncertainty estimation that jointly considers predictive confidence and variation in generated rationales.
Limitations
The study is limited to three English multiple-choice benchmarks and lexical diversity measures, leaving broader language and diversity generalization for future work.
- The study covers three English multiple-choice benchmarks and lexical diversity measures, not semantic or syntactic diversity across broader benchmarks or languages.
Ethical Considerations
The paper acknowledges that its findings may encourage overreliance on miscalibrated confidence, while leaving safety-sensitive and demographic-sensitive questions unevaluated.
- The study uses publicly available models and benchmark datasets under their intended-use and licensing terms.
- Instruction tuning may encourage reliance on incorrect predictions when increased confidence is interpreted as evidence of reliability without improved accuracy.
- Reduced cross-rationale diversity may limit how independently repeated generations support a model prediction.
- The study does not evaluate questions involving unsafe data or demographic-sensitive attributes.
- Whether these confidence and diversity patterns extend to safety-sensitive or demographic-sensitive prompts requires separate evaluation.
A Examples of Generated Rationales
Table 4 presents example rationales from Base and Instruct variants of Mistral-7B answering the same ARC-Easy question.
- The table compares rationales generated by Base and Instruct Mistral-7B variants for one shared ARC-Easy question.
B Implementation Details
The implementation evaluates repeated rationale generation, controlled lexical diversity, benchmark accuracy, and verbalized confidence using fixed procedures across models and benchmarks.
- Rationale Generation: Five rationales are sampled per question using fixed temperature, nucleus-sampling, token-limit, and prompting settings across models and benchmarks.Sampling uses K = 5, temperature T = 0.7, nucleus-sampling p = 1.0, and a maximum of 100 newly generated tokens.
- Controlled Lexical Analysis: Controlled lexical analysis retains same-answer examples while matching rationale counts and lengths across Base and Instruct variants.
- Implementation: Accuracy uses the LM Evaluation Harness, while paired t-tests assess per-example Instruct-Base differences and controlled-analysis results.
- Verbalized Confidence Evaluation: Verbalized confidence is elicited after likelihood-based answer selection, so the confidence prompt evaluates the already selected answer.
- Verbalized Confidence Evaluation: The confidence prompt asks for the probability that the selected answer is correct and requires a number between 0 and 1.
C Extended Results
Extended analyses show that uncertainty and lexical diversity can move in different directions, rationale length increases broadly, and calibration is summarized with ECE across model variants.
- Uncertainty-Diversity Directional Analysis: Decreases in answer uncertainty can coincide with either increases or decreases in lexical diversity, depending on model and diversity measure.
- Rationale Length Evaluation: Rationale length increases across all evaluated models and benchmarks, motivating a length-controlled analysis.
- Calibration Evaluation: Lower Expected Calibration Error indicates better alignment between predicted confidence and empirical correctness.
- Calibration Evaluation: Calibration results are reported separately for likelihood-based and verbalized confidence across all Base and Instruct models and benchmarks.
- Uncertainty-Diversity Directional Analysis: Table 6 reports percentages of questions in four directional-change quadrants for choice entropy and lexical diversity, highlighting each model-measure pair’s most frequent pattern.
- Calibration Evaluation: Table 7 reports ECE percentages for Base and Instruct variants, with Gap defined as ECEverb − ECElik.