R-Tuning: Instructing Large Language Models to Say `I Don't Know'

1 Introduction

• The paper attributes hallucination to a gap between instruction-tuning data and parametric knowledge, compounded by training that forces answers even to questions beyond the model’s knowledge.Training only on correct answers can teach guessing rather than admitting ignorance; the paper proposes explicitly training models to express uncertainty and refuse unknown questions.Source: §1 · pp. 1–2
• R-Tuning compares training-set predictions with labels to split uncertain D0 from certain D1, then appends uncertainty expressions to labels before fine-tuning.The approach constructs refusal-aware data from the intersection of parametric knowledge and instruction-tuning data.Source: §1 · p. 2
• R-Tuning refuses uncertain questions while improving accuracy on answered questions, and multi-task training generalizes this refusal ability to out-of-domain datasets as a task-agnostic meta-skill.The paper reports that this ability can benefit from more downstream tasks and joint inference.Source: §1 · p. 2
• Learning uncertainty during training outperforms direct test-time uncertainty filtering, improving uncertainty estimation and answering while reducing test-time computational overhead.An unsupervised variant identifies certain examples through consistent answers across repeated responses and treats low-consistency answers as uncertain.Source: §1 · p. 2

2 Refusal-Aware Instruction Tuning

• R-Tuning first separates instruction-tuning examples into certain and uncertain data, then uses them to train uncertainty expression, as illustrated in Figure 2.Figure 2 depicts construction of the certain and uncertain datasets D1 and D0.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Refusal-Aware Data Identification

• The supervised identification strategy assigns an example to D1 when the pretrained model’s prediction matches its label and to D0 when it does not.This split identifies the gap between the model’s parametric knowledge and the knowledge required by training questions.Source: §2.1 · p. 3

2.2 Refusal-Aware Data Construction

• For data construction, R-Tuning preserves the answer label and appends “I am sure” for D1 or “I am unsure” for D0, following a prompt about answering from internal knowledge.This padding method retains label knowledge while teaching the model to express uncertainty.Source: §2.2 · p. 3

2.3 Training and Inference

• R-Tuning fine-tunes with standard cross-entropy on answer and uncertainty tokens, excluding the question tokens from the loss.The model predicts the answer part based on each question in the training sequence.Source: §2.3 · p. 3
• At inference, the model answers a templated question, then receives the uncertainty prompt followed by “I am” to produce an uncertainty expression about its previous context.The prompt asks whether the answer was accurate based on the model’s internal knowledge.Source: §2.3 · p. 3
• For AP evaluation, R-Tuning calculates confidence using a weighted combination of the uncertainty-expression probability and the answer prediction.The paper uses this confidence value to calculate the AP score.Source: §2.3 · pp. 3–4

3.1 Datasets

• Experiments use question-answering and multiple-choice tasks, with single-task and multi-task designs to assess individual-task effectiveness and generalization.Question-answering datasets are ParaRel (Elazar et al., 2021), HotpotQA, SelfAware, HaluEval (Li et al., 2023a), FalseQA (Hu et al., 2023), and NEC; multiple-choice datasets are MMLU (Hendrycks et al., 2021), WiCE (Kamoi et al., 2023), and FEVER. Single-task experiments use ParaRel and MMLU, split by domain into in-domain and out-of-domain tests; multi-task training mixes ParaRel, MMLU, WiCE, HotpotQA, and FEVER, testing on corresponding in-domain sets and unseen HaluEval out-of-domain.Source: §3.1 · p. 4

3.2 Baselines

• The baselines are full-test-set pre-trained models (Pretrain-T), pre-trained models evaluated on fine-tuned models’ willingly answered questions (Pretrain-W), and conventional instruction tuning (Vanilla).Pretrain-W tests whether willingly answered questions fall within the base model’s knowledge; Vanilla fine-tunes on all questions and ground-truth labels.Source: §3.2 · p. 4

3.3 Evaluation

• For answer-or-unknown models, accuracy is evaluated on questions the model is willing to answer; R-Tuning is prompted for an answer and then uncertainty to assess precision-recall tradeoffs with AP.AP ranks predictions by confidence and averages precision across thresholds; it is high when correct answers receive high confidence and hallucinated answers relatively low confidence.Source: §3.3 · p. 4

3.4 Implementation

• Experiments use OpenLLaMA-3B, LLaMA-7B, and LLaMA-13B with LMFlow2, one epoch, a 2e−5 learning rate, batch size 4, and Nvidia A100-40GB GPUs.Source: §3.4 · p. 5

4.1 Single-task Experiments

• In single-task experiments, R-Tuning improves accuracy on questions it answers and outperforms Vanilla in AP, with greater AP gains at larger model sizes and better confidence calibration.Vanilla’s AP drops as model size grows on ParaRel (OOD) and MMLU (ID), whereas R-Tuning’s AP rises steadily; combining prediction and certainty confidence makes its predictions more reliable.Source: §4.1 · p. 5

4.2 Multi-task Experiments

• R-Tuning consistently achieves higher AP than all baselines across in-domain and out-of-domain multi-task evaluations, including unseen HaluEval-QA.Higher AP indicates that correct answers are ranked above incorrect ones; the results suggest refusal ability generalizes to other data distributions and can be enhanced with different datasets.Source: §4.2 · pp. 5–6, Figure 4 · p. 6
• Multi-task results indicate that refusal-aware answering can reduce hallucinations by disregarding questions outside the model’s knowledge domain while preserving performance on aligned inquiries.The paper characterizes refusal as a model meta-skill with generalization potential.Source: §4.2 · p. 6

5.1 Unsupervised Identification

• R-Tuning-U uses model uncertainty to identify the 50% most uncertain questions and generally achieves higher AP than R-Tuning on ParaRel and MMLU.It samples k = 10 predictions at temperature 0.7, and unlike supervised identification, does not require labels for uncertain questions.Source: §5.1 · pp. 6–7, Table 2 · p. 7

5.2 Uncertainty Learning

• Learning uncertainty during instruction tuning yields better AP and question-answering accuracy than filtering answers by test-time uncertainty.The authors hypothesize that predicting uncertainty embeds confidence in hidden representations, helping the model answer easier questions; learning uncertainty also avoids repeatedly calculating it at test time.Source: Table 2 · p. 7, §5.2 · p. 7
• R-Tuning-U improves calibration and accuracy over Vanilla-C, assigns higher confidence scores than Vanilla-C, and shows more salient score differences as model size increases.The paper interprets the increasing score differences as evidence that refusal ability is an emergent ability.Source: §5.2 · p. 7

5.3 Unanswerable Questions

• R-Tuning refuses nearly all unanswerable benchmark questions, while other fine-tuned and pretrained baselines answer most despite being prompted that refusal is allowed.The benchmarks include questions that contradict common sense or fall outside the model’s parametric knowledge.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 Perplexity and Entropy

• Certain data D1 has lower perplexity and lower answer entropy than uncertain data D0, supporting the split into easier questions to answer confidently and harder ones to approach conservatively.The perplexity is measured with pretrained models, while answer entropy is calculated from GPT-3.5-turbo solutions.Source: §5.4 · p. 8, Table 4 · p. 8

6.1 Hallucinations of LLMs

• Hallucinations can arise from training data, model training, and inference, while LLMs may confidently answer unknown questions or favor users over truth.Misinformation and bias in training data can be imitated (Penedo et al., 2023; Lin et al., 2022), and outdated data can cause temporal misalignment (Luu et al., 2022). Overconfidence and user-pleasing are also discussed (Kadavath et al., 2022; Perez et al., 2022; Radhakrishnan et al., 2023).Source: §6.1 · pp. 8–9
• Prior work detects hallucinations with internal-state classifiers, factuality benchmarks, and response-consistency checks, while mitigation includes retrieval and other factuality-improvement methods.Detection examples include Azaria and Mitchell (2023), Lee et al. (2023), and Manakul et al. (2023); mitigation includes retrieval-augmented methods (Peng et al., 2023; Lyu et al., 2023; Asai et al., 2023), knowledge-aware fine-tuning (Li et al., 2022), and other approaches (Chen et al., 2023; Elaraby et al., 2023).Source: §6.1 · p. 9
• R-Tuning targets hallucinations by teaching models to recognize knowledge gaps between instruction-tuning data and parametric knowledge and refuse out-of-knowledge instructions.This is presented alongside prior findings on the importance of early hallucination discovery and fine-tuning or external feedback for self-correction (Huang et al., 2023a).Source: §6.1 · p. 9

6.2 Uncertainty Quantification of LLMs

• Uncertainty research identifies predictive-confidence miscalibration in deep networks and LLMs and uses uncertainty for exemplar selection and knowledge assessment.Guo et al. (2017) identify calibration problems using ECE; Chen et al. (2022) observe miscalibration in pretrained LLMs. Active-Prompt uses uncertainty to select questions for chain-of-thought annotation (Diao et al., 2023b), and Knowledge Assessment for LLMs is also related (Dong et al., 2023).Source: §6.2 · p. 9

• R-Tuning outperforms traditional finetuning on AP score, indicating a good tradeoff between prediction and confidence.Source: §7 · p. 9
• R-Tuning identifies differences between instruction-tuning data and parametric knowledge, then appends uncertainty expressions to the uncertain training portion.Source: §7 · p. 9
• R-Tuning’s refusal ability extends from in-domain data to unseen tasks, suggesting refusal is a fundamental ability that can be abstracted through multi-task learning as a meta-skill.Source: §7 · p. 9

• R-Tuning expresses confidence only as sure or unsure; the authors propose quantitative confidence values as a more accurate alternative.Source: §8 · p. 10
• The evaluation uses answer checking and uncertainty quantification, but the authors note that comparing instruction-tuning and pre-training datasets could assess relevant knowledge more rigorously.Source: §8 · p. 10
• Entity linking of pre-training data, as in Kandpal et al. (2023), is another way to identify relevant knowledge, but its high computational cost motivates future efficiency improvements.Source: §8 · p. 10

A.1 Datasets

• The experiments use nine datasets spanning factual knowledge, broad subject tasks, textual entailment, question answering, and unanswerable or hallucination-prone questions.Source: §A.1 · p. 15
• ParaRel (Elazar et al., 2021) yields 25,133 prompt-answer pairs across 31 domains, split into 15 in-domain and 16 out-of-domain domains, with in-domain data equally divided for training and testing.Duplicate prompts with the same entities are removed, and the original mask-prediction format is converted to question answering.Source: §A.1 · p. 15
• MMLU (Hendrycks et al., 2021) is used directly as multiple choice, while WiCE (Kamoi et al., 2023) and FEVER are converted to multiple-choice NLI tasks.WiCE uses three answer choices; FEVER labels claims as SUPPORTED, REFUTES, or NOT ENOUGH INFO.Source: §A.1 · p. 15
• HotpotQA is evaluated on its development set using context documents and questions, and 10K training examples are sampled from both HotpotQA and FEVER.Source: §A.1 · p. 15
• SelfAware is used to test unanswerable questions; HaluEval (Li et al., 2023a) contributes only its question-answering part, while FalseQA and NEC contain questions with no correct answers.FalseQA questions conflict with common sense, and NEC questions contain made-up concepts.Source: §A.1 · p. 15
• For generated-answer tasks, evaluation checks whether the first few output tokens contain the ground-truth answer; multiple-choice tasks select the candidate with maximum probability from one generated token.Exact matching is not used because generation is not strictly controllable.Source: §A.1 · p. 15
• Original dataset details are provided in Appendix A.1 and Table 6, while Figure 6 presents the distributions of constructed refusal-aware data D0 and D1.Source: §A.1 · p. 15

• The experiments use OpenLLaMA-3B, LLaMA-7B, and LLaMA-13B as base models.Source: §A.2 · p. 15
• Instruction tuning uses LMFlow3 (Diao et al., 2023a) for 1 epoch, with a learning rate of 2e−5 and batch size 4.Source: §A.2 · p. 15
• Experiments run on Nvidia A100-40GB GPUs and sweep learning rates {1e−5, 2e−5, 5e−5} and batch sizes {2, 4, 8} on the training set.Source: §A.2 · p. 15

A.3 Label Replacement

• R-Tuning-R outperforms vanilla fine-tuning on willingly answered questions while refusing unknown ones; its accuracy also exceeds Pretrain-T on ID and OOD tests.Replacement discards ground-truth labels and cannot jointly express an answer and certainty, motivating the padding strategy used in the main approach.Source: Figure 5 · p. 16, §A.3 · pp. 16–17, Table 7 · p. 16

A.5 Uncertainty Text

• R-Tuning-R uses 16 uncertainty expressions from Yin et al. (2023), including “The answer is unknown.”Source: §A.3 · p. 16, §A.5 · p. 17

A.4 Case Studies of R-Tuning-R

• Case studies show Vanilla inventing nonsensical answers to out-of-knowledge questions, whereas R-Tuning-R explicitly refuses them with uncertainty phrases.The appendix presents cases of both refused and willingly answered questions.Source: §A.3 · p. 16, Table 9 · p. 17

A.6 Perplexity of Datasets

• D1 questions have lower perplexity than D0, while larger models show lower question perplexity; the authors interpret D0 as less familiar and more prone to hallucination.R-Tuning is described as helping models recognize uncertainty on relatively random questions while answering less random ones confidently.Source: Table 10 · p. 18, §A.6 · p. 18

A.7 Entropy of Answers

• GPT-3.5-turbo produces higher answer entropy on uncertain D0 questions than on certain D1 questions, and R-Tuning encourages confident answers to easier questions and caution on difficult ones.The entropy comparison is higher for D0 in most cases.Source: Table 11 · p. 18, §A.7 · p. 19

A.8 Min-Loss Training

• Min-loss training minimizes cross-entropy for either the ground-truth answer or an uncertainty expression; it outperforms R-Tuning-R on small models and ID settings but underperforms it OOD.The approach uses a two-stage training strategy and considers the first token of the answer label.Source: Table 12 · p. 19, §A.8 · pp. 19–20

A.10 AP Scores of Each Dataset and Model Size with Figures

• The appendix reports AP scores across datasets and model sizes in multi-task experiments, with AP curves shown for ParaRel, MMLU, WiCE, HotpotQA, FEVER, and HaluEval-QA.Source: Table 13 · p. 19, §A.10 · p. 20, Figure 8 · p. 23

A.9 Confidence Distribution of Training Dataset

• A larger share of certain training questions falls in high-confidence intervals, indicating that correct answers generally come with greater confidence.Source: §A.6 · p. 18, §A.9 · p. 20, Figure 7 · p. 22