PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan

arXiv:2606.32032v1cs.CLcs.AI

TL;DR

LLMs can misrepresent their uncertainty, making faithful calibration a challenge. The paper uses metacognitive-feedback training and a two-stage calibration framework, achieving state-of-the-art faithful calibration across tasks while RLMF improves post-training results by up to 63% over standard RL.

  • Problem

    LLMs can fail to recognize knowledge boundaries, hallucinate with high confidence, and misrepresent their internal uncertainty.

    Source: §1 · p. 1

  • Method

    RLMF uses self-judgments to refine completion rankings, paired with metacognitive data selection; a decoupled second stage maps calibrated numerical confidence to linguistic uncertainty.

    Source: §1 · p. 2

  • Results

    Across multiple LLMs and 10 tasks, the framework achieves state-of-the-art faithful calibration while preserving accuracy and factual calibration; RLMF improves post-training results by up to 63% over standard RL.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings support metacognitive performance as a promising RL signal for improving LLM self-assessment and uncertainty communication.

    Source: §1 · p. 2, §6 · p. 9

    Improvement in self-assessment of performance is not equivalent to broad metacognitive awareness, which encompasses many other capabilities.

    Source: §5 · p. 8

Abstract

from arXiv · show

Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes. Yet LLMs exhibit systemic deficiencies in key metacognitive faculties: they hallucinate with high confidence, fail to recognize knowledge boundaries, and misrepresent their internal uncertainty--undermining trustworthiness and reliability. Since monitoring task performance and adapting behavior accordingly are central to metacognition, we posit that models capable of accurately judging their own performance are better positioned to improve it. We operationalize this idea via two novel mechanisms: reinforcement learning with metacognitive feedback (RLMF), a paradigm to refine completion rankings during preference optimization based on the quality of a model's self-judgments of performance, and metacognitive data selection, which uses similar self-judgments to identify high-value training examples, outperforming naive active learning. We apply these innovations to the problem of faithful calibration (FC), a task that is itself fundamentally metacognitive: the goal is to align expressed with intrinsic uncertainty, difficult even for frontier LLMs. We adopt a two-stage, decoupled approach, first using these methods to calibrate the faithfulness of models' self-reported confidence scores, then mapping to natural, context-adaptable linguistic uncertainty via targeted output editing. Extensive experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks while preserving accuracy. Further, RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits. This positions RLMF as a promising paradigm to enhance LLM metacognition toward improved abilities and alignment, and suggests metacognitive performance as an effective RL signal to overcome limits of prior intrinsic feedback methods.

1 Introduction

The introduction presents metacognitive feedback and data selection as ways to improve models’ self-assessment, then applies them to faithful calibration of numerical and linguistic uncertainty. It reports broad calibration gains while preserving task accuracy and factual calibration.

Motivation

  • LLMs often fail to recognize knowledge boundaries, hallucinate with high confidence, and misrepresent internal uncertainty, undermining trustworthiness and reliability in high-stakes uses.The introduction frames metacognition as monitoring, assessing, and regulating one’s own cognitive processes.Source: §1 · p. 1
  • The approach treats metacognitive performance as a post-training signal because accurately judging task performance may help models improve it.The proposed aim is to encode metacognitive awareness while improving task performance.Source: §1 · p. 2

Faithful calibration

  • Faithful calibration aligns expressed uncertainty with models’ estimated intrinsic confidence, unlike factual calibration, which aligns confidence with empirical accuracy.The introduction characterizes faithful calibration as largely unresolved and notes that existing approaches do not holistically address numerical and linguistic uncertainty.Source: §1 · p. 2
  • The proposed two-stage pipeline first calibrates sentence-level numerical confidence, then maps scores to context-adaptable linguistic hedges and revises responses for coherence and fluency.The stages are decoupled: Stage 1 can run once, while Stage 2 can adapt to user preferences and contexts without repeating costly RL training.Source: §1 · pp. 2–3

Results

  • Across 10 tasks and 6+ content domains, the framework achieves state-of-the-art faithful calibration, generalizes from a single training dataset, and preserves task accuracy and factual calibration.Human evaluation reports an average 96% win rate over the strongest baseline for diversity, naturalness, helpfulness, and contextual suitability of linguistic uncertainty.Source: §1 · pp. 2–3

RLMF

  • RLMF strengthens post-training results by up to 63% over standard RL by using the quality of a model’s self-judgments to refine completion rankings.During RL, metacognitive advantage scaling uses self-judgment accuracy to scale each completion’s advantage.Source: §1 · pp. 2–3

Metacognitive data selection

  • Metacognitive data selection uses self-assessments to select examples from both high- and low-scoring ends, outperforming naive and active-learning-style selection of poorly handled examples.The two ends provide complementary learning signals.Source: §1 · pp. 2–3

2 Related Work

The related work motivates ranking completions using models’ metacognitive performance and presents RLMF as a new internal-feedback signal. It identifies gaps in prior faithful-calibration methods, particularly numerical uncertainty and adaptive hedging across full responses.

Metacognition in LLMs

  • Metacognition has been linked to cognition and uncertainty communication [88], while deficiencies in LLMs have been noted across tasks [125] [41].The paper proposes prioritizing completions with stronger metacognitive performance, subject to task-level reward, to support both task improvement and recognition of capability limits.Source: §2 · p. 3

Reinforcement Learning with Internal Feedback

  • RLMF adds metacognitive performance to internal-feedback RL, which has used confidence, self-certainty, entropy, and calibration metrics as reward signals [54] [97] [116] [1] [59] [16].The authors describe this as the first use of metacognitive feedback during RL for LLMs, and report that it outperforms standard RL while improving metacognitive awareness.Source: §2 · p. 3

Faithful Calibration of LLMs

  • Prior faithful-calibration work focused on linguistic uncertainty, with metacognitive prompting [62], steering [42], and template-based SFT each limited in accuracy, extensibility, or linguistic diversity.These approaches did not address faithful numerical uncertainty, and prior work did not target natural, coherent hedging across an entire response or explore RL for faithful calibration.Source: §2 · pp. 3–4

3 Method

The method uses metacognitive feedback to improve preference optimization and training-data selection, first calibrating numerical confidence and then rewriting outputs into linguistic uncertainty. This decoupling allows linguistic expressions to be tailored to context and varied without repeating costly RL training.

3.1 Reinforcement Learning Setup

  • GRPO samples groups of sentence-level confidence-scored completions and uses composite rewards for faithful alignment, correctness, factual calibration, and format adherence to guide policy updates.The framework uses targeted rewards to optimize numerically expressed uncertainty and exploits sampling-based completions to assess faithful calibration.Source: §3.1 · p. 4

3.2 Reinforcement Learning with Metacognitive Feedback (RLMF)

  • RLMF refines completion rankings by giving greater weight to task-strong completions for which the model more accurately judges its own faithful-calibration performance.The model’s self-judgment is compared with an estimated gold FC level; metacognitive scaling applies to above-average faithfulness while other quality constraints remain in effect.Source: §3.2 · p. 5

3.3 Metacognitive Data Selection

  • Metacognitive data selection scores each training example for perceived alignment between linguistic and internal confidence, then selects equal numbers of highest- and lowest-scoring examples.The model first generates responses with linguistic hedging when uncertain, then scores its perceived alignment from 0 to 100; selection is made for a target training size.Source: §3.3 · p. 6

3.4 Rewriting Protocol

  • The rewriting protocol maps sentence-level faithful confidence scores to candidate hedges and prompts a strong, cost-effective LLM to edit outputs for task- or user-specific context.Inputs include the original response, candidate hedges, and contextual guidance such as style, audience, or domain conventions, enabling uncertainty expression to adapt without retraining.Source: §3.4 · p. 6

4 Experimental Setup

The experimental setup evaluates metacognitive methods for faithful calibration across numerical and linguistic uncertainty expression and diverse tasks. It specifies the datasets, models, training conditions, baselines, and metrics used for evaluation.

4 Experimental Setup

  • Evaluation covers 10 datasets and samples 1,000 test examples per dataset to avoid dataset-size bias, following prior work [62].The datasets span diverse formats, content domains, and difficulty levels.Source: §4 · p. 6
  • Experiments use instruction-tuned Qwen3 (1.7B, 4B, 8B) [109] and Llama3.1-Instruct (8B) [29], with pre-RL SFT to teach the custom output format.The SFT stage also targets cross-task generalization and task-specific output lengths; RLMF is applied to each model’s best checkpoint.Source: §4 · p. 6
  • RLMF trains on 2,000 metacognitively selected PopQA examples, chosen for comparability to prior work, and evaluates out-of-distribution transfer across tasks and domains.SelfAware, UMWP, and HaluEval are alternative training datasets used to test robustness to training-data choice.Source: §4 · p. 6
  • Ablations remove metacognitive advantage scaling or compare metacognitive data selection with no special selection and an active-learning-style baseline.The study also ablates the pre-SFT stage; Gemini-2.5-Flash-Lite is used for rewriting and compared with GPT-5-Mini [84] in §C.4.3.Source: §4 · p. 6
  • Because prior work studied only linguistic FC, comparisons with MetaFaith [62] and Faithful Uncertainty Tuning are restricted to linguistic FC; frontier models are also tested with MetaFaith prompting [62].The frontier-model baselines are Gemini-3.1-Pro, Gemini-3-Flash, and GPT-5.Source: §4 · p. 7

implementation of

  • The proposed cMFG* addresses cMFG’s sensitivity to intrinsic-confidence distributions and ranges from 0 to 1, with 1 indicating perfect FC.Accuracy, scored with LLM-as-a-Judge, and Brier Score are also reported as reference metrics.Source: §4 · p. 7, implementation of · p. 7

5 Results

RLMF improves faithful calibration across tasks and model families while preserving task accuracy and factual calibration. Metacognitive feedback also improves RL outcomes and supports effective data selection and linguistic uncertainty rewriting.

5 Results

  • 29% and 25% average cMFG* gains over prior prompting and SFT methods, respectively, with cMFG∗≥0.80 in each setting, establish state-of-the-art FC across models and datasets.The approach also generalizes to MATH and SimpleQA despite training only on PopQA, unlike FUT, whose efficacy is largely confined to similar QA tasks.Source: §5 · p. 7
  • RLMF works similarly well across intrinsic confidence levels, while FUT and original models systematically struggle at low confidence.The reliability diagrams compare expressed with intrinsic confidence on PopQA, using FUT and RLMF trained on PopQA.Source: Figure 3 · p. 8, §5 · p. 8
  • Metacognitive data selection yields the best numerical FC cMFG* among random and ground-truth-FC-based active selection while preserving accuracy and factual calibration.The findings suggest limited intrinsic self-improvement through identifying effective training data, analogous to choosing study materials.Source: Table 3 · p. 8, §5 · pp. 8–9
  • 37%, 17%, and 25% average gains over GPT-5, Gemini-3.1-Pro, and Gemini-3-Flash, respectively, show that small models can outperform proprietary LLMs in FC.These gains hold even when the proprietary models use specialized prompting, and the observations extend to sub-8B models.Source: §5 · p. 8
  • Up to 63% better performance than standard RL shows that RLMF improves numerical FC and metacognitive monitoring, with consistent gains across diverse training tasks.Training tasks include math reasoning (UMWP), hallucination detection (HaluEval), and answerability (SelfAware); improvement in self-assessment is not equivalent to broad metacognitive awareness.Source: §5 · p. 8, Table 2 · p. 8
  • 98%, 98%, 95%, and 96% human-annotation win rates show that rewritten linguistic uncertainty is more diverse, natural, helpful, and contextually suitable than FUT.Inter-annotator agreement was 0.93; manual inspection found FUT's hedge phrases and sentence structures repetitive, especially in long-form settings.Source: §5 · p. 9

6 Conclusion

The paper introduces RLMF and metacognitive data selection, then applies them in a two-stage framework for holistic faithful calibration. Experiments show strong, generalizable calibration while preserving accuracy and improving models’ self-assessment and uncertainty expression.

6 Conclusion

  • The framework achieves strong, generalizable faithful calibration across diverse models and tasks, outperforming prior state-of-the-art while preserving task accuracy and factual calibration.It also enables more faithful self-reported confidence scores and naturalistic, context-appropriate linguistic uncertainty.Source: §6 · p. 9
  • RLMF refines completion rankings during preference optimization using models’ implicit performance judgments, while metacognitive data selection uses similar judgments to identify more effective training data than simple active learning.The paper presents these as contributions to post-training and as internal feedback for RL.Source: §6 · p. 9
  • The paper’s two-stage, decoupled framework aligns numerically and linguistically expressed uncertainty with intrinsic confidence, and enables models to improve self-assessment of performance.The paper describes it as the first end-to-end framework for holistic faithful calibration of LLMs.Source: §6 · p. 9
  • The authors introduce cMFG*, a metric that removes estimation bias for models whose intrinsic confidence occupies a limited range.They suggest metacognitive performance as an effective internal RL feedback signal that may overcome limitations of prior intrinsic feedback methods, with implications for alignment and self-directed learning.Source: §6 · p. 9

Ethics Statement · A Additional Related Work · B Experimental Details

The paper situates faithful uncertainty expression as an underexplored part of confidence calibration and discusses its potential benefits alongside the need for safeguards and continued verification. Related work is distinguished by the paper’s focus on aligning expressed uncertainty with intrinsic uncertainty in both numerical and linguistic forms.

Ethics Statement

  • Faithful calibration and metacognitive awareness may improve abstention and reliability, including in high-stakes settings, but do not replace verification.The authors call for safeguards against misuse and misinformation and increased oversight for more autonomous or self-directed behavior.Source: Ethics Statement · p. 9
  • The approach’s generalizability may support reliability across settings, while its decoupled strategy allows linguistic uncertainty to adapt to cultural communication norms.The authors warn that faulty communication of intrinsic uncertainty in high-stakes settings such as AI-assisted scientific planning and discovery could cause significant setbacks or other negative consequences.Source: Ethics Statement · p. 9

A Additional Related Work

  • Prior confidence-calibration methods generally align estimates with externally judged accuracy, whereas this work targets faithful numerical and linguistic expression of uncertainty.The stated aim is to enable richer and more complex patterns of uncertainty expression.Source: A Additional Related Work · p. 23
  • Numerical confidence methods include white-box approaches using model internals and black-box approaches using outputs, but they do not incorporate linguistic uncertainty into model outputs.The passage lists token probabilities, representation probes, and uncertainty-augmented training as white-box approaches; black-box examples include response consistency, semantic variability, verbalized scores, and auxiliary predictive models.Source: A Additional Related Work · p. 23
  • Linguistic-confidence approaches prompt or train models to verbalize confidence, map numerical scores to phrases, or combine these strategies, but overlook alignment with intrinsic confidence.The passage cites [3] [91] [111] [92] [100] for integration of linguistic markers and [11] for combined strategies.Source: A Additional Related Work · p. 23
  • Existing linguistic-confidence methods can oversimplify uncertainty expressions, require costly domain-specific training, lack zero-shot verbalization, or constrain verbalized confidence to narrow ranges.Examples include limited scoring scales, domain-specific training without zero-shot verbalization [60], simplified marker sets [128], and restricted verbalized-confidence ranges [120].Source: A Additional Related Work · p. 23

B.1 Model & Training Details … B.3 Prompts

The appendix specifies the models, training and prompting procedures, and English benchmarks used to evaluate faithful calibration, emphasizing challenging tasks where uncertainty expression matters. It also documents dataset abbreviations and the prompts used for standard responses, metacognitive judgments, and response editing.

B.1 Model & Training Details

  • Experiments apply the framework and MetaFaith and FUT baselines to Qwen3 and Llama3.1-Instruct, and evaluate proprietary Gemini and GPT models with a metacognitive prompt.Inference generally used a 256-token limit and temperature 1.0 without thinking mode; reasoning-constrained Gemini-3.1-Pro used its minimum thinking level and up to 512 tokens. Rewriting used temperature 0.5 and up to 2000 tokens for Gemini-2.5-Flash-Lite.Source: §B.1 · pp. 24–25
  • RLMF training modifies GRPO advantages and uses LoRA adapters, with τ = 0.10, 32 completions per prompt, and 2000 training samples by default.Training used 1500 steps, effective batch size 64, and selected checkpoints by in-domain faithful-calibration performance; models were trained on four GPUs, with two additional GPUs for sampling and judging.Source: §B.1 · pp. 24–25
  • Pre-SFT teaches a sentence-tagged output format with per-sentence confidence, using 1800 training examples after a 10% validation split from 2000 samples.For each task sample, one response supplied the official output and 20 additional responses estimated intrinsic confidence; LoRA SFT ran for five epochs with validation-loss checkpoint selection.Source: §B.1 · p. 25

B.2 Datasets

  • Benchmarks include PopQA for tail-entity knowledge, SelfAware for answerable and unanswerable questions, adversarial SimpleQA [102], and hallucination-focused HaluEval [56].The datasets span domains and task types, with emphasis on challenging settings where uncertainty expression is expected to matter.Source: §B.2 · p. 26
  • Other benchmarks cover broad knowledge and reasoning with MMLU [35], SciQ [45], MATH [36], UMWP [90], ARC-Challenge [14], and SuperGLUE [99].They include multiple-choice science questions, answerable and unanswerable math questions, and knowledge-intensive science questions requiring integrated information.Source: §B.2 · p. 26

B.2.1 Dataset Abbreviations

  • Table 4 defines the dataset abbreviations used in cMFG* results tables, and experiments sample equally from four SuperGLUE subsets.The subsets are boolq, copa, wic, and wsc.Source: Table 4 · p. 26, §B.2.1 · p. 26

B.3 Prompts

  • Standard response prompts pair a shared system prompt with task-specific queries, randomize multiple-choice ordering, and use a metacognitive prompt for original models and baselines.The prompt set also includes templates directing pre-SFT output length.Source: §B.3 · p. 27, Figure 4 · p. 27
  • Separate prompts elicit RLMF judgments of faithful-calibration performance, support metacognitive data selection, and edit responses in the framework’s second stage.An alternate two-step editing approach uses separate prompts.Source: §B.3 · p. 27

B.4 Metrics · C Methodological Details · C.1 GRPO Details

The section defines metrics for factual accuracy, intrinsic and expressed confidence, and faithful calibration, then introduces cMFG* to address limitations of cMFG. It also outlines GRPO’s advantage-based policy updates and regularization controls.

B.4 Metrics

  • The Brier Score is the average squared error between confidence and correctness; zero indicates perfect factual calibration, with correctness measured by LLM-as-a-Judge.The accuracy judge used during RLMF training was Qwen/Qwen3-32B-FP8 with no thinking.Source: §B.4 · pp. 27–28
  • Intrinsic confidence is estimated from sentence-level consistency across 20 additional sampled responses using NLI judgments mapped to inconsistency scores; sentence-level scoring follows Eikema et al..The method follows Liu et al. [62], adapts Manakul et al. [66], and avoids assertion extraction used in the initial paradigm of Yona et al..Source: §B.4 · p. 28
  • Faithful calibration measures alignment between expressed and intrinsic confidence, unlike traditional calibration, which aligns confidence with factual accuracy; a score of 0.5 is random or constant expression and 1 is perfect alignment.Linguistic confidence is scored as sentence-level decisiveness by an LLM judge using the prompt from Liu et al. [62] with few-shot examples from Eikema et al..Source: §B.4 · pp. 29–30
  • cMFG* addresses cMFG’s empty-bin and restricted-support penalties using equal-mass bins and integration over each model’s intrinsic-confidence support.Equal-mass bins give each bin the same sample count, while integration over model-specific support avoids penalizing models for confidence values outside their operating range.Source: §B.4 · pp. 31–32
  • Under cMFG*, RLMF retains the same qualitative ordering as under original cMFG and remains stronger than original models, MetaFaith, FUT, and standard RL.The authors use cMFG* to report results across experiments.Source: §B.4 · p. 33

C.1 GRPO Details

  • In GRPO, advantage scores capture the relative quality of candidate completions for a prompt and guide policy updates; β and ϵ control divergence regularization and update magnitude.The objective is defined relative to the pre-update policy πold and reference policy πref.Source: §C.1 · p. 34

C.1.1 Reward Design … C.1.4 Impact of GRPO Advantage Normalization

These sections specify the reward design and compare training choices for faithful calibration, including reward formulations, prompts, and GRPO advantage normalization. The reported findings motivate the choices used in subsequent experiments.

C.1.1 Reward Design

  • The weighted reward combines faithful and factual calibration, correctness, and strict and soft format feedback, with faithful calibration prioritized.Default weights were wstrict = 3, wsoft = 3, wfactual_calib = 1, wacc = 1, and wfaith = 12; the design ensures faithfully calibrated outputs score higher than unfaithfully calibrated ones.Source: §C.1.1 · pp. 34–36
  • A binary sentence-count length penalty was unfruitful across tested weightings, so the authors favored pre-SFT to teach the target format before GRPO.The penalty applied −1.0 when completions exceeded the dataset’s maximum sentence count.Source: §C.1.1 · p. 35
  • A binary gibberish penalty proved unfruitful, whereas RLMF minimized gibberish in post-trained outputs without such penalties.Standard RL often produced nonsensical text after unclosed tags despite hyperparameter searches; tested penalty thresholds and weights did not resolve this.Source: §C.1.1 · pp. 35–36
  • Reward ablations showed that changing faithfulness weight or removing other rewards impaired calibration, format validity, or task performance.Decreasing or overly increasing wfaith worsened cMFG*; removing format rewards caused malformed outputs, removing factual calibration worsened factual calibration, and removing accuracy meaningfully lowered task performance.Source: Table 6 · p. 36, §C.1.1 · p. 36

C.1.2 Impact of Faithfulness Reward Formulation

  • The quadratic faithfulness reward performed best among the tested formulations and was used in all experiments.Alternatives included linear absolute difference and two cross-entropy formulations; the comparison trained Llama3.1-8B-Instruct on PopQA with standard GRPO.Source: §C.1.2 · p. 36, Table 7 · p. 37

C.1.3 Impact of System Prompt

  • Prompting about internal confidence signals, metacognitive awareness, and a rough confidence-range rubric helped, while verbose interpretations and a reward rubric hurt versus the best approaches.The authors used system prompt D in their main experiments because it yielded markedly better performance.Source: Table 8 · p. 37, §C.1.3 · p. 37

C.1.4 Impact of GRPO Advantage Normalization

  • Removing GRPO’s standard deviation-based advantage normalization raised numerical faithful calibration results by 0.06 on average across tasks.The comparison used Llama3.1-8B-Instruct with system prompt D and without metacognitive data selection, metacognitive advantage scaling, or pre-SFT; the authors adopted this choice subsequently.Source: §C.1.4 · p. 37, Table 9 · p. 38

C.1.5 Impact of Training Data Size … C.2.4 Impact of τ Value on RLMF

The appendices test RLMF’s data, training, and feedback design choices, finding that outcomes depend on how metacognitive information is used. The official advantage-scaling setup performs best among the tested formulations, while pre-SFT and the selected hyperparameters support faithful calibration and generalization.

C.1.5 Impact of Training Data Size

  • Using 1000 or 4000 rather than 2000 metacognitively selected samples yields slightly worse performance, but still outperforms the Liu et al. [62] and Eikema et al. baselines.The comparison uses Llama3.1-8B-Instruct trained on PopQA, with a hyperparameter search repeated for each sample count.Source: §C.1.5 · p. 38

C.1.6 Contribution of Pre-SFT Stage

  • Without pre-SFT, models show weaker numerical faithful calibration and worse cross-task generalization, whether RLMF and metacognitive data selection are used together or separately.The authors also report that pre-SFT helps models learn the target output format and use sentence and confidence tags as specified.Source: §C.1.6 · p. 38, Table 11 · p. 39

C.2.1 Implementation of RLMF

  • RLMF is implemented through simple modifications to the trl GRPOTrainer class.The paper points to its code for further implementation details.Source: §C.2.1 · p. 38

C.2.2 Alternative Formulations of Metacognitive Feedback

  • The official quadratic metacognitive-performance formulation and original Zg scaling perform best, while alternative Zg formulations yield worse or sometimes degenerate results.Across datasets for Llama3.1-8B-Instruct trained on PopQA, the quadratic variant beats alternative mathematical formulations; the original Zg also beats mean- and/or standard-deviation-normalized variants across tested k settings. The main advantage-scaling setup also gives the best faithful calibration among the tested ARLMF variants, though suitability may vary by task or setup.Source: §C.2.2 · pp. 39–40, Table 12 · p. 39, Table 13 · p. 40
  • Using Zg as an additional reward helps versus no metacognitive signal but does not achieve state-of-the-art faithful calibration while preserving task accuracy.Online inference for Fpred consistently matches or outperforms direct generation; combining metacognitive and faithfulness rewards is necessary, and the metacognitive reward works better as a lower-weight accessory than at equal weight. Applying it only above a faithfulness threshold can improve results and reduce reward-hacking-related degradation, but raising the threshold too far yields diminishing returns; lowering τ to 0.05 did not necessarily help because the signal may be sparse and inconsistent.Source: §C.2.2 · p. 41, Table 15 · p. 42
  • Metacognitive advantage scaling is comparable to an additional metacognitive reward while requiring fewer hyperparameters, and the way metacognitive information is used matters.The authors conclude that metacognitive signals can improve completion rankings, with advantage scaling offering a simpler design than manually optimized reward schemes.Source: §C.2.2 · p. 41

C.2.3 Impact of k Value on RLMF

  • Larger k values lead to empirical collapse, while k = 0 worsens results, supporting the official k = 1 selection and its role in countering potential faithfulness harms.The comparison uses Llama3.1-8B-Instruct trained on PopQA and evaluated across datasets.Source: §C.2.3 · p. 42, Table 16 · p. 42

C.2.4 Impact of τ Value on RLMF

  • For RLMF advantage scaling, τ = 0.05 produces a larger difference in overall average cMFG* and slightly worse out-of-distribution performance than τ = 0.10.Per-dataset faithful calibration fluctuates, and the authors infer that the larger τ provides more consistent and helpful feedback.Source: §C.2.4 · p. 42, Table 17 · p. 43

C.3 Metacognitive Data Selection Details … D.1 Full Experimental Results (+More Models)

The appendices detail metacognitive data selection, construction of a numerical-to-linguistic confidence map, output-editing and rewriting-model comparisons, and expanded experimental results.

C.3 Metacognitive Data Selection Details

  • Combining high- and low-scoring examples stabilized faithful calibration across most confidence bins, but gains did not reliably transfer to SciQ and MMLU.High-scoring-only selection improved high-confidence bins, while low-scoring-only selection improved low-confidence bins; on SimpleQA results were similar or better, unlike the low-confidence-bin degradation on SciQ and MMLU.Source: §C.3 · p. 43, Table 19 · p. 44, Table 20 · p. 45
  • Selecting equal numbers of highest- and lowest-scoring samples yielded the best Llama3.1-8B-Instruct performance, with smarter selection comparable and smarter active learning better than its basic variant.The selection process scores model responses and takes half the target training set from each score extreme; the comparison used 2,000 PopQA samples.Source: §C.3 · p. 43, Table 18 · p. 44

C.4.1 Constructing the Numerical-Linguistic Confidence Map

  • 0.99 precision and 1.0 recall were achieved by Gemini-2.5-Flash-Lite extracting hedges from sentences against author annotations; human-rated hedges from Fagen-Ulmschneider and Tao et al. [92] formed the confidence map.Hedges were sorted into 0.05-wide confidence bins, and rewriting randomly sampled 20 hedges from the target bin; bin sizes 5, 10, and 20 were comparable, with 20 best.Source: §C.4.1 · p. 44

C.3 Metacognitive Data Selection Details

  • Qwen3-8B improved markedly with smarter metacognitive selection, while the main RLMF experiments retained the simpler, more efficient strategy despite model-specific selection differences.The authors report that the simpler strategy can still yield significant gains even when training data is not the most efficacious for a particular model.Source: §C.3 · pp. 43–44, Table 21 · p. 45

C.4.2 Impact of Output Editing Approach

  • The single-pass comprehensive editing approach was comparable to two-step editing and was adopted because it best preserved numerical-setting faithful calibration while remaining cost-effective.The two-step alternative applied sentence-level revisions before comprehensive editing of the full response.Source: §C.4.2 · p. 45

C.4.3 Impact of Rewriting Model

  • Gemini-2.5-Flash-Lite, GPT-5-Mini, and Qwen3-8B were similarly effective rewriting models, supporting the approach’s compatibility beyond one editor.Comparable linguistic faithful calibration with Qwen3-8B also supports attributing the gains to RLMF rather than rewriting alone.Source: §C.4.3 · p. 45

D.1 Full Experimental Results (+More Models)

  • Full main-experiment results include Qwen3-1.7B and Qwen3-4B, while expanded tables report training-dataset and data-selection analyses.The full results are in Table 25; expanded dataset and selection-strategy results are in Tables 26 and 27, respectively.Source: §D.1 · p. 45

D.2 Analysis of Faithful Calibration Level · D.3 Analysis of Metacognitive Performance · D.4 Example Generations

RLMF produces faithful confidence across confidence levels and improves metacognitive performance during training. Example generations illustrate numerical alignment and more diverse, naturalistic linguistic uncertainty than FUT.

D.2 Analysis of Faithful Calibration Level

  • The hedge-sampling analysis evaluates how the number of sampled hedges per confidence bin affects Llama3.1-8B-Instruct’s PopQA faithful calibration.The supplied passage identifies the table’s scope but does not provide its results.Source: Table 22 · p. 46
  • A single-pass rewriting approach is compared with a two-step approach for linguistic faithful calibration averaged across datasets.The supplied passage identifies the comparison but does not provide its outcome.Source: Table 23 · p. 46
  • RLMF is consistently faithful across confidence levels, unlike the original and FUT-trained models, which misalign intrinsic and expressed confidence at low-confidence bins.Representative plots show the faithfulness-score distributions for Llama3.1-8B-Instruct with and without Stage 1 RLMF or FUT training.Source: §D.2 · pp. 45–46

D.3 Analysis of Metacognitive Performance

  • Metacognitive performance, measured by Zg, increases throughout RLMF training for Llama3.1-8B-Instruct and Qwen-3-8B.Representative plots show mean Zg across groups of completions sampled during GRPO.Source: §D.3 · p. 46

D.4 Example Generations

  • Numerical uncertainty is well aligned when expressed and intrinsic confidence are both high (> 0.5) or both low (< 0.5); opposite confidence levels indicate misalignment.Source: §D.4 · p. 46
  • Compared with FUT’s generic formulations, the approach produces more diverse, naturalistic linguistic uncertainty, a difference confirmed by human evaluation.Source: §D.4 · p. 46

E Human Evaluation Study Details

Expert annotators preferred the two-stage approach over FUT across four dimensions of uncertainty expression. The study compared responses to context-conditioned queries from three datasets and two models.

E Human Evaluation Study Details

  • The study covered 120 randomly sampled examples: 20 each from Natural Questions, SciQ, and SelfAware, with responses from Llama3.1-8B-Instruct and Qwen3-8B.Natural Questions used an educational-assistant context, SciQ a research and fact-checking context, and SelfAware a high-stakes professional context; SciQ responses were elicited without answer choices, and prompts framed context in the user’s first person.Source: E Human Evaluation Study Details · pp. 46–47
  • Three expert annotators compared randomized sets of three FUT responses and three RLMF-plus-Stage-2 responses per example, rating helpfulness, naturalness, coherence, and context-appropriate uncertainty communication.Annotators completed 10 held-out examples first, provided informed consent, and received no compensation.Source: E Human Evaluation Study Details · pp. 47–48
  • 98%, 98%, 95%, and 96% win rates over FUT on diversity, naturalness, helpfulness, and context suitability, respectively, accompanied by Krippendorff’s alpha of 0.93.Win rates count absolute wins with half-weight for ties.Source: E Human Evaluation Study Details · p. 48

Related papers

Loading 2606.32032v1…