PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan

arXiv:2606.32032v1cs.CLcs.AI

TL;DR

LLMs often misrepresent their uncertainty and fail to recognize their capability limits, motivating methods that use self-assessed performance as training supervision. The paper introduces RLMF and metacognitive data selection within a decoupled numerical-to-linguistic calibration framework. Across diverse tasks, it reports state-of-the-art faithful calibration, preserved accuracy, and improvements of up to 63% over standard RL.

  • Problem

    LLMs exhibit high-confidence hallucinations, fail to recognize knowledge boundaries, and misrepresent internal uncertainty, undermining trustworthiness and reliability.

  • Method

    RLMF scales preference-optimization signals using self-judgment quality, while metacognitive data selection chooses training examples from model self-assessments within a decoupled numerical-to-linguistic calibration pipeline.

  • Results

    The framework achieves state-of-the-art faithful calibration across diverse models and tasks while preserving task accuracy and factual calibration, and RLMF improves post-training results by up to 63% over standard RL.

  • Takeaways & Limitations

    Metacognitive performance provides an effective training signal for improving uncertainty calibration and models’ ability to assess and express their capability limits.

  • Takeaways & Limitations

    Improved self-assessment of performance does not constitute broad metacognitive awareness, which includes many other capabilities.

Abstract

from arXiv · show

Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes. Yet LLMs exhibit systemic deficiencies in key metacognitive faculties: they hallucinate with high confidence, fail to recognize knowledge boundaries, and misrepresent their internal uncertainty--undermining trustworthiness and reliability. Since monitoring task performance and adapting behavior accordingly are central to metacognition, we posit that models capable of accurately judging their own performance are better positioned to improve it. We operationalize this idea via two novel mechanisms: reinforcement learning with metacognitive feedback (RLMF), a paradigm to refine completion rankings during preference optimization based on the quality of a model's self-judgments of performance, and metacognitive data selection, which uses similar self-judgments to identify high-value training examples, outperforming naive active learning. We apply these innovations to the problem of faithful calibration (FC), a task that is itself fundamentally metacognitive: the goal is to align expressed with intrinsic uncertainty, difficult even for frontier LLMs. We adopt a two-stage, decoupled approach, first using these methods to calibrate the faithfulness of models' self-reported confidence scores, then mapping to natural, context-adaptable linguistic uncertainty via targeted output editing. Extensive experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks while preserving accuracy. Further, RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits. This positions RLMF as a promising paradigm to enhance LLM metacognition toward improved abilities and alignment, and suggests metacognitive performance as an effective RL signal to overcome limits of prior intrinsic feedback methods.

1 Introduction

The paper frames LLM metacognition as deficient but proposes using models’ self-assessed performance to improve both task outcomes and faithful uncertainty communication. It introduces RLMF, metacognitive data selection, and a decoupled numerical-to-linguistic calibration pipeline.

  • LLMs often fail to recognize knowledge boundaries, produce high-confidence hallucinations, and faithfully represent internal uncertainty.
  • RLMF rewards strong outputs and accurate self-judgments of performance, while metacognitive data selection chooses examples using those self-assessments.RLMF scales completion advantages according to self-judgment quality; data selection samples from both high- and low-scoring examples.
  • Faithful calibration aligns expressed uncertainty with intrinsic confidence, distinct from factual calibration based on empirical accuracy.
  • The two-stage framework first calibrates numerical confidence faithfulness, then maps scores to natural, context-adaptable linguistic uncertainty through targeted rewriting.The stages are decoupled so linguistic preferences and contexts can be handled without repeating costly RL training.
  • Across 10 tasks and 6+ content domains, the framework achieves state-of-the-art faithful calibration while preserving task accuracy and factual calibration.The evaluation reports robust cross-task generalization despite training on a single dataset.
  • RLMF strengthens post-training results by up to 63% over standard RL while improving models’ metacognitive awareness.

2 Related Work

Prior work identifies metacognition and faithful calibration as important but incompletely addressed capabilities in LLMs. The paper extends this literature by prioritizing metacognitively stronger completions and targeting holistic numerical and linguistic uncertainty calibration.

  • Metacognition in LLMs: Metacognition concerns monitoring and controlling one’s cognitive processes, and prior studies link its deficiency in LLMs to hallucinations and misaligned expressions.
  • Metacognition in LLMs: The paper prioritizes RL completions for which models exhibit stronger metacognitive capabilities, subject to task-level reward signals.
  • Reinforcement Learning with Internal Feedback: Internal-feedback methods derive unsupervised RL rewards from models themselves, including confidence-related signals such as self-certainty and entropy.
  • Faithful Calibration of LLMs: Faithful calibration addresses models whose expressed confidence diverges from internal beliefs, a problem that threatens user reliance and safe AI use.
  • Faithful Calibration of LLMs: Existing linguistic-calibration approaches provide modest, limited-scope improvements and do not fully address naturalness, coherence, or dynamically varied hedging.

3 Method

The method integrates metacognitive feedback into reinforcement learning and data selection, then uses targeted rewriting to express calibrated uncertainty linguistically. Its RL objective combines faithful alignment with correctness, factual calibration, and formatting constraints.

  • Method overview: The framework combines metacognitive feedback, targeted rewriting, and RL to calibrate numerical uncertainty and map it into linguistic expressions.
  • Reinforcement learning setup: Each query produces grouped candidate completions containing sentence-level confidence scores.
  • Reinforcement learning setup: A composite reward combines faithful alignment, correctness, factual calibration, and format adherence before computing relative completion advantages.
  • RLMF: RLMF scales advantages using how accurately the model predicts each completion’s faithful-calibration performance.
  • RLMF: Above-average faithful completions receive additional scaling by metacognitive accuracy, while remaining stronger than below-average completions under the stated design.
  • Metacognitive data selection: Metacognitive data selection scores training examples by perceived alignment between linguistic and internal confidence, then selects equal-sized high- and low-scoring groups.
  • Targeted rewriting: Targeted rewriting supplies original responses, score-matched hedge candidates, and task or user context to guide coherent linguistic uncertainty expression.

4 Experimental Setup

The experiments evaluate faithful calibration across diverse datasets, model families, and baselines using cMFG*, with training primarily conducted on PopQA. The setup reports both numerical and linguistic FC and examines data-selection and pre-SFT choices.

  • FC is evaluated on 10 datasets spanning diverse formats, content domains, and difficulty levels, using 1,000 test examples per dataset.
  • The study applies its approach to Qwen3 models at 1.7B, 4B, and 8B parameters and Llama3.1-Instruct at 8B.
  • Models are trained primarily on 2,000 metacognitively selected PopQA training samples to assess out-of-distribution generalization across tasks and domains.
  • Table 1 reports results against baselines and includes an RL ablation without metacognitive advantage scaling.
  • The comparison includes numerical and linguistic FC results, while prior-method comparisons are restricted to linguistic FC because earlier work studied only that setting.
  • cMFG* is proposed as a refinement of cMFG that reduces sensitivity to intrinsic-confidence distributions and ranges from 0 to 1, with 1 indicating perfect FC.

5 Results

RLMF improves faithful calibration across models, tasks, and uncertainty-expression modes while preserving task accuracy and factual calibration. Results also support better self-assessment, effective metacognitive data selection, and more natural linguistic uncertainty.

  • 29% and 25% average cMFG* gains over prompting and SFT baselines, respectively, yield cMFG*≥0.80 in every evaluated setting.
  • RLMF generalizes from PopQA training to complex MATH reasoning and out-of-distribution SimpleQA while preserving task accuracy and factual calibration.
  • 37%, 17%, and 25% average FC gains over GPT-5, Gemini-3.1-Pro, and Gemini-3-Flash, respectively, are reported even with specialized prompting.
  • Up to 63% improvement over standard RL is reported, alongside progressively improved metacognitive performance during RLMF training.
  • Metacognitive data selection achieves the best cMFG* among random and ground-truth-FC active selection while preserving accuracy and factual calibration.
  • Improved self-assessment of performance should not be equated with broad metacognitive awareness, which includes many other capabilities.
  • Human evaluations report 98%, 98%, 95%, and 96% win rates over FUT for diversity, naturalness, helpfulness, and contextual suitability, with inter-annotator agreement of 0.93.

6 Conclusion

The paper presents RLMF and metacognitive data selection as a two-stage route to generalizable faithful calibration that preserves accuracy and improves uncertainty communication. It also introduces cMFG* to reduce estimation bias in restricted-confidence settings.

  • RLMF refines preference-optimization rankings using models’ implicit performance judgments, while metacognitive data selection identifies more effective training data than simple active learning.
  • The two-stage framework aligns numerical and linguistic uncertainty with intrinsic confidence across diverse models and tasks while preserving task accuracy and factual calibration.
  • The framework improves self-assessment, produces faithful confidence scores, and modulates linguistic uncertainty naturally and context-appropriately.
  • cMFG* improves on cMFG by removing estimation bias when models’ intrinsic confidence occupies a limited range.

Ethics Statement

The paper frames faithful uncertainty communication as important for trustworthy AI, while distinguishing it from factual calibration and emphasizing adaptable linguistic expression.

  • Faithful uncertainty expression is presented as important for improving the trustworthiness and reliability of LLMs.
  • Existing calibration work primarily aligns confidence with externally judged accuracy, whereas this work targets alignment with models’ internal beliefs.
  • Black-box numerical confidence methods assess outputs but do not incorporate the expressive, variable linguistic uncertainty needed for human communication.
  • Linguistic uncertainty approaches include prompting or training models to verbalize confidence and mapping numerical scores to uncertainty phrases.

B.1 Model & Training Details

The experiments use open and proprietary LLMs across diverse datasets, with supervised preparation, LoRA-based GRPO/RLMF training, and fixed inference settings.

  • Models: The study evaluates Qwen3 models from 1.7B to 8B, Llama3.1-Instruct 8B, and proprietary Gemini and GPT-5 baselines.
  • Inference: Inference uses a 256-token maximum, temperature 1.0 unless specified, and no thinking mode where disabling it is possible.
  • Training: RLMF modifies GRPO advantage computation and uses LoRA adapters to alter uncertainty expression without fundamentally changing downstream capabilities.
  • Training: Each RLMF experiment uses 6 GPUs: 4 for training and 2 for completion sampling and judge-model confidence estimation.
  • Training: Before RLMF, models undergo supervised fine-tuning to learn the output format, cross-task generalization, and task-specific length constraints.
  • Datasets: Benchmarks span knowledge-intensive QA, hallucination evaluation, broad knowledge and reasoning, and science multiple-choice tasks.

B.2.1 Dataset Abbreviations

This appendix documents dataset-result abbreviations and the prompting format used to elicit sentence-level numerical uncertainty across tasks.

  • Dataset abbreviations: Table 4 provides dataset-name abbreviations used in result tables reporting per-dataset cMFG* scores.
  • Prompting: The numerical-uncertainty prompt requires concise answers with each sentence enclosed in tags and followed by a confidence score from 0.00 to 1.00.
  • Prompting: Task prompts share a base query format while adding task-specific output instructions, including randomized answer ordering for multiple-choice tasks.
  • Prompting: The pipeline uses separate prompts for eliciting responses, judging FC performance during RLMF and data selection, and performing Stage 2 response editing.

B.4 Metrics

The paper measures faithful calibration by comparing expressed and intrinsic confidence, introduces cMFG* to address cMFG’s support and empty-bin problems, and reports accuracy-related metrics separately.

  • Additional metrics: Accuracy is measured with an LLM-as-a-Judge setup, while the Brier Score quantifies squared-error alignment between confidence and correctness.
  • Confidence estimation: Intrinsic confidence is estimated from consistency across sampled responses using sentence-level NLI judgments.
  • Faithful calibration: Faithful calibration measures alignment between expressed confidence and intrinsic confidence rather than confidence with factual accuracy.
  • cMFG: The cMFG metric aggregates sentence- or assertion-level faithfulness across dataset samples while conditioning on intrinsic confidence.
  • cMFG limitations: cMFG can produce unreliable estimates for empty bins and penalize models whose confidence occupies only part of the [0, 1] range.
  • cMFG*: cMFG* uses equal-mass confidence bins with width-proportional weighting to improve statistical reliability and uniformity over the confidence axis.

C.1.1 Reward Design

The reward design combines faithful calibration, factual calibration, correctness, and format adherence, while experiments compare reward formulations, weighting, prompts, and optimization choices.

  • Reward components: Faithfulness reward minimizes the squared gap between predicted and gold confidence for each sentence.It is an inverted Brier Score analog, maximized when expressed and intrinsic confidence align.
  • Reward components: Factual calibration and correctness rewards preserve factual calibration and task performance alongside faithful confidence alignment.The factual signal uses a Brier Score-based formulation, while correctness is binary completion accuracy.
  • Reward components: Format rewards encourage exact target-output conformity while providing partial penalties for lesser violations.The strict reward gives 1 for perfect conformity and -1 otherwise; the soft reward supplies finer-grained penalties.
  • Penalty design: Alternative length and gibberish penalties were abandoned after unfruitful experiments, while RLMF reduced gibberish without associated penalties.Standard RL frequently produced malformed text after unclosed tags despite extensive hyperparameter searches.
  • Reward weighting: The final reward is a weighted sum that prioritizes faithful calibration with wfaith = 12, above weights of 3 for strict and soft format rewards.Factual calibration and correctness each receive weight 1.
  • Ablations: Ablations examine reward removal, alternative faithfulness formulations, system prompts, advantage normalization, and training-data size.The supplied passages identify these comparisons but do not report their complete numerical tables.

C.1.5 Impact of Training Data Size

This section evaluates how RLMF design choices affect faithful calibration, including data size, pre-SFT, metacognitive scaling, reward composition, and reward-hacking safeguards.

  • Training data size: RLMF main experiments use 2000 metacognitively selected training samples, while 1000 and 4000 samples produce slightly worse but still above-baseline performance.The comparison uses Llama3.1-8B-Instruct trained on PopQA.
  • Pre-SFT: Pre-SFT strengthens numerical faithful calibration and cross-task generalization for both combined and isolated metacognitive methods.It also helps models learn the target sentence-and-confidence output format.
  • Reward composition: Using the metacognitive signal alone helps, but does not achieve state-of-the-art faithful calibration while preserving task accuracy.Combining faithfulness and metacognitive rewards is necessary, with the metacognitive reward more effective as a lower-weight accessory.
  • Reward safeguards: Very small τ values can make metacognitive supervision sparse and inconsistent, and whether this limitation extends to larger models remains open.The authors specifically identify τ = 0.05 as potentially too small when near-matches are rare.
  • Metacognitive feedback: Metacognitive advantage scaling is comparable to an additional metacognitive reward while requiring fewer hyperparameters.The main formulation applies scaling when faithfulness exceeds the group-level mean.

C.3 Metacognitive Data Selection Details

Metacognitive data selection ranks examples using model self-judgments and compares this strategy with active learning, sampling variants, and model-dependent scoring approaches.

  • Selection procedure: The selection procedure asks the model for a response and metacognitive score, then chooses equal numbers of highest- and lowest-scoring samples.Scores range from 0–100, rather than 0.0–1.0, because early experiments found the latter clustered tightly.
  • Baselines: Active learning selects examples with poor faithful calibration, whereas metacognitive selection uses the model’s judgment of its own performance.Averaged scores over 20 responses provide a more robust active-learning comparison.
  • Selection results: Among tested strategies, combining highest- and lowest-scoring samples gives the best performance for Llama3.1-8B-Instruct.The comparison trains on 2000 PopQA samples using standard GRPO with pre-SFT and no metacognitive advantage scaling.
  • Selection results: Combining extremes stabilizes faithful calibration across confidence bins, while highest-only and lowest-only sampling specialize at high and low intrinsic confidence.The bin-level analysis reports complementary strengths for the two single-sided strategies.
  • Model dependence: Qwen3-8B benefits markedly more from the smarter selection variant than Llama3.1-8B-Instruct.This suggests self-performance judgments for identifying useful training data vary across models and inference settings.
  • Model dependence: The simpler official selection strategy remains computationally efficient and can still yield significant gains despite not being optimal for every model.This conclusion is reported for the main RLMF experiments.

D.2 Analysis of Faithful Calibration Level

The analysis examines numerical and linguistic faithful calibration, training-time metacognitive performance, and the effect of rewriting choices and editing models.

  • Numerical calibration: RLMF produces confidence alignment across all model-confidence levels, unlike the original and FUT-trained models’ low-confidence misalignment.Histograms and violin plots provide qualitative support for the numerical calibration results.
  • Rewriting: Single-pass comprehensive rewriting and two-step rewriting achieve comparable linguistic faithful calibration.The single-pass approach is preferred because it better preserves the numerical calibration level.
  • Training dynamics: Metacognitive performance, quantified by Zg, increases throughout RLMF training for Llama3.1-8B-Instruct and Qwen3-8B.The analysis tracks mean Zg across completion groups sampled during GRPO.
  • Linguistic calibration: The linguistic outputs from the proposed approach are more diverse and naturalistic than FUT’s generic formulations.Human evaluation confirms the difference in linguistic expression.

E Human Evaluation Study Details

The human evaluation compared responses from FUT with outputs from the proposed RLMF-based framework, focusing on how naturally and appropriately they communicated uncertainty across contexts. Three expert annotators rated 120 examples, and the proposed approach achieved very high win rates across all evaluated dimensions.

  • Study setup: Three expert annotators evaluated 120 randomly sampled examples from Natural Questions, SciQ, and SelfAware.Responses came from Llama3.1-8B-Instruct and Qwen3-8B.
  • Results: Krippendorff’s alpha was 0.93, indicating high inter-annotator agreement.The study was conducted with three expert annotators who provided preference annotations.
  • Study setup: Annotators compared FUT responses with responses trained using RLMF and metacognitive data selection, then rewritten through Stage 2.Response ordering and model-set naming were randomized to reduce presentation effects.
  • Evaluation criteria: The annotation task measured diversity, naturalness, helpfulness, and contextual suitability of communicated uncertainty.Each example included the original query and its associated user context.
  • Results: 98%, 98%, 95%, and 96% win rates favored the proposed approach over FUT for diversity, naturalness, helpfulness, and context suitability, respectively.Win rates counted absolute wins and assigned half-weight to ties.
Loading 2606.32032v1…