PaperlayerView exact revision on arXiv

Source-linked AI summary

EpiCaR: Knowing What You Don't Know Matters for Better Reasoning in LLMs

Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Taesup Kim

arXiv:2601.06786v1cs.CL

TL;DR

Iterative self-training can improve reasoning accuracy while making LLMs overconfident and less able to represent uncertainty. EPICAR reframes training as epistemic learning, jointly optimizing reasoning and self-evaluation, and reports Pareto-superior accuracy and calibration across capable Llama-3 and Qwen-3 models, with broader generalization and lower inference cost.

  • Problem

    Positive-only reasoning self-training can incur a calibration cost, causing overconfidence and loss of uncertainty representation.

  • Method

    EPICAR jointly trains reasoning performance and explicit self-evaluation within an iterative supervised fine-tuning framework.

  • Results

    EPICAR achieves Pareto-superiority in accuracy and calibration across Llama-3 and Qwen-3, generalizes to GSM8K and MBPP, and matches STaR’s K=30 performance with K=10 samples in capable models.

  • Takeaways & Limitations

    Internalized calibration can improve reliability while making inference-time scaling more efficient in models with sufficient reasoning capacity.

  • Takeaways & Limitations

    The evaluation covers objectively verifiable mathematics and code domains, while applicability to subjective or ambiguous tasks remains uncertain.

Abstract

from arXiv · show

Improving the reasoning abilities of large language models (LLMs) has largely relied on iterative self-training with model-generated data. While effective at boosting accuracy, existing approaches primarily reinforce successful reasoning paths, incurring a substantial calibration cost: models become overconfident and lose the ability to represent uncertainty. This failure has been characterized as a form of model collapse in alignment, where predictive distributions degenerate toward low-variance point estimates. We address this issue by reframing reasoning training as an epistemic learning problem, in which models must learn not only how to reason, but also when their reasoning should be trusted. We propose epistemically-calibrated reasoning (EpiCaR) as a training objective that jointly optimizes reasoning performance and calibration, and instantiate it within an iterative supervised fine-tuning framework using explicit self-evaluation signals. Experiments on Llama-3 and Qwen-3 families demonstrate that our approach achieves Pareto-superiority over standard baselines in both accuracy and calibration, particularly in models with sufficient reasoning capacity (e.g., 3B+). This framework generalizes effectively to OOD mathematical reasoning (GSM8K) and code generation (MBPP). Ultimately, our approach enables a 3X reduction in inference compute, matching the K=30 performance of STaR with only K=10 samples in capable models.

1 Introduction

LLMs can achieve strong reasoning performance while remaining poorly calibrated, confidently producing incorrect conclusions. Existing positive-only self-training may improve accuracy but impose a calibration cost, whereas EPICAR jointly targets reasoning and reliability.

  • EPICAR: EPICAR drives Llama-3 and Qwen-3 models into a Pareto-superior region with simultaneous gains in task performance and uncertainty calibration.The figure compares relative accuracy improvement and ECE reduction against the base model.
  • The calibration problem: LLMs may assign near-certainty to incorrect reasoning outcomes, creating a gap between accuracy and self-knowledge.This mismatch is especially concerning when knowing when not to trust a model matters alongside producing correct answers.
  • The calibration cost: Positive-only iterative self-training reinforces successful reasoning paths but can drive predictive behavior toward low-variance point estimates.The resulting model may discard distributional tails needed for reliable uncertainty estimation.
  • Existing approaches: Slow Thinking methods can improve confidence expression through self-verification and backtracking, but their reliability depends on increased inference-time computation.These approaches therefore address reliability with additional inference overhead rather than solely by changing the base policy.
  • EPICAR: EPICAR reframes reasoning training as epistemic learning, jointly optimizing reasoning performance and calibration through explicit self-evaluation signals.Within iterative SFT, the framework aims to navigate the trade-off between accuracy and overconfidence.

2 Related Work

Related work improves reasoning, verification, confidence estimation, or calibration through separate mechanisms. EPICAR instead internalizes calibration within the generator’s unified reasoning objective.

  • Iterative reasoning and self-improvement: STaR and ReST fine-tune models on self-generated correct paths, while V-STaR also uses incorrect solutions to train a separate verifier.MASA targets metacognitive signals by aligning predicted solution attributes with rollout statistics.
  • Calibration cost: Alignment research has identified calibration cost as a pervasive rise in overconfidence, even when capability effects vary across benchmarks.This frames reliability degradation as distinct from conventional accuracy-focused alignment-tax analyses.
  • Calibration and uncertainty estimation: Calibration methods range from temperature scaling and verbalized confidence to inference-time ensembles, but verbalized confidence is prompt-sensitive and ensembles are computationally expensive.Long-form reasoning further challenges verbalized confidence approaches.
  • Calibration and uncertainty estimation: Slow Thinking and hidden-state correctness signals motivate methods that improve confidence through self-verification and reasoning-aware representations.EPICAR differs by balancing reasoning and self-evaluation directly within a unified SFT task.

3 Preliminaries

The paper formulates calibrated reasoning as maximizing answer correctness while producing reliable confidence, then evaluates calibration, predictive quality, and discrimination. It frames positive-only training as epistemic signal truncation that can cause model collapse.

  • Problem formulation: Calibrated reasoning maximizes Pθ(y|x, r) while requiring a reliable confidence score c ∈[0, 1].The formulation treats reasoning correctness and confidence quality as simultaneous objectives.
  • Problem formulation: Verbalized confidence estimates correctness by normalizing the model’s affirmative and negative answers to a binary self-evaluation query.The confidence is defined from the probabilities of “yes” and “no” given the problem, reasoning path, and predicted answer.
  • Evaluation metrics: ECE measures the weighted discrepancy between average accuracy and average confidence across confidence bins, with ECE = 0 representing perfect calibration.Predictions are partitioned into equally spaced intervals according to confidence.
  • Evaluation metrics: The Brier Score is the mean squared error between predicted confidence and binary correctness, with lower values indicating better-calibrated and more accurate predictions.It provides a holistic view by combining calibration, resolution, and uncertainty.
  • Evaluation metrics: AUROC evaluates whether correct answers receive higher confidence than incorrect answers, independently of logit scaling or temperature shifts.Unlike ECE, it measures discriminative reliability rather than absolute alignment.
  • Epistemic cost: Positive-only training truncates epistemic information by exposing the model to correct reasoning paths without the decision boundary separating correct and incorrect paths.This can produce low-variance point estimates and pathologically high confidence in flawed generations.

4 Methodology

EPICAR adds epistemic calibration to iterative self-training by combining reasoning reinforcement with explicit self-evaluation. Correct and incorrect generated paths provide differentiated training signals while formatting controls reduce mislabeled examples.

  • Framework: EPICAR internalizes epistemic calibration in iterative self-training without auxiliary models or inference-time compute scaling.Its unified objective balances problem-solving with self-evaluation.
  • Internalizing evaluation: Correct generated paths train reasoning and receive “yes” self-evaluation labels, whereas incorrect paths are excluded from reasoning reinforcement and receive “no” labels.This preserves incorrect generations as negative evidence for the evaluation task.
  • Internalizing evaluation: Training on both correct and incorrect paths is intended to expose signals needed for self-evaluation that standard reasoning models may encode but non-reasoning models often lack.The approach is motivated by evidence that correctness information can reside in hidden states.
  • Data quality: Rigid completion strings and stateful decoding handling reduce formatting failures that could incorrectly label valid reasoning paths as “no.”The implementation addresses premature termination and unclosed formatting tags during generation.

5 Experimental Setup

The experiments evaluate EPICAR across MATH, OOD GSM8K, and MBPP using Llama-3 and Qwen-3 families, with accuracy and calibration metrics against several baselines. Inference-time scaling tests compare self-consistency aggregation with confidence-informed weighted voting.

  • Datasets: Experiments use MATH1 for three iterative-training rounds, with GSM8K and MBPP testing out-of-distribution and cross-domain generalization.MATH-500 is used for scaling analysis, while MBPP uses 3-shot prompting and sandboxed functional-correctness evaluation.
  • Models and Baselines: The study evaluates Llama-3 and Qwen-3 families, focusing on 8B variants for MBPP and ablations.Baselines include the Base Model, STaR, Slow Thinking, and post-hoc Model Merging.
  • Inference-time Scaling Protocol: Inference-time scaling samples K ∈ {1, 10, 30} reasoning paths and compares majority-vote Self-Consistency with confidence-weighted CISC.CISC weights paths using the model’s self-assessed confidence V(r_i).
  • Implementation: Performance is measured with Accuracy, AUROC, ECE, and Brier Score, including ECE after Temperature Scaling.Training uses LoRA on 4x NVIDIA H100 GPUs, with hidden-feature updates supporting verbalized confidence.

6 Results and Analysis

EPICAR improves the accuracy–reliability trade-off across model families, with strongest gains at larger scales, and generalizes to OOD mathematics and code generation. Its calibrated confidence also improves inference-time scaling efficiency, enabling comparable performance with fewer reasoning paths.

  • Main Findings: Accuracy and Reliability Synergy: EPICAR enables models to navigate the reasoning-reliability frontier more effectively, particularly as model scale increases.This pattern appears across both Llama-3 and Qwen-3 families.
  • Main Findings: Accuracy and Reliability Synergy: STaR drops Llama-3-1B AUROC to 0.491, whereas EPICAR recovers it to 0.573 despite a slight accuracy trade-off at this scale.For Llama-3-3B, EPICAR reduces ECE from 0.376 to 0.108 and reaches a Brier Score of 0.097.
  • Main Findings: Accuracy and Reliability Synergy: EPICAR’s calibrated framework stabilizes Slow Thinking on Qwen-3-8B, achieving 55.56% performance, while Qwen-3-4B gains no improvement over STaR.The results indicate that leveraging self-evaluation signals depends on sufficient reasoning capacity.
  • Weight-space Interpolation & Merging: Ours + Merging achieves the highest Llama-3-8B accuracy of 15.02%, while Qwen-3-1.7B reaches ECE (+TS) of 0.018.These results show compatibility between EPICAR reliability signals and weight-space interventions.
  • Out-of-Distribution Generalization: EPICAR improves reasoning accuracy while maintaining or enhancing AUROC on zero-shot GSM8K, despite training exclusively on MATH.GSM8K therefore tests OOD mathematical generalization.
  • Discriminative Power vs. Absolute Calibration: EPICAR’s dual-objective training improves discriminative reliability, but absolute verbalized probabilities can remain misaligned with empirical accuracy.Explicit optimization of verbalized probability logits is identified as future work.
  • Generalization to Other Tasks: On MBPP, EPICAR recovers Llama-3-8B AUROC from STaR’s 0.523 to 0.538 and lowers Brier Score from 0.387 to 0.246.ECE (+TS) reaches 0.113 for Llama-3-8B and 0.059 for Qwen-3-8B.
  • Inference-time Scaling Performance: With CISC, EPICAR’s 8B model matches or exceeds STaR’s K = 30 performance using K = 10 samples, reducing inference compute by 3×.The weighted ensemble uses internalized confidence to suppress erroneous consensus paths.

7 Conclusion

The paper presents EPICAR as epistemic learning for reasoning models, combining accuracy reinforcement with explicit self-evaluation. Across Llama-3 and Qwen-3, it reports Pareto-superior performance and calibration, OOD generalization, and 3× lower inference compute for comparable scaled reasoning performance.

  • Conclusion: EPICAR reframes reasoning training as epistemic learning by internalizing a dual objective of accuracy reinforcement and explicit self-evaluation.The framework jointly targets reasoning performance and calibration.
  • Conclusion: Across Llama-3 and Qwen-3, EPICAR achieves Pareto-superiority in performance and calibration while generalizing to GSM8K and MBPP.The conclusion reports robustness across OOD mathematics and code generation.
  • Conclusion: Confidence-weighted ensembling with CISC reduces inference compute by 3× while matching high-sample-count performance.The reported comparison matches STaR’s K = 30 performance with significantly fewer reasoning paths.

Limitations

The paper’s limitations concern domain scope, model capacity, OOD absolute calibration, reliance on verbalized confidence, and comparisons with alternative training or verification methods.

  • Domain Scope and Verification: Evaluation is centered on mathematics and code generation with objective, automated verification, leaving subjective or ambiguous domains untested.
  • Model Scale and Capacity Constraints: Experiments cover models up to 8B parameters, and very low-capacity models may lack enough successful paths to learn accurate self-evaluation.For Llama-3-1B, sparse successful reasoning paths are associated with diminished benefits relative to STaR.
  • Generalization Gap in Absolute Calibration: OOD performance can preserve relative ranking while absolute verbalized probabilities remain more volatile than in-distribution calibration.The limitation distinguishes superior AUROC from higher Raw ECE volatility on OOD tasks such as GSM8K.
  • Comparison with RL and Verifier-based Methods: Unlike verifier-based alternatives, EPICAR uses one model for generation and self-evaluation, making its integration dependent on that model’s internal capacity.The unified design is associated with a reported 3× reduction in inference compute, while the single-model approach may remain capacity-bounded.
  • Sensitivity of the Verbalization Paradigm: Verbalized confidence improves calibration but remains sensitive to prompting, elicitation templates, and linguistic context.The authors identify prompt sensitivity as a boundary on the robustness of self-evaluation signals.

Ethical Considerations

The paper highlights risks from over-reliance, bias amplification, and training-related environmental costs, while also examining compute-performance scaling behavior.

  • Ethical Considerations: High reported confidence could encourage over-reliance, and confident hallucinations may still occur in high-stakes decisions.The authors state that EPICAR reduces overconfidence but is not infallible.
  • Ethical Considerations: Iterative self-training on model-generated data may amplify biases from the pretrained state or training distribution.The authors recommend rigorous bias-detection and mitigation protocols alongside the method.
  • Ethical Considerations: Training uses multiple rounds of generation and fine-tuning, increasing computational costs relative to single-pass SFT.The paper contrasts this training cost with reduced inference-time compute from efficient scaling.
  • Scaling Evaluation: The scaling evaluation varies sampled reasoning paths across K ∈ {1, 10, 30} to map the compute-performance frontier.
  • Scaling Results: At 8B parameters, pairing EpiCaR with CISC reaches 25.40% for Llama-3 and 59.80% for Qwen-3 at K = 30.The paper presents these results as state-of-the-art and links them to internalized reliability signals.
  • Error Suppression and Saturation: For Qwen-3 8B, STaR saturates at 57.80% for K = 30, whereas the calibrated model continues scaling by downweighting common erroneous paths.
  • Critical Mass Threshold: Calibration benefits are not uniform across scales: for Llama-3 1B with baseline accuracy ∼1.80%, the gap over STaR diminishes.The authors associate this regime with too few correct paths for learning discriminative self-evaluation features.

A.2 Full Reliability Analysis of Scaling: SC vs. CISC

The scaling analysis compares unweighted and confidence-weighted self-consistency across accuracy and reliability, finding that STaR reliability can collapse as sampling increases while EpiCaR preserves a stronger reliability-compute trade-off.

  • Confidence Estimation: CISC derives ensemble confidence by combining answer frequency with each path’s internalized verbalized confidence through temperature-scaled normalization.As T approaches infinity, CISC becomes uniform-weight self-consistency; as T approaches zero, it selects the highest-confidence path.
  • Reliability Scaling: STaR’s AUROC drops from 0.7895 at K = 5 to 0.7387 at K = 30 for Llama-3 8B.The authors attribute this degradation to incorrect reasoning paths forming deceptively high-confidence consensus as more samples are added.
  • Metric Divergence: ECE generally improves with larger K, whereas AUROC tends to decay, producing divergent reliability trends.The paper links this pattern to statistical smoothing benefiting ECE and consensus-driven noise narrowing the gap between correct and incorrect paths.
  • EpiCaR Scaling: EpiCaR mitigates AUROC decay relative to baselines as inference compute scales.Its confidence is tied to the intrinsic logic of each reasoning path rather than only to answer frequency.

J Ablation Study: Impact of Adaptive Injection Decoding

The ablation shows that Adaptive Injection Decoding is structurally important because removing it severely degrades reasoning accuracy and calibration by turning formatting errors into incorrect training labels.

  • Ablation Result: Removing AID causes Llama-3-8B accuracy to collapse to 2.56%.The ablation compares the full method with a variant that treats formatting errors as ordinary incorrect samples.
  • Failure Mechanism: Without AID, minor formatting errors in valid reasoning paths are mislabeled as negative self-evaluation signals.The resulting label noise confuses self-evaluation and destabilizes the reinforcement loop.
  • AID Function: AID filters formatting noise so negative signals reflect logical failures rather than syntactic errors.The authors characterize AID as a sanitization process rather than a source of reasoning bias.

K Full Results for Model Merging

Model merging evaluates interpolation between base and fine-tuned weights across Llama-3 and Qwen-3 families, finding that intermediate coefficients often improve the accuracy-calibration balance and that EpiCaR remains more reliable than STaR.

  • Interpolation Effects: Intermediate λ values, often 0.6 ≤ λ ≤ 0.8, provide a superior balance between task accuracy and probabilistic calibration for most model scales.The study treats weight interpolation as an effective regularizer between base and fine-tuned models.
  • Comparison: EPICAR consistently outperforms STaR in AUROC, ECE, and Brier Score across tested architectures.The advantage is especially pronounced in discriminative reliability at the 4B and 8B scales.
  • Calibration Stability: EPICAR tends to show more stable or improving Brier scores as λ increases, while STaR’s ECE often fluctuates.This pattern is reported as evidence that the calibration objective better aligns confidence with predictive correctness.
  • Scaling Trends: Larger 8B models benefit more from weight-space intervention, while smaller Llama-3.2-1B models show mixed calibration-error results but improved AUROC.The authors describe EpiCaR weights as a more robust initialization for merging than standard self-training weights.
  • Reliability Diagrams: Reliability diagrams across MATH, GSM8K, and MBPP compare verbalized confidence with empirical accuracy using ten confidence bins.EPICAR shows better alignment and higher AUROC than STaR, whose lower-accuracy bins frequently contain overconfident clusters.
Loading 2601.06786v1…