PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Reasoning Models More Prone to Hallucination?

Zijun Yao, Yantao Liu, Yanxu Chen, Jianhui Chen, Junfeng Fang, Lei Hou, Juanzi Li, Tat-Seng Chua

arXiv:2505.23646v1cs.CLcs.LG

TL;DR

The paper asks whether long-CoT reasoning reduces hallucination in fact-seeking tasks and studies the question through evaluation, behavior analysis, and uncertainty probing. It finds that factuality depends strongly on the post-training pipeline: complete cold-start SFT plus RL generally helps, whereas SFT-only and RL-only pipelines are associated with more hallucination.

  • Problem

    The paper asks whether reasoning models are more prone to hallucination in fact-seeking tasks and why reported results differ across models.

  • Method

    The paper evaluates LRMs on fact-seeking benchmarks, analyzes Flaw Repetition and Think-Answer Mismatch, and probes uncertainty encoded in hidden states.

  • Results

    Complete cold-start SFT plus verifiable-reward RL generally improves factuality, while SFT-only and RL-only pipelines are associated with increased hallucination and miscalibrated uncertainty.

  • Takeaways & Limitations

    Hallucination patterns are linked to post-training strategy, cognitive reasoning behaviors, and misalignment between model uncertainty and factual accuracy.

  • Takeaways & Limitations

    The study mainly analyzes released LRMs statically rather than controlling training data, update steps, and hyperparameters through its own post-training runs.

Abstract

from arXiv · show

Recently evolved large reasoning models (LRMs) show powerful performance in solving complex tasks with long chain-of-thought (CoT) reasoning capability. As these LRMs are mostly developed by post-training on formal reasoning tasks, whether they generalize the reasoning capability to help reduce hallucination in fact-seeking tasks remains unclear and debated. For instance, DeepSeek-R1 reports increased performance on SimpleQA, a fact-seeking benchmark, while OpenAI-o3 observes even severer hallucination. This discrepancy naturally raises the following research question: Are reasoning models more prone to hallucination? This paper addresses the question from three perspectives. (1) We first conduct a holistic evaluation for the hallucination in LRMs. Our analysis reveals that LRMs undergo a full post-training pipeline with cold start supervised fine-tuning (SFT) and verifiable reward RL generally alleviate their hallucination. In contrast, both distillation alone and RL training without cold start fine-tuning introduce more nuanced hallucinations. (2) To explore why different post-training pipelines alters the impact on hallucination in LRMs, we conduct behavior analysis. We characterize two critical cognitive behaviors that directly affect the factuality of a LRM: Flaw Repetition, where the surface-level reasoning attempts repeatedly follow the same underlying flawed logic, and Think-Answer Mismatch, where the final answer fails to faithfully match the previous CoT process. (3) Further, we investigate the mechanism behind the hallucination of LRMs from the perspective of model uncertainty. We find that increased hallucination of LRMs is usually associated with the misalignment between model uncertainty and factual accuracy. Our work provides an initial understanding of the hallucination in LRMs.

1 Introduction

The paper asks whether long-CoT reasoning makes large reasoning models more prone to hallucination and evaluates this question through factuality, behavior, and uncertainty analyses. It finds that post-training pipelines have divergent effects on factuality, with cognitive behaviors and uncertainty calibration linked to hallucination.

  • Research Question: The paper evaluates whether reasoning models are more prone to hallucination on fact-seeking tasks after post-training for formal reasoning.The motivation arises from conflicting observations: DeepSeek-R1 improved SimpleQA performance, while OpenAI-o3 showed more severe hallucination.
  • Evaluation: Full cold-start SFT followed by verifiable-reward RL generally improves factuality, whereas SFT-only or RL-only pipelines can reduce it.The evaluation compares LRMs with non-reasoning backbones across SimpleQA and TriviaQA.
  • Behavior Analysis: Flaw Repetition and Think-Answer Mismatch are two cognitive behaviors that directly affect LRM factuality.Flaw Repetition repeatedly follows the same underlying flawed logic, while Think-Answer Mismatch produces a final answer that does not faithfully match the reasoning process.
  • Behavior Analysis: RL-only and SFT-only training encourage repetitive reasoning or shallow reasoning formats, while the complete cold-start-then-RL pipeline shows fewer such non-factual behaviors.The complete pipeline is associated with more fact-faithful model outputs.
  • Mechanism Analysis: RL-only and SFT-only models show corrupted calibration between output probability and answer correctness, with partial loss of uncertainty information in hidden states.The paper probes hidden states to examine whether uncertainty remains internally recoverable.
  • Mechanism Analysis: The investigation relates hallucination patterns to post-training strategy and argues that hallucination cannot be explained simply by catastrophic forgetting.Parameter-update volume is examined as part of the proposed initial understanding of LRM hallucination.

2 Related Works

The related work frames LRMs as models that use extra thinking tokens and post-training to improve long-CoT reasoning. It distinguishes cold-start SFT with RL, zero-style RL, and reasoning-distillation SFT, while defining hallucination as unverified fabricated information.

  • Large Reasoning Models: LRMs decompose complex tasks into sub-goals and generate extra tokens to think before producing a final answer.This chain-of-thought approach has been associated with improved reasoning performance.
  • Large Reasoning Models: Modern LRMs add a computationally intensive post-training stage that incentivizes long-CoT reasoning during inference.The resulting models commonly use a thinking phase before generating the final token.
  • Post-training Pipelines: Cold-start SFT with reasoning RL first primes long-thought capability on CoT data, then uses verifiable-reward RL to search formal-task reasoning paths.This is described as the de facto pipeline for several flagship LRMs.
  • Post-training Pipelines: Zero-style RL omits cold-start SFT and attempts to incentivize long-CoT reasoning directly from a non-reasoning model.The approach is motivated by AlphaGo-Zero-style reinforcement learning from scratch.
  • Post-training Pipelines: Reasoning-distillation SFT trains non-reasoning models on high-quality long-CoT data, often distilled from stronger models to reduce model size.This pipeline is presented as an alternative when RL is computationally expensive and long-CoT annotation is difficult.
  • Hallucination: The paper focuses on hallucinations as fabricated, unverified information and studies them using fact-seeking benchmarks.The scope emphasizes outputs lacking trustworthiness or factual grounding rather than creative hallucination.

3 Initial Investigation: Hallucination Evaluation

The initial investigation evaluates LRMs from three post-training pipeline categories against accessible non-reasoning counterparts on SimpleQA and TriviaQA. Full SFT+RL improves factuality, while RL-only and SFT-only models generally show factuality degradation.

  • Experiment Setup: The study evaluates hallucination on SimpleQA and TriviaQA, using questions that require retrieving and combining parametric facts.TriviaQA validation has 17,944 QA pairs, while SimpleQA contains 4,326 adversarially collected questions.
  • Experiment Setup: LLM-as-a-Judge evaluates both benchmarks because exact match cannot reliably handle semantically equivalent generative answers.Qwen3-32B with reasoning enabled serves as the judging model, and accuracy is reported in percentile.
  • Experiment Setup: The model collection distinguishes SFT+RL, RL-only, and SFT-only LRMs and compares each with an accessible base or instruct counterpart.SFT+RL combines cold-start SFT with verifiable-reward RL; RL-only omits cold-start SFT; SFT-only uses distillation-style SFT.
  • Evaluation Results: All four evaluated SFT+RL LRMs improve accuracy on both SimpleQA and TriviaQA relative to their non-reasoning counterparts.The paper attributes this pattern to long-CoT reasoning explicitly verbalizing induced knowledge and easing knowledge reasoning.
  • Evaluation Results: Most RL-only LRMs lose performance on both benchmarks; DeepMath-Zero is the exception on SimpleQA, improving by 0.6% while dropping 20.1% on TriviaQA.The authors associate the broader degradation with RL instability and tendencies toward repetition or Think-Answer Mismatch.
  • Evaluation Results: SFT-only LRMs consistently lose average accuracy across SimpleQA and TriviaQA, despite isolated gains such as DPSK-LLaMA-70B rising from 74.1% to 75.2% on TriviaQA.The paper suggests SFT-only training teaches shallow reasoning format without effective retry and reflection for factual search.

Repetition and Think-Answer Mismatch

The paper identifies flaw repetition and think-answer mismatch as cognitive behaviors associated with hallucinated outputs, especially in RL-only and SFT-only LRMs. Statistical analysis and case studies illustrate how repeated flawed reasoning and divergence between reasoning and final answers produce factual errors.

  • Behavior definitions: Flaw repetition traps an LRM in semantically similar reasoning loops, even when the wording changes.The behavior involves repeatedly following the same underlying line of thought instead of exploring alternatives.
  • Statistical analysis: Flaw repetition and think-answer mismatch are linked to hallucination in RL-only and SFT-only LRMs, while non-reasoning counterparts do not exhibit these behaviors.The analysis uses judged outputs from GLM-4-Z1-32B, DeepMath-Zero, and DPSK-Qwen-32B to compare the behaviors across pipelines.
  • Statistical analysis: SFT-only DPSK-Qwen-7B shows a clear tendency toward flaw repetition, whereas RL-only DeepMath-Zero exhibits flaw repetition and a higher think-answer mismatch rate than the SFT+RL comparison model.These frequencies are reported for hallucinated outputs on SimpleQA and TriviaQA.
  • Behavior definitions: Think-answer mismatch occurs when an LRM’s final answer fails to faithfully match its preceding reasoning process.The paper’s case study shows reasoning toward “Freddie Keppard” followed by the final answer “Fred Hager.”
  • Case study: The case studies contrast repeated false statements from RL-only DeepMath-Zero with a final answer that contradicts the reasoning of SFT-only DPSK-Qwen-32B.Table 5 presents one example of each cognitive behavior.

5 Mechanism Analysis: An Uncertainty Perspective

The paper analyzes hallucination through calibration and hidden-state uncertainty, comparing LRMs with non-reasoning counterparts. It finds that SFT+RL improves calibration and probing performance, whereas SFT-only and RL-only pipelines disrupt the alignment between uncertainty and correctness.

  • Model calibration: Model confidence P(a) is estimated from answer frequencies across N=10 samples, while correctness probability P(c|a) is measured against golden answers using LLM-as-a-Judge.Expected Calibration Error (ECE) quantifies the mismatch between uncertainty and factual correctness.
  • Model calibration: SFT+RL improves calibration, while SFT-only and RL-only post-training corrupt it.The calibration analysis compares LRMs with their non-reasoning counterparts using confidence, correctness probability, and ECE.
  • Model calibration: GLM-4-Z1-32B has ECE 0.12 versus 0.146 for GLM-4-32B-Base, while DeepMath-Zero has ECE 0.156, 0.042 higher than its non-reasoning counterpart.SFT-only Qwen3-14B and DPSK-Qwen-14B also exceed their non-reasoning counterparts by 0.073 and 0.015, respectively.
  • Mechanism analysis: The analysis tests whether hallucination reflects corrupted calibration or uncertainty that remains encoded in hidden states but is poorly verbalized.A probing experiment evaluates whether hidden states reveal whether the model can produce a correct answer.
  • Uncertainty probing: Probes for RL-only and SFT-only LRMs perform worse than probes for their non-reasoning counterparts, whereas probes for SFT+RL LRMs show a significant performance gain.The result indicates that uncertainty information is lost during RL-only and SFT-only post-training but better preserved or represented after SFT+RL.
  • Uncertainty probing: The paper argues that the mis-calibration of RL-only and SFT-only LRMs is rooted in their hidden states rather than being explained simply by parameter-update volume.Table 7 compares scaled parameter differences, while the probing results directly assess hidden-state uncertainty.

6 Discussion

The paper finds that hallucination cannot be explained by parameter-update magnitude alone. Static analysis of released models limits causal interpretation and motivates more controlled studies.

  • Methodological scope: The analysis relies mainly on static comparisons of publicly released LRMs rather than reproducing their post-training pipelines.The authors note that controlling training data, update steps, and hyperparameters would require substantial computational resources.
  • Parameter analysis: MAE measures the mean absolute parameter difference between each reasoning model and its corresponding base model.The calculation averages |θ_i − θ_i,b| across parameters.
  • Parameter analysis: GLM-Z1-32B has the largest reported parameter shift, MAE = 1.1529, yet exhibits relatively mild hallucination.This observation challenges the hypothesis that larger parameter changes necessarily produce more hallucination.
  • Parameter analysis: DPSK-Qwen-14B and Qwen3-14B show smaller shifts, MAE = 0.3374 and 0.1590, yet exhibit more significant hallucination behaviors.The comparison suggests that update magnitude alone does not account for hallucination differences.

7 Conclusion.

The paper concludes that post-training has inconsistent effects on hallucination in fact-seeking tasks. It identifies incomplete SFT-only and RL-only pipelines, two reasoning behaviors, corrupted calibration, and parameter-update analysis as central findings.

  • Conclusion: Post-training consistently improves formal-task performance but has inconsistent effects on hallucination in fact-seeking tasks.
  • Conclusion: SFT-only and RL-only pipelines are identified as the main causes of hallucination, whereas SFT+RL can alleviate it.
  • Conclusion: Hallucination is reflected in flaw repetition and think-answer mismatch during reasoning.
  • Conclusion: Hallucinating LRMs show corrupted calibration even when uncertainty is probed through their inner representations.
  • Conclusion: SFT-only and RL-only pipelines lead to a significant drop in parameter-update volume, which may explain their hallucination.

A Limitations

The study's evidence is constrained by its reliance on publicly released models, judge-based factuality evaluations, and primarily short-form fact-seeking tasks. It also does not claim to cover every cause of hallucination.

  • Scope and controls: The analysis cannot control confounds such as pretraining data, instruction-tuning data, or post-training schedules because it uses publicly released models.
  • Evaluation: Behavioral and mechanistic evaluations use LLM-as-a-Judge, which may introduce subjectivity or judge-specific bias.The authors use Qwen3-32B but call for robustness checks across judges and domains.
  • Task scope: The study focuses mainly on SimpleQA and TriviaQA, so its findings may not generalize to long-form synthesis, retrieval-augmented generation, or multi-turn dialogue.
  • Causal scope: The examined factors do not exhaustively cover hallucination causes; alignment methods, dataset quality, and prompt structure may also matter.

B Broader Impact

The broader-impact discussion links incomplete reasoning-model post-training to factual-error concerns while emphasizing that identifying risky pipelines and behaviors can inform more responsible LRM development.

  • Broader impact: RL-only or SFT-only reasoning models can exhibit more factual errors than their non-reasoning counterparts.
  • Broader impact: These findings raise societal concerns because LRMs are increasingly used in education, legal assistance, and medical decision support.
  • Responsible development: Identifying hallucination-associated training pipelines and behavioral patterns provides actionable insights for reducing factual inconsistencies in future LRM development.
  • Responsible development: The paper highlights combining supervised fine-tuning with verifiable-reward reinforcement learning as an effective approach for reducing hallucination.

E Details of Decoding Parameters

This section specifies the decoding parameters used in the experiments and identifies Table 9 as their location. It also records the deployment framework and version.

  • The experiments use decoding parameter settings listed in Table 9.Table 9 is titled “Sampling parameter settings for different model configurations.”
  • The models are deployed with the vLLM framework, version 0.8.3.
  • Table 9 organizes sampling parameter settings by model configuration.
Loading 2505.23646v1…