Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models

• Outcome-driven RL can produce fluent, confident reasoning containing unsupported intermediate claims, undermining reliability and interpretability.Source: §1 · p. 1
• High-variance gradients, entropy-induced randomness, and spurious local optima help explain why outcome-based RL can induce hallucinations.Source: §1 · pp. 1–2
• FSPO adjusts token-level advantages using automated evidence-based verification of each reasoning sentence, rewarding factual tokens and penalizing incorrect ones.Source: §1 · p. 2
• Across mathematical reasoning and hallucination benchmarks, FSPO improves reasoning performance, reduces hallucinations, and enhances reasoning-step factuality without compromising generated-content quality.Source: §1 · p. 2

Models and Datasets

• The benchmarks define hallucination rates through untruthful answers or false judgments, depending on the task format.Source: §2 · pp. 2–3

Results and Analysis

• Models receiving large-scale RL or long-CoT training show dramatically higher hallucination rates across TruthfulQA, HaluEval, and HalluQA.Source: §2 · pp. 2–3
• Among 200 HaluEval-QA cases where DeepSeek-V3 was correct but DeepSeek-R1 was not, most errors originated in incorrect reasoning steps.Source: §2 · p. 3

Reinforcement Learning

• Language generation is modeled as a token-level Markov decision process in which each generated token transitions the state.Source: §3 · p. 3
• Binary outcome rewards assign 1 to correct outputs and 0 to incorrect outputs, creating sparse and discontinuous learning signals.Source: §3 · p. 3

Group Relative Policy Optimization

• PPO optimizes a clipped surrogate objective using token advantages, while GRPO replaces a learned value-model baseline with the mean reward of sampled outputs.Source: §3 · p. 3
• GRPO normalizes each sampled output’s reward relative to the group mean and standard deviation, reducing memory and computational burden by avoiding a value model.Source: §3 · pp. 3–4

• When correct-answer probability p is small, binary-reward policy gradients are mostly zero with occasional large spikes, producing unstable updates.The variance scales as p(1 − p)||∇θ log πθ(y∗)||2.Source: §4 · p. 4
• Avoiding zero-reward local optima requires maintaining sufficiently high entropy, creating an exploration–exploitation trade-off that increases hallucination risk.Source: §4 · p. 4
• Binary rewards admit spurious local optima where a policy confidently outputs an incorrect answer, receives zero reward, and has no gradient incentive to change.Source: §4 · pp. 4–5

5 Approach

• FSPO integrates step-wise factuality signals into policy optimization to encourage more factual and faithful reasoning steps.An automated verifier checks whether each reasoning sentence is entailed by the evidence, producing step-wise factuality scores.Source: §5 · p. 5, §5.1 · p. 5

5.1 Reward Function Design

• The reward design combines a rule-based final-answer correctness reward with sentence-level factuality rewards, providing denser feedback than sparse outcome rewards.Factuality rewards assign 1 to entailed sentences, 0 to neutral sentences, and −1 to contradicted sentences; neutral connective or exploratory phrases are treated as unrelated to the evidence.Source: §5.1 · pp. 5–6

5.2 Factuality-Aware Policy Optimization

• Factuality-aware advantage adjustment reweights tokens according to both the response-level advantage and their sentence-level factuality, encouraging verified tokens and penalizing contradicted ones.GRPO computes response advantages from group-relative rewards, after which token advantages are adjusted rather than assigning every token the same response-level value.Source: §5.2 · p. 6
• When a reasoning sentence is entailed or contradicted, its adjusted token advantages remain nonzero even for an incorrect answer, providing a direction that can move learning away from spurious local optima.The resulting gradient pushes probability mass away from factually incorrect tokens and toward factually verified tokens.Source: §5.2 · p. 6

6.2 Main Results

• FSPO based on Qwen2.5-7B-Base achieves the highest hallucination-benchmark scores among open-source and reasoning models.The comparison covers closed-source, distilled, and open-source alternatives across multiple benchmarks.Source: §6.2 · p. 7, Table 1 · p. 8
• GSM8K accuracy reaches 89.5% with FSPO, versus 73.2% for Qwen2.5-7B-Instruct and 77.5% for Llama3.1-8B-Instruct.FSPO also maintains competitive performance on MATH-500.Source: §6.2 · p. 7
• FSPO scores 20.0% on AIME 2024 and 13.3% on AIME 2025, surpassing smaller base models but trailing reasoning and larger API-based models.These results demonstrate competitive performance on advanced mathematical reasoning tasks.Source: §6.2 · p. 7

6.3 Further Analysis

• Across training steps on MATH-500 and HaluEval-QA, FSPO outperforms GRPO and GRPO with factuality scores while improving reasoning and reducing hallucinations.The ablation compares answer-only GRPO, GRPO with factuality rewards, and FSPO.Source: §6.3 · pp. 7–8, Figure 4 · p. 8
• FSPO with Reinforce++ improves substantially over Reinforce++ alone, while FSPO with GRPO remains more accurate across training steps on both benchmarks.The findings support FSPO’s adaptability beyond GRPO.Source: §6.3 · p. 8, Figure 5 · p. 8
• Using 1K–4K samples yields similar MATH-500 accuracy, whereas HaluEval-QA improves through 2K samples and performs less well at 4K and 8K.With 8K samples, MATH-500 accuracy diminishes, suggesting that more factual QA data can lower mathematical reasoning performance.Source: §6.3 · p. 8, Figure 6 · p. 9
• FSPO and GRPO generate similarly long responses, but FSPO achieves significantly higher factuality scores.The analysis attributes this difference to explicit optimization of factual correctness during intermediate reasoning steps.Source: §6.3 · p. 9, Figure 7 · p. 9

Large Reasoning Models

• Large reasoning models use reinforcement learning and long chain-of-thought training to improve complex reasoning, planning, and problem-solving.Examples include OpenAI’s o1, DeepSeek-R1, Kimi k1.5, and QwQ [15] [9] [17].Source: §7 · p. 9
• Prior reasoning-model research has largely emphasized capability improvements while overlooking hallucinations arising during reasoning.The paper positions its analysis and mitigation method as an early effort addressing this issue.Source: §7 · p. 9

Hallucination

• Hallucination mitigation spans dataset curation, supervised fine-tuning, reinforcement learning with human feedback, and post-processing verification.Detection methods differ depending on whether model internal states are accessible or only external responses can be queried.Source: §7 · p. 9

• FSPO integrates step-wise factual verification into RL training, providing denser feedback to address hallucinations in generated reasoning.The method adjusts token-level advantages using factuality information during training.Source: §8 · p. 9
• FSPO significantly mitigates hallucinations across multiple benchmarks without compromising the models’ reasoning capabilities.Source: §8 · p. 10

• Checklist completion is mandatory because papers without the checklist will be desk rejected.Source: NeurIPS Paper Checklist · p. 13
• The checklist requires a short justification after each Yes, No, or NA response.Source: NeurIPS Paper Checklist · p. 13
• The checklist asks whether the abstract and introduction accurately reflect the paper’s contributions and scope.Source: NeurIPS Paper Checklist · p. 13
• Claims should match theoretical and experimental results while reflecting how broadly the findings can be expected to generalize.Source: NeurIPS Paper Checklist · p. 13

• The paper reports that its limitations are discussed in Appendix A.Source: §2 · p. 13
• The checklist requires authors to articulate strong assumptions, robustness to assumption violations, and the practical implications of those violations.Source: §2 · p. 13
• Authors are asked to discuss scope, performance factors, computational efficiency, scaling, and applicable privacy or fairness limitations.Source: §2 · p. 14
• The paper states that it provides dataset and implementation details, training and test information, compute resources, and reproducibility material in specified sections and appendices.Source: §2 · pp. 14–16
• The checklist covers ethics, societal impacts, safeguards, licensing, and human-subject considerations for research assets and experiments.Source: §2 · pp. 17–19

A Limitations

• The theory focuses on simplified binary rewards, while experiments are constrained by the cost of evaluating much larger models such as 14B and 32B systems.Source: A Limitations · p. 20

B Broader Impact

• FSPO may improve trustworthiness in critical applications, but misuse could support persuasive systems for propaganda or misinformation.Source: B Broader Impact · p. 20

C Datasets

• Training uses 2K randomly selected samples from a mixed HotpotQA/2WikiMultiHopQA subset and 8,523 SimpleRL mathematical samples, with evaluation on hallucination and mathematical-reasoning datasets.Source: C Datasets · p. 20

D Training Infrastructure

• Training uses 4×H100 80G GPUs for about 24 hours per run, with batch size 8, eight rollouts per prompt, and a 4e-7 learning rate.Source: D Training Infrastructure · p. 20

E Preliminary Experiments

• After large-scale RL or long-CoT training, models showed significantly more hallucinations across three benchmarks and twelve LLMs.The study examined 200 HaluEval-QA samples where DeepSeek-V3 was correct but DeepSeek-R1 was not.Source: E Preliminary Experiments · p. 20
• Intermediate reasoning errors were the most common source, followed by cases where correct reasoning produced an incorrect final answer.Source: E Preliminary Experiments · p. 20
• Overthinking, incomplete outputs, and answer–ground-truth mismatches accounted for additional hallucination cases.Source: E Preliminary Experiments · p. 21

F Prompt Templates

• The experiments used separate prompts for mathematical reasoning, question answering, and HaluEval-QA evaluation.These prompts are presented in Tables 2, 3, and 4, respectively.Source: F Prompt Templates · p. 21, Table 2 · p. 21, Table 3 · p. 21