Source-linked AI summary
FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision
Qiming Xie, Wenjie Zheng, Xiangqing Shen, Rui Xia
TL;DR
Outcome-driven RL can reinforce factually flawed reasoning, and existing factual supervision can mismatch verification granularity with token-level policy updates or rely on uncertain verifier signals. FARCA aligns atomic facts with originating tokens and weights their signals by counterfactual evidence dependence; experiments show improved factuality while preserving reasoning capabilities.
Problem
Outcome-driven rewards can reinforce factually flawed intermediate reasoning, while existing factual supervision leaves credit localization and verifier reliability insufficiently addressed.
Method
FARCA aligns independently verifiable atomic facts with their originating token spans and uses counterfactual evidence attribution to weight factual rewards and local policy advantages.
Results
FARCA significantly improves model factuality across models and hallucination scenarios while maintaining strong reasoning performance and training stability.
Takeaways & Limitations
Fact supervision is incorporated more accurately and robustly when verification signals are transformed into fine-grained, reliability-aware credit for policy optimization.
Abstract
from arXiv · showhide
To reduce the hallucination risk caused by outcome-driven rewards in large language models trained through reinforcement learning with verifiable rewards, existing mitigation approaches introduce process-level factual supervision. However, due to coarse-grained aggregation of factual signals and the lack of reliability assessment for these signals, they create a mismatch between fact verification and policy updates. We term this noisy factual credit assignment and decompose it into two aspects: credit localization ambiguity and credit reliability ambiguity. To address these issues, we propose FARCA (Fact-Aligned Reliability-Aware Credit Assignment), a policy optimization framework that transforms factual supervision into localized, reliability-weighted token-level training signals. FARCA achieves fine-grained credit localization by aligning the granularity of fact verification with that of policy updates. It further introduces counterfactual evidence attribution, which uses the dependence of a factual judgment on key evidence as an empirical proxy for verification reliability to compute reliability weights. These weights modulate factual rewards and local policy advantages, reducing the influence of potentially unreliable signals on policy optimization. Experiments across different models and multiple factual reasoning benchmarks show that FARCA significantly improves model factuality while preserving general reasoning capabilities.
Introduction
Outcome-driven RL can reinforce factually flawed reasoning, while existing factual supervision suffers from ambiguous credit localization and uncertain verifier reliability. FARCA addresses both by routing evidence-weighted factual signals to the responsible tokens.
- Outcome-level rewards can reinforce factually flawed reasoning because they assess final answers without penalizing factual errors in intermediate reasoning.
- Existing factual-supervision methods introduce trajectory- or reasoning-step-level signals, but coarse aggregation can misalign verification outcomes with token-level policy updates.
- Noisy factual credit assignment comprises credit localization ambiguity and credit reliability ambiguity.Localization concerns where credit is assigned; reliability concerns whether the verification signal is trustworthy.
- FARCA decomposes trajectories into independently verifiable atomic facts, anchors each fact to its originating token span, and routes continuous signed factual scores through token provenance.This alignment differentiates optimization signals for correct and hallucinated content within the same reasoning process.
- Counterfactual evidence attribution estimates verification reliability from dependence on key evidence, producing weights that modulate factual rewards and local policy advantages.
- FARCA improves model factuality across models and hallucination scenarios while maintaining strong reasoning performance and training stability.
Related Work
RLVR has improved complex reasoning through automatically verifiable outcome rewards, but these rewards do not check intermediate factual accuracy. Factual-supervision methods address this gap with increasingly fine-grained signals, while FARCA adds fact–token alignment and reliability modeling.
- RLVR optimizes automatically verifiable outcome-level rewards to elicit complex reasoning, but final-answer rewards do not verify intermediate factual accuracy.
- KnowRL aggregates atomic-fact verification into a trajectory-level factual reward, while FSPO verifies each reasoning step to adjust policy advantages.
- FARCA addresses these limitations through fact–token alignment and reliability modeling, providing fact-guided signals for accurate and robust policy optimization.
Methodology
FARCA converts factual supervision into localized, reliability-weighted token-level signals by aligning atomic facts with their source tokens and measuring verifier dependence on evidence. It then uses these signals in reward construction and advantage reshaping within token-level policy optimization.
- Fact Extraction and Provenance: FARCA decomposes reasoning into independently verifiable atomic facts and records each fact’s token provenance in the originating reasoning text.Token provenance identifies the source token span that receives the corresponding factuality signal.
- Factual Verification: An NLI verifier produces a continuous signed factual score, where positive values indicate evidence support, negative values indicate contradiction, and zero indicates neutrality.The score is computed as 2h_i,j,k − 1 and preserves the verifier’s support strength rather than thresholding it.
- Counterfactual Evidence Attribution: FARCA estimates verification reliability by removing relevant evidence, re-verifying the fact, and measuring the absolute change between the original and counterfactual signed scores.A larger score change indicates stronger dependence on identifiable evidence and therefore greater reliability.
- Counterfactual Evidence Attribution: The reliability-weighted factual score multiplies the signed factual score by a continuous reliability weight, separating the supervision signal’s direction from its strength.The weight is derived from evidence-dependency strength using a calibration-set median and a smoothness parameter.
- Reward Design: FARCA combines format, answer, and reliability-weighted factual rewards before computing group-relative advantages for rollouts of the same question.The factual reward averages reliability-weighted scores over verifiable atomic facts in the reasoning text.
- Reliable Fact-Guided Advantage Reshaping: Token-level advantage reshaping assigns factual corrections to provenance-covered tokens, blends them with original advantages according to reliability, and leaves uncovered tokens unchanged.Multiple facts covering one token contribute their average reliability-aware advantage; the reshaped advantage is then substituted into the token-level PPO-clip objective.
- Training Objective: FARCA injects factual supervision into token-level policy optimization through fact-token alignment and reliability modeling.The framework is designed to make factual credit assignment more precise and robust.
Experiments
Experiments evaluate FARCA across factuality and mathematical reasoning benchmarks, component ablations, and analyses of localization, reliability, and advantage shaping. FARCA improves factuality while preserving reasoning ability, with supporting analyses showing broad fact–token coverage and stable, reliability-aware credit corrections.
- Main Results: FARCA achieves the best results across four hallucination benchmarks and mathematical reasoning benchmarks while preserving or enhancing reasoning performance.Against FaithRL, average factuality gains are 1.75 percentage points on Qwen2.5-3B-Instruct and 2.21 points on Llama-3.2-3B-Instruct.
- Main Results: Compared with FaithRL, FARCA improves average factuality by 1.75 percentage points on Qwen2.5-3B-Instruct and 2.21 points on Llama-3.2-3B-Instruct.TruthfulQA and HalluQA gains are 2.09 and 2.67 points for Qwen2.5-3B-Instruct, and 3.31 and 4.00 points for Llama-3.2-3B-Instruct.
- Ablation Studies: Token provenance routing raises the average score from 24.40 without routing to 25.36 with FARCA by restricting factual signals to atomic-fact token spans.This addresses sentences containing reliable and hallucinated information simultaneously.
- Ablation Studies: Removing reliability estimation or replacing continuous signed factual scores with discrete values degrades performance, supporting reliability weighting and continuous supervision.Without reliability estimation, all factual-judgement weights are set to 1; discretization uses {-1, 0, +1} with a 0.5 threshold.
- Ablation Studies: FARCA is robust to reliability temperature settings from τ = 0.10 to 0.30, with average scores of 25.31, 25.36, and 25.47 and a maximum difference of 0.16 percentage points.The default τ = 0.20 preserves discrimination without an overly sharp mapping.
- Further Analysis: About 98.3% of extracted facts align to source token spans, while matched-fact coverage rises early in training and fluctuates at a higher level thereafter.The results indicate both reliable localization and broad factual-supervision coverage.
- Further Analysis: Counterfactual attribution yields 36.3% of facts with ∆ > µ, 1.1% fallback, and mean reliability weight w = 0.512, while soft interpolation shifts fact-span advantages by 48.8% on average.In advantage outcomes, 49.6% retain direction, 39.5% retain direction with rescaled magnitude, 10.7% reverse, and 0.2% are neutral.
- Further Analysis: Token provenance enables local–global credit correction, with contradiction correction and support rescue averaging 77.1% and 75.6% strength and mixed-sentence separation reaching 45.9%.These corrections modulate global advantages using the direction and reliability of local facts.
Conclusion
FARCA addresses noisy factual credit assignment by aligning factual verification with policy optimization and estimating verification reliability. Experiments confirm improved model factuality and stability.
- FARCA addresses noisy factual credit assignment through fact–token alignment and counterfactual evidence attribution.The framework matches verification granularity to policy optimization and uses reliability estimates to reshape rewards and advantages.
- FARCA improves model factuality with reported effectiveness and stability across experiments.
Implementation Details
The implementation combines factual and mathematical reasoning data, evaluates hallucinations on four factuality benchmarks, and uses controlled component ablations. Ablation variants selectively remove token localization or reliability estimation while retaining other FARCA signals.
- Training and evaluation data: Training combines knowledge-intensive question answering with mathematical reasoning problems.The factuality-oriented subset contains 4,761 HotpotQA samples and 3,737 2WikiMultiHopQA samples.
- Training and evaluation data: Hallucinations are evaluated on SimpleQA, TruthfulQA, HalluQA, and HaluEval-QA.The benchmarks use metrics including F1 and truthful ratio for factuality assessment.
- Component ablations: Ablations modify only the investigated component, while atomic facts and token provenance localize fact-corrected advantages to source tokens.This configuration isolates the contribution of reliability estimation and other individual design choices.
- Component ablations: The no-token-provenance variant broadcasts sentence-level factual advantages to every token in the sentence.It averages advantages from atomic facts extracted from the same sentence before broadcasting them across that sentence’s token positions.
- Component ablations: The no-reliability-estimation variant treats every verification outcome as fully reliable by bypassing counterfactual evidence attribution.Its reliability-weighted factual score therefore reduces to the unweighted form.
- Component ablations: The no-continuous-score variant thresholds verifier output at 0.5 while preserving token provenance and reliability estimation.It replaces the continuous signed score with −1, 0, or +1 only when constructing factual rewards and fact-corrected advantages.