PaperlayerView exact revision on arXiv

Source-linked AI summary

MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Models

Xinming Wang, Jian Xu, Bin Yu, Sheng Lian, Hongzhu Yi, Yi Chen, Yingjian Zhu, Boran Wang, Hongming Yang, Han Hu, Xu-Yao Zhang, Cheng-Lin Liu

arXiv:2510.24794v2cs.CL

TL;DR

Factual QA can fail when a model surfaces the correct answer in its reasoning but does not carry it into the final response. MR-ALIGN aligns reasoning transitions using meta-reasoning annotations, without external retrieval or verifier signals. The paper reports improved factual QA accuracy and reduced reasoning–answer mismatch across its evaluations.

  • Problem

    In factual QA, models may identify correct facts during reasoning but fail to incorporate them into the final response, creating a reasoning–answer hit gap.

    Source: §1 · p. 1

  • Method

    MR-ALIGN aligns meta-reasoning transitions using annotated reasoning segments, without external retrieval or verifier signals.

    Source: §1 · p. 2

  • Results

    MR-ALIGN improves factual QA accuracy and reduces reasoning–answer mismatch, with its effect reported as most stable on NQ-Open and SciQ and effective on TruthfulQA and LongFact.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The paper concludes that learning transition probabilities and using a transition-aware advantage can encourage more faithful responses in factual QA and long-form factuality.

    Source: §5 · p. 8

    The authors do not know whether other models would introduce biases into the LLM-generated meta-reasoning annotations.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large reasoning models (LRMs) show strong capabilities in complex reasoning, yet their marginal gains on evidence-dependent factual questions are limited. We find this limitation is partially attributable to a reasoning-answer hit gap, where the model identifies the correct facts during reasoning but fails to incorporate them into the final response, thereby reducing factual fidelity. To address this issue, we propose MR-ALIGN, a Meta-Reasoning informed alignment framework that enhances factuality without relying on external verifiers. MR-ALIGN quantifies state transition probabilities along the model's thinking process and constructs a transition-aware implicit reward that reinforces beneficial reasoning patterns while suppressing defective ones at the atomic thinking segments. This re-weighting reshapes token-level signals into probability-aware segment scores, encouraging coherent reasoning trajectories that are more conducive to factual correctness. Empirical evaluations across four factual QA datasets and one long-form factuality benchmark show that MR-ALIGN consistently improves accuracy and truthfulness while reducing misleading reasoning. These results highlight that aligning the reasoning process itself, rather than merely the outputs, is pivotal for advancing factuality in LRMs.

1 Introduction

Factual QA can exhibit a reasoning–answer hit gap: a model may identify the correct answer during reasoning but omit it from its final response. MR-ALIGN aligns reasoning transitions using meta-reasoning annotations and requires neither external retrieval nor verifier signals.

1 Introduction

  • In factual QA, correct answers may appear during reasoning but be discarded before the final response, undermining factual faithfulness.The paper describes this reasoning–answer hit gap as a problem in evidence-centric factual reasoning.Source: §1 · p. 1
  • The paper proposes that factuality improvement requires aligning how models reason, not merely what they predict.It notes that prolonged reasoning can lead models to negate an earlier correct candidate.Source: §1 · pp. 1–2
  • A cognitive-grounded annotation pipeline identifies and categorizes the principal strategies in LLM problem-solving.The paper presents transition visualizations as revealing shifts associated with factuality improvements.Source: §1 · p. 2
  • MR-ALIGN uses meta-reasoning transition probabilities for fine-grained preference alignment, without external retrieval or verifier signals.The paper says it guides models toward more factual reasoning patterns.Source: §1 · p. 2

2 Related Works

Prior work addresses reliability in reasoning models through structural controls, supervision, retrieval, memory, and factuality-alignment methods. The paper motivates targeting intermediate reasoning transitions because many existing factuality methods rely on post-hoc correction or external verification.

Reasoning Models

  • Analyses describe errors in extended reasoning traces, including compounding errors and confident hallucinations, as arising within reasoning chains.Proposed responses include reasoning selection and pruning, atomic-level supervision, and retrieval or memory mechanisms.Source: §2 · p. 2

Factuality Alignment

  • Short-form factuality methods include evidence retrieval, critic-based alignment, preference tuning, factuality rewards, knowledge-graph verification, and uncertainty estimation.The paper contrasts these with long-form methods for multi-sentence factual explanations.Source: §2 · p. 2
  • The paper says many existing factuality methods rely on post-hoc correction or external verification rather than directly improving underlying reasoning.This motivates integrating meta-reasoning into reasoning dynamics.Source: §2 · p. 3

3 Method

MR-ALIGN builds a transition-aware preference-alignment signal from annotated reasoning segments. Its method generates positive and negative examples, annotates meta-reasoning strategies, estimates their transitions, and uses those estimates to reweight implicit rewards.

3.1.1 Training Data Candidates Generation

  • The training corpus uses THINKON and THINKOFF outputs from the same base model, with positive examples where reasoning recovers a correct answer and negative examples where it degrades accuracy.Correctness is judged using exact match.Source: §3.1.1 · p. 3

3.1.2 Meta-reasoning Labels Annotation

  • The annotation pipeline converts free-form segment labels into a 15-label taxonomy grouped into four macro-strategies.Two annotators propose up to two labels per segment, with GPT-5 adjudicating when consolidation leaves fewer than two labels; reported entailment-setting agreement is κ = 0.7855.Source: §3.1.2 · pp. 3–4

3.2.1 Kahneman-Tversky Optimization

  • KTO uses single-sample binary labels and evaluates chosen and rejected responses separately relative to a baseline.The paper contrasts this with DPO's pairwise preferences and notes KTO's decoupled treatment of positive and negative labels.Source: §3.2.1 · p. 4

3.2.2 Atomic Reasoning Process Modeling

  • The transition model uses a row-stochastic matrix over meta-reasoning states, augmented with start and stop states, to represent probabilities of moving between states.The stop state is absorbing.Source: §3.2.2 · p. 4
  • Each annotated segment has one or two unordered labels, and an observed set-to-set move is modeled as arising from one latent base-level transition.The method estimates the transition matrix using EM in the presence of a latent variable.Source: §3.2.2 · p. 5

3.2.3 Alignment with meta-reasoning transitions

  • The alignment method defines atomic implicit rewards by measuring changes in meta-reasoning transition probabilities.The weight reflects local transition probability relative to the global transition pattern and is clipped to mitigate small-sample artifacts.Source: §3.2.3 · p. 5
  • MR-ALIGN reweights token-level contributions according to meta-reasoning transition dynamics, increasing the contribution of locally divergent transitions and reducing that of near-global transitions.The resulting signal is a probability-aware reward over atomic reasoning segments.Source: §3.2.3 · p. 5
  • Using the global transition matrix as an anchor raises the contribution of segments whose local transitions diverge from the global pattern and lowers that of near-global transitions.This reshapes a uniform token-level signal into a transition-modulated reward over atomic reasoning segments.Source: §3.2.3 · p. 5

4 Experiments

Experiments compare MR-ALIGN with alternative decoding and tuning approaches across factual QA and long-form factuality. Results show broad improvements, while ablations and stratified analyses locate stronger effects in transition design and reasoning–answer inconsistencies.

4.1 Experiments Setup

  • Evaluation compares THINKON/OFF, Self-Refine, SFT, KTO, and open-search baselines across factual QA and long-form factuality tasks.NQ-OPEN, SCIQ, and SIMPLEQA use accuracy and misleadingness; FULQA is judged for truthfulness and helpfulness, while LONGFACT uses F1@K and claim counts.Source: §4.1 · pp. 5–6

4.2 Main Result

  • Without retrieval, MR-ALIGN improves factual QA accuracy and lowers misleadingness and reasoning–answer mismatch; gains are stable on NQ-Open and SciQ and extend to TruthfulQA and LongFact.With a retriever, it also improves over the original model.Source: Table 2 · p. 6, §4.2 · p. 7

4.3 Ablation Study

  • Performance drops rapidly when reject ratio λd exceeds 1.5, and negative-sample suppression is more pronounced than with vanilla KTO.The ablation varies λd to temper loss aversion from imbalanced positive and negative subsets.Source: §4.3 · p. 7, Table 4 · p. 7
  • NQ-OPEN+SCIQ training gives the strongest overall accuracy and mismatch results, especially on SIMPLEQA; SCIQ-only training yields limited gains.With training data fixed, EM transition estimation improves accuracy and lowers mismatch on NQ-OPEN and SCIQ; disabling label-divergence modeling degrades performance across datasets.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Futher Analysis

  • After alignment, evidence-seeking, quality-control, and synthesis-driven transitions gain mass, while reasoning chains become shorter and more targeted.The comparison uses 977 sampled NQ-OPEN instances and element-wise transition differences between MR-ALIGN and the vanilla model.Source: §4.4 · p. 8, Figure 4 · p. 8
  • On INCONSIST examples, Qwen3-8B and Qwen3-4B improve by more than 10% on NQ-OPEN and SCIQ, while BOTH WRONG performance changes little.The authors conclude MR-ALIGN reduces reasoning–answer discrepancies on near-boundary questions rather than expanding intrinsic knowledge coverage.Source: §4.4 · p. 8, Figure 5 · p. 8
  • MR-ALIGN improves accuracy and reduces misinformation across three benchmarks, with similar trends on the Llama-3.1-Nemotron-Nano-4B-v1.1 backbone.The paper presents the latter backbone’s performance in Table 5.Source: §4.4 · p. 8, Table 5 · p. 8

5 Conclusion

The paper examines the reasoning–answer hit gap in factual QA and long-form factuality, then proposes MR-ALIGN to encourage more faithful responses through transition-informed alignment.

5 Conclusion

  • The work studies the reasoning–answer hit gap in factual QA and long-form factuality from a cognitive perspective.It frames this gap as part of the limitations of prevailing reasoning paradigms for factual adherence.Source: §5 · p. 8
  • MR-ALIGN learns transition probabilities from positive samples and uses a transition-aware advantage to encourage more faithful responses.Source: §5 · p. 8
  • The authors hope this perspective motivates broader research on principled, process-level alignment for factual domains.Source: §5 · p. 8

Limitations

The authors identify uncertainty about annotation bias across LLMs and limited experimental verification on larger or MoE models.

Limitations

  • The annotations rely on LLMs, and whether other models introduce bias in this process remains unknown despite the current annotator’s controllable consistency.Source: Limitations · p. 9
  • Computational limits prevented experiments on models larger than 14B or on MoE models, leaving their behavior in this setting unverified.Source: Limitations · p. 9

Ethical Considerations

The authors report that the datasets and models used are open-source and state that their research raises no human-ethics or values risks.

Ethical Considerations

  • The study reports no licensing risks for its open-source datasets and models, and no human-ethics or values risks for its stated mathematical-domain research.Source: Ethical Considerations · p. 9

A Datasets and Metrics

The appendix describes five factuality benchmarks and their evaluation metrics, then details how meta-reasoning labels are clustered and distributed in training data.

A.1 Dataset Details

  • Five datasets span short-answer factual QA, multiple-choice science, truthfulness, and long-form claim-level factuality evaluation.NQ-Open, SciQ, SimpleQA, TruthfulQA, and LongFact cover distinct answer formats and evaluation targets.Source: §A.1 · p. 11

A.2 Metrics Details

  • Exact Match uses gold-answer containment, while factual-QA evaluation reports answer Accuracy and Misleading reasoning based on gold-answer presence in exactly one output.For reasoning-enabled responses, the EM definition is refined per example; Misleading counts cases where the gold answer appears in the thought trace or final answer, but not both.Source: §A.2 · pp. 11–12
  • TruthfulQA uses fixed GPT-4o binary truthfulness and informativeness judgments, while LongFact evaluates extracted claims against retrieved evidence with F1@K at median claim count.Truthfulness requires factual claims to be consistent with established evidence; K is the median number of extracted facts.Source: §A.2 · p. 12

B.1 Meta-reasoning label clustering

  • 2,473 distinct labels from 2,000 annotated samples were clustered using frequency-filtered bge-m3 embeddings and OPTICS, yielding 15 categories and a 0.4861 silhouette score.GPT-5 generated canonical cluster labels.Source: §B.1 · p. 12

B.2 Meta-reasoning label statics in training data

  • Training labels show a long-tail distribution: Knowledge Operations and Meta-cognitive Regulation dominate, while explanatory and problem-solving categories are sparser.The authors say this structure separates knowledge manipulation from reasoning regulation and validation, supporting interpretable supervision and targeted analysis.Source: §B.2 · pp. 12, 14, Table 7 · p. 14

B.3 Inter-Annotation Agreement Analysis

The agreement analyses find that entailment-based matching captures substantial consistency across annotators despite only moderate strict label-set matches. Annotation differences also yield small deviations in estimated transition matrices, supporting their use for modeling meta-reasoning dynamics.

B.3 Inter-Annotation Agreement Analysis

  • The annotation pipeline reached 0.7855 entailment agreement with DEEPSEEK-CHAT, while macro-strategy agreement was close to 0.9702.Strict exact-match agreement remained moderate; subset-based agreement treats different label granularity as compatible.Source: §B.3 · pp. 14–15

B.4 Human Annotation Agreement Statistics

  • Human–human agreement was 0.513 under strict matching and 0.834 under entailment matching, and human–pipeline agreement was comparable or higher under entailment.The study sampled 100 examples across four NQ-OPEN and SCIQ positive/negative subsets, yielding 775 labeled steps.Source: §B.4 · p. 15, Table 9 · p. 15

B.5 Influence of Annotation Agreement

  • Transition matrices estimated under different annotation settings differed by typically 1–2% in ℓ1 distance (about 10−2 on average), suggesting step-level non-agreement is substantially attenuated when aggregated into transition statistics.The authors characterize annotation noise as controlled and adequate for modeling meta-reasoning transition dynamics.Source: §B.5 · pp. 15–16

B.6 Illustration of Meta-reasoning labels

  • The illustrated taxonomy includes meta-cognitive regulation, backtracking, self-verification, and evaluation as named meta-reasoning labels.These examples describe framing a problem, revisiting a reasoning branch, checking intermediate claims, and assessing candidate reasoning products.Source: Meta-cognitive Regulation · p. 16

Problem-Solving Operations

  • Problem-solving operations include decomposition, which splits a task into subproblems, and chaining, which links intermediate inferences..Source: Problem-Solving Operations · p. 16

Knowledge Operations

  • Knowledge-operation labels include causal reasoning, retrieval, analogy, synthesis, comparison, categorization, and case analysis.They cover hypothesis testing, acquiring evidence, transferring a schema, integrating evidence, contrasting alternatives, classifying instances, and using precedents.Source: Knowledge Operations · p. 16

Explanatory & Communication

  • The explanatory and communication labels are explanation and summarization, focused on articulating supported reasoning and compressing content faithfully..Source: Explanatory & Communication · p. 16

C More Results

On Qwen3-14B, MR-ALIGN remained competitive across accuracy and misinformation measures, although its gains were relatively limited. The transition visualization contrasts forward-progressing patterns in positive traces with self-loops and regressions in negative traces, while noting an imbalance-related measurement effect.

C More Results

  • Positive transitions concentrate on forward progress, whereas negative transitions show self-loops and regressions toward backtracking.The positive examples include categorization → decomposition and chaining → synthesis; the global mixture and a roughly 2:1 positive-to-negative data imbalance make the positive panel appear milder and the negative panel heavier-tailed.Source: C More Results · p. 17
  • On Qwen3-14B, MR-ALIGN remained consistently competitive on accuracy and misinformation, while SFT and KTO were less stable across benchmarks and could trade off the two outcomes.The authors describe MR-ALIGN's improvements as relatively limited on this stronger backbone; the table reports its performance.Source: C More Results · p. 17, Table 12 · p. 17

D.1 Training Details

The appendix specifies the training and inference setup, outlines the EM estimation pseudocode, and presents annotation prompts and Qwen3-8B case-study material.

D.1 Training Details

D.2 Sampling Details

  • Inference sampling parameters are provided in a table, and inference used vLLM with one Nvidia A800 (40 GB) GPU.The setup follows the recommended Qwen3-8B implementation.Source: §D.2 · p. 17, Table 15 · p. 17

E Pseudo Code of EM Estimation

  • The EM pseudocode alternates responsibility computation with Dirichlet-smoothed row-wise updates under structural masks, then uses a driver to set constraints and invoke the estimator.Figure 7 depicts transition advantages for positive and negative subsets relative to the full training set.Source: E Pseudo Code of EM Estimation · pp. 17–18, Figure 7 · p. 18

F.2 Case Study

Loading 2510.24794v2…