PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Correctness: Learning Robust Reasoning via Transfer

Hyunseok Lee, Soheil Abbasloo, Jihoon Tack, Jinwoo Shin

arXiv:2602.08489v1cs.LGcs.CL

TL;DR

RLVR improves LLM reasoning through verifiable final answers but does not directly test whether the reasoning process is robust and reusable. RLTR adds a transfer reward based on whether another model can complete truncated reasoning correctly, improving accuracy and sampling consistency while reaching comparable performance in fewer training steps.

  • Problem

    RLVR focuses on final-answer correctness without directly ensuring that reasoning remains robust, reusable, or transferable across models.

  • Method

    RLTR augments RLVR with a transfer reward that evaluates whether a receiver model can reach the correct answer from a truncated generator reasoning prefix.

  • Results

    RLTR improves average accuracy and majority-voting consistency across reasoning benchmarks, while matching RLVR performance with about 2.5× fewer training steps on MATH-500.

  • Takeaways & Limitations

    Transferability provides a training signal for reasoning that is more consistent, reusable, and sample efficient than final-answer supervision alone.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has recently strengthened LLM reasoning, but its focus on final answer correctness leaves a critical gap: it does not ensure the robustness of the reasoning process itself. We adopt a simple philosophical view, robust reasoning should remain useful beyond the mind that produced it, and treat reasoning as a form of meaning transfer that must survive truncation, reinterpretation, and continuation. Building on this principle, we introduce Reinforcement Learning with Transferable Reward (RLTR), which operationalizes robustness via transfer reward that tests whether a partial reasoning prefix from one model can guide a separate model to the correct answer. This encourages LLMs to produce reasoning that is stable, interpretable, and genuinely generalizable. Our approach improves sampling consistency while improving final answer accuracy, and it reaches comparable performance in substantially fewer training steps. For example, on MATH500, RLTR achieves a +3.6%p gain in Maj@64 compared to RLVR and matches RLVR's average accuracy with roughly 2.5x fewer training steps, providing both more reliable reasoning and significantly more sample efficient.

1. Introduction

RLTR addresses RLVR’s focus on final-answer correctness by training reasoning that can transfer across models. Evaluations report better consistency, accuracy, and training efficiency.

  • RLVR verifies final-answer correctness but does not explicitly ensure robust, reusable, or interpretable reasoning trajectories.
  • RLTR defines reasoning transferability by testing whether a separate model can complete a truncated reasoning prefix to a correct answer.
  • RLTR augments RLVR with a transfer reward that directly incentivizes reasoning traces transferable across models.
  • RLTR improves majority-voting consistency and average accuracy across mathematical and scientific reasoning datasets.
  • On AMC23, RLTR improves Maj@64 from 61.7 to 67.5 compared to RLVR.
  • On MATH-500, RLTR reaches RLVR’s average accuracy with about 2.5× fewer training steps.

2. Related Work

Prior LLM reinforcement-learning approaches rely on learned reward models, while RLVR replaces them with verifiable final-answer rewards. Process reward models instead provide dense step-level supervision through learned verifiers.

  • RLHF-style reinforcement learning commonly optimizes language models against learned reward models that approximate human preferences.
  • Neural reward models introduce challenges including reward hacking and increased training complexity.
  • RLVR derives rewards from ground-truth correctness, such as mathematical answers or code execution, rather than a learned proxy.
  • Process Reward Models provide dense step-level supervision using learned verifiers trained on annotated reasoning traces.

3. RLTR: Reinforcement Learning with Transferable Reward

RLTR extends RLVR by evaluating whether another model can recover the correct answer from truncated generator reasoning. Its transfer reward is combined with existing RLVR rewards to train more reusable reasoning.

  • RLTR augments RLVR with a transfer reward that evaluates whether a receiver model can recover the correct answer from a truncated reasoning prefix.
  • RLVR rewards generator completions primarily for final-answer correctness and well-formed formatting.
  • Conventional RLVR does not explicitly account for step-level noise or encourage reasoning that other models can reliably continue.
  • RLTR treats transferability as stable, informative partial reasoning that enables a receiver to complete the solution correctly.
  • The generator’s reasoning is truncated at length ℓ = ⌊τ |ygen|⌋ and passed to a receiver model for continuation.
  • Transfer reward computation samples τ uniformly from 0.3 to 0.9 to reduce overfitting to fixed truncation boundaries.

4. Experiments

Across mathematical reasoning benchmarks, RLTR improves accuracy and multi-sample consistency over RLVR, including under distribution shift and harder tasks. Transfer-reward weighting, receiver choice, and training dynamics analyses associate these gains with more transferable reasoning and improved efficiency.

  • Performance of RLTR: RLTR consistently outperforms RLVR across moderate and challenging benchmarks on average accuracy and majority-voting metrics.The evaluation covers MATH-500, GSM8K, AIME2024, and AMC23.
  • Performance of RLTR: On GSM8K, RLTR improves accuracy from 89.1 to 92.0 and consistently outperforms RLVR across all K.RLVR slightly improves single-sample accuracy but degrades in larger-K majority voting relative to the base model.
  • Performance of RLTR: On AIME2024, RLTR raises average accuracy from 9.8 to 14.8 and Maj@64 from 16.7 to 21.1.The gains are more pronounced on harder benchmarks, where RLVR provides smaller high-K improvements.
  • Training Dynamics of RLTR: RLTR reaches comparable average accuracy to RLVR with about 2.5× fewer training steps and improves Maj@64 as training proceeds.RLVR’s Maj@64 performance degrades during training, while RLTR’s consistency and transferability grow more strongly.
  • Component ablations: Increasing transfer-reward weight raises high-K performance, with the setting (a=0.1, t=1.0) increasing Maj@64 from 82.2 to 84.2.When transfer reward is downweighted, results closely resemble RLVR and show limited scaling with K.
  • Component ablations: A stronger receiver improves high-K consistency, although RLTR still outperforms RLVR with the smallest tested receiver.Qwen-3B achieves Maj@16 of 83.8 and Maj@64 of 84.2, while Qwen-1.5B achieves the best Maj@4 at 79.6.

5. Conclusion

RLTR optimizes reasoning transferability through a transfer-augmented RLVR framework, improving accuracy, sampling consistency, sample efficiency, and robustness across settings.

  • RLTR optimizes reasoning transferability by augmenting RLVR with a transfer reward.The reward targets whether truncated reasoning can be completed across models to a verifiably correct answer.
  • RLTR improves average accuracy and sampling-based consistency across multiple reasoning benchmarks.Majority voting is used as an indicator of reasoning robustness.
  • RLTR reaches comparable performance with fewer training steps than the baseline.The conclusion reports significantly improved sample efficiency without specifying a numerical reduction here.
  • RLTR generalizes across base models and out-of-distribution settings.

Impact Statement

The work presents transferability as a training signal for robust LLM reasoning, with potential relevance to computationally costly and consistency-sensitive applications.

  • RLTR introduces transferability as a training signal to improve robust LLM reasoning.
  • Improving consistency and sample efficiency may benefit research and industry amid rising computation costs and sustainability concerns.
  • Domains requiring consistency and heavily annotated training samples may adopt RLTR by augmenting existing RLVR frameworks.Examples include Vision Language Action and Computer Use Agent applications.

A.4. Evaluation Details

The evaluation framework uses Math Verify and SimpleRL, with hyperparameters specified for RLTR and RLVR training.

  • Math Verify and SimpleRL are used as the evaluation frameworks.
  • Table 8 specifies the hyperparameters used for RLTR and RLVR training.

B.1. Rigorous Analysis of Computation Efficiency of RLTR

RLTR adds receiver-model computation per training step, but its improved training-step efficiency reduces the total computation needed to reach comparable performance.

  • Table 9 describes the Qwen2.5-7B-Instruct and Qwen2.5-3B-Instruct architectures used for the computation analysis.
  • RLTR has higher per-step computational cost because it adds a transfer rollout with a smaller receiver model.The analysis measures training cost using FLOPs and includes an additional receiver rollout phase.
  • 927.48 PFLOPs versus 993.96 PFLOPs per step corresponds to approximately 7.2% overhead for RLTR over RLVR.
  • RLTR reaches the same accuracy level with approximately 60% fewer training steps than RLVR.
  • 57% lower total computational cost is reported for RLTR compared with the baseline.The reduction is attributed to training-step efficiency outweighing per-step overhead.

B.2. Additional Analysis for Transferability

Across truncation ratios from τ = 0.3 to 0.9, RLTR’s transferability advantage over RLVR grows consistently, suggesting more robust reasoning.

  • Transferability gaps between RLVR and RLTR consistently increase across truncation ratios τ ∈ {0.3, 0.5, 0.7, 0.9}.The main analysis used τ = 0.7; this extension varies τ while keeping other settings unchanged.

B.3. Additional Analysis on Reward Ratio

Larger transfer weights improve Pass@K diversity, with the reported setting increasing Pass@64 from 92.2 to 95.0 and addressing diversity degradation associated with RLVR.

  • Larger transfer weights yield stronger improvements in Pass@K diversity.The analysis varies the transfer weight while examining diversity through Pass@K.
  • 92.2 to 95.0: setting (a=0.1, t=1.0) increases Pass@64.
  • Transfer rewards help retain diversity, addressing the diversity degradation problem suggested for RLVR.

B.4. Additional Analysis on Reveiver Model

Receiver models preserve similar large-K diversity while affecting single-sample accuracy, and RLTR produces more coherent reasoning than RLVR in the generation examples.

  • Receiver Model Analysis: 95.2 vs. 95.0 vs. 95.0: receiver models achieve similar Pass@64 performance.This indicates comparable diversity in producing at least one correct sample.
  • Receiver Model Analysis: 74.0/77.6/77.0: average accuracy varies more across receiver models.The results suggest receiver choice can influence single-sample correctness while preserving diversity.
  • Transfer as a Verifier: Transferability is proposed as a test-time verification signal that prefers partial reasoning robust under cross-model continuation.The proposed extension uses transferability during verification rather than only during training.
  • Generation Examples: RLTR produces coherent and logically consistent reasoning where RLVR contains an intermediate-trajectory inconsistency, despite both final answers being correct.The examples compare generation trajectories from RLVR-trained and RLTR-trained models on the same problem.
Loading 2602.08489v1…