PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn Hard Problems During RL with Reference Guided Fine-tuning

Yangzhen Wu, Shanda Li, Zixin Wen, Xin Zhou, Ameet Talwalkar, Yiming Yang, Wenhao Huang, Tianle Cai

arXiv:2603.01223v2cs.LGcs.CL

TL;DR

RL for mathematical reasoning can stall when hard problems produce no correct trajectories, and direct fine-tuning on human references may mismatch the model’s reasoning distribution. ReGFT uses partial references to elicit model-generated, solution-consistent trajectories before RL. Across benchmarks, it improves supervised accuracy, accelerates DAPO training, and raises final RL performance.

  • Problem

    RL can stall on hard problems when the base model samples no correct trajectories, while direct reference-solution fine-tuning often mismatches the model’s reasoning patterns.

  • Method

    ReGFT prompts the model with partial reference solutions and fine-tunes on the resulting model-generated reasoning trajectories before RL.

  • Results

    Across OmniMath, AIME 2024, AIME 2025, and Beyond-AIME, ReGFT improves supervised pass rates, accelerates DAPO training, and yields higher final performance than raw and ReFT baselines.

  • Takeaways & Limitations

    ReGFT mitigates reward sparsity by converting references into model-derived correct trajectories, providing a stronger initialization for downstream RL.

  • Takeaways & Limitations

    ReGFT is evaluated with DAPO and assumes access to reference solutions in the training data; its scope therefore depends on these experimental conditions.

Abstract

from arXiv · show

Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity: for challenging problems, LLM fails to sample any correct trajectories, preventing RL from receiving meaningful positive feedback. At the same time, there often exist human-written reference solutions along with the problem (e.g., problems from AoPS), but directly fine-tuning on these solutions offers no benefit because models often cannot imitate human proofs that lie outside their own reasoning distribution. We introduce Reference-Guided Fine-Tuning (ReGFT), a simple and effective method that utilizes human-written reference solutions to synthesize positive trajectories on hard problems and train on them before RL. For each problem, we provide the model with a partial reference solution and let it generate its own reasoning trace, ensuring the resulting trajectories remain in the model's reasoning space while still benefiting from reference guidance. Fine-tuning on these reference-guided trajectories increases the number of solvable problems and produces a checkpoint that receives more positive rewards during RL. Across three benchmarks (AIME24, AIME25, BeyondAIME), ReGFT consistently improves supervised accuracy, accelerates DAPO training, and raises the final performance plateau of RL. Our results show that ReGFT effectively overcomes reward sparsity and unlocks stronger RL-based mathematical reasoning.

1 Introduction

RLVR improves mathematical reasoning by reinforcing verified correct trajectories, but reward sparsity stalls learning when hard problems yield no correct samples. ReGFT addresses this gap by using reference-guided, model-generated trajectories before RL, improving initialization and downstream performance.

  • Motivation: RLVR optimizes sampled reasoning trajectories using rewards based on verified final-answer correctness.This provides automatic supervision without human labels or reward-model training.
  • Motivation: Hard problems can produce zero correct trajectories, leaving RL without useful learning signals and causing training to stall.The limitation arises when problems exceed the base model’s current reasoning capability.
  • Motivation: ReFT helps on problems the model can already solve but does not improve harder problems when no correct trajectories can be sampled.Those problems therefore continue to provide absent reinforcement signals.
  • Contribution: ReGFT uses reference solutions to elicit model-generated, solution-consistent reasoning traces for hard problems rather than directly fine-tuning on reference chains of thought.The resulting trajectories remain aligned with the model’s own reasoning process.
  • Results: ReGFT-initialized models outperform raw checkpoints during RL, with faster early gains and higher final accuracy across benchmarks.The results indicate improved reward availability and optimization efficiency.
  • Results: ReGFT produces more stable pass@k improvements across inference budgets, whereas ReFT gains are mainly confined to low-k regimes.This comparison suggests ReGFT scales more effectively with additional inference computation.

2 Related Work

Prior work seeks to expand RL beyond the model’s initial competence through greater exploration, question augmentation, or SFT–RL combinations. ReGFT instead raises competence before RL by using targeted pre-RL supervised fine-tuning on model-generated trajectories guided by references.

  • Motivation: The literature identifies pass@k saturation and occasional underperformance on initially unsolvable problems as limitations of naïve RL.These observations suggest standard RL often amplifies existing behaviors rather than inducing new reasoning capabilities.
  • Existing directions: One research direction scales exploration through longer training horizons or adaptive rollout allocation toward harder instances.The goal is to obtain more informative trajectories and learning signals.
  • Existing directions: Another direction augments difficult questions with partial solutions, scaffolding, or hints to increase rewarding trajectories.These interventions transform some zero-reward cases into learnable examples.
  • Existing directions: A third direction interleaves or jointly optimizes SFT and RL to introduce solutions on hard questions and improve training stability.The approaches differ in whether supervised and reinforcement objectives alternate or are combined in one stage.
  • Our difference: ReGFT differs by improving model competence before RL instead of modifying RL through supervised objectives or expert-prefixed rollouts.Its central comparison is with ReFT and RL Teachers, but it uses reference guidance with the model’s own reasoning.

3 Approach

ReGFT extends on-policy fine-tuning to initially unsolved problems by prompting the model with partial reference solutions and generating its own reasoning traces. Its stronger initialization yields better RL outcomes than raw, and consistently beyond ReFT, checkpoints.

  • ReFT: ReFT fine-tunes only on correct, model-generated trajectories to align supervision with the model’s intrinsic reasoning patterns.This avoids the mismatch associated with directly imitating human-written reference solutions.
  • ReGFT: ReGFT prompts the model with partial references that provide structural guidance while requiring reasoning from the beginning through its own generation process.This design avoids copying or completing the reference trajectory.
  • ReGFT: ReGFT is intended to generate correct reasoning trajectories on problems initially unsolvable by the base model.The method extends supervised learning beyond the model’s initial capability boundary.
  • RL initialization: ReGFT checkpoints provide stronger RL starting points because they generate more positively rewarded trajectories on previously unsolvable questions.RL is then applied to checkpoints produced by ReFT or ReGFT.
  • RL results: Across three benchmarks, ReGFT outperforms raw and ReFT initializations in final RL accuracy, although both methods accelerate early training.The comparison attributes additional final gains to reference-guided demonstrations beyond self-generated trajectories.
  • DAPO integration: ReGFT continues to improve results when paired with DAPO, whose dynamic sampling already addresses uninformative gradients under sparse rewards.This supports compatibility between ReGFT and improvements to RL optimization or sampling.

4 Experiments

The experiments use Qwen3-4B-2507-Instruct on OmniMath, a verified Olympiad-level mathematics dataset designed to induce severe reward sparsity, with AIME benchmarks used for evaluation.

  • Setup: Qwen3-4B-2507-Instruct serves as the base model because it offers strong reasoning and reliable instruction following in this setting.The authors also note sensitivity to spurious reward signals in Qwen2.5 models.
  • Setup: OmniMath contains 4,428 Olympiad-level mathematics problems with verified reference solutions and severe reward sparsity.This makes it suitable for studying verifier-based reinforcement learning.

4.2 Main Results

ReGFT strengthens reinforcement learning by using partial human references to produce model-derived reasoning trajectories, yielding faster training and higher final performance. It also improves inference-time scaling, whereas direct fine-tuning on raw human solutions performs substantially worse.

  • RL training: ReGFT-initialized models outperform raw checkpoints throughout RL training, with faster early and middle-stage improvement and higher accuracy at convergence.This indicates a stronger initialization that enables more effective progress with fewer RL updates.
  • RL training: ReGFT alleviates reward sparsity by increasing the likelihood that sampled trajectories contain correct solutions, helping RL converge faster and reach a higher performance ceiling.The method improves both optimization efficiency and attainable final performance under DAPO.
  • Impact of reference-guided demonstrations: ReGFT outperforms ReFT at convergence across benchmarks, suggesting reference-guided demonstrations expand reasoning capabilities beyond what self-exploration alone discovers.ReFT accelerates learning, but ReGFT also raises asymptotic accuracy and extends competence to previously unsolvable problems.
  • Necessity of model-derived reasoning: Direct fine-tuning on raw human solutions produces substantially weaker supervised and RL performance than ReGFT.ReGFT instead generates reasoning trajectories in the model’s own inference style under partial reference guidance.
  • Inference-time scaling: ReGFT maintains stronger pass@k scaling across small and large k regimes, while ReFT’s gains diminish as k increases.The reported comparison attributes ReGFT’s advantage to improved coverage of the solution space rather than gains limited to a few samples.

4.3 Additional Experimental Evidence

Additional experiments show that partial reference guidance unlocks correct trajectories on problems standard sampling misses, while ReGFT improves training-set performance and supports stronger RL initialization. Larger sampling budgets help, but do not replace reference-guided fine-tuning.

  • Reference-guided generation: Reference-guided sampling provides partial human solutions while withholding the final-answer portion, requiring the model to generate its own reasoning trajectory.The experiment supplies the first 80% of reference sentences and withholds the remaining 20%.
  • Reference-guided generation: 70.82% of OmniMath problems are solved with reference-guided sampling, versus 68.58% with standard sampling.Reference-guided sampling additionally solves 5.85% of problems never solved under standard sampling, while 3.61% are solved only by standard sampling.
  • Supervised evaluation: Both ReFT and ReGFT improve over the base model on OmniMath and the AIME24, AIME25, and BeyondAIME evaluation benchmarks.The results indicate that supervised fine-tuning on self-generated trajectories strengthens mathematical reasoning across training and out-of-distribution tasks.
  • Supervised evaluation: ReGFT achieves a higher OmniMath training-set pass rate than ReFT, while external benchmark performance remains largely comparable.The training-set improvement reflects the additional correct trajectories made available through reference-guided sampling.
  • Sampling scale: Increasing responses per problem from 16 to 64 improves performance across settings, but ReGFT+DAPO remains strongest across all benchmarks.The results indicate that improved initial competence and increased exploration scale are complementary.

5 Conclusion

ReGFT addresses reward sparsity by using partial reference solutions to generate model-aligned, verifiable trajectories before RL. Across benchmarks, it improves supervised performance, accelerates RL training, and raises final accuracy while supporting stronger inference-time scaling.

  • ReGFT conditions models on partial reference solutions and requires them to generate their own reasoning trajectories before RL.The resulting trajectories remain aligned with the model’s inference distribution while benefiting from reference guidance.
  • ReGFT increases the density and diversity of verifiable trajectories, providing meaningful RL signals on previously unsolvable problems.
  • Across OmniMath, AIME 2024, AIME 2025, and Beyond-AIME, ReGFT improves supervised pass rates, accelerates DAPO training, and outperforms raw and ReFT-based baselines.
  • ReGFT’s advantage persists as k increases in pass@k evaluation, indicating more reliable correct-solution discovery under additional test-time compute.
  • ReGFT mitigates reward sparsity by converting reference solutions into model-derived correct trajectories, yielding a stronger initialization for downstream RL.

A RL Training Implementation Details.

The RL training setup follows DAPO with specified optimizer, learning-rate, warm-up, and rollout batch-size settings. The supplied passage provides implementation details but is truncated before all configurations are stated.

  • The reinforcement-learning setup follows the DAPO framework and adopts its recommended hyperparameter configuration unless otherwise stated.
  • Training uses AdamW with a constant learning rate of 1 × 10^-6 and a linear warm-up over the first 20 rollout steps.
  • With 16 sampled responses per prompt, the rollout prompt batch size is 512 prompts.

B Prompts.

The appendix contrasts ordinary solution generation with reference-guided sampling. Reference-guided prompts provide a partial correct solution as a hint while requiring the model to derive its own answer.

  • The ordinary solution-generation prompt asks the model to reason step by step and place its final answer within boxed notation.
  • Both solution-generation prompts request step-by-step reasoning and a final answer within boxed notation.
  • The reference-guided sampling prompt supplies a partial reference solution known to be correct as a hint.
  • The reference-guided prompt requires the model to solve independently while allowing it to follow ideas from the hint.

C Problem Examples.

The appendix presents mathematical problem examples and partial reference solutions used to illustrate reference-guided sampling. The examples cover projection inequalities, Hölder’s inequality, modular residues, and additive-set counting.

  • Partial Reference Solutions: The reference solutions are concise and omit detailed reasoning, motivating reference-guided sampling that encourages the model to construct its own trajectories.
  • Problem Examples: One example defines sets S and T from triples whose coordinate sums are respectively below 1 and above 2, with each set required to contain at least 2018 elements.
  • Problem Examples: For that example, projections onto coordinate planes satisfy |Sxy| + |Txy| ≤ n^2, |Syz| + |Tyz| ≤ n^2, and |Szx| + |Tzx| ≤ n^2.
  • Partial Reference Solutions: The accompanying partial solution applies the Projection Inequality and Hölder’s Inequality to relate |S| and |T| to their coordinate projections.
  • Problem Examples: A second example asks for the smallest number of pairwise sums ai + aj when 0 = a0 < a1 < ··· < an = 2^n − 1.
  • Partial Reference Solutions: Its partial solution shows that interior sums cover all residues modulo 2^n − 1 using the Pigeonhole Principle, then combines this with endpoint sums to establish additional distinct values.
Loading 2603.01223v2…