Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding

• RGPO first tests each problem without hints, reinforces a correct response, and introduces progressively stronger rationale scaffolds after failure.Source: Figure 1 · p. 2
• When all sampled rollouts are incorrect, rewards provide little information about how to reach a correct solution, and training can stagnate on hard examples.The capacity–difficulty mismatch is especially severe for weak or small models.Source: §1 · p. 2
• RGPO uses reference rationales as temporary scaffolds for hint-assisted refinement, rather than as direct imitation targets.The refined response is trained conditioned only on the original prompt when it improves on the initial rollout.Source: §1 · p. 2
• An adaptive hint mechanism adjusts the amount of revealed guidance to the model’s current ability, concentrating help on difficult examples while preserving exploration on solvable ones.Source: §1 · p. 2
• The paper reports improved training sample efficiency over standard RLVR and better performance than hybrid baselines combining on-policy and off-policy training.Source: §1 · p. 3

• PPO [6] uses a clipped policy-gradient objective, GRPO [7] uses group-relative reward normalization, and DeepSeek-R1 [16] showed large-scale RLVR can elicit sophisticated reasoning.Source: §2 · p. 3
• Hybrid approaches include off-policy reasoning traces in LUFFY [8], a dynamically weighted SFT objective in CHORD [17], and heuristic hints in HINT.Source: §2 · p. 3
• RGPO generates auxiliary examples online through rationale-guided refinement and trains them under the original prompt distribution, unlike methods using external traces or prefixes.Source: §2 · p. 3
• Cognitive studies motivate attempting problems before guidance and fading support, while warning that too little or too much assistance can hinder learning or transfer.The cited work also notes that hint abuse can let learners obtain answers without understanding the reasoning.Source: §2 · p. 3

3.1 Supervised Fine-Tuning

• SFT trains on prompt–response pairs by maximizing the conditional likelihood of the reference response.Source: §3.1 · p. 3
• SFT can initialize a policy and teach reasoning formats, but stylistically mismatched or difficult expert traces can lead to superficial imitation or memorization.The text notes this limitation can be pronounced for smaller models.Source: §3.1 · p. 4

3.2 Policy Optimization for Language Model Reasoning

• For reasoning tasks, RL optimizes expected reward, which can be verified by checking whether the final answer matches the ground truth.Source: §3.2 · p. 4
• GRPO samples a group of responses per prompt, normalizes their rewards into group-relative advantages, and optimizes a PPO-style clipped objective.Source: §3.2 · p. 4
• REINFORCE++ normalizes advantages over a global batch rather than within prompt-local groups, aiming for more stable estimates; the described pipelines reduce or remove KL for exploration and simplicity.Source: §3.2 · p. 4

4.1 Problem setup

• RGPO starts with an unguided epoch, then adjusts each example’s revealed rationale segments according to whether any rollout solves it.Unsolved examples receive one more segment; solved examples lose one, creating a curriculum toward the smallest hint level that keeps each example learnable.Source: §4.1 · p. 5

4.2 Adaptive rationale-guided rollout

• Each epoch pairs ordinary rollouts from the original prompt with guided revisions, retaining guided candidates only when they improve on unguided attempts under the reward gate.In the exact-reward setting, the margin is δ = 0; guidance is treated as a proposal rather than assumed to help.Source: §4.2 · pp. 5–6

4.3 Policy optimization objective

• RGPO combines on-policy RL on unguided rollouts with an auxiliary supervised objective on accepted guided responses, conditioned on the original prompt.The total objective is Ltotal(θ) = LRL(θ) + λtLSFT(θ).Source: §4.3 · p. 6

4.4 Why RGPO helps under sparse rewards

• At equal response budgets, guided search finds a successful trajectory more often exactly when guided success probability exceeds unguided success probability.On the all-failure event, RGPO obtains an accepted positive target with probability (1 − p)^K[1 − (1 − s)^Kg].Source: §4.4 · p. 6
• Unguided GRPO learns easy problems but makes no progress on hard problems in the controlled simulation.Accuracy without hints is evaluated on training and held-out problems, with mean ± s.e. over five seeds.Source: Figure 2 · p. 7

5.1 Controlled Simulation

• 99.1% easy and 0.0% hard test accuracy for GRPO contrast with RGPO’s 99.5% hard accuracy while preserving easy-problem performance.Direct training on the imperfect reference reached 12.8% hard accuracy, while full-hint performance fell to the 1% chance level without hints at evaluation.Source: §5.1 · p. 7

5.2 RGPO on Language Models

• 26.8 vs. 21.8 in-distribution average (RGPO vs. HINT) and 32.2 vs. 29.9 out-of-distribution average show RGPO leading across language benchmarks.RGPO achieves the best result on every in-distribution benchmark and the highest scores on ARC, GPQA, and MMLU.Source: Table 1 · p. 8, §5.2 · p. 8

5.3 RGPO on Vision-Language Models

• On Qwen2-VL-2B with GRPO, RGPO reaches baseline performance in 2.95× fewer training steps and achieves higher final accuracy.Across the tested Qwen2-VL-2B and Qwen2-VL-7B settings, RGPO improves data efficiency and final accuracy over corresponding RLVR baselines.Source: §5.3 · p. 9, Figure 3 · p. 9
• 70.1 on MMStar-Math and 73.65 on MathVista-Math are RGPO’s reported overall scores, with strongest GEO, ALG, and GPS subset scores but a modest Textbook QA gain.The MathVista-Math scores are 73.47 for GEO, 73.62 for ALG, and 73.89 for GPS; the authors suggest possible training-set domain overlap for Textbook QA.Source: §5.3 · p. 9, Table 2 · p. 9
• Full hints quickly raise training reward but yield poorer no-hint validation performance; RGPO’s failure-conditioned hinting learns more gradually and performs better without hints.The full-hint policy can use the hint to answer without independently reasoning; RGPO’s optimization is smoother and maintains longer reasoning responses.Source: Figure 4 · p. 10, §5.3 · p. 10
• With the concise DeepSeek-R1-style prompt, RGPO exceeds GRPO’s reward while maintaining comparable entropy; with the Mulberry-style template, AHPO’s reward rise accompanies sharp entropy collapse.The authors use the simpler format in their main experiments to make the reward signal easier to optimize while avoiding premature loss of policy diversity.Source: Figure 5 · p. 10, §5.3 · p. 10

6 Conclusion

• RGPO uses rationales as temporary scaffolds instead of direct imitation targets to improve reasoning.The conclusion describes the framework as an RL approach that uses rationales as instructional interventions to turn failed attempts into useful learning signals.Source: §6 · p. 11
• The design reduces hint dependence by not providing rationales for every example and avoids rationale memorization by not training directly on rationale text.The conclusion frames rationales as instructional interventions rather than fixed training targets.Source: §6 · p. 11
• RGPO shows superior post-training performance in both language and vision-language models.The conclusion reports this empirical scope across text and multimodal models.Source: §6 · p. 11

A.1 Training Procedure

• With δ = 0, RGPO accepts a guided response only when its verifier reward exceeds that of every unguided rollout for the example.The experiments use τcorrect = 1.0 for determining whether an unguided rollout solves a problem, and tune λt over 10^-2, 10^-3, and 10^-4.Source: §A.1 · p. 16
• RGPO uses prompt-level rationale scaffolds to generate refined candidates with the trainable policy, accepts reward-improving candidates, then distills them under the original no-hint prompt.This avoids requiring teacher logits or an additional teacher model during training and permits black-box, human-written, or dataset-provided rationales.Source: §A.1 · p. 17
• The appendix examples illustrate guided revisions using partial solutions, prior responses, and verifier feedback, followed by training to reproduce corrected responses without auxiliary context.The examples describe cases where direct attempts were incorrect and auxiliary information steered a refined response toward a correct solution.Source: §A.1 · p. 17

B Theoretical Analysis of Rationale-Guided Learning

• The theoretical setup samples K unguided responses from the decoding distribution, then generates additional guided candidates conditional on that group, the rationale prefix, and verifier feedback.The passage distinguishes sampling quantities defined by ρθ from likelihood-based quantities defined by πθ.Source: B Theoretical Analysis of… · p. 22

B.1 Recovering supervision from failed unguided groups

• When every unguided response fails, accepted guided successes provide positive supervision even though group-centered GRPO has zero reward-advantage signal.For binary rewards, the proposition identifies guidance as a source of signal on the all-failure event.Source: §B.1 · p. 22
• GRPO’s expected correctness signal is O(p) as the pass rate p approaches zero, and its update is exactly zero on the all-failure event.The all-failure event has probability (1 − p)^K.Source: §B.1 · pp. 22–23
• With equal response budgets and independent guided candidates of success probability s, guided sampling discovers a success more often than unguided sampling when s > p.Here p is the unguided success probability; when p = 1, both quantities are zero.Source: §B.1 · pp. 22–23
• When guided success varies with the failed unguided group, the equal-budget comparison depends on the guided group’s conditional failure probability, not only its mean success rate.The stated qualification applies when there is more than one guided candidate.Source: §B.1 · p. 23

B.2 Transfer to the original prompt

• For binary-reward transfer, a reduction in loss on successful responses greater than the initial divergence implies an increased success probability.The stated condition is ΔL > Dθ0.Source: §B.2 · p. 24
• When reward-weighted proposal mismatch is small, the auxiliary supervised-learning direction is close to the gradient of log expected reward.The result is stated as a local-alignment condition.Source: §B.2 · p. 24

B.3 Adaptive assistance from unguided competence

• The supplied result specifies boundary drift: at hint level g_t = 0 it is (1 − p_t)K, and at g_t = N it is −[1 − (1 − p_t)K].Source: §B.3 · p. 25

B.4 Learning time in a two-response model

• In the two-response model, blocks of visits have a lower-bounded chance of accepted guided progress or reaching the target; the rare-success theorem compares learning time with same-budget GRPO.The block guarantee relies on the useful-hint condition, including a positive chance of a guided success after unguided failure.Source: §B.4 · p. 26

C.1 Language-Model Experiments

• Language-model post-training uses GRPO in verl, four on-policy responses per prompt, and up to 512 hint tokens; input prompts over 1,024 tokens are filtered.The setup also specifies a 256-prompt global batch and responses up to 4,096 tokens.Source: §C.1 · p. 27

C.2 Vision-Language Training

• Vision-language training uses multimodal GRPO/RGPO with programmatic verification, four responses per prompt, and up to 512 tokens of rationale guidance.It uses 512 prompts per global batch, with over-length inputs filtered rather than truncated.Source: §C.2 · p. 28

C.3 Baselines

• Comparisons include GRPO and REINFORCE++ for on-policy RLVR, alongside methods that use external traces, adaptive guidance, or other reasoning signals.The listed baselines include LUFFY, CHORD, GHPO, QuestA, BREAD, and HINT.Source: §C.3 · p. 28
• General-purpose VLMs are reference points rather than strictly matched baselines because their backbones, training corpora, and post-training recipes differ.The multimodal comparisons also include supervised reasoning variants and open reasoning systems.Source: §C.3 · pp. 28–29

C.4 Controlled Simulation Setup

• The controlled simulator makes hard problems rare to solve initially without hints, while references are useful but deliberately incorrect for three of eight hard decision types.The reference is available at no-hint, partial, and full levels; guided generation follows a revealed action with probability ρ = 0.7.Source: §C.4 · p. 29, Table 4 · p. 30
• RGPO raises a problem’s hint level after an epoch with no correct unguided response and retains guided responses only when they are correct and all corresponding unguided responses fail.Accepted responses are optimized under the original problem; evaluation is also performed without hints.Source: §C.4 · p. 30

C.6 Prompt templates

• The refinement prompt combines the original problem, a revealed solution segment, and feedback, while requiring a new standalone solution rather than a revision of the prior response.Feedback addresses format and accuracy, and the instruction uses the training and evaluation answer format.Source: §C.6 · p. 31, Refinement prompt · p. 31

DAPO’s system prompt

• The vision-language system prompt uses <think> tags for reasoning and a boxed final answer, with fewer structural constraints than the Mulberry-style format.The Mulberry-style prompt separates image description, rationale, step-by-step reasoning, and final answer.Source: DAPO’s system prompt · p. 31, DeepSeek-R1’s system prompt · p. 32

D.1 Controlled Simulation

• 0.0% hard test accuracy: GRPO learns the easy subset to 99.1% test accuracy but remains at zero on hard problems after three epochs.The reported hard-problem NLL indicates that hint-conditioned RL begins improving individual decisions, while hard accuracy reaches only 0.4%.Source: §D.1 · p. 33, Figure 6 · p. 33
• 12.8% hard test accuracy: both reference-SFT variants and RGPO without the verifier-based gate reach this result.The passage attributes this comparison to the importance of checking guided responses before using them as training targets.Source: §D.1 · p. 33
• 99.5% hard test accuracy and 99.4% overall test accuracy: RGPO also retains 99.3% easy-problem performance after three epochs.These are no-hint results; Table 5 reports final accuracy and Figure 6 shows learning dynamics.Source: Figure 6 · p. 33, Table 5 · p. 34, §D.1 · p. 34
• 1% hard test accuracy: full-hint training reaches chance accuracy when evaluated without the reference, while RGPO with full hints always reaches 1% on hard test problems despite 71.2% hard training accuracy.The comparison distinguishes performance under guidance from transfer to held-out problems without it.Source: §D.1 · pp. 33–34

D.2 Fine-Grained Benchmark Results

• On MathVista, RGPO scores 40.7 versus GRPO’s 39.6 and the base model’s 36.7; textbook question answering reaches 53.2, 6.4 points above GRPO and 8.0 above the base model.The largest category gain is in textbook question answering.Source: Figure 7 · p. 34, §D.2 · pp. 34–35
• On MMStar, RGPO reaches 45.2 versus GRPO’s 41.2 and the base model’s 36.8, with its largest category gain in geometry at 34.5.RGPO also scores 58.3 on numeric commonsense and calculation, versus 56.2 for GRPO and 50.0 for the base model; it matches GRPO’s 52.3 on statistical reasoning.Source: Figure 8 · p. 35, §D.2 · p. 35

D.3 Dynamics of Adaptive Rationale Guidance

• Rationale guidance is absent in the probing epoch, reaches around 20% of examples in the second epoch, then falls to roughly 15% in the third.The passage attributes the decline to more examples becoming solvable without hints.Source: §D.3 · p. 35, Figure 9 · p. 36

D.4 Beyond Expert Rationale Scaffolds

• On MM-HELIX 24-Points, RGPO-7B reaches 40.00% using mechanical solver traces, while RGPO-7B without CoT reaches 36.67% using verifier feedback alone.Both variants outperform MM-HELIX-7B at 33.33%; the final learning signal comes from verifier-approved policy responses.Source: Table 7 · p. 38, §D.4 · p. 38

D.5 Effect of Prompt Format

• The DeepSeek-R1-style prompt learns better than the Mulberry-style prompt while producing shorter responses and reducing runtime per training step.The Mulberry-style prompt produces around 380–400 tokens with nearly flat reward, while the simpler prompt’s responses drop below 100 tokens as reward rises.Source: Figure 11 · p. 38, §D.5 · p. 38

• Under noisy hints, the full-hint baseline’s score drops to 20.4, while RGPO scores 25.5, indicating greater resistance to corrupted guidance.The passage attributes the baseline’s decline to its tendency to trust and follow the misleading rationale.Source: §D.6 · p. 39, Figure 12 · p. 39
• With correct CoT hints, the full-hint baseline scores 95.2 versus RGPO’s 93.8, but RGPO leads in no-hint evaluation, 38.4 to 33.6, and noisy-hint evaluation, 25.5 to 20.4.The noisy setting shuffles the CoT and permits the model to use it if helpful.Source: Figure 12 · p. 39, §D.6 · p. 39
• RGPO provides hints after unguided failure, verifier-filters guided outputs, and distills accepted responses back to the no-hint prompt.This design treats rationales as temporary scaffolds rather than permanent dependencies.Source: §D.6 · p. 39

• Compared with the base model, GRPO and RGPO often produce more focused reasoning processes.The examples appear in Figures 13–15.Source: §D.7 · p. 39, Figure 13 · p. 40, Figure 14 · p. 41
• RGPO can use temporary rationale scaffolds to identify a relevant solution path.Source: §D.7 · p. 39
• RGPO answers questions more concisely than the compared responses by using rationales as temporary scaffolds.Source: §D.7 · p. 39

E Limitations

• Standard RGPO assumes a reference rationale or other informative guidance, and effectiveness depends on its quality and granularity.Source: E Limitations · p. 43
• Too much answer-specific guidance may encourage shortcut learning or memorization; too little or stylistically distant guidance may fail to rescue hard examples.Source: E Limitations · p. 43
• Reward-gated acceptance guarantees improvement only under the available verifier, not genuine reasoning quality, and can inherit verifier bias or reward hacking.Source: E Limitations · p. 43
• Guided rollouts and auxiliary SFT updates add compute compared with single-pass PPO or GRPO, although rollout counts match GRPO for a fair comparison.Source: E Limitations · p. 43

F Broader Impacts

• Improved reasoning may also assist harmful planning, persuasion, misinformation, or cyber misuse; the paper calls for safeguards and robustness evaluation.Source: F Broader Impacts · p. 43